diff --git a/.github/.copilot/breadcrumbs/2026-08-12-2353-delete-stage-tasks.md b/.github/.copilot/breadcrumbs/2026-08-12-2353-delete-stage-tasks.md new file mode 100644 index 000000000..4b624e9b1 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-08-12-2353-delete-stage-tasks.md @@ -0,0 +1,42 @@ +# Support Before-Stage Tasks on the Delete Stage + +## Overview + +Allow staged update strategies to gate the delete stage with an Approval task, the same before-stage task the +update stages support, so that removing the resources from the clusters that left a placement can be held until +a sign-off is given. + +## Plan + +1. Add an optional `deleteStage` to the shared update strategy API, with `beforeStageTasks`. +2. Snapshot the tasks and initialize their statuses as the `beforeStageTaskStatus` of the delete stage. +3. Evaluate the tasks with the same code that evaluates the before-stage tasks of the update stages. +4. Add API validation, unit, integration and E2E coverage for both cluster-scoped and namespaced runs. +5. Regenerate API code and CRDs, then run the tests and the repository quality checks. + +## Decisions + +- The delete stage is configured as a stage (`deleteStage.beforeStageTasks`) rather than with a delete stage + specific task list, so that the API, the status and the approval requests use the same vocabulary as the + update stages, and so that the delete stage can take more configurations later. +- The before-stage tasks of the delete stage follow the update stages: at most one task, and it must be an + Approval. A soak time before the removal is already possible with a TimedWait in the afterStageTasks of the + last update stage, as the delete stage only starts once the last update stage, including its after-stage tasks, + completes. +- The approval request is named `-before-delete-stage` with the generic before-stage name format. + `delete-stage` also is the stage label value of the approval request, as the name of the delete stage, + `kubernetes-fleet.io/deleteStage`, is not a valid label value. An update stage cannot have this name. +- The tasks only gate the start of the deletion and they are skipped if there is no cluster to delete, the same + as an update stage with no clusters. +- An update run that is stopped while it waits for the tasks stops without deleting any binding. + +## Success Criteria + +- [x] No delete stage configuration preserves immediate deletion. +- [x] The approval must be given before any binding is deleted. +- [x] Cluster-scoped and namespaced runs keep the bindings until the tasks pass. +- [x] Generated API and CRD artifacts are current. + +## Follow-ups + +- Document the delete stage configuration in the staged update docs (website repository). diff --git a/.github/.copilot/breadcrumbs/2026-09-10-0000-cleanup-leftover-works.md b/.github/.copilot/breadcrumbs/2026-09-10-0000-cleanup-leftover-works.md new file mode 100644 index 000000000..27a2ab336 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-09-10-0000-cleanup-leftover-works.md @@ -0,0 +1,32 @@ +# Cleanup Leftover Works + +## Goal + +Implement `cleanupLeftOverWorks` so manifests previously applied or prepared by leftover Work objects are removed unless still referenced by the current linked Work set. + +## Plan + +- [x] Collect manifest identifiers whose Applied condition is true or has the preparing-to-process reason. +- [x] Exclude identifiers present in `seenManifIDs`. +- [x] Remove the collected manifests in parallel through `removeOneLeftOverManifest`. +- [x] Aggregate per-manifest failures and run focused validation. + +## Success Criteria + +- Current manifests are preserved. +- Only previously applied or pre-processed leftover manifests are selected. +- Selected manifests are removed through the existing ownership-aware helper in one parallel batch. + +## Validation + +- `gofmt` completed successfully. +- Editor diagnostics report no errors in `cleanup.go`. +- `go test ./pkg/v1/controllers/workapplier` is currently blocked by pre-existing unused `leftOverWorks` and `seenManifestIDs` variables in `controller.go`, where the surrounding cleanup call site is still unfinished. + +## Placement Deletion Follow-Up + +- [ ] Preserve explicit sync strategy values while applying defaults. +- [ ] Delete linked AppliedWork objects with foreground or orphan propagation according to `WhenPlacementDeleted`. +- [ ] After `foregroundDeletionWaitTime`, explicitly clean tracked manifests blocking foreground deletion. +- [ ] Remove Work cleanup finalizers only after the corresponding AppliedWork is gone. +- [ ] Wire placement and leftover cleanup into reconciliation and validate the package. diff --git a/.github/.copilot/breadcrumbs/2026-09-23-1617-hub-condition-timestamp.md b/.github/.copilot/breadcrumbs/2026-09-23-1617-hub-condition-timestamp.md new file mode 100644 index 000000000..4058a809d --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-09-23-1617-hub-condition-timestamp.md @@ -0,0 +1,26 @@ +# Fix: Preserve UpdateRun Condition Transition Timestamp + +## Overview + +Prevent hub-agent reconciliation and restarts from restamping the UpdateRun status metric when the selected Kubernetes condition has not transitioned. + +## Plan + +1. Trace UpdateRun condition reconciliation and metric emission. +2. Emit the selected condition's persisted `LastTransitionTime` instead of reconciliation time. +3. Add focused unit tests for unchanged failed conditions and actual status transitions. +4. Run the smallest relevant UpdateRun unit test suite. + +## Success Criteria + +- [x] Re-emitting an unchanged failed condition preserves the metric timestamp. +- [x] A condition status transition emits the new transition timestamp. +- [x] Focused UpdateRun tests pass. + +## Implementation Notes + +- Kubernetes `meta.SetStatusCondition` already preserves `LastTransitionTime` when `Status` is unchanged and advances it when `Status` changes. +- The restamping occurred in metric emission through `SetToCurrentTime()`, independently of the persisted condition. +- `emitUpdateRunStatusMetric` now sets the gauge from the selected condition's `LastTransitionTime`. +- Added table-driven unit coverage for failed, initialized, executing, and succeeded runs. Each case emits twice to prove reconciliation does not restamp the selected condition's timestamp. +- Validation: `go test ./pkg/controllers/updaterun -run 'TestEmitUpdateRunStatusMetricUsesConditionTransitionTime|TestDetermineFailureType|TestIsFailureReason' -count=1`. diff --git a/.github/.copilot/breadcrumbs/2026-10-02-0001-add-work-owner-binding-field-index.md b/.github/.copilot/breadcrumbs/2026-10-02-0001-add-work-owner-binding-field-index.md new file mode 100644 index 000000000..d0431a8e1 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-02-0001-add-work-owner-binding-field-index.md @@ -0,0 +1,39 @@ +# Add Work Owner Binding Field Index + +**Date**: October 2, 2026 00:01 AEST +**Task**: Add a composite field index for Work objects keyed by owner binding namespace and name. + +## Requirements + +1. Add the `workOwnedByBinding` custom field. +2. Format indexed values as `%s/%s`. +3. Extract the owner namespace from `WorkOwnerNamespaceLabelKey`. +4. Extract the owner binding name from `WorkOwnedByPlacementBindingLabelKey`. +5. Register the Work field extractor in `SetupWithHubControllerManager`. +6. Use the custom field index when listing Work objects by owner binding. +7. Use the existing owner-and-index custom field when listing secondary placement resource snapshots. + +## Plan + +1. Add exported field-name and value-format constants to `pkg/v1/utils/fieldindexers/hub.go`. +2. Add a Work owner-binding field extractor using the owner namespace and binding labels, while distinguishing a missing owner-namespace label from a valid empty cluster-scoped namespace. +3. Register the extractor for `placementv1alpha1.Work` in `SetupWithHubControllerManager`. +4. Update `listWorksByOwnerBinding` to query the composite custom field while retaining the Work namespace filter. +5. Update secondary placement resource snapshot retrieval to query the existing composite owner-and-index field. +6. Run the targeted field-indexer and workgenerator package tests and formatting. + +## Decisions + +1. Use the owner-binding label as the indexed binding-name source. +2. Permit an empty owner namespace only when the owner-namespace label is present, preserving support for cluster-scoped bindings. +3. Unit tests are omitted for now at the user's request. + +## Status + +Implementation complete. + +## Validation + +- `gofmt -w pkg/v1/utils/fieldindexers/hub.go pkg/v1/controllers/workgenerator/retrieval.go` +- `go test ./pkg/v1/utils/fieldindexers ./pkg/v1/controllers/workgenerator` (passed; packages currently have no test files) +- `git diff --check` (passed) diff --git a/.github/.copilot/breadcrumbs/2026-10-08-1729-selected-resource-authorization.md b/.github/.copilot/breadcrumbs/2026-10-08-1729-selected-resource-authorization.md new file mode 100644 index 000000000..ae7a3d325 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-08-1729-selected-resource-authorization.md @@ -0,0 +1,165 @@ +# Enforce selected-resource authorization + +**Date**: October 8, 2026 17:29 UTC +**Task**: Prevent placement and override APIs from using Fleet's controller permissions to access resources the requester could not access directly. + +## Requirements + +1. Enforce selected-resource authorization for ClusterResourcePlacement, ResourcePlacement, ClusterResourceOverride, and ResourceOverride create and update requests. +2. Build SubjectAccessReview requests from the admission request's complete user information: username, UID, groups, and extras. +3. Fail closed when resource resolution or authorization cannot be completed. +4. Require `get` for resources selected by name. +5. For CRP/RP label and all-of-kind selectors, require `list` in the relevant scope, without individual `get` checks for enumerated objects. +6. For namespace selection, authorize the Namespace selection using `get` by name or `list` for label/all-of-kind selection. For NamespaceWithResources, require `list` per eligible expanded namespaced resource type, without individual `get` checks for their objects. +7. Ignore resources that begin matching dynamic selectors after admission in this implementation. +8. Require `patch` for JSONPatch overrides and `delete` for Delete overrides. +9. Require Kubernetes RBAC privilege-protection checks when overrides affect Role, ClusterRole, RoleBinding, or ClusterRoleBinding resources. +10. For CRP/RP, authorize creates and reauthorize the complete selected set on spec-changing updates, even if selectors are unchanged. Overrides retain the original every-update reauthorization requirement. +11. For CRP/RP, skip selected-resource authorization on updates with semantically unchanged specs, including label, annotation, and finalizer changes regardless of deletion state. Preserve existing validation behavior. For overrides, retain the deletion/finalizer cleanup exception. +12. Enable enforcement by default and provide a temporary feature flag to disable it. + +## Plan + +### Phase 1: Authorization foundation and tests + +- [x] Add a selected-resource reviewer that creates Kubernetes SubjectAccessReviews and preserves all admission user information. +- [x] Add RESTMapper-based conversion from selector GVKs to the plural GVRs required by SubjectAccessReview. +- [x] Deduplicate identical checks within one authorization batch and use bounded concurrency for object-level reviews. +- [x] Add unit tests for allowed, denied, no-opinion, API-error, timeout, identity-copying, and duplicate-review cases. +- **Success criterion:** Authorization behavior is independently testable and every unresolved or unsuccessful review fails closed. + +### Phase 2a: ResourcePlacement authorization (current slice) + +- [x] Build RP permission checks from selectors and REST mappings without fetching source objects. +- [x] Require `get` for named selections and `list` for label/all-of-kind selections in the RP namespace. +- [x] Authorize creates and spec-changing updates; preserve existing validation and deletion cleanup behavior. +- [x] Add RP constructor, scope, selector, overlapping-review, identity, denial, and failure tests. +- **Success criterion:** RP authorization is independently testable, fails closed, and does not require live object or informer dependencies. + +### Phase 2b: ClusterResourcePlacement authorization (follow-up slice) + +- [ ] Add direct cluster-scoped resource authorization. +- [ ] Resolve selected Namespaces using live hub API reads. +- [ ] Add NamespaceOnly, NamespaceWithResourceSelectors, and NamespaceWithResources authorization. +- [ ] Add CRP webhook tests for namespace modes, scope, overlapping selectors, and failure cases. +- **Success criterion:** CRP authorization follows the agreed get/list policy without expanding the RP-only review scope. + +### Phase 3: Override authorization + +- [ ] Authorize every currently selected CRO and RO target. +- [ ] Require `patch` for JSONPatch rules, including `remove` operations on fields, and `delete` when a Delete rule removes the entire selected resource from a member cluster. +- [ ] Apply or inspect override patches sufficiently to perform correct `escalate` checks for Role/ClusterRole rule changes and `bind` checks against the resulting RoleBinding/ClusterRoleBinding role reference. +- [ ] Add CRO and RO webhook tests for patch, delete, escalate, bind, namespaced scope, update reauthorization, and fail-closed behavior. +- **Success criterion:** Overrides cannot mutate, suppress, escalate, or bind selected RBAC resources beyond the requester's direct authority. + +### Phase 4: Wiring, RBAC, and rollout + +- [ ] Add a default-enabled hub-agent feature flag that can temporarily disable selected-resource authorization. +- [ ] Pass the RESTMapper, uncached/live resource reader, and SAR client into all four validating webhooks. +- [ ] Grant the hub-agent service account `create` on `authorization.k8s.io/subjectaccessreviews`. +- [ ] Update directly related Helm values and user-facing flag documentation. +- **Success criterion:** Default installations enforce authorization, while operators have an explicit temporary rollback switch. + +### Phase 5: Verification + +- [ ] Run formatting and focused authorization, webhook, and option tests. +- [ ] Run the smallest relevant integration tests using real Kubernetes RBAC authorization. +- [ ] Add focused RP webhook authorization wiring and update-gating coverage. +- [ ] Run `make reviewable` before submission. +- [ ] Update this breadcrumb with implementation details, changed files, and validation results. +- **Success criterion:** Focused and repository-required checks pass with no authorization bypass or unrelated behavior regression. + +## Decisions + +1. Cover all four v1beta1 APIs in separate reviewable slices: RP first, then CRP, CRO, and RO. +2. Dynamic resources that match after admission are an acknowledged limitation and are not blocked because dynamic selection is a core Fleet feature. +3. Placement label and all-of-kind selection requires `list` only. Named selection requires `get` for the named object. +4. NamespaceWithResources authorizes the Namespace selection and requires `list` for each relevant namespaced GVR, without individual `get` checks for enumerated objects. +5. Delete overrides require `delete`; JSONPatch overrides require `patch`. +6. CRP/RP creates and spec-changing updates authorize the complete selected set using the current requester's identity. Unchanged-spec updates skip selected-resource authorization but still undergo existing validation. Override updates retain every-update reauthorization. +7. Finalizer removal during deletion bypasses selected-resource authorization. +8. Enforcement is enabled by default behind a temporary disable flag. + +## Non-goals + +1. Continuous reauthorization when a new resource begins matching an existing dynamic selector. +2. Persisting creator identity for controller-time authorization. +3. Replacing Kubernetes RBAC or supporting label-restricted authorization policies beyond what SubjectAccessReview reports. + +## Status + +Plan approved by the user. Phase 1 and the Phase 2a RP-only slice are complete on +`feat/selected-resource-authorization-phase-2`. Phase 2b and Phases 3-5 remain. +On October 19, the user requested a smaller review scope. The full implementation +is preserved in pushed commit `e640d9b2`; the follow-up slice retains RP only. + +## Phase 2a implementation details + +- Added `ResourcePlacementAuthorizer` and `NewResourcePlacementAuthorizer` in + `pkg/utils/authorization/placement.go`, with private mapper, reviewer, and + eligibility-checker dependencies. No reader, informer, or Namespace resolution + is needed. An explicit RP/CRP type switch routes RP to its typed resolver; + the CRP branch returns a clear not-implemented error for the follow-up slice. +- Build get checks for named selectors and list checks for label/all-of-kind + selectors in the RP namespace. No source objects are fetched. Reuse the Phase 1 + reviewer's identity preservation, deduplication, and bounded concurrency. +- Use explicit Allowed, Denied, and Indeterminate decisions. Denials take + precedence over additional review errors; malformed results fail closed. +- Add an optional RP webhook authorizer through shared placement validation. + Authorize creates and spec-changing updates only. Unchanged-spec updates still + run existing validation, and the old invalid-placement deletion exception is + unchanged. Authorizer unit tests cover scope, selectors, denials, and SAR errors; + shared validator tests cover admission decision mapping. The dedicated RP + webhook authorization test file was removed at the user's request; its wiring + and update-gating coverage is deferred. +- Defer CRP authorization and its tests. Its handler passes nil to the shared + authorization parameter, preserving existing behavior. The controller's resource + selector resolver is unchanged. +- Production registration still does not inject the RP authorizer. Default-on + enforcement, rollout wiring, and SAR RBAC remain Phase 4. + +## Phase 2a validation + +- Focused race tests passed for `pkg/utils/authorization`, `pkg/utils/validator`, + `pkg/webhook/clusterresourceplacement`, and `pkg/webhook/resourceplacement`. +- Existing CRP webhook tests passed unchanged. +- `go vet` passed for the authorization, validator, and placement webhook packages. +- `go build ./cmd/hubagent/...` passed. +- Before committing, `make lint GOLANGCI_LINT=/home/weiweng/go/bin/golangci-lint` + passed repository-wide with the installed repository-version linter (v2.13.2). +- Repository-configured lint passed with `--new-from-rev=main`, + `--max-issues-per-linter=0`, and `--max-same-issues=0` for the affected packages. + Unfiltered package lint also reports pre-existing issues in the CRP mutating + webhook tests; these were not changed. +- `git diff --check` passed. Go commands use + `MS_GOTOOLCHAIN_ALLOW_NON_LOCAL=1 GOTOOLCHAIN=auto`, as in Phase 1. +- Full `make reviewable` and real Kubernetes RBAC integration remain Phase 5 + verification, before PR submission. + +## Phase 1 implementation details + +- Added `pkg/utils/authorization` with a reusable, stateless batch Reviewer. +- Preserved username, UID, groups, and deep-copied extras in every SubjectAccessReview. +- Added RESTMapper-based GVK-to-GVR conversion for correctly pluralized authorization attributes. +- Deduplicated identical attributes within each authorization batch; separate calls always perform fresh reviews. +- Limited concurrent SubjectAccessReviews per request, collected results in stable input order, and recorded explicit failures for reviews not submitted after request cancellation. +- Returned structured results that aggregate explicit denials separately from joined evaluation, API, and malformed-response errors; conclusive denials take precedence while errors remain available for logging. +- Added focused unit coverage for resource mapping, identity propagation, aggregated denial and error results, timeout handling, deduplication, and bounded concurrency. + +## Phase 1 validation + +- `MS_GOTOOLCHAIN_ALLOW_NON_LOCAL=1 GOTOOLCHAIN=auto go test -race ./pkg/utils/authorization` passed. +- `MS_GOTOOLCHAIN_ALLOW_NON_LOCAL=1 GOTOOLCHAIN=auto go vet ./pkg/utils/authorization` passed. +- Automatic toolchain selection was required because `go.mod` requires Go 1.26.6 while the installed local Microsoft Go toolchain is 1.26.5. + +## References + +- `pkg/utils/validator/placement.go` - shared CRP/RP admission validation. +- `pkg/webhook/clusterresourceplacement` - CRP mutating and validating webhooks. +- `pkg/webhook/resourceplacement` - RP validating webhook. +- `pkg/webhook/clusterresourceoverride` - CRO validating webhook. +- `pkg/webhook/resourceoverride` - RO validating webhook. +- `pkg/utils/controller/resource_selector_resolver.go` - current placement selector resolution semantics. +- `cmd/hubagent/options/featureflags.go` - hub-agent feature flag conventions. +- `charts/hub-agent/templates/rbac.yaml` - hub-agent service account permissions. +- No applicable specification directory exists under `.github/.copilot/`, and no selected-resource authorization entry was found in the domain knowledge files. diff --git a/.github/.copilot/breadcrumbs/2026-10-16-0232-placementresourcesnapshot-integration-test-plan.md b/.github/.copilot/breadcrumbs/2026-10-16-0232-placementresourcesnapshot-integration-test-plan.md new file mode 100644 index 000000000..26a94844a --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-16-0232-placementresourcesnapshot-integration-test-plan.md @@ -0,0 +1,1166 @@ +# Placement Resource Snapshot Manager: Integration and Unit Test Plan + +**Date**: October 16, 2026 02:32 AEDT +**Revised**: October 24, 2026 07:20 AEDT, for leftover snapshot detection and UID-based snapshot names (see +[Revision: leftover snapshots and UID-based names](#revision-leftover-snapshots-and-uid-based-names)). +**Revised**: October 26, 2026 16:20 AEDT, for latest-index checks and secondary snapshot GC, and to add a unit test +plan (see [Revision: latest-index checks, secondary GC, and unit tests](#revision-latest-index-checks-secondary-gc-and-unit-tests)). +**Revised**: October 27, 2026 02:13 AEDT, for Service and Job sanitization (PP-13 and PP-14; see +[E. Manifest sanitization](#e-manifest-sanitization-snapshotresource)). +**Revised**: October 27, 2026 03:35 AEDT, for the API server fallback on cache misses in name-based selection (see the +October 27, 03:19 [Status](#status) entry). +**Task**: Plan additional envtest-based integration tests for `pkg/v1/managers/placementresourcesnapshot/`. +Revised October 26: unit tests are planned as well. + +## Baseline + +Measured with `go test ./pkg/v1/managers/placementresourcesnapshot/... -coverprofile` (envtest 1.33.0): **72.9%** of statements. + +| Function | Coverage | +|---|---| +| `resources.go` `retrieveResourceByName` | 47.1% | +| `resources.go` `retrieveResourcesByLabelSelector` | 53.1% | +| `resources.go` `snapshotResource` | 72.7% | +| `resources.go` `lookUpGVKGVRAndNamespace` | 75.0% | +| `resources.go` `retrieveAndHashSelectedResources` | 77.3% | +| `uniquename.go` (both functions) | 20.0% | +| `snapshots.go` `placementPolicyOwnerLabelVal` | 50.0% | +| `ops.go` `retrieveLatestSnapshot` / `isSnapshotUpToDate` / `cleanUpDanglingSecondarySnapshots` | 74–79% | + +The existing suite (`manager_integration_test.go`, ~3.5k lines) covers the snapshot lifecycle for both scopes: +create, up-to-date, stale, split by count/size, oversized resource, dangling secondaries, and default-limit GC. +Every test selects only ConfigMaps/Deployments in the `app` namespace, so these paths are never run: + +- The resource eligibility checks (new, uncommitted): namespace, GVK, and object-level checks. +- Selector validation errors (scope/namespace mismatches, missing name/label selector, bad label selector, unknown kind). +- The informer (cache) read path: the suite's informer manager never registers or starts any informer. +- Manifest sanitization of annotations (last-applied, deployment revision), plus status stripping when status is set. +- Deduplication across selectors, cluster-scoped kinds (Namespace, ClusterRole), and label selections that match nothing. +- Long or dotted policy names (hashed snapshot names, truncated owned-by label). +- Custom `ResourceRevisionHistoryLimit`, GC of multi-group snapshots, and stale-cache branches in `ops.go`. + +## Layout + +All files share the existing suite in `suite_test.go`. As implemented: + +- `ops_gc_integration_test.go`: the former `manager_integration_test.go` (renamed at 02:48), plus a + `Describe("GC variants")` for P2. The P2 specs were planned for a separate `gc_integration_test.go`. +- `resources_integration_test.go`: selection, eligibility, dedup, and sanitization (P0-A to P0-E). +- `corner_cases_integration_test.go`: stale-cache and inconsistent-state branches, and long and dotted names (P1). + It replaces the planned `ops_integration_test.go` and `snapshots_integration_test.go`. +- `utils_integration_test.go`: all shared helpers (consolidated at 06:52). It replaces the planned + `helpers_integration_test.go`; the helper names differ from the original sketch. +- The planned `resources_cache_integration_test.go` (P0-F) was not created; see section F. + +## P0: `resources.go` + +### A. Selector validation (DescribeTable, both scopes where relevant) + +Each entry creates the policy, calls `SnapshotResourcesIfStale`, asserts the error category and message, +and asserts no snapshot was created. + +| # | Setup | Want | +|---|---|---| +| A1 | PP selector `namespace` differs from the PP namespace | User | +| A2 | PP selector `namespace` equals the PP namespace | Success (positive control) | +| A3 | PP lives in `default` (and in a `kube-*` namespace) | User (policy namespace ineligible) | +| A4 | CPP selects a ConfigMap in `kube-system` | User (selector namespace ineligible) | +| A5 | CPP selects a namespaced kind without `namespace` | User | +| A6 | CPP selects ClusterRole with `namespace` set | User | +| A7 | PP selects ClusterRole | User (cluster-scoped via PP) | +| A8 | Denied GVK: `v1/Pod`, `coordination.k8s.io/v1/Lease`, `placement.kubefleet.dev/v1alpha1/PlacementPolicy` | User | +| A9 | Neither `name` nor `labelSelector` (allowed by CEL) | User | +| A10 | Invalid label selector (`In` with no values; unknown operator) | User | +| A11 | Unknown kind (typo) | Currently Unexpected (see open question 1) | +| A12 | Named resource does not exist; then create it | User error ("selected resource does not exist"), then success on retry (an APIServer error before October 27) | +| A13 | In-memory accessor with empty selectors (CRD enforces MinItems=1); nil accessor | Unexpected | + +### B. Object eligibility, name-based (explicit selection fails) + +Use a fresh namespace, because envtest runs no controller-manager and these objects must be created by hand. + +- ConfigMap `kube-root-ca.crt`, ServiceAccount `default`, a Secret of type `kubernetes.io/service-account-token`: + each returns a User error whose message carries the reason. Control: an Opaque Secret succeeds. +- A ConfigMap with `ownerReferences` returns a User error. +- A terminating ConfigMap (finalizer + delete) returns a User error. +- A CPP selecting Namespace `kube-system` by name returns a User error ("selected namespace is not eligible"). +- A CPP selecting a fresh Namespace by name succeeds. Verify the manifest: status stripped, spec matches the live object. + +### C. Object eligibility, label-based (ineligible objects are skipped) + +- A fresh namespace holds eligible ConfigMaps plus `kube-root-ca.crt`, an owned ConfigMap, and a terminating ConfigMap, + all carrying the same label. The snapshot contains only the eligible ones. +- Dynamic case: add an ownerRef to an eligible ConfigMap. The snapshot goes stale and the new one excludes it. + Remove the ownerRef and the ConfigMap is included again. +- A CPP selects Namespaces with `kubernetes.io/metadata.name In [default, kube-system, kube-public, ]`. + Only `` is snapshotted. +- A label selector that matches nothing creates a primary snapshot with zero resources (pin; open question 4). + Adding a matching object makes it stale and produces a new snapshot. + +### D. Deduplication and ordering + +- The same ConfigMap selected by name and by label appears once. +- An HPA selected via `autoscaling/v1` and then `autoscaling/v2` appears once with `Identifier.APIVersion == v1` + (the first selector wins). Swapping the selector order changes the hash, and the new snapshot carries v2. +- Reordering selectors for distinct resources keeps the same hash, so `SnapshotResourcesIfStale` returns the + existing snapshot (no new index). +- A CPP selecting cluster-scoped kinds (ClusterRole, Namespace) plus namespaced resources across two namespaces: + resources are sorted by `group/kind/namespace/name`. + +### E. Manifest sanitization (`snapshotResource`) + +Assert on raw JSON (`map[string]any`) so that the absence of keys is checked, not just typed zero values. + +- ConfigMap with `kubectl.kubernetes.io/last-applied-configuration` plus a custom annotation: only the custom one remains. +- ConfigMap with only last-applied: `metadata.annotations` is absent. +- Deployment with a `deployment.kubernetes.io/revision` annotation and a status set via the status subresource: + both are stripped, and `uid`, `resourceVersion`, `generation`, `creationTimestamp`, `managedFields` are absent. +- Finalizers on a selected object: pin the current behavior, which is that they are retained (open question 2). +- Changes that must not trigger a new snapshot: + - Updating only the last-applied annotation keeps the hash unchanged. + - A status-only update keeps the hash unchanged. +- Services and Jobs (added October 27). `snapshotResource` strips the values that the hub API server fills in, as + they would conflict with the ones each member cluster's API server generates. `TestSnapshotResource` (unit) covers + every branch with hand-built objects, including the paths that change nothing (a headless Service, a Job with + `manualSelector: true`). The two contexts below check the stripping against what envtest's API server actually + fills in, and that re-creating the object with the same spec does not produce a new snapshot. They are appended + to the end of the first `Describe`, so the existing positional IDs do not shift. + - **PP-13, `manifest sanitization (services)`**: create a `LoadBalancer` Service with two ports and + `externalTrafficPolicy: Local`, so that the API server fills in `clusterIP`, `clusterIPs`, a `nodePort` per + port, and `healthCheckNodePort`. Assert that the live object carries all of them, so the spec can't pass + trivially. + 1. Snapshot. These keys are absent from the raw manifest; the rest of the spec equals the live spec with them + cleared, and the metadata keeps the user's labels. + 2. Delete the Service and re-create it with the same spec. The API server fills in new values. + 3. `SnapshotResourcesIfStale` returns index 0, and there is no index 1. + - **PP-14, `manifest sanitization (jobs)`**: create a Job without `manualSelector`. Assert that the API server + added `batch.kubernetes.io/controller-uid` to the selector and to the pod template labels, and the legacy + `controller-uid` to the pod template labels. + 1. Snapshot. Neither `controller-uid` nor `batch.kubernetes.io/controller-uid` is left in the selector or the + pod template labels, and the pod template has no `creationTimestamp`. The user's labels and the `job-name` + labels are kept, and the rest of the spec equals the live spec. + 2. Delete the Job and re-create it with the same spec. It gets a new UID, and so new controller-uid labels. + 3. `SnapshotResourcesIfStale` returns index 0, and there is no index 1. + - Envtest constraint: `batch/v1` Jobs default to orphan propagation, which adds the `orphan` finalizer, and + envtest runs no garbage collector to remove it. Jobs are therefore deleted with background propagation: + `deleteObjectsAndWait` takes optional `client.DeleteOption`s. + +### F. Informer (cache) path, with a dedicated `Manager` + +**Not implemented as planned.** At 07:30 the shared suite started reading Deployments through the cache instead (see +Status). That covers name-based and label-based reads from the cache, and mixed policies that read ConfigMaps from +the API server. Since October 27, a name-based cache miss is confirmed with a read from the API server (see the +October 27, 03:19 Status entry): PP-2b covers a name in neither the cache nor the API server, and the unit test +`TestRetrieveResourceByName` covers the fallback itself and an unsynced informer. These planned checks are still +uncovered: identical hashes from the cache and API server paths for the same policy, and ineligible objects read +through the cache. + +The original plan: + +- Build a separate `informer.Manager` and `AddStaticResource` for `v1/ConfigMap` and `v1/Namespace`. + Then call `Start` and `WaitForCacheSync`, and construct a second `Manager` with `New(hubMgr, ...)`. + It does not need to be added to `hubMgr`. +- Name-based and label-based selection through the cache must yield resources and a hash identical to the + API-server-path manager for the same policy. This is the key invariant, and it also guards the informer transform. +- A name not in the cache returns an APIServer error. (Superseded on October 27: a cache miss falls back to the API + server, and a name missing there too is a User error.) +- One representative ineligible object each for name-based (error) and label-based (skip). +- A mixed policy reads the ConfigMap from the cache and the Deployment via the API server fallback. +- An informer that is registered but not started falls back to the API server and succeeds. +- After a ConfigMap update, `Eventually` the hash changes (cache convergence). + +## P1: `ops.go` stale-cache and inconsistency branches + +**Technique**: snapshot objects without the owned-by label are not indexed (see `fieldindexers/hub.go`), so the cached +field-index lists never see them, while the uncached quorum reads by name do. This drives the +"cache is stale" branches deterministically. + +1. A decoy named `uniqueNameForPrimary(P, latest+1)` makes `SnapshotResourcesIfStale` return Transient. + Also pin that `SnapshotResourcesIfNoSnapshotExists` fails the same way (open question 3). +2. A decoy at the primary name for index 0 on a fresh policy returns an APIServer error (AlreadyExists) and creates no + secondaries. After the decoy is removed, the call succeeds. +3. A decoy at the secondary name `(0, 1)` on a multi-group policy returns an APIServer error and the primary is not created. + After the decoy is removed, the call succeeds. +4. A labeled secondary at the next index plus an unlabeled primary decoy at the same index returns Transient, + and the secondary is **not** deleted. +5. A dangling secondary with a finalizer returns Transient repeatedly while it is terminating. + After the finalizer is removed, creation proceeds. +6. An extra labeled secondary beyond the count label at the latest index: the call returns exactly `count` + snapshots, sorted by sub-index, with no error. (Revised October 26: the extra must be owned by the policy, since + an ownerless one is now a leftover, and it is also enqueued for GC and deleted; see SC-6 in the October 26 + revision.) +7. Deleting one secondary after creation returns Unexpected ("fewer sub-indexed ... than the count label"). +8. Hand-crafted labeled primaries with a count of `abc`, a count of `0`, or an index of `abc` return Unexpected. + The index case needs at least 2 primaries so that the sort comparator runs. (Revised October 26: the count-label + messages were reworded, and a non-contiguous sub-index case is added; see SC-8 in the October 26 revision.) +9. Concurrency: N goroutines call `SnapshotResourcesIfStale` on one policy. + - Exactly one snapshot exists per index. + - Any errors are retryable (Transient or APIServer). + - The policy `Eventually` converges to up-to-date. + +## P1: Long and dotted names + +- A PP name of about 240 characters (owned-by label truncated and hashed; snapshot names truncated), with + multi-group snapshots: + - Names match `uniquename.PrimaryPlacementResourceSnapshotName` and `SecondaryPlacementResourceSnapshotName`, + computed with the policy UID (revised October 24; names were `uniqueNameFor*()` before). + - The label matches `placementPolicyOwnerLabelVal`. + - The annotation keeps the full name. + - The stale path produces index 1, and GC plus the binding in-use check work, since they rely on the annotation. +- A dotted name (`app.policy.v1`) produces dot-free snapshot names, and the flow works end to end. +- Two long names sharing their first 48 characters do not interfere: each policy sees only its own snapshots. +- One CPP variant. + +## P2: GC variants + +- `ResourceRevisionHistoryLimit=1`: the previous snapshot is GC'd after the next retrieval. + With a limit of 5, five snapshots are kept. Lowering the limit from 5 to 1 triggers bulk GC. +- GC of multi-group snapshots removes the primary and all of its secondaries. + Existing GC tests cover only single-group snapshots. (Revised October 26: GC-2 adds its extra secondary only once + that index is no longer the latest, since an extra at the latest index is now collected on its own.) +- A terminating snapshot (with a finalizer) is not re-enqueued, and GC completes once the finalizer is removed. + +## Revision: leftover snapshots and UID-based names + +Added October 24. It revises the plan for the snapshot manager changes that handle placement policies deleted and +re-created with the same name. + +### What changed in the code + +- `retrieveLatestSnapshot` (`ops.go`) now runs `checkForInconsistentResourceSnapshots` (`gc.go`) on the primary + snapshots, right after sorting them. Both entry points are affected. For each primary: + - **Owned by a same-named policy with another UID** (the previous policy is still being garbage-collected): + Transient, "found a placement resource snapshot owned by a different placement policy of the same name (e.g., one + that has been deleted); waiting for garbage collection to complete". The scan stops. + - **No owner references** (left over from a policy deleted with orphan propagation): a GC request is enqueued, + unless the snapshot is already marked for deletion. Either way, after the scan, the call fails as Transient, + "orphaned resource snapshots are detected; waiting for garbage collection to complete"; the error names the + orphaned snapshots. + - **Other owner references** (a differently named owner, an extra owner, the wrong kind): User, "found a placement + resource snapshot (NAME) that has unexpected owner references". The name is in the message, so it reaches the + policy status. + - While blocked, no history GC requests are enqueued and no snapshot is created. +- `garbageCollect` no longer requires the snapshots at the primary's index to be owned by the policy; orphaned groups + are deleted too. The comment at the top of `garbageCollect` explains why this is safe in normal operations: the block + above, plus `cleanUpDanglingSecondarySnapshots`. As a sanity check against heavy cache lag, it skips (and logs) any + listed snapshot that is another primary, or whose policy owner (or lack of one) differs from the primary's. Since + snapshot names include the policy UID, the API server no longer stops two policy generations from sharing an index. +- `isSnapshotInUseByAnyPlacementBinding` takes the policy's namespaced name from the GC request. + - A snapshot without an owner reference to the policy is never in use, so orphans are collected even when a + binding still names them. + - Binding owners are no longer checked; any binding that names the policy and the snapshot keeps it. + - So "in use" never produces an error any more. GC returns nil and the request is dropped, not retried, until the + next retrieval enqueues it again. +- Snapshot names moved to the exported `uniquename` package. Primary and secondary names always end in a + 12-character hash of the policy name, index, sub-index (secondaries only), and **policy UID**. Long or dotted names + are truncated and sanitized; there is no longer a separate "hashed" form. +- Known gap, accepted and documented on `checkForInconsistentResourceSnapshots`: an ownerless secondary without a + primary is not detected. It lingers until the policy creates snapshots at that index. (Revised October 26: the gap + is narrower; ownerless secondaries at the latest index are now detected. See LO-6.) + +### Existing tests: already adapted, no action needed + +The current tests pass: `go test -race ./pkg/v1/managers/placementresourcesnapshot/...` took 195s, and the +`uniquename` package about 3s. + +| Area | Adaptation | +|---|---| +| Every name expectation | Uses `uniquename.*` with the policy UID, computed after the policy is created | +| Indexed decoys (SC-1, SC-8) | Owned by the policy (`createOwnedDecoySnapshot`, `decoyOwnerRefs`); without an owner they would now count as leftovers and be garbage-collected | +| NM-1 name length boundaries | 216, 217, 218, 219, and 253 characters (previously 229 to 232, and 253); "hashed" is now "truncated" | +| `autorollout` and `stagedupdaterun` suites | Their name helpers take the policy instead of its name | + +Unindexed decoys (no owned-by annotation) are never scanned and need no owner references. + +### Removed + +- **GC-5, "retryable GC failures" and "non-retryable GC failures"**, with the constants `gcStaleOwnerInUseErrMsg`, + `gcMissingOwnerInUseErrMsg`, and `staleOwnerUID`. The errors they checked no longer exist. They are already gone + from `ops_gc_integration_test.go`. + - They were the only integration coverage of two things, both re-added below: + - a binding blocking GC until it goes away; + - the GC worker's retry (backoff) and drop paths in `Start`. +- The out-of-scope entry for "the GC owner-UID mismatch case" (rewritten in Out of Scope). + +### Unit test coverage (do not duplicate in integration) + +(Revised October 26: this table lists the unit tests as of October 24, plus the two added October 27 on `main`. The +full unit test plan, including new tests, is in the [October 26 revision](#unit-test-plan).) + +| Unit test | Covers | +|---|---| +| `uniquename/uniquename_test.go` | Name format, a different UID giving a different name, dots, truncation boundaries, uniqueness | +| `TestCheckForInconsistentResourceSnapshots` | Every owner variant (stale UID, orphan, other name, extra owner, wrong kind, namespaced), scan order, returned leftovers, enqueued requests, orphans already marked for deletion (still leftovers, not re-enqueued) | +| `TestIsSnapshotInUseByAnyPlacementBinding` | Orphan bypass, other-name and wrong-kind owners, cross-namespace bindings, list failure | +| `TestGarbageCollect` | Owned and orphaned groups, in use, re-created primary, primary not found, Get failure, skipped snapshots at the same index (another primary, a different or missing policy owner) | +| `TestStart` | The GC worker loop: retry with backoff, drop on Unexpected errors, stop on context cancellation (added with this revision; see below) | +| `TestSnapshotResource` | Service and Job sanitization in `snapshotResource`: assigned cluster IPs and node ports, a headless Service, a Job with and without `manualSelector`, and that the input is not modified (added October 27) | +| `TestRetrieveResourceByName` | Name-based retrieval: a cache hit; the API server fallback on a cache miss (both scopes); a name missing from both (a User error wrapping NotFound); non-NotFound errors from either side; an unsynced informer (added October 27) | + +**Not covered anywhere:** `IsSnapshotSynchronized` (a gap that predates this change, including its owner-UID check). +At planning time, the `Start` worker loop was not covered either; `TestStart` now covers it. + +The integration cases below therefore take one representative per owner variant. They focus on what unit tests +can't show: real field indexes and the cache, the running GC worker, recovery once the block clears, and both entry +points. + +### Envtest constraints + +- **There is no kube-controller-manager, so no Kubernetes GC.** + - Deleting a policy leaves its snapshots in place with the old owner UID. That is exactly the "previous policy + still being collected" state. Deleting the old snapshots by hand stands in for that collection finishing. + - Orphan propagation can't complete: the policy would hang on the `orphan` finalizer. Simulate it by removing the + snapshots' owner references, then deleting the policy normally. +- **Some helpers don't work while a previous policy's snapshots exist.** `snapshotResourcesAndExpectErr` and + `verifyNoPlacementResourceSnapshots` match by the owned-by label, so they would find those snapshots. Use + `verifyNoPrimarySnapshotAtIdx`, which builds the name with the policy's current UID, or a UID-aware variant. + +### Cases to add + +- **Placement:** LO-1 to LO-5 go in a new `Describe("corner cases (snapshots of previous placement + policies)")` in `corner_cases_integration_test.go`. GC-5 goes back into `Describe("GC variants")`, reusing the freed + slot. +- Wherever a call should fail, check **both entry points**. + +**LO-1. Re-created policy; the previous policy's snapshots are not collected yet** (PP) + +1. Create the policy and a snapshot at index 0. +2. Delete the policy and re-create it with the same name. +3. Both entry points fail as Transient ("owned by a different placement policy of the same name"): + - nothing is created under the new UID's names (`verifyNoPrimarySnapshotAtIdx(0)`); + - the old snapshot is untouched; + - a paused manager's queue stays empty, so no history GC happens while blocked. +4. Delete the old snapshot by hand. The call then creates index 0, owned by the new UID, with a name different from + the old index-0 name. + +**LO-2. Re-created policy; the previous policy's snapshots are orphaned** (PP and CPP, looped like SC-1) + +1. Create the policy and snapshots at index 0 and 1, with two groups at index 1 so secondaries are covered. +2. Create a binding with no owner that names the old index-1 primary (an orphaned binding). +3. Remove the owner references from all snapshots, delete the policy, and re-create it. +4. With a paused manager, the call fails as Transient ("orphaned resource snapshots are detected"). The drained queue + holds exactly one request per old primary, each carrying the policy's namespaced name. +5. With the suite manager, every old snapshot, primary and secondary, is `Eventually` gone, even though the binding + still names one. +6. The call then `Eventually` succeeds with index 0 under the new UID. No old name is reused, and the binding is left + untouched. + +**LO-3. An orphaned primary pending deletion** (PP) + +1. Create the policy and a snapshot at index 0. Give the snapshot a finalizer, orphan it, and re-create the policy. + (Implemented with a single snapshot; the LO-2 setup, as first planned, adds nothing here.) +2. Once the GC worker has marked the snapshot for deletion, calls stay Transient ("leftover resource snapshots are + detected", `Consistently`), and nothing is created. +3. Remove the finalizer; the call then succeeds. + +**LO-4. Unexpected owner references** (PP) + +1. Create the policy and a snapshot at index 0. +2. Add a second owner reference, to the selected ConfigMap, on the primary. +3. Both entry points fail as User errors whose message contains the snapshot name: + - a change to the selected resources still creates nothing; + - no GC request is queued (paused manager). +4. Remove the extra owner reference. The call returns index 0 again, or creates index 1 if the resources changed. + +The other owner variants are unit-tested; one representative is enough here. + +**LO-5. Leftover secondaries without a primary** (PP; pins the documented limitation) + +1. Before creating the policy, create two ownerless indexed secondaries for its name, named with an old UID: index 0 + sub-index 1, and index 1 sub-index 1. +2. Create the policy. The leftovers don't block the first call. That call removes the index-0 leftover and fails as + Transient ("cleaned up dangling secondary ..."); the next call creates index 0. The index-1 leftover is still there. +3. Change the selected resources. The index-1 leftover is cleaned up the same way, then index 1 is created. + +**GC-5. Snapshots in use by placement bindings** (PP; replaces the removed GC-5) + +1. Create a policy with `ResourceRevisionHistoryLimit` 1 and a snapshot at index 0, plus a binding owned by the + policy that names index 0. +2. Create index 1, then call again; that retrieval sees both indices and enqueues index 0. +3. Once the GC worker has picked up the request (its workqueue is empty), index 0 is kept (`Consistently`), and + `NumRequeues` for its request stays 0: the request is dropped, not retried. +4. With a paused manager, `garbageCollect` returns nil and index 0 is kept. +5. Point the binding to index 1. The next retrieval enqueues index 0 again, and it is collected. +6. A binding with no owner that names an outdated snapshot also keeps it, since binding owners are not checked. + +**GC worker loop (`Start`) — a unit test, not integration.** Add `TestStart` to `gc_test.go` (or a new +`manager_test.go`), using the fake client with an interceptor: + +- A retryable failure (a Delete error) is retried with backoff (`NumRequeues` of at least 1). Once Delete succeeds, + the snapshots are gone and `NumRequeues` drops back to 0. +- An unexpected failure (a malformed index label) is dropped: `NumRequeues` stays 0, and later requests are still + processed. +- Canceling the context stops the loop. + +Integration can no longer produce a retryable GC error deterministically without an admission policy, which is why +this moves to a unit test. + +**Optional (a gap that predates this change):** cover `IsSnapshotSynchronized`, as a unit or integration test: + +- current and stale hashes; +- a non-primary snapshot; +- a snapshot owned by another policy; +- a stale owner UID (Transient). + +## Revision: latest-index checks, secondary GC, and unit tests + +Added October 26. It revises the plan for the working-tree changes to `ops.go` and `gc.go` (not yet committed), and +adds a unit test plan for the package; earlier revisions left most of the package to integration tests. + +### What changed in the code + +- `retrieveLatestSnapshot` (`ops.go`) now ends in a new `allSnapshotsFor`, which always lists every snapshot at the + latest primary's index. The fast path that returned the primary alone when its count label was `1` is gone. On + that listing, in order: + 1. Sort by sub-index; a malformed sub-index is Unexpected (unchanged). + 2. **`checkForInconsistentResourceSnapshots` now also runs on the listing**, so the owner checks cover secondaries + at the latest index: an ownerless one is orphaned (Transient "orphaned resource snapshots are detected", and + enqueued for GC unless it is pending deletion); one owned by a stale policy UID is Transient; one with + unexpected owners is a User error. + 3. **New:** the given primary must be the first listed snapshot (same name and UID); otherwise Transient + ("…is not found as the first snapshot listed at its index; the client cache might have changed between reads"). + 4. The count label is parsed; the messages now read "count label **value** to integer" and "count label **has a + value** less than 1" (Unexpected). + 5. Fewer snapshots than the count: Unexpected (unchanged). + 6. **Extras beyond the count are now enqueued for GC** (`enqueueLeftoverResourceSnapshotsForGC`; extras pending + deletion are skipped), instead of only being ignored. The call still succeeds and logs a warning. + 7. **New:** the returned snapshots must have contiguous sub-indices `0` to `count-1`; otherwise Unexpected + ("…the sub-indices are not contiguous"). The contents-hash check follows (unchanged). + - Ordering: history GC (`enqueueOldResourceSnapshotsForGC`) runs **before** `allSnapshotsFor`. The two never + overlap: history GC only takes primaries at older indices (the latest is excluded), while `allSnapshotsFor` only + enqueues snapshots at the latest index, in practice always secondaries. So a block at the latest index leaves + history GC alone, whereas leftover primaries block it, as they would skew the history itself (Q10, resolved). +- `garbageCollect` (`gc.go`) accepts requests for **secondary** snapshots. After the shared Get, UID, and + pending-deletion checks, a target whose sub-index is not `0` is deleted right away, with a UID precondition and no + in-use check. Requests for primaries behave as before. Log messages on the shared path no longer say "primary". +- `cleanUpDanglingSecondarySnapshots` deletes with a UID precondition, as GC already did, and (since 16:31) ignores + NotFound, like GC (Q11, resolved). +- Refactor, no behavior change: `enqueueResourceSnapshotForGC` is the single place that builds GC requests; + `checkForInconsistentResourceSnapshots`, `enqueueOldResourceSnapshotsForGC`, and the new + `enqueueLeftoverResourceSnapshotsForGC` call it. +- Docs: the rationale at the top of `garbageCollect` was restored and extended to secondaries, and + `checkForInconsistentResourceSnapshots` documents the narrower known gap (ownerless secondaries at indices other + than the latest). + +### Existing tests: adapted October 26 (done) + +| Spec | Change | +|---|---| +| SC-6 | The extras are owned decoys (`createOwnedDecoySnapshot`); both "extra" specs also assert that the extra is deleted. The count-1 spec was renamed (it was "should ignore extra secondary snapshots when the count label is 1") | +| SC-8 | The two count-label messages match the new wording | +| GC-2 | The extra secondary at index 1 is added after index 2 exists, so it survives until index 1 is collected; the index-0 expectation no longer lists it | +| SC-7, LO-5 | Comments corrected: the manager can delete a single secondary, but never one within the count; leftover secondaries are detected at the latest index only | + +Validation: `ginkgo -p` (without `--race`) passed 347 of 347 specs, none skipped, in about 38s. The `gc_test.go` unit +tests pass. + +### Cross-reference: plan items vs. the change + +| Plan item (spec) | Affected? | Action | +|---|---|---| +| P1-1 to P1-3, P1-9 (SC-1 to SC-3, SC-9) | No | None | +| P1-4, P1-5 (SC-4, SC-5): dangling-secondary cleanup | Only the UID precondition, which envtest can't observe deterministically | U3, U8 | +| P1-6 (SC-6): extras | Yes | Adapted; extend (SC-6 below) | +| P1-7 (SC-7): fewer than the count | No; the primary-first check passes before the count check | None | +| P1-8 (SC-8): malformed labels | Messages reworded; new contiguity check | Adapted; extend (SC-8 below) | +| GC-1, GC-3 to GC-6 | No; GC-3's queue assertions only involve primaries | None; the SC-6 extension covers requests for secondaries | +| GC-2: multi-group GC | Yes | Adapted | +| LO-1 to LO-4 | No; they are blocked at the primary level, before `allSnapshotsFor` | None; the owner variants on secondaries go to U1 | +| LO-5, and the known gap | The gap is narrower; LO-5's leftovers are never at the latest index (none exists yet, or they sit at the next index), so it still holds | Comment adapted; add LO-6 | +| `TestCheckForInconsistentResourceSnapshots` | No; the function never reads sub-indices, and its fixtures carry none | None; secondaries are covered through U1 | +| `TestGarbageCollect` | Every case targets a primary | U2, U3 | +| `TestIsSnapshotInUseByAnyPlacementBinding`, `TestStart` | No | None | +| The primary missing from the latest-index listing | Not reproducible in envtest (the cache must change between two reads) | U1 | + +### Integration cases to add + +(Implemented at 16:40; see Status.) + +Wherever a call should fail, check both entry points (as before). + +**SC-6 (extend). Extras beyond the count label** (PP; in the existing context, before the suite manager's call +collects the extra) + +1. With a paused manager, the call succeeds with the `count` snapshots, and the drained queue holds exactly one + request: for the extra, with its namespaced name and UID and the policy's namespaced name. No request targets the + primary or the sub-index-1 secondary. +2. A second extra with a finalizer, marked for deletion: the call still succeeds, since an owned extra does not block + (unlike an ownerless leftover), and no request is enqueued for it. Remove the finalizer; it is gone. + `createOwnedDecoySnapshot` takes no finalizers, so this needs a variant or a follow-up update. + +**SC-8 (extend). Non-contiguous sub-indices** + +- A primary with a count of `2`, plus a secondary at sub-index `2` and none at `1`: Unexpected, "the sub-indices are + not contiguous". Append it to the end of the table, so that `staleMalformedIndexTestCaseIdx` stays valid. + +**LO-6. An orphaned secondary at the latest index** (PP; a new context in `Describe("corner cases (snapshots of +previous placement policies)")`, after LO-5) + +The suite manager's GC worker would collect the leftover right away and race the assertions, so steps 3 to 5 use a +paused manager. + +1. Create the policy and a two-group snapshot at index 0. +2. Create an ownerless indexed secondary at index 0, sub-index 1, named with an old UID. It duplicates a sub-index, + mimicking a previous policy's secondary that slipped past the dangling-secondary cleanup under cache lag. +3. Both entry points fail as Transient ("orphaned resource snapshots are detected"), and the error names the + leftover. The drained queue holds exactly one request, for the leftover. +4. Change the selected resources: the calls still fail the same way, and nothing is created at index 1 + (`verifyNoPrimarySnapshotAtIdx(1)`). +5. Process the request with the paused manager's `garbageCollect`: only the leftover is deleted, and the index-0 + primary and its owned secondary keep their UIDs. +6. `SnapshotResourcesIfNoSnapshotExists` returns the original index-0 set (stale), and `SnapshotResourcesIfStale` + creates index 1. + +That history GC still runs while the latest index is blocked (Q10) is covered in U5, not here. + +### Unit test plan + +(Implemented at 16:59, U1 to U15; see Status.) + +Unit tests cover branch-level behavior and what envtest can't observe: delete options, call order, divergence between +the cache and the API server, and a primary missing from a listing. Integration tests keep the end-to-end flows. +Conventions as in `CLAUDE.md`: table-driven, `cmp.Diff` and no assert libraries, `want` and `got`, and messages of +the form `"Func(...) = %v, want %v"`. + +**Techniques and helpers** + +- **Field indexes.** `newFakeHubClientWithIndexes` (`gc_test.go`) registers only the "owned by and indexed" snapshot + index. Tests that go through `retrieveLatestSnapshot` also need the "owned by and sub-indexed" index; extend the + helper or add a variant. +- **Cache lag.** `hubClient` and `hubUncachedReader` are separate fields, so a test can back them with two fake + clients holding different objects; for example, a primary that only the "API server" store has, for the quorum-read + branches. +- **Delete options.** The fake client enforces `ResourceVersion` preconditions but not `UID` ones. An interceptor + records each Delete's `client.DeleteOptions`, and the test compares their `Preconditions.UID`s. +- **Create order.** An interceptor records the names of Create calls, in order. +- **GC requests.** Use `newTestGCQueue`; drain the queue and compare with `cmp.Diff`, as + `TestCheckForInconsistentResourceSnapshots` does. +- **Fixtures.** A snapshot builder with the index, sub-index, count label (primaries only), contents hash, owner + references, and an optional deletion timestamp, for both scopes. `TestGarbageCollect`'s local `snapshot` builder can + be promoted. The fake client rejects an object with a deletion timestamp but no finalizers, so pending-deletion + fixtures need a finalizer. +- **Layout.** New tests go in `ops_test.go`, `snapshots_test.go`, `resources_test.go`, and `manager_test.go`, matching + the source files; the GC additions stay in `gc_test.go`. New shared unit helpers go in a new `utils_test.go`; the + existing ones stay in `gc_test.go`, as decided at 06:52. + +**UT-P0: the October 26 changes** + +U1. `TestAllSnapshotsFor` (`ops_test.go`, new). Cluster-scoped unless noted: + +| Case | Want | +|---|---| +| Primary only, count 1 | `[primary]`; no requests | +| Primary and 2 secondaries, listed out of order, count 3 | Sorted by sub-index, primary first | +| Same, for a namespaced policy | Same; covers the namespaced list branch | +| Extras at sub-indices 2 and 3, count 2 | The first 2; one request per extra, carrying the policy's namespaced name; none for the primary or sub-index 1 | +| An extra pending deletion | Success; no request for it | +| An ownerless secondary (old-UID name, duplicate sub-index 1) | Transient, naming the leftover; one request, for it | +| An ownerless secondary pending deletion | Transient; no request | +| A secondary owned by a stale policy UID | Transient (stale owner); no requests | +| A secondary with an extra owner | User | +| The given primary is not in the store | Transient ("not found as the first snapshot listed"); no requests | +| The stored primary has a different UID from the given one | Transient | +| A count label of `abc` or `0`, or none | Unexpected | +| Fewer snapshots than the count | Unexpected | +| Sub-indices `0` and `2`, count 2 | Unexpected ("not contiguous") | +| A secondary with a different contents hash | Unexpected | +| A malformed sub-index label | Unexpected (sort) | + +U2. `TestGarbageCollect` (`gc_test.go`, extend). Give each case an optional GC request; the default stays the +primary's. New cases, each targeting a secondary: + +| Case | Want | +|---|---| +| An owned extra (sub-index 2), next to an owned primary and sub-index 1 | Only the extra is deleted | +| An ownerless secondary next to an owned primary | Only it is deleted | +| An owned extra while a binding uses the primary | The extra is deleted; secondaries skip the in-use check | +| The secondary was re-created (UID mismatch) | Kept; no error | +| The secondary is pending deletion | Kept; no error and no Delete call | +| Delete fails (interceptor) | APIServer error; kept | +| Delete returns NotFound (interceptor) | No error | +| A namespaced secondary | Deleted; covers the namespaced Get branch | + +A secondary that is not found takes the same path as the existing "primary snapshot not found" case; no new case. + +U3. **Delete preconditions** (`gc_test.go`, a subtest of U2; and a U8 case). With the recording interceptor, every +Delete carries `Preconditions.UID` equal to the object's UID, on all three paths: a secondary target; the primary +path's secondaries and primary; and `cleanUpDanglingSecondarySnapshots`. + +U4. `TestEnqueueResourceSnapshotForGC` (`gc_test.go`, new; one table that also covers +`enqueueLeftoverResourceSnapshotsForGC`): + +- A cluster-scoped and a namespaced snapshot: the request carries the snapshot's namespaced name and UID, and the + policy's namespaced name. +- A snapshot pending deletion: no request. +- `enqueueLeftoverResourceSnapshotsForGC` with three snapshots, one of them pending deletion: two requests. + Enqueueing the same snapshot twice yields one request (the workqueue deduplicates). + +**UT-P1: functions near the change that have no unit tests** + +U5. `TestRetrieveLatestSnapshot` (`ops_test.go`, new). Integration covers most branches; this covers the ordering. + +- No primaries: `nil` and no error. +- Primaries out of order: the highest index wins. +- A leftover primary: Transient, and **no** history requests, only the leftover's own (blocked before history GC). +- The latest primary pending deletion: Transient, and history requests **are** enqueued (expected; Q10). +- An ownerless secondary at the latest index: Transient; history requests are enqueued, plus one for the leftover + (expected; Q10). +- A history limit of 1 with three indices: requests for the two older primaries only. + +U6. `TestEnqueueOldResourceSnapshotsForGC` (`gc_test.go`, new; pure): + +- No limit set means the default of 3; a limit of `0` or less means 1. +- History within the limit: no requests. A limit of 1 with four primaries: the three oldest. +- Primaries pending deletion are skipped. + +U7. `TestIsSnapshotUpToDate` (`ops_test.go`, new; two stores): + +- Hash mismatch: false, with no quorum read (an interceptor on the uncached reader sees no Get). +- Hash match, nothing at the next index: true. +- Hash match, and the next primary exists only in the uncached store: Transient ("a newer snapshot already exists"). +- A malformed index label: Unexpected. An uncached Get failure: APIServer. + +U8. `TestCleanUpDanglingSecondarySnapshots` (`ops_test.go`, new; two stores): + +- Nothing at the index: false. +- A primary at the index in the cache: Transient; nothing is deleted. +- A primary only in the uncached store: Transient; nothing is deleted. +- Owned, ownerless, and stale-owner secondaries alike (there is no owner check): all deleted, with UID preconditions; + true. +- Only secondaries pending deletion: nothing is deleted, and it still returns true, so the caller requeues. +- Delete fails: APIServer. Delete returns NotFound: no error, and it still returns true, so the caller requeues + (Q11). + +U9. `TestCreateResourceSnapshotAnyway` (`ops_test.go`, new; a Create-recording interceptor and a small +`maxPerSnapshotResourceCnt`): + +- No latest primary: index 0. A latest primary at index 4: index 5. A malformed latest index: Unexpected. +- Three groups: Creates run in the order sub-index 2, sub-index 1, then the primary. The primary has a count of 3, + names come from `uniquename`, and every snapshot has a controller owner reference to the policy. +- Dangling secondaries at the next index are deleted, the call is Transient ("cleaned up dangling secondary …"), and + nothing is created. +- A secondary Create fails (AlreadyExists): APIServer, and the primary is not created. The primary Create fails: + APIServer. + +U10. `TestIsSnapshotSynchronized` (`ops_test.go`, new; the checks that run before any resource is read): + +- A nil snapshot or policy: Unexpected. A non-primary snapshot: Unexpected. +- Not owned by the policy, or in another namespace: Unexpected. +- Owned by a stale policy UID: Transient. +- No contents hash: Unexpected. +- The hash comparison itself reads the selected resources (REST mapper, dynamic client), so it stays with integration + (optional, as before). + +U11. `TestPlacementPolicyOwnerRef` and `TestIsSamePlacementPolicyOwner` (`gc_test.go`, new; pure): + +- Owner selection: a malformed API version is skipped; the kind must match the scope (`ClusterPlacementPolicy` for + cluster-scoped objects); the name must match; the matching reference is returned from among several; no match + returns nil. +- Same owner: both nil is true, one nil is false, the same UID is true, and different UIDs are false. + +**UT-P2: earlier gaps elsewhere in the package (optional)** + +U12. `TestSnapshotResource` (`resources_test.go`): strips the last-applied and deployment-revision annotations +(dropping `metadata.annotations` when nothing else is left), owner references, managed fields, read-only metadata, +finalizers, and status; keeps other annotations, labels, and the spec; and sets the identifier from the GVK. Assert on +raw JSON, as P0-E does. + +U13. `TestSplitResourcesIntoSizeControlledGroups` (`resources_test.go`): empty input gives one empty group; splits by +count and by size (`AdditionalInfo` counts toward the size); a resource exactly at the size limit fits; one over the +limit is a User error. + +U14. `TestPrimaryPlacementResourceSnapshot`, `TestSecondaryPlacementResourceSnapshot`, and +`TestPlacementPolicyOwnerLabelVal` (`snapshots_test.go`): labels (owned-by value, index, sub-index, and the count on +primaries only), annotations (contents hash, full owned-by name), the controller owner reference, and the type per +scope; the owned-by label is truncated and hashed at its length limit. + +U15. `TestNew` (`manager_test.go`): each argument check fails with the right category (mutex slot count, size and +count bounds, a nil eligibility checker). The checks run before the controller manager is used, so a nil manager +works for the failure cases. + +### Open questions (October 26) + +10. **History GC while blocked at the latest index.** `retrieveLatestSnapshot` enqueues history GC before + `allSnapshotsFor`, so leftovers at the latest index (and, as before, a latest primary pending deletion) do not + stop history GC, whereas leftover primaries do. Should `enqueueOldResourceSnapshotsForGC` move after + `allSnapshotsFor`? **Resolved at 16:30**: no; the order does not matter, as the two never overlap. + - History GC only takes primaries (from the primary scan) at older indices; the latest is excluded, and + collecting a primary touches only its own index. + - `allSnapshotsFor` only enqueues snapshots at the latest index, and in practice only secondaries: extras always + are (the count is at least 1, and the primary is verified to come first), and an ownerless primary there would + already have been caught by the primary scan, barring a cache change between the two reads. + - Leftover primaries block history GC for a different reason: they would skew the history itself. + - U5 covers the behavior as expected, not as a pin. +11. **NotFound in `cleanUpDanglingSecondarySnapshots`.** Its Delete fails with an APIServer error on NotFound (the + caller requeues), while every GC delete ignores NotFound. Pin it (U8), or align it with GC? **Resolved at + 16:31**: aligned with GC; the Delete now ignores NotFound (`!apierrors.IsNotFound(err)`). The method still + returns true, so the caller still requeues. A re-created snapshot (new UID) still fails the UID precondition with + a conflict, not NotFound. U8 expects no error. + +## Out of Scope + +These are unit-test territory or unreachable: + +- `New()` argument validation. (Revised October 26: now planned as unit test U15.) +- Hash, marshal, and `SetControllerReference` failures. +- API server transport errors. +- `AdditionalInfo` sizing. +- The owner-reference variants of the leftover check, the in-use check, and GC, which `gc_test.go` covers, and + snapshot name formatting, which `uniquename/uniquename_test.go` covers. (Revised October 24: this entry used to + name the GC owner-UID mismatch case, which no longer exists.) + +## Open Questions (all resolved; see Status) + +1. A typo in the selector Kind fails REST mapping and returns an **Unexpected** error. Should it be a User error + (`meta.IsNoMatchError`)? **Resolved at 03:07**: it is now a User error. +2. `snapshotResource` keeps `metadata.finalizers`, so they would be propagated to member clusters. Is that intended? + **Resolved at 03:07**: finalizers are now stripped. +3. `SnapshotResourcesIfNoSnapshotExists` still runs `isSnapshotUpToDate` and can fail with Transient even though + existing snapshots are present. Is that intended? **Resolved at 05:33**: yes; a retry succeeds once the cache + catches up. +4. A label selection that matches nothing produces an empty primary snapshot. Is that intended? + **Resolved at 03:07**: an empty selection is now a User error. + +Q5 to Q9 came up during the P1 and P2 planning; their questions and answers are in the Status log. Q10 and Q11 came +up during the October 26 revision and are resolved; see [Open questions (October 26)](#open-questions-october-26). + +## Validation + +- `KUBEBUILDER_ASSETS=$(hack/tools/bin/setup-envtest-release-0.22 use 1.33.0 -p path) go test ./pkg/v1/managers/placementresourcesnapshot/... -coverprofile=...` +- Targets: `resources.go` at 90% or more for every function except unreachable branches; package at 85% or more. + Dropped at 06:41: per the user, coverage is no longer measured. +- Suite runtime increase of about 60s or less (baseline is about 58s). +- `make reviewable`. +- Final runs use `ginkgo -p --race` instead of `go test`. +- Unit tests (added October 26) need no envtest when filtered by name, for example + `go test -race -count=1 -run '^Test(AllSnapshotsFor|GarbageCollect|EnqueueResourceSnapshotForGC)$' ./pkg/v1/managers/placementresourcesnapshot/`. + Without `-run`, `go test` also starts the envtest suite (`TestAPIs`), which needs `KUBEBUILDER_ASSETS`. Run the + queue-based tests with `-count=30` as well, as was done for `TestStart`. +- From October 27, per the user: integration runs are always focused (`ginkgo -p --race --focus=...`) on the specs + being changed. The whole suite is not run. + +## Status + +The detailed P0, P1, and P2 plans mentioned below were working documents and were not committed. Their spec IDs map +to test contexts by position, in file order: PP-n and CPP-n are the n-th `Context` of the two `Describe`s in +`resources_integration_test.go`; SC-n and NM-n are those of the two `Describe`s in `corner_cases_integration_test.go` +(SC-1 runs once per scope, as the first two contexts); and GC-n are those of `Describe("GC variants")` in +`ops_gc_integration_test.go`. Contexts added after the IDs were assigned are skipped when counting: in the PP +`Describe`, "resources with non-controller owner references" (not in the plan) and "named resource not found +(informer-backed)" (October 27, referred to as PP-2b). PP-13 and PP-14 ("manifest sanitization (services)" and +"(jobs)", October 27) are the exception: they were numbered by their positions in the file at the time, which counted +the non-controller owner references context; they are the last two contexts of the PP `Describe`. + +- 02:32: High-level plan drafted. +- 02:48: The existing IT file has been renamed to `ops_gc_integration_test.go`. + A detailed plan for P0-A to P0-E was written, with resource selection and eligibility tests going in + `resources_integration_test.go`. + Its environment facts, error messages, and manifest shapes were verified with a throwaway envtest probe spec, + which has since been removed. + - Implementation then waited for approval and for answers to Q1, Q2, and Q4. +- 03:07: The user resolved Q1, Q2, and Q4 in code (`resources.go`, `ops.go`), and the detailed plan was updated: + - Q1: unknown kinds, versions, and groups are now user errors. Added cases A11c (unknown group) and + A11d (unknown kind through a label selector, which covers the label-path lookup error). + - Q2: finalizers are now stripped. PP-9 asserts their absence and that a finalizer-only change creates no snapshot; + CPP-2 asserts that the Namespace's `spec.finalizers` is kept. + - Q4: an empty selection is now a user error. PP-5 covers it both with and without an existing snapshot, and + CPP-3 adds an "every match is ineligible" case. The empty branch of the size-controlled split is now + unreachable through the public entry points. + - The existing `ops_gc_integration_test.go` suite still passes (62s). + - Q3 (`createOnlyWhenMissing` failing on a quorum-read transient error) was left open, as it is outside P0-A to + P0-E. It was resolved at 05:33. +- 03:12: P0-A to P0-E were implemented in `pkg/v1/managers/placementresourcesnapshot/resources_integration_test.go`. + - The file contains 14 ordered contexts and 78 specs. + - It adds shared helpers for error and category checks, snapshot reads from the cache, absence checks, and cleanup. + - Validation: + - The focused run passed (78/78, about 22s). + - The full package run with `-race` passed (90s), and coverage rose from 72.9% to 78.7%. + - `make lint` (`--fast-only`) reported 0 issues. Under `lint-full`, the new file has only `goconst` findings + on string literals, the same style as `ops_gc_integration_test.go`. + - A mutation check confirmed that the intended specs fail when finalizer stripping, deduplication, or the + label-path eligibility check is disabled. +- 04:49: A detailed P1 plan was written. + - It covers the stale-cache and inconsistent-state branches (SC-1 to SC-9) and long and dotted names (NM-1 to NM-5). + - It targets `corner_cases_integration_test.go`. The requested `..._tests.go` name would not be compiled as a test file. + - Every behavior was verified with a throwaway envtest probe, since removed. + - **Finding (Q5)**: concurrent calls can hit the `unexpected` "found a primary placement resource snapshot at the + given index while cleaning up orphaned secondary snapshots" error under ordinary cache lag (1 in 8 calls in the + probe). Recommendation: make it transient. + - New questions: Q6 (a missing secondary is unrecoverable) and Q7 (a known label-collision limitation). +- 05:26: The user resolved Q5 and Q7: + - **Q5**: the sanity-check failure in `cleanUpOrphanedSecondarySnapshots` is now `transient`. Verified with a probe: + 12 races in 192 concurrent calls, all transient, with no `unexpected` errors and exactly one set of snapshots per + index. + - **Q7**: ownership lookups now key on the owned-by annotation (the full name). + - Changed code: the `hub.go` extractors, the `ops.go` field matchers, `gc.go`, and `workgenerator/retrieval.go`. + - The final code is consistent; checked statically. + - A mid-edit state (index still keyed on the label, GC on the full name) leaked secondaries in GC for names over + 61 characters, as the probe showed. + - Plan updates: indexed decoys need the annotation, `verifyOwnedSnapshots` filters by annotation, and the new + NM-6 guards the former collision. + - Follow-up: one `workgenerator` suite run during the Q7 edits failed its 10 "multiple placement resource snapshots" + specs. A full re-run on the final code was interrupted at the time; the suite passed in the 07:50 review. +- 05:33: The user resolved Q3 and Q6 as expected behavior: + - **Q3**: the transient error is fine; a retry finds the newer snapshot once the cache catches up. Verified with a + probe that simulates the catch-up by turning the decoy into an indexed snapshot: both entry points then return + index 1 as up to date, and no index 2 is created. SC-1 now ends with this catch-up instead of removing the decoy. + - **Q6**: a missing secondary needs manual recovery. Verified with a probe: + - A resource change alone does **not** recover; the call still returns `unexpected`, because + `retrieveLatestSnapshot` fails before the selected resources are read. + - Deleting the broken primary does recover; the next call re-creates the set. + - SC-7 now checks all three stages. + - No open questions remain for the P1 plan. +- 06:24: A detailed P2 plan was written. + - It targets a new `Describe("GC variants")` in `ops_gc_integration_test.go`, with seven contexts (GC-1 to GC-7): + limit changes, multi-group GC (CPP, plus an extra secondary), the GC request queue, snapshots pending deletion, + retryable and non-retryable GC failures, and same-named policies across namespaces and scopes. + - New technique: a *paused manager*, a second `Manager` built with `New(hubMgr, ...)` that is never started. Its + workqueue can be drained to assert the exact GC requests, deduplication, and in-memory limit overrides. + - Overlap is avoided: NM-2 already covers limit 1 with two groups (PP), and `gc_test.go` covers the UID and owner + mismatch cases. + - Every behavior was verified with a tag-guarded throwaway envtest probe (`//go:build gcprobe`), since removed. + With the probe, coverage moved as follows: + - `enqueueOldResourceSnapshotsForGC`: 84.6% to 100% + - `garbageCollect`: 82.1% to 87.5% + - `Start`: 65.5% to 93.1% + - Package: 86.7% to 89.1% + - New questions: + - **Q8**: an out-of-band deletion of an old primary leaks its secondaries; the plan pins the current behavior. + - **Q9** (ops.go, outside P2): a terminating latest primary is returned as up to date. + - Minor: the owned-by annotation check in `garbageCollect` is duplicated, so the second check is dead code. + - Implementation then waited for approval and for an answer to Q8. +- 06:36: The user resolved the P2 questions, and the plan was updated: + - **Q8** is expected behavior: it does not happen in normal operation, and a guard would be too expensive. GC-4 + step 5 pins it. + - **Q9**: a sanity check in `retrieveLatestSnapshot` (ops.go:184–188) now returns a transient error when the latest + primary is marked for deletion. It was verified with a tag-guarded probe, since removed: + - Both entry points fail as transient, including after a resource change, and no new index is created. + - GC-4 steps 1 to 8 are unchanged. + - Added GC-4 step 9 and the constant `latestSnapshotPendingDeletionErrMsg`. + - The duplicated owned-by check in `garbageCollect` was removed. The coverage figures were recomputed: + `garbageCollect` 83.6% to 89.1%, and the package about 86.7% to 89.2%. + - No open questions remained for P2, and implementation waited for approval. +- 06:41: P2 was implemented as `Describe("GC variants")` at the end of `ops_gc_integration_test.go`. + - It has seven ordered contexts (GC-1 to GC-7) and 57 specs. + - New constants were added to the existing `const` block. New helpers, including the paused manager, `checkOwnedSnapshots`, and + `createPlacementBinding`, sit just before the new `Describe`. + - Validation: + - `ginkgo -p --race .` passed 316 of 316 specs, with about 49s of spec time. + - `make lint` reported 0 issues. + - Per the user, coverage is no longer measured, and the planned mutation checks were skipped (no repeated runs). +- 06:52: All helper functions were moved out of the three `*_integration_test.go` files into a new `utils_integration_test.go`. + - 61 functions moved, plus the helper-only types `snapshotErrTestCase` and `snapshotCallResult`. + - `suite_test.go` and `gc_test.go` were left untouched, per the user. + - Constants and spec-only types (`decoySnapshotSpec`) stay in their files. + - The move was done mechanically with an AST-based tool, so the bodies and doc comments are byte-for-byte identical. The + source files only lost lines, and the set of functions is unchanged. + - Validation: + - `ginkgo -p --race .` passed 316 of 316 specs. + - golangci-lint (`--fast-only`) on the package reported 0 issues. +- 07:50: A code review of the branch found several issues, which have been fixed: + - Review fixes: + - `resourceeligibility.New` now takes a `CheckerOptions` struct instead of three positional bools. + `Baseline(restMapper)` always runs the object checks; the v0 default uses an unexported variant that skips them. + - `AllowsNamespace` allows `""` (cluster-scoped resources) before honoring the skip flag, so a skipped check in + allow-list mode no longer denies cluster-scoped resources. + - The manager's `New` rejects a nil eligibility checker. + - `retrieveAndHashSelectedResources` checks the placement policy's namespace once, before the selector loop. + - `gc.go`: a stale "owned-by label" comment was corrected, and a redundant alias was removed. + - Deployments are now read through the cache: + - `suite_test.go` registers a Deployment informer with the dynamic informer manager (`CreateInformerForResource`, + `Start`, `WaitForCacheSync`). It replaces a controller-runtime watch on Deployments that nothing read from. + ConfigMaps and all other kinds are still read from the API server. + - This exposed cache-lag races: 1 of 316 specs failed. + - Fix: two new helpers in `utils_integration_test.go`, `waitForDeploymentsInCache` and + `waitForDeploymentsRemovedFromCache`. They are called after every Deployment write in `ops_gc_integration_test.go` + and `resources_integration_test.go`. + - The informer applies events in order, so waiting for a context's own writes also covers earlier contexts' + deletions. The `AfterAll` blocks need no waits. + - Validation: + - `ginkgo -p --race` passed for `placementresourcesnapshot`, `workgenerator`, `resourceeligibility`, + `resourcewatcher`, and `utils/controller`. + - `golangci-lint --fast-only` reported 0 issues. + - The user confirmed the final build and test runs. + - Still open (not code): + - The PR description must list this branch's behavior changes: + - Ownership lookups key on the owned-by annotation. + - GC deletes use UID preconditions. + - Finalizers are stripped from snapshotted manifests. + - Error categories changed: + - An unknown GVK is a User error. + - An empty selection is a User error. + - A terminating latest primary is Transient. + - The orphan-cleanup sanity check is Transient. + - A contents-hash mismatch is Unexpected. + - A resource selected by name that does not exist is a User error (added October 27; a cache miss is + confirmed with a read from the API server first). Auto-rollout then waits for its next rollout window + instead of retrying, and a staged update run fails its initialization. + - The `Checker` API changed: `AllowsObject` also returns a reason, and `New` takes `CheckerOptions`. + - PR #974 also edits `guard_test.go`, so a merge conflict is expected. +- October 24, 07:20: The plan was revised for leftover snapshot detection and UID-based snapshot names; see + [Revision: leftover snapshots and UID-based names](#revision-leftover-snapshots-and-uid-based-names). + - Existing tests were already adapted: names use `uniquename.*` with the policy UID, indexed decoys are owned by + the policy, and the NM-1 boundaries moved. `go test -race` passes for the package (195s) and for `uniquename`. + - Removed: GC-5 (retryable and non-retryable GC failures), whose errors no longer exist. + - To add: LO-1 to LO-5 (leftover snapshots from previous policies), a new GC-5 (snapshots in use by bindings), and + a unit test for the GC worker loop. Optional: `IsSnapshotSynchronized`. + - More entries for the PR description: + - Snapshot creation is blocked while leftover primaries exist. A stale owner UID or an orphan is Transient, and + unexpected owners are a User error. + - GC no longer checks ownership, and the in-use check is by name only. (Revised at 07:56: GC now skips snapshots + whose policy owner differs from the primary's.) + - Snapshot names always carry a hash that includes the policy UID; the exported `uniquename` package builds them. + - Implementation is waiting for approval. +- October 24, 07:35: The revision was implemented (commit `f2e74e10`). + - **LO-1 to LO-5** are in a new `Describe("corner cases (leftover snapshots from previous placement policies)")` at + the end of `corner_cases_integration_test.go`. There are six contexts (LO-2 runs once per scope) and 33 specs. + Three error message constants were added: `staleOwnerSnapshotErrMsg`, `leftoverSnapshotsErrMsg`, and + `unexpectedOwnerRefsErrMsg`. + - **GC-5** ("snapshots in use by placement bindings", 8 specs) is in `Describe("GC variants")`, at the position of + the removed contexts. Because the IDs are positional, the "same-named placement policies" context is now GC-6 + (it was GC-7). + - **`TestStart`** is a new unit test in `gc_test.go` with three subtests: a retryable failure is retried with backoff + and then forgotten, an unexpected failure is dropped while later requests still run, and canceling the context + stops the loop. It uses a test workqueue with a 1ms backoff, the fake client with an interceptor, and the helpers + `newTestGCQueue`, `startGCWorker`, and `pollUntil`. + - **New shared helpers** in `utils_integration_test.go`: + - `recreatePlacementPolicy` + - `setSnapshotOwnerRefs`, which also waits until the cache has observed the change + - `updatePlacementBindingSnapshot` + - The optional `IsSnapshotSynchronized` coverage was not done. + - Validation: + - `ginkgo -p --race ./...` passed: 347 of 347 specs (9 processes, 53s), plus the `uniquename` unit tests. The + focused run of the new specs matched 41. + - `go test -race -count=30 -run TestStart` passed. + - `golangci-lint --fast-only` reported 0 issues. In the new code, the full run flags only a `goconst` finding on + the `ConfigMap` literal, the same style as the rest of the file. +- October 24, 07:56: Following review, `garbageCollect` gained a sanity check in its delete loop. It skips (and logs at + `V(2)`, as a cache-lag symptom rather than an error) any snapshot at the primary's index that is another primary, or + whose policy owner (or lack of one) differs from the primary's. The check uses the new helper + `isSamePlacementPolicyOwner`. + - Why: names now include the policy UID, so only the cache-based block keeps two policy generations from sharing an + index. If the cache lags, GC of an orphaned primary could otherwise delete a re-created policy's snapshots at that + index. Losing only its secondaries would leave the policy stuck on the "fewer sub-indexed snapshots" Unexpected error. + - `TestGarbageCollect` has four new cases: another primary at the index; an orphaned primary alongside a re-created + policy's snapshots; a secondary owned by another UID; and an ownerless secondary under an owned primary. + - `checkForInconsistentResourceSnapshots` no longer re-enqueues ownerless primaries that are already marked for + deletion, but they still block snapshotting. `TestCheckForInconsistentResourceSnapshots` has two new cases for + this. + - Doc and wording fixes from the review: + - `checkForInconsistentResourceSnapshots` now returns the leftover primaries (as `klog.ObjectRef`s) instead of a + bool, so the "leftover resource snapshots are detected" error names them, along with the policy. + - The stale-owner error now reads "owned by a different placement policy of the same name (e.g., one that has + been deleted)"; `staleOwnerSnapshotErrMsg` was updated to match. + - Comments now credit the Kubernetes garbage collector (kube-controller-manager), not the API server. + - The doc comment of `isSnapshotInUseByAnyPlacementBinding` says that binding owners are not checked. +- October 24, 08:05: Minor test fixes from the review, and breadcrumb consistency fixes. + - `gc_test.go` builds snapshot names with `uniquename` instead of the pre-UID format. The "primary snapshot has been + re-created" case now re-creates the snapshot under the same policy, since only the same policy can reuse a name. + - GC-5's `verifySnapshotsKeptWithoutRetries` first waits until the GC worker has picked up the request (the + workqueue is empty). Before, `NumRequeues` of 0 also held for a request that had not been processed at all. + - Breadcrumb fixes: + - The implementation entry is now timed 07:35, matching the commit, instead of 07:50. + - The unit test table lists `TestStart`, and "not covered anywhere" no longer lists the `Start` loop. + - The LO-3 plan matches the single-snapshot implementation. + - The leftover check and GC-5 descriptions match the current behavior. + - Validation: `gofmt` and `go vet` are clean, and the `gc_test.go` unit tests pass with `-race`. The integration + suite was not re-run. +- October 26, 16:20: The plan was revised for latest-index checks and secondary snapshot GC (working-tree changes to + `ops.go` and `gc.go`), and a unit test plan was added; see + [Revision: latest-index checks, secondary GC, and unit tests](#revision-latest-index-checks-secondary-gc-and-unit-tests). + - Before the revision, the three specs that the change broke (SC-6, SC-8, GC-2) were adapted, and two outdated test + comments (SC-7, LO-5) were corrected. `ginkgo -p` passed 347 of 347 specs. + - To add: + - Integration: extend SC-6 and SC-8, and add LO-6. + - Unit, by priority: U1 to U4 for the change itself; U5 to U11 for nearby functions without unit tests; U12 to + U15 (optional) for earlier gaps elsewhere in the package. + - Open: Q10 (recommendation: pin) and Q11 (recommendation: align with GC). + - More entries for the PR description: + - Ownerless or stale-owner secondaries at the latest index now block snapshot creation (Transient), and + ownerless ones are garbage-collected; unexpected owners on them are a User error. + - Extra secondaries beyond the count label are garbage-collected. + - New errors: the primary missing from the latest-index listing (Transient), and non-contiguous sub-indices + (Unexpected). + - The count-label error messages were reworded. + - `garbageCollect` deletes secondary targets directly, and orphan cleanup deletes with UID preconditions. + - Implementation is waiting for approval. +- October 26, 16:30: The user resolved Q10: the order of history GC and the latest-index checks does not matter. + History GC only takes primaries at older indices, while `allSnapshotsFor` only enqueues snapshots at the latest + index (in practice, secondaries), so the two never overlap. U5 now covers the behavior as expected rather than + pinning it. Q11 is still open. +- October 26, 16:31: The user resolved Q11 by aligning orphan cleanup with GC. + - Code: the Delete in `cleanUpOrphanedSecondarySnapshots` (`ops.go`) now ignores NotFound + (`!apierrors.IsNotFound(err)`; `ops.go` gained the `apierrors` import that `gc.go` already uses). The method + still returns true, so the caller still requeues. + - Plan: U8 now expects no error on NotFound. + - More for the PR description: orphan cleanup ignores NotFound on delete. + - Validation: `gofmt` and `go vet` are clean, the unit tests pass, and `ginkgo -p` passed 347 of 347 specs. + - No open questions remain; implementation is waiting for approval. +- October 26, 16:40: The integration cases of the October 26 revision were implemented (the 11 new specs are all in + `corner_cases_integration_test.go`). The unit tests (U1 to U15) are not implemented yet. + - **SC-6** (extended; 4 specs in the existing context): with a paused manager, the queue holds only the extra's + request; a second extra (sub-index 3) pending deletion neither blocks the calls nor is enqueued; once it is gone, + the calls return the same set. The paused manager is never called at the same time as the manager under test, + which still collects the first extra. + - **SC-8** (extended; 1 table entry, appended): non-contiguous sub-indices (a primary with a count of 2 and a + secondary at sub-index 2) return Unexpected. New constant: `nonContiguousSubIdxsErrMsg`. + - **LO-6** ("orphaned secondary snapshot at the latest index"; 6 specs, at the end of the leftover `Describe`): uses + a paused manager throughout. Both entry points fail as Transient; the error's `leftoverPlacementResourceSnapshots` + attribute names the leftover (read with `errAttrValue`); each call enqueues exactly one request, for the leftover. + The block holds after a resource change. The paused manager's `garbageCollect` deletes only the leftover, and the + owned snapshots keep their UIDs. The calls then return the original set (stale) and create index 1. + - Helper change: `gcRequestFor` (`utils_integration_test.go`) is documented for any snapshot, primary or secondary, + and its parameter is renamed to `snapshot`. The test file now imports `k8s.io/klog/v2`. + - Validation: `go vet` is clean; the focused run passed 24 of 24 specs; `ginkgo -p --race` passed 358 of 358 specs + (about 43s), with no data races; `golangci-lint --fast-only` reported 0 issues. +- October 26, 16:59: The unit test plan (U1 to U15, including the optional U12 to U15) was implemented. There are 17 + new test functions and 8 new `TestGarbageCollect` cases; with the 4 existing functions, the package has 21 unit test + functions and 149 subtests. + - Files: + - `ops_test.go` (new): U1 `TestAllSnapshotsFor` (18 cases), U5 `TestRetrieveLatestSnapshot`, U7 + `TestIsSnapshotUpToDate`, U8 `TestCleanUpOrphanedSecondarySnapshots`, U9 `TestCreateResourceSnapshotAnyway`, and + U10 `TestIsSnapshotSynchronized`. + - `gc_test.go`: U2 (8 `TestGarbageCollect` cases; each case can now name its GC request, and the remaining + snapshots are listed across both scopes), U3 `TestGarbageCollectDeletePreconditions` (the cleanup path is a U8 + case), U4 `TestEnqueueResourceSnapshotForGC`, U6 `TestEnqueueOldResourceSnapshotsForGC`, and U11 + `TestPlacementPolicyOwnerRef` and `TestIsSamePlacementPolicyOwner`. + - `resources_test.go` (new): U12 `TestSnapshotResource` (it also checks that the given resource is not + modified) and U13 `TestSplitResourcesIntoSizeControlledGroups`. + - `snapshots_test.go` (new): U14 `TestPrimaryPlacementResourceSnapshot` and `TestSecondaryPlacementResourceSnapshot` + (each also covers a `SetControllerReference` failure: a namespace-scoped owner of a cluster-scoped snapshot), and + `TestPlacementPolicyOwnerLabelVal`. + - `manager_test.go` (new): U15 `TestNew`. + - `utils_test.go` (new): shared unit helpers: the `snapshotFixture` builder (`newSnapshotFixture` and `with*` + modifiers), `newTestPlacementPolicy`, `testPolicyOwnerRef`, the Delete recorder (`recordDeletes`, `deleteCall`), + `listSnapshotNames`, and `verifyErrCategoryAndMsgSubstr`. + - Helper changes: `newFakeHubClientWithIndexes` (`gc_test.go`) also registers the "owned by and sub-indexed" + snapshot index. The unit tests reuse the Ginkgo-free integration helpers `drainGCRequests`, `gcRequestFor`, and + `snapshotNamesOf`, and the `testFinalizer` constant. + - Small additions beyond the plan: namespace-scoped cases in U5 and U9; a malformed index label case and a + multi-group case in U5; a "latest snapshot only" case in U6. + - Mutation checks (reverted afterwards; the sources were restored byte-for-byte), each caught by the intended cases: + orphan cleanup failing on NotFound (U8); no primary-first check (U1, two cases); no contiguity check (U1); the + secondary GC path disabled (U2, two cases; U3). + - Validation: `go vet` and `gofmt` are clean; all unit tests pass with `-race`; the queue-based tests pass with + `-race -count=30`; `golangci-lint --fast-only` reported 0 issues; `ginkgo -p --race` passed 358 of 358 specs. + - Not built: the optional integration backlog (F.1 to F.4, and `IsSnapshotSynchronized`'s hash comparison). +- October 26, 17:08: Lint fixes in the new unit tests: `fmt.Appendf` in `ops_test.go`, and preallocated slices in + `ops_test.go` (`summarize`) and `utils_test.go` (`listSnapshotNames`). +- October 26, 17:40: Renamed `cleanUpOrphanedSecondarySnapshots` to `cleanUpDanglingSecondarySnapshots`, at the user's + suggestion. "Orphaned" now means only "no owner reference" (as with Kubernetes' orphan propagation), and "dangling" + means "a secondary snapshot at an index with no primary snapshot". + - Code (`ops.go`, `gc.go`): the function, its doc comment (which now defines "dangling" and the return value), the + related comments, and the messages, e.g., "cleaned up dangling secondary placement resource snapshots; requeue + before creating new snapshots". + - Tests: U8 is now `TestCleanUpDanglingSecondarySnapshots`. The U9 case, the SC-3 to SC-5 contexts and specs, the + LO-5 and LO-6 comments, and the `verifyDanglingSecondariesCleanedUp` helpers in `ops_gc_integration_test.go` use + the new term. The error-message constant is now `danglingSecondariesCleanedUpErrMsg`. Uses of "orphaned" that mean + "no owner" (LO-2, LO-3, LO-6, and the GC unit tests) are unchanged. + - This plan: the sections above the status log use the new names; earlier status entries keep the old ones. + - The user changed `checkForInconsistentResourceSnapshots` (`gc.go`) to name its result `orphaned` and to return nil + with an error. The `TestCheckForInconsistentResourceSnapshots` case "orphaned snapshot followed by a snapshot owned + by a deleted placement policy with the same name" now wants no snapshots returned; it still wants the GC request. + - Validation: `go build`, `go vet`, and `gofmt` are clean; `golangci-lint --fast-only` reported 0 issues; all unit + tests pass with `-race`; `ginkgo -p --race` passed 358 of 358 specs. +- October 26, 17:47: Consistency fixes from the latest review round (items 4 to 7). + - Error/log keys: "observedIdx" (`garbageCollect`) is now "observedIndex", "snapshotIdx" + (`cleanUpDanglingSecondarySnapshots`) is now "snapshotIndex", and "actualCount" (`allSnapshotsFor`) is now + "observedCount". The workgenerator's `retrieval.go` still uses "actualCount" for its own count check. + - Messages: "found via quorum reads" (`isSnapshotUpToDate`) is now "found via quorum read", and + `IsSnapshotSynchronized` now says "of the same name", like the GC check. The tests follow: + `quorumReadNewerSnapshotErrMsg` and the U10 case. + - The `klog.Warningf` in `allSnapshotsFor` is now a structured `klog.InfoS` (same visibility), with the extra + snapshots listed (`extraPlacementResourceSnapshots`). + - The index-parse errors in `isSnapshotUpToDate` and `createResourceSnapshotAnyway` now carry the primary snapshot + and the observed index. + - Validation: `go build`, `go vet`, and `gofmt` are clean; `golangci-lint --fast-only` reported 0 issues; + `ginkgo -p --race` passed 358 of 358 specs. +- October 26, 17:58: Settled the snapshot terminology (item 2 of the latest review round): "orphaned" means "no owner + reference", "leftover" means "an extra secondary snapshot beyond the count label", and "dangling" means "a secondary + snapshot at an index with no primary snapshot". + - Code: the user renamed the result of `checkForInconsistentResourceSnapshots` to `orphaned` (17:31). Its callers + in `ops.go` follow: the variable is `orphaned`, the message is "orphaned resource snapshots are detected; waiting + for garbage collection to complete", and the error key is `orphanedPlacementResourceSnapshots`. The docs of + `checkForInconsistentResourceSnapshots` (which also notes that no snapshots are returned with an error), + `allSnapshotsFor`, and `garbageCollect` use the new terms. `enqueueLeftoverResourceSnapshotsForGC` keeps its name. + - Tests: `leftoverSnapshotsErrMsg` is now `orphanedSnapshotsErrMsg`. The `Describe` for LO-1 to LO-6 is now + "corner cases (snapshots of previous placement policies)" (it covers snapshots still owned by a deleted policy + too). LO-5 ("orphaned secondary snapshots without a primary snapshot") and LO-6 use "orphaned" in their names, + variables (`orphanedSnapshotName*`, `verifyBlockedByOrphan`), and comments. U1/U5 messages and the U5 case + "orphaned primary snapshot", and the `TestCheckForInconsistentResourceSnapshots` fields (`wantOrphaned`, + `orphanedRef`) follow. "Leftover" in `TestEnqueueResourceSnapshotForGC` still means extras. + - This plan: the sections above the status log use the new message and `Describe` names; the LO-* labels stay. + - Validation: `go build`, `go vet`, and `gofmt` are clean; `golangci-lint --fast-only` reported 0 issues; + `ginkgo -p --race` passed 358 of 358 specs. +- October 26, 18:05: Typo and wording fixes from the latest review round, plus one trivia item. + - `ops.go`: the `snapshotResources` doc list now reads "a) …; b) …; or c) …"; "yet … yet" in `isSnapshotUpToDate` + is gone; "a few secondary … snapshots" and "(their contents might be outdated, and their specs are immutable + anyway)" in `createResourceSnapshotAnyway`; "match the current state" in `IsSnapshotSynchronized`. + - `manager.go`: "to shut down the GC workqueue"; the `New` error message says "mu slot count" (no test matched it). + - Tests: "clean-up(s)" is now "cleanup(s)" in the integration tests (14 places, including the SC-4 context name), + matching the production code. + - `snapshots.go`: the local imports now sit in their own group, as in the other files. + - Left as they are, per the user: item 1 (history GC runs before the pending-deletion check; only reachable by + tampering), the `TestGarbageCollect` case names, the `retrieveLatestSnapshot` result names, and `actualCount` in + the workgenerator's `retrieval.go`. + - Validation: `go build`, `go vet`, and `gofmt` are clean; `golangci-lint --fast-only` reported 0 issues; + `ginkgo -p --race` passed 358 of 358 specs. +- October 27, 02:13: `snapshotResource` now strips Service and Job fields filled in by the hub API server, matching + `generateRawContent` in `pkg/utils/controller/resource_selector_resolver.go`: + - Services: `clusterIP` and `clusterIPs` (unless headless), `healthCheckNodePort`, and each port's `nodePort`. + - Jobs without `manualSelector`: the `controller-uid` and `batch.kubernetes.io/controller-uid` labels in the + selector and the pod template, and the pod template's `creationTimestamp`. + - The new unit test `TestSnapshotResource` (in the new file `resources_test.go`) passes with `-race`. + - The plan adds PP-13 (Services) and PP-14 (Jobs) to section E; implementation follows. + - Another entry for the PR description: Service and Job manifests in snapshots no longer carry these fields. +- October 27, 02:20: PP-13 and PP-14 were implemented at the end of the first `Describe` in + `resources_integration_test.go`. There are two ordered contexts with 4 specs each. + - New constants: `serviceNameTemplate`, `jobNameTemplate`, and `legacyJobControllerUIDLabel`. + - New helpers in `utils_integration_test.go`, next to `newDeployment`: `newLoadBalancerService`, + `verifyServiceFieldsFilledIn`, `serviceSpecWithoutFilledInFields`, `newJob`, `verifyJobControllerUIDLabelsAdded`, + and `jobSpecWithoutControllerUIDLabels`. + - `deleteObjectsAndWait` takes optional `client.DeleteOption`s; existing callers are unchanged. The Job context + deletes with background propagation. + - Envtest facts confirmed by the preconditions: the API server (1.33) fills in all of the Service fields, and it + adds both the current and the legacy controller UID labels to the Job's pod template. + - Validation: + - `ginkgo -p --race --focus='manifest sanitization \((services|jobs)\)'` passed 8 of 355 specs (347 skipped). + Per the user, the whole suite was not run. + - `gofmt` and `go vet` are clean. `golangci-lint --fast-only` reported 0 issues. In the new code, the full run + flags only `goconst` findings, the same style as the rest of the files. +- October 27, 03:19: A cache miss in name-based selection is now confirmed with a read from the API server + (`resources.go`). + - `retrieveResourceByName`: when the informer cache returns NotFound, the resource is read again through the + dynamic client by `retrieveResourceByNameWithNoCache` (renamed from `getSelectedResourceFromAPIServer` at 03:28), + which the no-informer path now shares. A NotFound from that read is a User error ("selected resource does not + exist") that still wraps the NotFound; any other error stays an APIServer error. Other cache errors are unchanged. + - The cache and the direct read produce the same snapshot: the informer transform only strips `managedFields`, + which `snapshotResource` strips anyway. + - Test updates: + - A12 (PP-2, and the not-found case in CPP-1's table) now expects a User error with the new message. + - New context PP-2b "named resource not found (informer-backed)": a nonexistent Deployment (read through the + cache) is a User error; once the Deployment is created, a snapshot is taken right away, without waiting for the + cache. + - New unit test `TestRetrieveResourceByName` (9 cases, in `resources_test.go` next to `TestSnapshotResource`), + with `test/utils/informer.FakeManager` and a fake dynamic client; see the unit test table. + - `placementpolicy/controller_test.go`: the "selected resource gone" fixture uses the new error shape. + - Callers: the placement policy controller already treated NotFound as a selection error (no change); auto-rollout + now reports `FailedToSnapshotResourcesDuringAutoRollout` and waits for the next rollout window instead of + retrying; a staged update run fails its initialization (B11 in the staged update run plan). + - Validation: `ginkgo -p --race ./pkg/v1/managers/placementresourcesnapshot`: 350 of 350 specs pass; the focused + unit tests pass with `-race`; `go vet` and `golangci-lint --fast-only` report 0 issues. + - 03:35: breadcrumb fixes: A12, section F, the unit test table, the PR description list, and the note on contexts + skipped by the positional IDs (which also corrects PP-5 and PP-9, shifted by the non-controller owner references + context). + - 03:45: merged `main` (#1018, the Service and Job sanitization in the 02:13 and 02:20 entries above). + `resources_test.go` was added on both sides; it now holds both `TestRetrieveResourceByName` and + `TestSnapshotResource`. Validation: both unit tests pass with `-race`; `ginkgo -p --race + ./pkg/v1/managers/placementresourcesnapshot` passes 358 of 358 specs (347, plus 3 for PP-2b and 8 for PP-13 and + PP-14). +- October 27, 04:10: Merged `main` (#1018 and #1019) into `feat/handle-incomplete-snapshots-in-work-generator`. Only + this plan conflicted; both sides' additions were kept, in time order. + - The October 26 entries (16:20 to 18:05) and the October 27 entries (02:13 to 03:45) were made on separate + branches. The October 26 code is on `feat/clean-up-orphaned-secondary-snapshots`, which `main` does not have yet. + Spec counts in each set are relative to its own branch. + - Expected conflict: that branch's `resources_test.go` (U12 and U13) and the one on `main` both define + `TestSnapshotResource`. Combine the two when that branch merges `main`. diff --git a/.github/.copilot/breadcrumbs/2026-10-20-1723-stagedupdaterun-integration-test-plan.md b/.github/.copilot/breadcrumbs/2026-10-20-1723-stagedupdaterun-integration-test-plan.md new file mode 100644 index 000000000..a797fcaff --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-20-1723-stagedupdaterun-integration-test-plan.md @@ -0,0 +1,855 @@ +# Staged Update Run Controller (v1) Integration Test Plan + +**Date**: October 20, 2026 17:23 AEDT +**Task**: Plan and implement the integration tests for `pkg/v1/controllers/stagedupdaterun`, built on the envtest +suite in [suite_test.go](../../../pkg/v1/controllers/stagedupdaterun/suite_test.go). All planned scenarios are +implemented; see [Status](#status) for the progress log and the follow-ups from the review. + +## Requirements + +1. Cover the new controller end to end against a real API server (envtest), for both `ClusterStagedUpdateRun` and + `StagedUpdateRun`. +2. Do not port the old `pkg/controllers/updaterun` tests as-is; the new controller behaves differently (see + [Old tests → new plan](#old-tests--new-plan)). +3. Follow repo conventions: Ginkgo/Gomega, `_integration_test.go`, a new `Context` per scenario group, + whole-struct `cmp.Diff` comparisons, `want` naming. +4. Keep the package fast enough for `make local-unit-test` (estimate below). + +## Controller behavior under test + +Reconcile flow (see `controller.go`): + +1. Deleting → `cleanup`: relinquish the binding manager role on the linked policy (skipped if the policy is gone), + drop finalizer `rollout.kubefleet.dev/staged-update-run-cleanup`. +2. `shouldSkipProcessing` → `cleanup` and stop when `Initialized=False`, `Completed` exists, or + `Initialized=True && spec.suspended`. The suspended path sets `Suspended=True/Suspended`; otherwise a previously + suspended run gets `Suspended=False/Resumed`. Skips resync `ObservedGeneration` of `Initialized`/`Completed`. +3. Add the cleanup finalizer. +4. Get the linked policy (same namespace as the run). Not found → terminal: `Initialized=False/Failed` + ("No linked placement policy") before init, `Completed=True/Failed` after init. +5. Claim the binding manager role (`policy.status.bindingManager`, controller name `staged-update-run`). Several runs + may co-hold it; if another controller holds it, back off with the claim rate limiter and write no status. +6. Initialize **once** (never re-validated afterwards). User errors → `Initialized=False/Failed` (terminal): + - strategy lookup (missing → user error); + - bindings listed by the `spec.placementPolicyName` field index, deleting ones excluded; + - member clusters fetched by name; missing/deleting → binding silently dropped; none left → user error; + - grouping: first matching stage wins; `nil` selector = empty stage; empty selector = all remaining clusters; + clusters matched by no stage are left out of the run; sort by integer `sortingLabelKey` (name tie-break) or by + name; missing / non-integer sort label → user error; + - `maxConcurrency`: int or percent (rounded down), min 1, default 1; + - snapshot: explicit `resourceSnapshotName` must exist, not be deleting, be primary (sub-index `"0"`), and be + controlled by the policy; otherwise `SnapshotResourcesIfStale` (create index 0 / reuse latest / create next), + whose user errors also fail the initialization, e.g., a resource selected by name that does not exist (the + snapshot manager confirms a cache miss with a read from the API server before reporting it). +7. Re-check skip (a run created suspended is initialized, then suspended and cleaned up). +8. `process`, bounded by `refreshPeriod` (30s minimum in `New`; the suite overrides it to 5s, see decision 1): + - run `Started`; per stage: `StartedTimestamp`, `Started`, then before-stage tasks (approval only); + - clusters via a worker pool of `min(maxConcurrency, 10)`; per cluster: set + `binding.spec.resourceSnapshotName`, set `Started`, poll (cached client, availability rate limiter) until + `Synchronized` and `Available` are True at the post-update binding generation; + - shortcuts / failures: `binding.spec.suspended` → success; snapshot changed by someone else → `FailedToUpdate` + (conflict); binding missing / deleting / cluster-name mismatch → `FailedToUpdate`; elapsed > + `maxWaitTimePerClusterMinutes` → `FailedToUpdate` (timeout); + - failures are counted across stages; more than `maxAllowedClusterFailures` (default 1) → cancel in-flight + clusters, stage `Failed`, run `Completed=True/Failed`. Every cluster in the failing stage with `Started` but no + `Completed` is marked `Completed=True/UpdateCanceled` (`markAllOngoingUpdatesInStageAsCanceled`); clusters that + never started get no conditions; + - then after-stage tasks (approval and/or timed wait, both must clear); stage and run complete as + `Succeeded` or `PartiallySucceeded`. +9. Status is written **once at the end of Reconcile** (also on errors). Requeue: 200ms by default, the remaining + time for timed waits, and **no requeue** while waiting on approvals (the approval request watch re-triggers). + +Watches: runs (`GenerationChangedPredicate`, i.e. only spec changes such as suspend toggles) and +`ApprovalRequest`/`ClusterApprovalRequest` (create with `Approved` present, `Approved` added/changed, delete). +Bindings, policies, strategies, member clusters and snapshots are **not** watched. + +## Test environment constraints and design implications + +1. **Only this controller runs.** Tests create bindings by hand (no placement policy controller), simulate binding + status (no work generator/applier), and clean up owner-referenced objects themselves (no GC in envtest). The lack + of GC also makes the "stale approval request from a same-named run" path testable naturally. +2. **Run status visibility.** A reconcile blocks in the cluster wait loop for up to `refreshPeriod` and persists + status only at the end. With the suite's 5s override, in-progress run status (e.g., a cluster `Started` but not + `Completed`) shows up within ~5s of a cluster stalling, so: + - assert ordering and progress primarily on side effects written immediately: binding spec, approval requests, + the policy's binding manager claim, the finalizer; + - assert run status at rest points (waiting on approval or timed wait, suspended, completed, failed, init failed), + and for in-progress states after a budget expiry (≤5s + polling), as in C1 and C13. +3. **Writing to a run mid-reconcile** (e.g., toggling `suspended`) makes the in-flight reconcile's status update + conflict and be dropped; the next reconcile recomputes. Assertions must not depend on that lost write. +4. **Suspension is not immediate.** An in-flight reconcile keeps processing (and may start new clusters or stages) + until it blocks or the budget runs out. Suspension tests keep the in-flight cluster unavailable until + `Suspended=True` is observed. +5. **Simulated availability.** The helper waits until the controller has updated + `binding.spec.resourceSnapshotName` (generation bumped), then sets `Synchronized` and `Available` to True with + `ObservedGeneration = generation`. Writing binding status earlier can conflict with the controller's spec update + (shared `resourceVersion`) and produce an erred cluster and a requeue. +6. **Timeouts are expensive.** The CRD minimum for `maxWaitTimePerClusterMinutes` is 1, so the timeout path costs at + least 60s. Failure-policy tests use cheap failure injection instead (conflict, missing binding, deleting binding, + cluster-name mismatch); one context (C12) covers real timeouts with four runs (two per scope) in parallel. +7. **Fast rate limiters** (250ms–2s in the suite) make claim retries and availability polling converge within ~2s. +8. **Eventually budgets:** 10s (`eventuallyDuration`) for rest-point assertions, 20s + (`eventuallyDurationForBudgetExpiry`) for anything that waits on a budget expiry, and ~120s for real timeouts. +9. **Cache lag on fixtures.** The controller reads policies, strategies, bindings and member clusters from the cache it + shares with the tests; a strategy or binding the cache has yet to see fails initialization terminally. Wait for all + fixtures to be cached (`waitForObjectsToBeCached`) before creating a run; for objects marked for deletion, wait + until the cache sees the deletion timestamp (`markObjectForDeletion` does this). +10. **Unblocking in-flight rollouts at cleanup.** A run waiting on an unavailable binding holds its reconcile for up + to `refreshPeriod`, which would delay its deletion. Deleting the bindings unblocks it: the wait loop fails to + find the binding on its next poll (≤2s with the suite's rate limiters), the reconcile returns, and the deletion + is processed. `removeObjects` deletes runs and bindings together (in its first phase), so contexts that leave a + run mid-rollout (e.g., B9) need no extra steps. +12. **Approval requests go last at cleanup.** Deleting an approval request makes the controller process the run it + references; if the cache still has the run, the controller may re-create the request, which then never goes + away (no GC). `removeObjects` therefore removes (and confirms the removal of) everything else, runs included, + before it deletes any approval request; the request's delete event then finds no run to process. +11. **Injecting snapshot conflicts.** A reconcile that reads a run whose cached copy has yet to record a cluster's + `Started` treats that cluster's binding as not yet updated, and points it back at the snapshot to roll out. Before + changing a binding's snapshot behind the controller's back (C7, C10), wait for the persisted status to show the + cluster `Started`; the cache is shared and monotonic, so the controller will not read an older copy afterwards. + +## Suite changes + +Added so far: + +- A second app namespace, `otherAppNamespaceName` (`other-app`), created in `BeforeSuite` alongside `app`, for the + namespace isolation checks (B5, B6). envtest cannot delete namespaces, so both are created once per process. +- `refreshPeriodForTesting` (5s, decision 1): the suite builds the reconciler with `New(..., minRefreshPeriod)` (which + rejects anything shorter) and then sets `reconciler.refreshPeriod` directly before `SetupWithManager`, with a + comment on the deviation from production. Added with C1; all of group B still passes with it. + +Still to add as needed: nothing so far. The timing constants (`eventuallyDuration`, `eventuallyInterval`, +`consistentlyDuration`, `consistentlyInterval`, `eventuallyDurationForBudgetExpiry`, +`consistentlyDurationAcrossBudgetExpiries`, `eventuallyDurationForClusterTimeout`) live in `utils_integration_test.go`, +as in the workgenerator tests. + +Dropped: exposing the placement resource snapshot manager. B8 creates its snapshots directly +(`createPlacementResourceSnapshot`) instead; the controller only checks a snapshot's existence, deletion timestamp, +sub-index label and controller owner, and the manager's garbage collection only runs after the manager itself +creates a snapshot, so hand-made snapshots are left alone. + +## Shared test utilities (`utils_integration_test.go`) + +Scope-agnostic helpers built on the accessor interfaces, parameterized by a namespace (`clusterScopedNS`, i.e. `""`, +for cluster-scoped objects, `appNamespaceName` for namespaced ones). Following the existing v1 integration tests +(e.g., workgenerator), every scenario group runs for both scopes (decision 2) as two `Describe` blocks, one per scope +(namespaced first, then cluster-scoped), each with its own `Ordered` contexts. Scope-specific checks live only in +the namespaced block (see the scope notes below). + +Implemented (all of A1–A10, B1–B10, C1–C13, D1–D8, E1–E3): + +- **Object stubs**: `newPlacementPolicyObj`, `newPlacementBindingObj`, `newPlacementResourceSnapshotObj`, + `newStagedUpdateStrategyObj`, `newStagedUpdateRunObj` (namespace + name, scope picked by the namespace), + `newConfigMapObj` (app namespace). +- **Creators**: `createConfigMap`, `createPlacementPolicy` (selects a ConfigMap by name), `createMemberCluster`, + `createPlacementBinding`, `createStagedUpdateStrategy(stages)`, `createStagedUpdateRun(spec)`. +- **Member cluster fixtures**: `memberClusterAndBindingNames(randStr, count)` (names `cluster--NN`, zero-based and + zero-padded so that lexical order matches index order), `createMemberClustersAndBindings(ns, policy, names, + bindingNames, labels)` (created in reverse name order; bindings point at a placeholder previous snapshot), and + `memberClusterAndBindingObjs` (stubs for cleanup). +- **Cache sync**: `waitForObjectsToBeCached(objs...)` (see constraint 9). +- **Checkers**: `stagedUpdateRunStatusUpdatedActual(ns, name, want)` (sets every condition's `ObservedGeneration` + to the run's generation), `stagedUpdateRunFailedToInitializeActual(ns, name, msgSubstr)` (only + `Initialized=False/Failed`, plus a message substring), `primaryPlacementResourceSnapshotCreatedActual(policy, + index)` (labels and owner ref), `objectRemovedActual(obj)`. +- **Want builders**: `wantStagedUpdateRunStatusInitializedAndSuspended(snapshot, stages)`, + `wantPerClusterStatuses(names, bindingNames, idxs...)` (expected cluster order by index). +- **Snapshots**: `createPlacementResourceSnapshot(name, ownerPolicy, index, subIndex, count)` (labels + controller + owner ref, no resources), `primaryPlacementResourceSnapshotName`, `secondaryPlacementResourceSnapshotName`. +- **Updaters**: `updateConfigMapData`, `updateMemberClusterLabels`, `updateStagedUpdateStrategyStages`, + `updateStagedUpdateRunSuspended` (all retry on conflicts). +- **Deletion fixtures**: `markObjectForDeletion(obj)` (adds `testFinalizer`, deletes, waits for the cache to see the + deletion timestamp) and `removeTestFinalizer(obj)` (call in `AfterAll` before `removeObjects`). +- **More checkers**: `stagedUpdateRunConditionActual(ns, name, cond)` (one run-level condition at the current + generation), `stagedUpdateRunStageLayoutActual(ns, name, stages)` (stages/clusters/tasks/concurrency, progress + ignored), `stagedUpdateRunCleanupFinalizerRemovedActual`, `bindingManagerRoleReleasedActual(policyNS, policyName)`, + `placementBindingResourceSnapshotActual` / `placementBindingsResourceSnapshotActual` (also used with the previous + snapshot name to assert bindings are untouched). +- **Message substrings**: the initialization failure substrings are constants in `initialize_integration_test.go` + (e.g., `strategyNotFoundMsgSubstr`), so the cluster-scoped copy of the tests cannot alter them. +- **Cleanup**: `removeObjects(objs...)` works in two phases (everything but approval requests, then approval + requests; see constraint 12); within each phase, it issues all deletions first, then waits for each to be gone, so + cleanup does not cost one polling interval per object. `AfterAll` blocks build the list with `slices.Concat` + (keeps the `prealloc` linter quiet). +- **Unique names** per context (random suffix via `utils.RandStr()`), so leftovers never collide; member clusters + are cluster-scoped, so their names carry the suffix too. +- **Rollout (C1)**: + - simulator `markPlacementBindingAsAvailable(ns, name, wantSnapshot)` (waits for the binding to point at the + snapshot, then sets `Synchronized`/`Available` True at the binding's generation; see constraint 5); + - condition builders `stagedUpdateRunInitializedCond`, `stagedUpdateRunStartedCond`, + `stagedUpdateRunCompletedCond(reason)`, `perStageStartedCond`, `perStageCompletedCond(reason)`, + `perClusterStartedCond`, `perClusterCompletedCond(reason)`, plus `perClusterUpdateSucceededConds()` / + `perStageSucceededConds()` and `wantPerClusterStatus(names, bindingNames, idx, conds...)`; + - checkers `stagedUpdateRunRolloutStatusActual(ns, name, want)` (full status, stage timestamps ignored; shares its + comparison core with `stagedUpdateRunStatusUpdatedActual`) and `stagedUpdateRunStagesCompletedInOrderActual` + (every stage started and completed; each stage starts no earlier than the previous one completed, at the + seconds precision of `metav1.Time`). +- **Rollout (C2–C5)**: + - `markPlacementBindingAsAvailable` is now a thin wrapper around `setPlacementBindingConditions(ns, name, + wantSnapshot, synchronizedStatus, availableStatus, observedGenerationOffset)` (C3 uses offset -1 for a stale + observation and `Available=False` for a synchronized-only binding); + - `updatePlacementBindingSuspended(ns, name, suspended)` waits until the cache observes the flag, so that the + controller deterministically takes the "suspended before the update" shortcut (C5); + - checkers `placementBindingGenerationActual(ns, name, gen)` (C4: spec never updated) and + `perClusterConditionMessageActual(ns, run, cluster, condType, substr)` (C5: both suspended shortcuts report + "binding is suspended"; the substring is a constant in `process_integration_test.go`). +- **Rollout (C6–C9)**: + - `updatePlacementBindingSpec(ns, name, mutate)` updates a binding spec and waits until the cache observes the new + generation; `updatePlacementBindingSuspended` is now a wrapper around it. C7 uses it for conflict injection + (snapshot name) and C9 for retargeting (cluster name); + - condition builders `stagedUpdateRunSuspendedCond` (now also used by + `wantStagedUpdateRunStatusInitializedAndSuspended`), `stagedUpdateRunResumedCond`, + `perStageTaskTimedWaitStartedCond`, `perStageTaskWaitTimeElapsedCond`, and `perClusterUpdateFailedConds()` + (`Started` + `Completed/FailedToUpdate`; failures detected before the binding is updated have no `Started`); + - checker `stagedUpdateRunStageLastedAtLeastActual(ns, name, stage, minDuration)` (stage timestamps; holds at the + seconds precision of persisted timestamps, as the controller measures waits against persisted timestamps too). +- **Rollout (C10–C13)**: + - `getPerClusterCondition(ns, run, cluster, condType)` (C13 reads the `Started` transition time; + `perClusterConditionMessageActual` now builds on it) and `perStageConditionMessageActual(ns, run, stage, + condType, substr)` (C10/C11 stage failure messages, via `stageFailedMsgSubstrFmt`); + - timing constants `consistentlyDurationAcrossBudgetExpiries` (`2*refreshPeriodForTesting + 2s`, C13) and + `eventuallyDurationForClusterTimeout` (2 min, C12); + - `slowTestLabel` (`"slow"`) on C12; skip it with `--label-filter='!slow'`. +- **Stage tasks (D1)**: + - `newApprovalRequestObj(ns, name)` (scope picked by the namespace) and `stagedUpdateRunKind(ns)`; + - simulator `approveApprovalRequest(ns, name)` (sets `Approved=True` at the request's generation, reason + `markedAsApprovedReason`); + - checkers `approvalRequestCreatedActual(runNS, runName, requestName, stage)` (spec `{stagedUpdateRunName, + stageName}` and a single controller owner ref to the run, with its UID) and + `approvalRequestApprovalAcceptedActual(ns, name)` (`Approved` + `ApprovalAccepted=True/ApprovalAccepted` at the + request's generation); + - condition builders `perStageTaskApprovalRequestCreatedCond`, `perStageTaskApprovalRequestApprovedCond`; + - the expected request name is built from a test-side template (`beforeStageApprovalRequestNameTemplate` in + `tasks_integration_test.go`), not from the controller's own formatter. +- **Stage tasks (D2–D8)**: + - `approveApprovalRequest` / `rejectApprovalRequest` now share `setApprovalRequestApproved(ns, name, status)` + (reasons `markedAsApprovedReason` / `markedAsRejectedReason`; waits until the cache observes the new status, so + that checks right afterwards, e.g., D3's and E1's Consistently checks, do not read a stale copy); `createManualApprovalRequest(ns, name, run, + stage)` (no owner refs) and `getApprovalRequest(ns, name)`; + - checkers `approvalRequestConditionsActual(ns, name, conds)` (exact conditions at the request's generation; + `approvalRequestApprovalAcceptedActual` now wraps it), `approvalRequestAbsentActual`, + `approvalRequestReplacedActual(ns, name, oldUID)` (exists with a different UID), and + `perStageTaskTimedWaitLastedAtLeastActual(ns, run, stage, min)` (after-stage TimedWait: `WaitTimeElapsed` at + least `min` after `TimedWaitStarted`, and the stage completed no earlier); + - builders `approvalRequestApprovedCond(status)`, `approvalRequestApprovalAcceptedCond`, + `perStageTaskApprovalRequestOwnershipMismatchCond`, `wantApprovalTaskStatus(name, conds...)`, + `wantTimedWaitTaskStatus(waitTime, conds...)`, and (in `tasks_integration_test.go`) + `wantSingleClusterCompletedStatus`; + - deleting an approval request that the controller re-creates (D6, D8) uses a plain `hubClient.Delete`: + `removeObjects` would wait for the name to disappear, which it never does. +- **Lifecycle (A1)**: + - checker `stagedUpdateRunCleanupFinalizerAddedActual(ns, name)` (counterpart of the "removed" one); + - checker `bindingManagerRoleHeldActual(policyNS, policyName, wantRefs)` (`status.bindingManager` equals + `{controllerName, wantRefs}`, refs in any order via `lessFuncObjectRef`; the controller name comes from the + package's `controllerName` constant), with `wantStagedUpdateRunObjectRef(ns, name)` building each ref + test-side (`rollout.kubefleet.dev` / `v1alpha1` / `StagedUpdateRun` or `ClusterStagedUpdateRun` / namespace / + name). +- **Lifecycle (A2–A10)**: + - `bindingManagerRoleHeldActual` now wraps a generic `bindingManagerActual(policyNS, policyName, want)` (also used + for another controller's claim in A6); `updatePlacementPolicyBindingManager(ns, name, bm)` writes + `status.bindingManager` directly and deliberately does not wait for the cache (the controller may claim a + released role right away; A6 waits for the other controller's claim explicitly before creating the run); + - `wantStagedUpdateRunOwnerRefs(ns, name, uid)` (now shared by `approvalRequestCreatedActual`) and + `approvalRequestOwnerRefsActual(ns, name, wantRefs)` (A8 checks the owner ref survives the run's deletion); + - `stagedUpdateRunConditionMessageActual(ns, name, condType, substr)` (A5's `Completed` message); + - in `lifecycle_integration_test.go`: `beforeStageApprovalStages()`, `wantWaitingForBeforeStageApprovalStatus(...)` + (A1 refactored onto both), `triggerApprovalRequestNameTemplate`, `noLinkedPlacementPolicyMsgSubstr`, and the + `otherBindingManager*` constants. + - Triggering a reconcile without changing a run: `triggerStagedUpdateRunReconciliation(ns, run, stage, + requestName)` creates an approval request whose spec names the run, waits for the cache to see it, deletes it + (the delete handler enqueues the named run), and waits for the cache to see the deletion. Used by A1, A3, A4, + and E2, before Consistently checks that would otherwise pass without the controller ever looking at the run. +- **Suspension (E1–E3)**: + - `stagedUpdateRunRolloutStatusAsIsActual(ns, name, want)`: like `stagedUpdateRunRolloutStatusActual`, but + compares condition `ObservedGeneration`s as given in the want (the shared core now takes a + `setObservedGenerations` flag); needed once a run's generation changes, as the controller refreshes some + conditions' generations but not others (see behavior to note 7); + - `observedAt(gen, cond)` sets the generation on a condition builder's output. + +Still to add: nothing; all planned scenarios are implemented. (Message substring checks, listed here earlier, exist: +`stagedUpdateRunFailedToInitializeActual`, `perClusterConditionMessageActual`, `perStageConditionMessageActual`, +`stagedUpdateRunConditionMessageActual`.) + +## Test plan + +File layout mostly mirrors the source files; the lifecycle and suspension tests (A and E) share +`lifecycle_integration_test.go`. Ordered contexts with step-wise `It`s. Every scenario group runs for both +scopes unless a scope note says otherwise. + +### A. `lifecycle_integration_test.go`: lifecycle and bookkeeping + +- **A1** While waiting on a before-stage approval: the finalizer is present and + `policy.status.bindingManager == {controllerName: staged-update-run, objectRefs: [run]}`. +- **A2** After a successful run: `Completed=True/Succeeded`, finalizer removed, binding manager cleared; the run + object remains. +- **A3** A completed run is not reprocessed: change a binding's snapshot or availability; the run's status and the + binding stay as they are (Consistently). +- **A4** Policy missing at creation → `Initialized=False/Failed` "No linked placement policy", finalizer removed; + creating the policy later does not revive the run. +- **A5** Policy deleted mid-run (while waiting on an approval) → `Completed=True/Failed` "No linked placement policy", + finalizer removed. +- **A6** Claim contention: `bindingManager` pre-set to another controller → finalizer added, the run's status stays + empty (binding manager conflicts are not reported in the status), the other claim stays, bindings untouched + (Consistently ~5s); clear it → the run initializes, claims, and gets `Started=True/UpdateStarted`. +- **A7** Co-holding: two runs on one policy (both waiting on approvals) → both refs listed; deleting one removes only + its ref, and the other continues. +- **A8** Delete while a cluster update is in flight → the run is gone, the claim is released, and bindings keep their + current spec (no rollback). Approval requests carry a controller owner ref to the run (asserted, then deleted + manually since there is no GC). +- **A9** Delete after the policy was deleted → the finalizer is still removed (NotFound path). +- **A10** Delete a run that failed to initialize → removed immediately. + +### B. `initialize_integration_test.go`: one-shot initialization + +Most cases create the run with `suspended: true`, which gives a stable, fully initialized rest point (and releases +the claim), then compare the full status. + +- **B1** Happy path: + - two label-selected stages, sorted by name; `status.stages` matches the want, including resolved + `maxConcurrency` and task statuses (`WaitTime` copied for timed waits, no request name yet); + - `Initialized=True/PreppedResourceSnapshotAndAllStages`; + - `resourceSnapshotNameToRollout == -resource-snapshot-0`, and that snapshot exists and is owned by the + policy. +- **B2** Grouping: overlapping selectors → the first stage wins; `nil` selector → empty stage; empty selector → all + remaining clusters; a cluster matched by no stage is left out of every stage. +- **B3** Sorting: integer ascending with name tie-break; missing label → init failed; non-integer → init failed. +- **B4** `maxConcurrency`: unset → 1; `3` with 2 clusters → 3; `"50%"` of 5 → 2; `"10%"` of 3 → 1; percent on an + empty stage → 1. +- **B5** Binding and cluster filtering: + - excluded: bindings of other policies (field index); deleting bindings (finalizer, then delete); bindings to + non-existent clusters; deleting clusters; + - namespaced scope only: bindings in another namespace with the same policy name are excluded; + - all excluded → init failed "no placement bindings are found". +- **B6** Strategy missing → init failed and terminal (creating the strategy later does not help). Namespaced scope + only: a strategy in another namespace is not found. +- **B7** Snapshot selection without an explicit name: none exists → index 0 created; up-to-date latest → reused + with no new snapshot; selected ConfigMap changed → index 1 created. +- **B8** Explicit snapshot: valid primary owned by the policy → used; not found / sub-index not `"0"` / owned by + another policy / marked for deletion → init failed. +- **B9** One-shot init: after init (suspended), change the strategy (add a stage, change a selector), cluster labels, + and add a binding; resume → stages unchanged. +- **B10** Created suspended → `Initialized=True`, `Suspended=True/Suspended`, no `Started`, bindings untouched, + claim released, finalizer removed. +- **B11** Selected resource missing: the policy selects (by name) a ConfigMap that does not exist → + `SnapshotResourcesIfStale` returns a user error ("selected resource does not exist") → init failed, claim released, + no snapshot created; creating the ConfigMap later does not help (terminal, like B6). + +### C. `process_integration_test.go`: rollout execution + +- **C1** Multi-stage happy path: canary (1 cluster) then prod (3 clusters, `maxConcurrency` 1, sorted). + - Stage 2 bindings stay untouched during stage 1, and cluster N+1's binding stays untouched until N is marked + available (Consistently). + - While cluster N is held unavailable, the persisted status shows it `Started` without `Completed` within one + budget (~5s). + - Final status: + - every cluster `Started` and `Completed/UpdateSucceeded`; + - stages `Completed/Succeeded` with ordered timestamps; + - the run `Started` and `Completed/Succeeded`; + - all bindings on the target snapshot. +- **C2** Concurrency: `maxConcurrency: 2` of 3 → two bindings updated together, the third waits for one to complete; + `"100%"` → all at once. +- **C3** Availability detection: stale True conditions (`observedGeneration` < generation) and + `Synchronized` without `Available` do not count; both True at the current generation do. +- **C4** A binding already on the target snapshot and available → completes without a spec update (generation + unchanged). +- **C5** Binding suspended: before processing → immediate success; suspended during the wait → success. Both report + "(binding is suspended)". +- **C6** Empty stage (`nil` selector) between two stages → `Started` and `Completed/Succeeded` with no clusters; its + after-stage tasks still run. +- **C7** Conflict: while waiting on cluster X, change its binding's snapshot → X `FailedToUpdate` (conflict); + default failure policy → stage and run `PartiallySucceeded`. +- **C8** Binding deleted after init (suspend, delete, resume) → `FailedToUpdate` "binding cannot be found". +- **C9** Binding deleting, or its `clusterName` changed after init → `FailedToUpdate`. +- **C10** Threshold within a stage (`maxAllowedClusterFailures: 0`, `maxConcurrency: 2`, three clusters A, B, C): + wait until A's and B's bindings are on the target snapshot (C still on the old one), keep B unavailable, then fail + A (conflict injection) → + - stage `Failed` (its message counts only `FailedToUpdate` clusters: "1 out of 3"), run `Completed/Failed`; + - A `Completed=True/FailedToUpdate`; B (in flight) `Started` plus `Completed=True/UpdateCanceled`; + - C (never started) has no conditions, and its binding stays on the old snapshot (Consistently); + - the next stage's bindings are untouched; + - claim released, finalizer removed. + - Keep B's binding unavailable throughout: if it became available right as the cancel happens, B could + legitimately finish as `UpdateSucceeded` instead. +- **C11** Threshold across stages (max 1): one failure in stage 1 (`PartiallySucceeded`), with a reconcile boundary + (a budget expiry while the next stage's first cluster is in flight) between the two stages, then one failure in + stage 2 → stage 2 `Failed`, run `Failed`. This exercises + `countClusterUpdateFailures` on persisted status. +- **C12** Real timeouts (`Label("slow")`), each run with one cluster that never becomes available: + - T1 (`maxWaitTimePerClusterMinutes: 1`, max 1) → `FailedToUpdate` (timeout), run `PartiallySucceeded`; + - T2 (max 0) → run `Failed`. + - Exception to the one-`Describe`-per-scope layout: one context creates T1 and T2 for both scopes at once (4 runs in parallel), so + the 1-minute minimum is paid once (~70–90s total). +- **C13** Budget exhaustion: one cluster never becomes available. After the 5s budget, the persisted status shows + run, stage and cluster `Started`, with no `Completed`. Mark it available → completes with no second spec update + (re-entry path). + +### D. `tasks_integration_test.go`: stage tasks and approval requests + +- **D1** Before-stage approval: + - request `-before-stage-` is created with a controller owner ref to the run and spec + `{stagedUpdateRunName, stageName}`; + - task `ApprovalRequestName` is set, with `ApprovalRequestCreated=True/RequestCreated`; + - the stage is already `Started`; bindings stay untouched (Consistently); + - approve → the request gets `ApprovalAccepted=True`, the task gets `ApprovalRequestApproved=True`, and the + rollout starts promptly (driven by the watch, not polling). +- **D2** After-stage approval: `-after-stage-` is created only after every cluster in + the stage is done; the stage does not complete and the next stage stays untouched until approved. +- **D3** `Approved=False` → no progress; flipping it to True → progress. +- **D4** After-stage timed wait (5s): `TimedWaitStarted`, then `WaitTimeElapsed` at least 5s later; the next stage + starts only after that. +- **D5** Approval and timed wait after one stage: both must clear (approve early → still waits for the timer; timer + done → still waits for the approval). +- **D6** Deleted approval request → re-created with the same name; the run proceeds once approved. +- **D7** Stale request from a deleted same-named run (no GC): create run R, wait for its request, delete R, + re-create R → the old request is deleted and re-created with the new run's UID as owner. +- **D8** Manually created request (no owner) with the expected name → `ApprovalRequestCreated=False/ + RequestOwnershipMismatch` and no progress; delete it → the controller creates its own and the condition returns to + `True/RequestCreated`. The manual request's `spec.stagedUpdateRunName` must name the run, so its delete event + re-enqueues the run. + +### E. Suspension (Context group in `lifecycle_integration_test.go`) + +- **E1** Suspend while waiting on an approval → `Suspended=True/Suspended`, claim released, finalizer removed; + approving while suspended makes no progress; resume → `Suspended=False/Resumed`, claim and finalizer re-acquired, + rollout proceeds. +- **E2** Suspend during a cluster update: + - keep the binding unavailable until `Suspended=True` is observed (within one ~5s budget); + - mark it available while suspended → no status change; + - resume → the cluster completes, and its `Started` `LastTransitionTime` is at or after the `Resumed` + transition (wait timer reset). +- **E3** `ObservedGeneration` resync: toggling `suspended` on a completed run bumps `Completed`'s + `observedGeneration` without adding `Suspended`; on an init-failed run it bumps `Initialized`'s. + +### Better as unit tests (out of scope here) + +- Approval request name truncation and hashing (`uniquename.go`). **Done (October 20, 2026)**: `uniquename_test.go` + covers both formatters with golden names (short, exactly 200 characters, 201 characters, a cut right after a dot, + and an over-long stage name), checks that truncated names stay apart (hash suffix) and that before/after names + differ, and runs every run-name length (1–253) with varying dot positions and three stage-name lengths, checking + the length limit, the hash suffix, and DNS-subdomain validity. `uniquename.go` is at 100% statement coverage; + removing the dot trimming makes all three relevant tests fail. +- The concurrency cap of 10 workers. +- The pre-loop quorum-read conflict path (racy to hit via envtest). +- Approval event handler filtering (`eventhandler.go`). +- Rejecting a before-stage `TimedWait` (the CRD already forbids it). +- The `UpdateCanceled` branches in `refreshOneCluster` (treated as failed) and `countClusterUpdateFailures` (counted + as a failure). The controller never reaches them: `UpdateCanceled` is only written into a stage that is marked + `Failed`, and failed stages are skipped before their clusters are looked at. + +## Old tests → new plan + +| Old area (`pkg/controllers/updaterun`) | New plan | +| --- | --- | +| Finalizer added | Adapted → A1 | +| Delete normal / failed / processing run | Adapted → A8–A10 | +| Delete removes approval requests | Changed: controller relies on owner-ref GC; assert owner ref (A8) | +| `validateCRP` (not found / strategy type / ApplyStrategy copy) | Not-found → A4; the rest dropped (no such fields) | +| `determinePolicySnapshot` (PickN / PickFixed / PickAll) | Dropped: no policy snapshots; bindings are listed by policy name | +| `collectScheduledClusters` (binding states, to-be-deleted, counts) | Dropped; replaced by B5 | +| Strategy not found | Adapted → B6 | +| After-stage task validation (duplicates, wait time) | Dropped: enforced by CRD CEL (belongs in `test/apis`) | +| Invalid sorting label | Adapted → B3 (plus missing label) | +| Cluster in multiple stages / in no stage → fail | Behavior changed → B2 (first match wins; unmatched left out) | +| Empty vs. nil selector | Adapted → B2 | +| Stage status generation | Adapted → B1 | +| `resourceSnapshotIndex` checks | Replaced by explicit `resourceSnapshotName` checks → B8 | +| Create / reuse / next-index snapshot | Adapted → B7 | +| Override snapshots | Dropped | +| `validation_integration_test.go` (re-validate every reconcile) | Dropped; replaced by B9 (one-shot init) | +| One-by-one rollout, approvals, timed waits | Adapted → C1, D1–D5 | +| Empty stage | Adapted → C6 | +| Stuck condition / metrics | Dropped (no metrics); replaced by timeout + failure policy (C10–C12) | +| Unexpected binding state abort | Replaced by C8 / C9 | +| Delete stage | Dropped (no delete stage) | +| Report-diff mode | Dropped (no report-diff in the new binding API) | +| Recreate deleted approval request | Adapted → D6 | +| State `Initialize` → `Run` | Replaced by created-suspended → resume (B10, E1) | +| `stop_integration_test.go` (`Stop` / `Run`) | Replaced by E1–E3 (suspend releases the claim; not immediate; wait timer reset on resume) | + +## Decisions (approved October 20, 2026) + +1. Override `refreshPeriod` to 5s in the suite (direct field override after `New`; see Suite changes). This makes + in-progress run status visible quickly, at the cost of deviating from the 30s production minimum. +2. Run every scenario group for both scopes, as one `Describe` block per scope (the existing v1 integration test + pattern, adopted instead of the originally proposed `DescribeTableSubtree`); scope-specific checks (B5 namespace + isolation, B6 cross-namespace strategy) run in the namespaced block only. +3. Include real timeouts in C12 only, with all four runs (T1 and T2 for both scopes) in parallel, labeled `slow`. +4. Simulate binding availability by hand in each test (no auto-responder), so ordering can be asserted. + +## Behaviors to note (documented only; no action for now) + +The tests should assert the current behavior below as-is. None of these items is to be changed as part of this work. + +1. An approval ownership mismatch returns a user error from Reconcile, so recovery uses controller-runtime's + exponential backoff; after the manual request is deleted, a fast re-trigger depends on that request naming this + run. +2. Clusters matched by no stage are silently left out of the run, with no condition or message. +3. Suspension relinquishes the binding manager role, so other managers may move bindings while a run is suspended; + on resume this surfaces as `FailedToUpdate` (conflict). +4. Suspension only takes effect after the in-flight reconcile ends; until then it may keep starting clusters and + stages. +5. A missing policy at creation is terminal, even if the policy appears later. The same holds for a resource selected + by name that is missing at initialization (B11). +6. **Resolved (October 20, 2026).** Previously, in a run that failed on the threshold, in-flight clusters kept + `Started` without `Completed`. They are now marked `Completed=True/UpdateCanceled` (C10). Verified against envtest + with a throwaway test (A conflict → `FailedToUpdate`, in-flight B → `UpdateCanceled`, unstarted C untouched). + Remaining nuances, documented only: + - An in-flight cluster whose binding becomes available right as the cancel happens may finish as + `UpdateSucceeded` instead of `UpdateCanceled`. + - Narrow, pre-existing race: if a binding spec update is on the wire when the context is canceled, the API server + may apply it while the controller sees an error. That cluster then has no conditions even though its binding + moved. Not reliably testable via envtest. +7. Stage-task conditions (`ApprovalRequestCreated`, `ApprovalRequestApproved`, `TimedWaitStarted`, `WaitTimeElapsed`) + keep the `ObservedGeneration` they were set at; unlike run-, stage- and cluster-level conditions, they are never + refreshed when the run's generation changes (e.g., after a suspend and resume). E1 asserts a completed run whose + approval task has `ApprovalRequestCreated` at generation 1 and `ApprovalRequestApproved` at generation 3. Related: + while a run is suspended, only `Initialized` (and `Suspended`) carry the new generation; on a completed run, a + generation change refreshes only `Completed` (E3). +8. Suspending a run mid-reconcile makes that reconcile's final status write fail with a conflict (the run changed), + so whatever it learned is dropped and the next reconcile works from the last persisted status. E2 therefore + suspends only after the persisted status shows the in-flight cluster `Started`. + +## Running the tests + +Use the Ginkgo CLI in parallel mode rather than `go test`, via `go run` so that the CLI matches the Ginkgo version +pinned in `go.mod` (v2.33.0; a v2.23.4 binary in `GOPATH/bin` is older): + +```bash +KUBEBUILDER_ASSETS="$(make -s kubebuilder-assets-path | tail -1)" \ + go run github.com/onsi/ginkgo/v2/ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun +``` + +Each parallel process runs its own `BeforeSuite` (its own envtest API server, manager, and controller), as the suite +already does; `Ordered` contexts stay within one process, and the random name suffixes keep contexts apart within a +process. Add `--focus=""` to run a subset, and `--label-filter='!slow'` to skip the ~1-minute +per-cluster timeout context (C12). Run each change once; repeated runs (flake hunting) are not needed for now. + +In CI (`.github/workflows/ci.yml`), the package is skipped in the `make test` step (`TestMain` in `suite_test.go` +exits early when `KUBEFLEET_CI_TEST_RUNNER_NAME` is `default`, as the work applier tests do) and runs in its own step +instead: `ginkgo -v -p --procs=4 --race --cover -coverprofile=staged-update-run-it-coverage.out +./pkg/v1/controllers/stagedupdaterun/` with `KUBEFLEET_CI_TEST_RUNNER_NAME=ginkgo`. Run locally the same way (with +the v2.23.4 CLI that CI installs): 610 of 610 specs pass, ~207s of spec time, ~3m44s wall, 85.9% coverage (measured +before B11, which brought the total to 620 specs). + +## Estimated runtime + +With the 5s refresh period, about 6 min per scope (A ~1 min, B ~1.5 min, C ~1.5–2 min without C12, D ~1 min, +E ~1 min), so ~12 min for both scopes, plus ~1.5 min for C12: roughly 13–14 min of serial spec time. With +`ginkgo -p`, wall time drops to roughly the longest context (C12, ~1.5 min) plus the share of the rest per process; +measured after group C: ~80s of spec time, ~1m33s wall (including compilation), dominated by C12; after group D: +~115s of spec time, ~2m07s wall; after group A: ~102s of spec time, ~1m55s wall; with everything (606 specs): ~99s of +spec time, ~1m51s wall; after B11 (620 specs, October 27): ~150s of spec time, ~2m39s wall. +Outside CI, `make local-unit-test` still runs this package serially under `go test` within the 30m timeout. + +## Status + +Plan approved with the decisions above. **All planned scenarios (A1–A10, B1–B11, C1–C13, D1–D8, E1–E3) are +implemented**; see the last entry below for the final validation. + +- October 20, 2026: updated for the per-cluster `Canceled` reason (behavior to note 6 resolved; C10 and the + unit-test list adjusted). +- October 20, 2026: **B1 implemented** in `initialize_integration_test.go` (context "label-selected stages", both + scopes), with the shared utilities in `utils_integration_test.go`. Five member clusters (two canary, three prod, + created in reverse name order); canary stage: before-stage approval, after-stage 10m timed wait, default + concurrency; prod stage: `maxConcurrency: 2`, after-stage approval + 5m timed wait. Asserts the full run status + (stages, cluster order, task statuses, resolved concurrency, `Initialized` and `Suspended` conditions, snapshot to + roll out) and the created primary snapshot (labels, owner ref). Validation: + - `go test -race ./pkg/v1/controllers/stagedupdaterun/...`: 4 of 4 specs pass (~11s) (later runs use + `ginkgo -p`; see Running the tests); + - negative check (expected prod concurrency changed to 3): fails with a clear status diff; + - golangci-lint: no findings in the new test files. +- October 20, 2026: **B2–B4 implemented** in `initialize_integration_test.go` (both scopes; B1 refactored onto the + new member cluster helpers). Contexts per scope: + - B2 "overlapping, nil, and empty label selectors": stages east (region=east), prod (env=prod), empty (nil + selector), rest (`{}` selector); member cluster 1 matches east and prod and lands in east; cluster 3 is + picked up by rest; the empty stage has no clusters. + - B2 "clusters selected by no stage": a cluster without an env label is left out of both stages. + - B3 "clusters sorted by an integer label": `order` values 10, 2, 1, 2 → order 2, 1, 3, 0 (numeric, not lexical; + tie broken by name). + - B3 "clusters missing the sorting label" / "clusters with a non-integer sorting label": the run (not + suspended) ends with only `Initialized=False/Failed`, message containing "yet the label is missing on a + cluster" / "failed to parse sorting label value as integer". + - B4 "max concurrency resolution": default → 1; 3 with 2 clusters → 3; "50%" of 5 → 2; "10%" of 3 → 1; "100%" + of an empty stage → 1. + - The cluster-scoped `Describe` is a mechanical copy of the namespaced one (namespace and wording only). + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 16 of 16 specs pass (~14s spec time); + golangci-lint: no findings in the test files. +- October 20, 2026: **B5–B10 implemented** in `initialize_integration_test.go` (both scopes unless noted). The + file now holds 16 namespaced and 14 cluster-scoped contexts; the cluster-scoped `Describe` is still a mechanical + copy, minus the namespaced-only contexts. Contexts added per scope: + - B5 "placement bindings and member clusters to leave out": 6 clusters; only 0 and 1 make it into the stage + (2: binding of another policy; 3: binding marked for deletion; 4: member cluster missing; 5: member cluster + marked for deletion). + - B5 "no placement bindings that count": all bindings filtered out → `noPlacementBindingsMsgSubstr`. + - B5 (namespaced only) "placement bindings in another namespace": a same-policy-name binding in `other-app` is + left out. + - B6 "missing staged update strategy": init fails; creating the strategy and then bumping the run generation + (suspend) yields a re-processed run that is still failed (condition observed at generation 2). + - B6 (namespaced only) "staged update strategy in another namespace": not found → init fails. + - B7 "resource snapshot selection (no resource snapshot specified)": run 1 creates index 0; run 2 reuses index 0; + after a ConfigMap change, run 3 creates and uses index 1. + - B8 "resource snapshot selection (resource snapshot specified)": hand-made snapshots; valid primary → used; + missing / secondary (sub-index 1) / other policy's / marked for deletion → init fails with the matching message. + - B9 "one-shot initialization": after init (suspended), the strategy (selector changed, stage added), a cluster's + labels and a new cluster + binding change so that a re-initialization would reorder clusters; on resume, + `Suspended=False/Resumed`, the stage layout is unchanged, cluster 0's binding moves to the snapshot, and the + others (including the new cluster's) stay on the previous snapshot (Consistently). + - B10 "staged update runs created as suspended": full status (no `Started`), finalizer removed, binding manager + role released, bindings untouched (Consistently). + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 68 of 68 specs pass (~25s spec time, + ~36s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **C1 implemented** in `process_integration_test.go` (both scopes; context "multiple stages, one + cluster at a time"), together with the 5s refresh period override in the suite. Four member clusters: 0 (canary), + 1-3 (prod, `order` labels 30/10/20, so the prod rollout order is 2, 3, 1); no stage tasks; run not suspended. + For each cluster in turn: its binding moves to the snapshot (Eventually); the bindings of the clusters that come + later stay on the previous snapshot (Consistently, 5s); the persisted status shows the cluster `Started` without + `Completed` (full status, within a budget expiry); then the binding is marked available. Finally: the run, both + stages, and every cluster are `Completed` as succeeded, the stages completed in order (timestamps), and all + bindings point at the snapshot. + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 102 of 102 specs pass (~35s spec time, + ~45s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **C2–C5 implemented** in `process_integration_test.go` (both scopes). Contexts per scope: + - C2 "concurrent updates within a stage": canary stage (clusters 0-2, `maxConcurrency: 2`) then prod stage + (clusters 3-5, `"100%"` → resolved to 3). Bindings 0 and 1 move together while 2-5 stay put (Consistently); + the persisted status shows clusters 0 and 1 `Started`; marking cluster 1 available frees a slot for cluster 2 + while the prod clusters stay put; once the canary stage completes, bindings 3-5 move together and all three + show `Started`; the run completes with stages in order. + - C3 "placement binding availability detection": with the cluster in flight, `Synchronized`/`Available` True at + the previous generation, and then `Synchronized` True with `Available` False at the current generation, each + leave the persisted in-progress status unchanged (Consistently, 5s); both True at the current generation + completes the run. + - C4 "placement bindings already up to date": cluster 0's binding is created pointing at the snapshot the run + will create (`-resource-snapshot-0`) and already available; it completes immediately (status shows it + succeeded while cluster 1 is in flight), and stays at generation 1 (never updated). + - C5 "suspended placement bindings": cluster 0's binding is suspended before the run (its binding still moves to + the snapshot, and the rollout moves on to cluster 1 without availability); cluster 1's binding is suspended + mid-wait; the run completes, and the `Completed` messages read "binding has been suspended" and "binding is + suspended" respectively. + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 158 of 158 specs pass (~38s spec time, + ~49s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **C6–C9 implemented** in `process_integration_test.go` (both scopes). Contexts per scope: + - C6 "empty stage with after-stage tasks": canary (cluster 0) → empty (nil selector, after-stage 5s timed wait) → + prod (cluster 1). Once the canary cluster is available, the persisted status shows the empty stage `Started` + with its task `TimedWaitStarted` (persisted right as the wait starts); the prod binding moves only after the + wait; final status has the task with `TimedWaitStarted` + `WaitTimeElapsed`, all stages succeeded and in order, + and the empty stage lasting at least 5s by its timestamps. + - C7 "resource snapshot conflicts": with cluster 0 in flight, the test waits for the persisted `Started` before + pointing its binding at another snapshot (see the race note below); cluster 0 ends `FailedToUpdate` ("resource + snapshot name conflict"), cluster 1 succeeds, stage and run end `PartiallySucceeded` (default failure policy), + and the binding keeps the conflicting snapshot. + - C8 "placement bindings deleted after initialization": suspended run initializes; binding 0 is deleted (cache + observes it); on resume, cluster 0 ends `Completed/FailedToUpdate` with no `Started` ("the binding cannot be + found"), cluster 1 succeeds; run conditions include `Suspended=False/Resumed`; `PartiallySucceeded`. + - C9 "placement bindings marked for deletion or retargeted after initialization": `maxAllowedClusterFailures: 2`; + after init, binding 0 is marked for deletion and binding 1's `clusterName` is changed; on resume both fail with + no `Started` ("marked for deletion or its cluster name does not match"), keep the previous snapshot, cluster 2 + succeeds; `PartiallySucceeded`. + - Race noted while writing C7 (not a controller change, a test-ordering rule): a reconcile that reads a run from + a cache that has yet to record a cluster's `Started` treats the binding as not yet updated and rewrites its + snapshot name. Tests that change a binding's snapshot behind the controller's back must first observe the + persisted `Started` (the shared cache is monotonic, so the controller sees it too). + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 218 of 218 specs pass (~42s spec time, + ~55s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **C10–C13 implemented** in `process_integration_test.go` — group C is complete. Contexts: + - C10 "cluster failures exceeding the threshold within a stage" (both scopes): canary stage (clusters 0-2, + `maxConcurrency: 2`) then prod (cluster 3); `maxAllowedClusterFailures: 0`. After the persisted status shows + clusters 0 and 1 `Started`, binding 0 gets a conflicting snapshot (binding 1 is never marked available). Final: + cluster 0 `FailedToUpdate`, cluster 1 `Started` + `Completed/Canceled`, cluster 2 no conditions; canary stage + `Failed` ("1 out of 3 clusters failed to update; total failures across stages have exceeded the threshold"); + run `Completed/Failed`; bindings 2 and 3 stay on the previous snapshot (Consistently); finalizer removed and + binding manager role released. + - C11 "cluster failures exceeding the threshold across stages" (both scopes): default policy (max 1). Cluster 0 + fails (conflict) and cluster 1 succeeds → canary `PartiallySucceeded`; the test then waits for the persisted + status to show the canary stage completed and cluster 2 `Started` (a reconcile boundary, so the next + reconcile counts the canary failure from persisted status), and injects a conflict on cluster 2 → prod `Failed` + ("1 out of 2 ..."), run `Failed`; cluster 3 never starts and its binding stays put. The plan's "after-stage + approval" boundary was replaced with this budget-expiry boundary (no approval helpers needed). + - C12 "member clusters that time out" (one context for both scopes, `Label("slow")`): four runs created at once, + `maxWaitTimePerClusterMinutes: 1`, stage concurrency `"100%"`; per scope, T1 (max 1, clusters 0 and 1; cluster 1 + marked available) → cluster 0 `FailedToUpdate` ("has failed due to timeout"), cluster 1 succeeded, stage and + run `PartiallySucceeded`; T2 (max 0, cluster 0 only) → stage and run `Failed`. Each stage lasted at least one + minute by its timestamps. + - C13 "processing time budget expiries" (both scopes): one cluster held unavailable; once the in-progress status + is persisted, it stays the same across two more budget expiries (Consistently, 12s), the cluster's `Started` + transition time is unchanged (no reset of the wait time limit), and the binding stays at generation 2 (no + second spec update); after marking it available, the run completes and the binding is still at generation 2. + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 274 of 274 specs pass (~80s spec time, + ~1m33s wall; C12 dominates); `ginkgo --dry-run --label-filter='!slow'` selects 270 of 274 (only C12 skipped); + go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **D1 implemented** in the new `tasks_integration_test.go` (both scopes; context "before-stage + approval"). One stage with a before-stage approval task and both clusters at `"100%"` concurrency. Checks: the + request `-before-stage-canary` exists with spec `{run, canary}` and a controller owner ref to the run (UID + included); the persisted status shows the stage `Started` with the task's `ApprovalRequestName` and + `ApprovalRequestCreated=True/RequestCreated`; bindings stay on the previous snapshot (Consistently, 5s); after the + test sets `Approved=True`, the request gets `ApprovalAccepted=True`, both bindings move within 10s (the run is not + requeued while waiting for approval, so this relies on the approval request watch), and once they are available, + the run completes with the task showing `ApprovalRequestCreated` + `ApprovalRequestApproved`. The cluster-scoped + copy is generated by the same script, which now also rewrites "approval request" to "cluster approval request". + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 290 of 290 specs pass (~76s spec time, + ~1m29s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **D2–D8 implemented** in `tasks_integration_test.go` (both scopes) — group D is complete. + Contexts per scope: + - D2 "after-stage approval": canary (clusters 0, 1; after-stage approval) then prod (cluster 2). The request + `-after-stage-canary` stays absent (Consistently, 5s) while cluster 0 and then cluster 1 are in flight, and + is created once both are done; the persisted status shows the canary stage `Started` (not completed) with the + task `ApprovalRequestCreated`; binding 2 stays put (Consistently) and the status is unchanged; after approval, + the request is accepted, prod rolls out, and the run completes with stages in order. + - D3 "rejected approval": after `Approved=False`, the request carries only that condition (no + `ApprovalAccepted`) and the binding stays put (Consistently); the waiting status is unchanged; flipping to True + → accepted → rollout → completed. + - D4 "after-stage timed wait": canary (after-stage 5s wait) then prod; status shows `TimedWaitStarted` right + after the canary cluster is done; prod rolls out only after the wait; the final task has `TimedWaitStarted` + + `WaitTimeElapsed` at least 5s apart, the canary stage completed no earlier, and stages are in order. + - D5 "after-stage approval and timed wait": canary `[Approval, TimedWait 10s]`, prod `[Approval, TimedWait 5s]`. + Canary approved early → status shows the approval task approved but the wait still running, prod untouched; + prod rolls out only after the 10s wait (checked via the task timestamps). Prod's wait elapses first → status + shows `WaitTimeElapsed` with the approval still pending, unchanged for 5s (no requeue); approving completes + the run. + - D6 "deleted approval request": deleting the request → re-created with the same name and a new UID, still owned + by the run; binding stays put until the new request is approved; then completes. + - D7 "stale approval request from a deleted staged update run of the same name": the run is deleted (its request + stays, no GC), the stale request is then approved, and the run is re-created. The controller replaces the stale + request (new UID, owner = the re-created run's UID, no conditions), so the old approval does not carry over; + the binding stays put until the new request is approved. + - D8 "approval request not owned by the staged update run": a manually created (and approved) request with the + expected name and no owner → task `ApprovalRequestCreated=False/RequestOwnershipMismatch`, no rollout + (Consistently); deleting it (the request names the run, so its delete event re-enqueues the run) → the + controller creates its own (owner = run) and the condition returns to `True/RequestCreated`; approve → completes. + - First run: 6 failures, all test bugs, fixed: (1) D6/D8 deleted the request through `removeObjects`, which waits + for the name to disappear while the controller re-creates it right away; (2) D5's in-progress want omitted the + prod stage's after-stage task statuses, which exist (with no progress) from initialization on. + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 430 of 430 specs pass (~115s spec time, + ~2m07s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **A1 implemented** in the new `lifecycle_integration_test.go` (both scopes; context "staged + update runs waiting for approval"). One stage with a before-stage approval that is never granted. Once the + persisted status shows the run waiting for approval, the run has the cleanup finalizer and the policy's + `status.bindingManager` is exactly `{controllerName: staged-update-run, objectRefs: []}` (Eventually), + and both stay that way while the run waits (Consistently, 5s). `AfterAll` deletes the run while it is still in + progress, which also exercises the deletion path (role released, finalizer removed) as part of cleanup. + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 438 of 438 specs pass (~93s spec time, + ~1m45s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **A2–A10 implemented** in `lifecycle_integration_test.go` (both scopes) — group A is complete. + Contexts per scope: + - A2 + A3 "completed staged update runs" (one context): the run completes (`Completed=True/Succeeded`), the + finalizer is removed, the binding manager role is released, and the run object stays (Consistently). Then + the binding is pointed at another snapshot and marked unavailable, and a reconcile is triggered (create + + delete an approval request naming the run); the run's status and the binding stay as they are (Consistently). + - A4 "missing placement policy at creation": `Initialized=False/Failed` ("No linked placement policy"), + finalizer removed; after the policy is created and the run's `suspended` flag is toggled (to force a + reconcile), the run is still failed at the new generation, with no claim on the new policy and the binding + untouched (Consistently). + - A5 "placement policy deleted while waiting for approval": the policy is deleted, and approving the request + (policies are not watched, so the approval is what triggers the run) yields `Completed=True/Failed` ("No linked + placement policy") with the approval task still only `ApprovalRequestCreated`; finalizer removed; binding + untouched. + - A6 "binding manager role held by another controller": `status.bindingManager` is pre-set to another + controller (and observed in the cache) before the run is created; the run gets the finalizer but its status + stays empty, the other claim stays, and the binding stays put (Consistently); after the claim is cleared, the + run initializes (waiting for approval) and claims the role for itself. (Superseded by the A6 update below: + the status is no longer empty.) + - A7 "multiple staged update runs of the same placement policy": both runs wait for approval and are both in the + claim; deleting run 1 leaves only run 2 in the claim; approving run 2 rolls out and completes it, releasing the + role. + - A8 "staged update runs deleted while a member cluster is being updated": after approval, cluster 0's binding + moves; deleting the run removes it (within a budget expiry), releases the role, leaves binding 0 on the new + snapshot and binding 1 on the previous one (Consistently), and leaves the approval request with its controller + owner ref to the deleted run (no GC). + - A9 "staged update runs deleted after their placement policy": with the run waiting (finalizer present), the + policy is deleted and then the run; the run is removed (NotFound path in cleanup). + - A10 "staged update runs that failed to initialize": missing strategy → init failed, finalizer removed; deleting + the run removes it right away. + - First run: 2 failures (A6, both scopes), a test bug, fixed: the binding-manager helper waited for the cache to + show the cleared claim, but the controller re-claimed the role within its retry backoff, so the cleared state + was never observed. The helper no longer waits; A6 waits for the other controller's claim explicitly instead. + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 538 of 538 specs pass (~102s spec time, + ~1m55s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **A6 updated** for the controller change that reports failed binding manager claims: when the + claim fails (another controller holds the role), the run gets `Started=False/FailedDueToBindingManagerConflicts` + ("... currently managed by another controller; will retry later"), and `process` now re-sets `Started` to + `True/UpdateStarted` whenever it is not True. A6 now first waits for the run's status to be exactly that one + condition (plus the message check), then keeps the previous Consistently check (status unchanged, other claim + intact, binding untouched) against that status instead of an empty one; the existing "initialize" check after the + claim is cleared verifies that `Started` flips to True. No other test hits a failed claim. + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 540 of 540 specs pass (~105s spec time, + ~1m57s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **E1–E3 implemented** in `lifecycle_integration_test.go` (both scopes) — all groups complete. + Generations: each run is created at 1, suspended at 2, resumed at 3. Contexts per scope: + - E1 "staged update runs suspended while waiting for approval": once suspended, the status is `Initialized@2`, + `Started@1`, `Suspended=True@2` (stage and task conditions still @1), the finalizer is removed, and the role is + released; approving while suspended changes nothing (status as is, request without `ApprovalAccepted`, binding + untouched; Consistently). On resume: `Suspended=False/Resumed`, the approval is accepted, the binding moves, and the + finalizer and the claim come back; the completed status has every run/stage/cluster condition @3 but the + approval task's `ApprovalRequestCreated` @1 (behavior to note 7). + - E2 "staged update runs suspended while a member cluster is being updated": the test waits for the persisted + status to show cluster 0 `Started` (budget expiry), then suspends; the suspension lands once the in-flight + reconcile ends (cluster 0 `Started@1`, cluster 1 untouched), finalizer removed and role released. Cluster 0 is + marked available while suspended — no change (Consistently). On resume, cluster 0 completes without a new + binding update, cluster 1 rolls out, and the run completes (all @3); cluster 0's `Started` transition time is + not before the `Resumed` one (its wait time limit was reset). + - E3 "suspending and resuming finished staged update runs": a completed run and an init-failed run (missing + strategy) on the same policy; suspending and then resuming each moves only `Completed` (completed run: @2, @3; + everything else stays @1) or `Initialized` (init-failed run) to the new generation, with no `Suspended` + condition either way; the binding stays on the rolled-out snapshot. + - The assembly script now checks that regenerating the cluster-scoped block from the namespaced one reproduces + the existing cluster-scoped block; the only difference found (and accepted) was one comment in A6 now reading + "cluster staged update run". + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 606 of 606 specs pass (~99s spec time, + ~1m51s wall); go vet and golangci-lint: no findings in the test files. +- October 20, 2026: **tests updated for the review follow-up controller changes**: + - binding manager conflicts are no longer reported as a condition (the `Started=False/ + FailedDueToBindingManagerConflicts` condition, its builder, and `bindingManagerConflictsMsgSubstr` are gone). A6 + is back to asserting that the run's status stays empty while it retries the claim (one `It` fewer per scope); + - the per-cluster cancellation reason is now `UpdateCanceled` (C10 picks it up through the constant); + - both suspended-binding shortcuts now report "(binding is suspended)"; C5 checks one substring + (`bindingSuspendedMsgSubstr`) for both clusters, which also removes its dependence on which shortcut is taken; + - the strategy-not-found message is now "staged update strategy cannot be found" (`strategyNotFoundMsgSubstr`). + - Validation: `ginkgo -p --race --focus=...` on A6 and C5 (18 specs) and on C10 and C11 (38 specs): all pass; go + vet and golangci-lint: no findings in the test files (604 specs in total). + - The cluster worker loop in `processOneStage` then moved into `doClusterProcessingWork` (with a shared + `perStageClusterProcessingTracker`), as the added cancellation check had pushed `processOneStage` over the + `gocyclo` limit. `ginkgo -p --race --label-filter='!slow'`: 600 of 600 specs pass (~104s spec time, ~2m wall). +- October 20, 2026: **review follow-up: Consistently checks that could not fail, and a stale-cache race**: + - A1 and E2: nothing reconciles a run that waits for approval or is suspended, so their Consistently checks held + regardless of what the controller would do. Both now trigger a reconciliation first (A1 also checks the run + status). Mutation check: with the controller changed to ignore a suspension after marking it, E2 fails at + "should not process ... while it is suspended" in both scopes, and passes without the trigger. + - A4: the reconciliation is now triggered without suspending the run (a suspension would also stop a run that the + controller wrongly re-initialized), and the Consistently check covers the run status as well; the Eventually + check that could pass on a stale (generation 1) cached copy is gone. + - A3 uses the new `triggerStagedUpdateRunReconciliation` helper. + - D3 and E1: `setApprovalRequestApproved` now waits for the cache, so their Consistently checks no longer race + with the informer. + - B6 had the same problem as A4 and got the same fix (trigger without suspending, Consistently check on the run + status, the binding manager role, and the binding), plus an explicit check that the role claimed for the failed + initialization is released. `triggerApprovalRequestNameTemplate` moved to `utils_integration_test.go`, as two + files use it now. `ginkgo -p --race --focus=...` on B6 (10 specs): all pass (610 specs in total). + - `removeObjects` now removes approval requests last, after everything else (runs included) is gone, so that a + run that the cache still has cannot re-create a request that cleanup has just deleted (constraint 12). + `ginkgo -p --race --label-filter='!slow'`: 606 of 606 specs pass (~96s spec time, ~1m49s wall). + - B9: the stage layout is now checked after the rollout has started (the Resumed condition is set before the + initialization step, so checking right after it could pass before the controller re-checks the stages). + - A2: dropped the check that the run object stays around (nothing reconciled the run during that check, and the + controller never deletes runs); A3's checks read the run anyway. + - D5: both approval requests (canary and prod) are now checked for `ApprovalAccepted`, like the other approvals. + - `createPlacementResourceSnapshot`'s comment no longer claims to match the snapshot manager's output (the + snapshots lack the manager's annotations, so they suit only runs that name their snapshot explicitly). + - Validation: `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun` (including the `slow` context): 610 of 610 + specs pass (~106s spec time, ~2m01s wall); go vet and golangci-lint: no findings in the test files. + - Validation: `ginkgo -p --race --focus=...` on A1, A3, A4, E1, E2, and D3 (108 specs): all pass; go vet and + golangci-lint: no findings in the test files (608 specs in total). +- October 27, 2026: **B11 implemented** in `initialize_integration_test.go` (context "selected resource missing", + both scopes, after B8), following the placement resource snapshot manager change that reports a resource selected + by name that does not exist as a user error (after confirming a cache miss with a read from the API server; the + suite registers no informers, so the manager reads ConfigMaps from the API server directly). The policy selects a + ConfigMap that is never created up front; the run (not suspended) ends with only `Initialized=False/Failed`, message + containing "selected resource does not exist" (`selectedResourceNotFoundMsgSubstr`), and the binding manager role is + released. Then the ConfigMap is created and a reconciliation triggered (without suspending the run, as in B6); a + Consistently check covers the run status, the role, the binding (still on the previous snapshot), and the absence + of the index 0 snapshot (which a re-initialization would create). + - No existing test covered this: B7 always creates the selected ConfigMap, and B8's "not found" is about an + explicitly specified snapshot. + - Mutation check: with the snapshot manager reporting the missing resource as an API server error again (the + previous behavior), the run is never marked as failed and both scopes fail at "should fail to initialize". + - Validation: `ginkgo -p --race --focus="selected resource missing"`: 10 of 10 specs pass; + `ginkgo -p --race ./pkg/v1/controllers/stagedupdaterun`: 620 of 620 specs pass (~150s spec time, ~2m39s wall); + go vet and golangci-lint: no findings. diff --git a/.github/.copilot/breadcrumbs/2026-10-21-2045-workapplier-v1-integration-test-plan.md b/.github/.copilot/breadcrumbs/2026-10-21-2045-workapplier-v1-integration-test-plan.md new file mode 100644 index 000000000..63a81bb26 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-21-2045-workapplier-v1-integration-test-plan.md @@ -0,0 +1,711 @@ +# Work Applier (v1) Integration Test Plan + +**Date**: October 21, 2026 20:45 AEDT +**Task**: Plan (not yet implement) the integration tests for `pkg/v1/controllers/workapplier`, building on the +envtest suite in [suite_test.go](../../../pkg/v1/controllers/workapplier/suite_test.go) and the two existing IT files +([core_integration_test.go](../../../pkg/v1/controllers/workapplier/core_integration_test.go), +[failures_integration_test.go](../../../pkg/v1/controllers/workapplier/failures_integration_test.go)). The old work +applier ITs (`pkg/controllers/workapplier`) are used as a reference only; many do not apply as-is because of API and +behavior changes (see [Old tests → new plan](#old-tests--new-plan)). + +**Status**: In progress. The [Decisions](#decisions) are adopted. All P0 contexts are implemented, plus T8, W1–W3, R2, +R3, L2, L3 (P1) and L5 (P2); the remaining P1/P2 contexts (including the V section and W4) are pending. See +[Status](#status) for details. + +## Requirements + +1. Cover the behaviors of the new work applier that the existing ITs do not cover yet, against real API servers + (envtest hub + member). +2. Do not port the old ITs as-is; adapt the scenarios that still apply to the `v1alpha1` API (`SyncStrategy`, linked + work objects, `DiffDetails`), and drop the ones whose features are gone. +3. Follow repo conventions: Ginkgo/Gomega, `_integration_test.go`, one `Context` per scenario (`Ordered`, + `BeforeAll`/`AfterAll`), random names and a fresh namespace per context (envtest cannot delete namespaces), + whole-struct `cmp.Diff`, `want` naming, existing `*Actual` helper style. +4. Always run with `ginkgo -p` (see [Running the tests](#running-the-tests)). +5. Put **all** IT helpers (fixtures, builders, `*Actual` checks, GC simulation, recorder accessors, shared expected + messages/constants) in [utils_integration_test.go](../../../pkg/v1/controllers/workapplier/utils_integration_test.go) + (renamed from `utils_test.go` on October 21, 2026). The `*_integration_test.go` scenario files hold only + `Describe`/`Context` bodies. + +## Baseline (October 21, 2026) + +```bash +KUBEBUILDER_ASSETS="$(make -s kubebuilder-assets-path | tail -1)" \ + go run github.com/onsi/ginkgo/v2/ginkgo -p --race ./pkg/v1/controllers/workapplier +``` + +155 of 155 specs pass; 61.7s of spec time, 1m13s wall. + +## Controller behavior under test + +Reconcile flow ([controller.go](../../../pkg/v1/controllers/workapplier/controller.go)): + +1. Not joined (`ready=false`) → requeue after 5s, nothing else. `Join()` sets ready; `Leave(ctx)` sets not-ready, + deletes every appliedWork of a finalized work with **Orphan** propagation, `Forget`s both rate limiters, and strips + the cleanup finalizer (`placement.kubefleet.dev/work-cleanup`) from all works in the reserved namespace. +2. Get work; not found → done. Only **primary** works (with `placement.kubefleet.dev/linked-work-count`) are + reconciled; secondary-work events are mapped to the primary via its controller owner reference + ([eventhandler.go](../../../pkg/v1/controllers/workapplier/eventhandler.go)). Watch uses + `GenerationChangedPredicate` (label/annotation-only changes are not events). +3. Deleting → `cleanupWhenBindingDeleted` ([cleanup.go](../../../pkg/v1/controllers/workapplier/cleanup.go)): + delete the primary appliedWork with **Foreground** (`CleanUpResources`, default) or **Orphan** + (`OrphanResources`) propagation, using the **primary's** sync strategy. Wait; after `cleanupWaitTime` set + `BlockOwnerDeletion=false` on the appliedWork owner ref of every manifest that has an `Applied` condition, and + annotate the appliedWork with `placement.kubefleet.dev/applied-work-forced-deleted`; after `2×cleanupWaitTime` + proceed anyway. Then delete secondary appliedWorks, drop secondary finalizers, `Forget`, drop the primary finalizer. +4. Retrieval ([retrieval.go](../../../pkg/v1/controllers/workapplier/retrieval.go)): list works by the + `ownerNS/bindingName` field index; linked = same primary-snapshot annotation, leftover = different. Transient error + (no processing, no finalizer) if the linked count mismatches the annotation or any leftover work is not deleting. + Unexpected error if the binding label / snapshot annotation is missing or the count is not an integer. Secondaries + are sorted by name. +5. Add the cleanup finalizer to every linked work; ensure one appliedWork per linked work (transient error if one is + being deleted). **All** manifests across linked works are owned by the **primary** appliedWork + (`BlockOwnerDeletion=true`). +6. Pre-process ([preprocess.go](../../../pkg/v1/controllers/workapplier/preprocess.go)): decode (unknown kind / scope + mismatch → `DecodingErred`; transient REST mapping error → abort), reject `generateName` (`FoundGenerateName`) and + nameless (`NoNameSpecified`) manifests, reject duplicates across **all** linked works by `GK+ns+name` + (`Duplicated`; first occurrence in primary-then-name order wins). Write-ahead (skipped when the work `Applied` + condition already observes the current generation): carry over existing manifest statuses, add + `PreparingToProcess` for new ones, delete leftover manifests (in the old status, not seen in any linked work, with + an `Applied` condition, still owned by the primary appliedWork UID; co-owned → disown only). +7. Leftover works: remove their applied manifests not seen in linked works, delete their appliedWork, drop finalizer. +8. Default the sync strategy if nil (CSA, `WhenOwnedByOthers=ReportError`, `WhenDrifted=ApplyAnyway`, + `WhenAlreadyExists=ReportError`, `WhenPlacementDeleted=CleanUpResources`, + `WhenNamespaceDoesNotExist=CreateNamespace`, `PartialComparison`). The **primary's** strategy governs all linked + works' manifests. +9. Process in **waves** across all linked works ([waves.go](../../../pkg/v1/controllers/workapplier/waves.go); + unknown groups/resources go last), per manifest ([process.go](../../../pkg/v1/controllers/workapplier/process.go)): + - owned by an old-API (`placement.kubernetes-fleet.io/v1beta1`) AppliedWork → `OwnedByOtherKubeFleetAPIObjects` + (always); + - exists, not owned by us, `WhenAlreadyExists≠ReportError` → takeover + ([takeover_drift_detection.go](../../../pkg/v1/controllers/workapplier/takeover_drift_detection.go)): strip + owner refs to `v1alpha1` AppliedWorks whose work no longer exists; other owners + not `ShareOwnership` → + `FailedToTakeOver`; another `v1alpha1` AppliedWork owner → `FailedToTakeOver`; `TakeOverIfNoDiff` with diffs → + `FailedToTakeOver` + `DiffDetails` (partial = dry-run SSA; Invalid → degraded full comparison); else add owner ref; + - exists, not owned by us → `NotTakenOver` (covers `WhenAlreadyExists=ReportError`, the default); + - `WhenDrifted=ReportError` and live manifest-hash annotation equals the current manifest hash → pre-apply drift + detection → `FoundDrifts` / `FoundDriftsInDegradedMode` + `DiffDetails`; + - apply ([apply.go](../../../pkg/v1/controllers/workapplier/apply.go)): sanitize, set manifest-hash annotation, + validate owner refs (co-owned while `ShareOwnership` is off → `FailedToApply`); create (auto-creating the namespace + if `CreateNamespace`; the namespace is unmanaged) or CSA (strategic merge for built-ins, JSON merge for CRs; + falls back to non-forced SSA when the last-applied annotation exceeds the size limit) or SSA (`ForceConflicts`; + forced anyway when all field managers are KubeFleet/`before-first-apply`); optimistic lock iff + `WhenDrifted=ReportError`. API error → `FailedToApply`; + - `FullComparison` → post-apply drift detection; diffs are reported with `Applied=True`. +10. Availability ([availability_tracker.go](../../../pkg/v1/controllers/workapplier/availability_tracker.go)) for + applied manifests only: Deployment/StatefulSet/DaemonSet/Service/CRD/PDB rules; data objects available; others + `NotTrackable` (counted as available). Not applied → no `Available` condition (a stale one is removed). +11. Status ([status.go](../../../pkg/v1/controllers/workapplier/status.go)): per-manifest conditions carry the + member object's generation; `DiffDetails.FirstDiffedObservedTimestamp` is preserved while diffs persist and reset + once they clear; work `Applied`/`Available` (+ `SomeManifestsAreNotTrackable`); status writes are skipped when + nothing changed; appliedWork status lists applied resources (with UIDs) per work. +12. Requeue after `periodicRequeueRateLimiter.When(...)` + ([backoff.go](../../../pkg/v1/controllers/workapplier/backoff.go)); the counter resets when any linked work's + name/generation set or any manifest's processing result changes. Default: 5s once, then slow backoff from 2s + (×1.2, cap 15s), then fast (×1.5, cap 15m); skip to fast when everything is available. + +## What the existing ITs already cover + +| File / Context | Behaviors | +| --- | --- | +| core: single work, `core ops` | Finalizer, appliedWork, NS + Deployment apply, owner refs, status, appliedWork status | +| core: single work, `garbage collection` | Foreground appliedWork deletion, forced unblock (`BlockOwnerDeletion=false`), finalizer removal after `2×cleanupWaitTime` | +| core: single work, `leftover cleanup` | Manifest removed from work → object deleted | +| core: multiple works, `core ops` / `garbage collection` | Linked set apply, primary appliedWork owns all, per-work statuses, secondary finalizer/appliedWork cleanup | +| core: multiple works, `leftover cleanup (remove and shuffle…)` | Rollout in progress (inconsistent set → no processing), manifest moved between works kept, removed one deleted | +| core: multiple works, `scaling up` / `scaling down` | Linked count changes, leftover works cleaned up (manifests, appliedWork, finalizer) | +| failures: decoding / generateName / no name / scope mismatch | Pre-processing rejections + fix-later recovery | +| failures: duplicates (×5) | Within one work, same GK different versions, remove-all, primary↔secondary, two secondaries (name-sort winner), handover without re-creation | +| failures: `namespace absent with namespace auto-creation disabled` | `WhenNamespaceDoesNotExist=ReportError` → `FailedToApply`; namespace created outside → applied, unmanaged | + +**Not covered at all**: takeover (`WhenAlreadyExists`, `WhenOwnedByOthers`), drift detection (`WhenDrifted`, +`ComparisonOption`), SSA / CSA semantics and fallback, namespace auto-creation (default), `OrphanResources`, leftover +safety (not-owned / co-owned objects), availability beyond Deployment/data objects, waves ordering, requeue/backoff +wiring, status-write skipping, `Join`/`Leave`, namespaced-owner indexing, malformed primary works. + +## Test environment constraints and design implications + +1. **No GC controller.** Foreground/Orphan deletion of an appliedWork leaves it stuck with `foregroundDeletion` / + `orphan`. Dependents are never deleted and owner refs are never stripped. Tests simulate GC where needed + (`appliedWorkRemovedActual` already strips `foregroundDeletion`; add an `orphan` variant). Consequences: + - `Leave` leaves every appliedWork stuck with `orphan`; on `Join`, `ensureAppliedWorks` returns a transient + error until the test simulates GC (strip the AppliedWork owner refs from dependents, then the finalizer). + - Cleanup policy differences (`CleanUpResources` vs `OrphanResources`) are observable only through the propagation + finalizer on the appliedWork and the forced-unblock path, not through object deletion. +2. **No built-in controllers.** Availability is simulated by writing status (Deployment helper exists; add + StatefulSet/DaemonSet/PDB/LoadBalancer Service). The API server itself does allocate Service ClusterIPs and marks + CRDs `Established`/`NamesAccepted`. +3. **No namespace deletion.** Each context uses a fresh namespace; cluster-scoped fixtures (ClusterRole, PriorityClass, + CRDs) use random names. +4. **Periodic requeue is the only re-trigger for member-side changes.** The work applier does not watch member + objects; drift/takeover/fix-on-member scenarios rely on the default limiter (first requeue 5s, then 2–15s), so they + fit in `eventuallyDuration` (10s) for one requeue; use a 20s budget (`eventuallyDurationForRequeue`) where two or + more requeues are needed (e.g., asserting `FirstDiffedObservedTimestamp` stays stable). +5. **No per-reconcile timestamp in status.** The old `ObservationTime` fields are gone and status writes are skipped + on no-op reconciles, so reconcile timing and write counts are only observable through instrumented clients (see + [Suite changes](#suite-changes)). +6. **Default strategy is conservative.** With `nil` sync strategy, pre-existing objects are **not** taken over + (`NotTakenOver`). Takeover/drift tests must set the strategy explicitly. Fields the test leaves empty are filled + in by the API server's CRD defaults, except `ServerSideApplyOptions` (defaulted in code). +7. **Full comparison and server-populated fields.** Full comparison of objects with server defaults (Namespace + `/spec/finalizers`, Deployment defaults, Job selectors/labels) always yields diffs. Use ConfigMaps/Secrets/CRs for + deterministic full-comparison cases, and assert the full diff list (`cmp.Diff`) where defaults are involved. +8. **`Join`/`Leave` affect the whole reconciler.** `Leave` marks the reconciler not ready, and it strips the cleanup + finalizer from (and orphans the appliedWork objects of) **every** work object in the reconciler's reserved + namespace. L1 therefore uses a dedicated reconciler (Decision 2), so that a `Leave`, or a failed test stuck in the + left state, never affects the other test cases on the same process. `Serial` is not needed. Ginkgo runs specs one + at a time within a process (an `Ordered` container runs as a unit), and every process has its own envtest + environments and reconcilers. `Serial` only prevents overlap with specs on other processes, which share nothing + here. (The earlier note claiming that specs on the same process run in parallel was incorrect.) +9. **Shared member cluster across reconcilers** (Decision 1) is safe only if no test lets one reconciler touch objects + owned by another reconciler's appliedWorks (`removeLeftBehindAppliedWorkOwnerRefs` resolves works in its own + reserved namespace only). +10. **The first reconciliation of a work object can take seconds.** It adds the finalizer, creates the appliedWork + and applies the manifests, and it often runs while the suite is still starting up. Two causes were measured with + `ginkgo -p --race` (October 22, 2026): + - The first AppliedWork create on a fresh member API server (Kubernetes 1.33.0) sometimes takes ~2.0s. It is one + HTTP round trip that returns `201`, with no `Retry-After` and no client-go retries, so the wait is server-side. + Later creates take a few milliseconds. + - The CPU is starved while every Ginkgo process boots its envtest control planes; first reconciliations then take + 1–7s in wall time, even though their API calls are fast. + + Timing assertions must therefore not span the first reconciliation (R2/R3 count gaps from the second one). + +## Suite changes + +0. **Lifecycle reconciler** (`workApplierLifecycle`, hub reserved namespace `fleet-member-experimental-lifecycle`, own + hub manager `hubMgrLifecycle`). **Implemented (October 21, 2026).** It shares member env 1 (Decision 1), has its + own cleanup rate limiter, and shares `defaultRequeueRateLimiter`, which keys by namespace/name and so never + collides. The manager sets `Controller: config.Controller{SkipNameValidation: ptr.To(true)}`, as the work applier + always registers its controller under the same name (test-only setting). Used by L1; any future test case that + calls `Join`/`Leave` should use it too. +1. **Reconciler 2: waves** (`workApplierWaves`, hub reserved namespace `fleet-member-experimental-waves`, own hub + manager `hubMgrWaves`). **Implemented (October 21, 2026).** Same member env as reconciler 1 (Decision 1). Uses + `parallelizerWithFixedDelay` (port from the old `pkg/controllers/workapplier/suite_test.go`) that sleeps + `parallelizerFixedDelay` (3s, Decision 4) after every `ParallelizeUntil` whose name starts with + `processingManifestsInWave`. Its hub client is a `recordingClient` (`hubClientRecorderWaves`), so W5 can count + work status writes. +2. **Reconciler 3: instrumented** (`workApplierInstrumented`, `fleet-member-experimental-instrumented`, own hub manager + `hubMgrInstrumented`, same member env). **Implemented (October 21, 2026).** Wraps: + - the hub (cached) client (`hubClientRecorderInstrumented`): records the timestamps of `Get` calls on `Work` + objects (= reconcile starts per primary work) and counts `Status().Update` per work; + - the member client (`memberClientRecorderInstrumented`): counts `Status().Update` per appliedWork. + + It uses a fast custom periodic limiter (`instrumentedRequeueRateLimiter`: 1 fixed attempt of 2s; slow ×1.5 from + 2s, cap 4s; fast ×2, cap 8s; skip-to-fast on), so backoff stages are visible within ~30s. Ported and extended the + old `clientWrapperWithStatusUpdateCounter` as `recordingClient`. +3. Register the extra field indexes per manager (`newHubManager`), create the extra hub namespaces in + `setupResources`, start/stop the extra managers in `BeforeSuite`/`AfterSuite`. **Implemented.** +4. Add constants: `eventuallyDurationForRequeue = 20s`, `parallelizerFixedDelay = 3s`. Only `parallelizerFixedDelay` + was added; specs use multiples of `eventuallyDuration` instead. +5. **CI wiring** (Decision 5): add a `TestMain` that skips the package when `KUBEFLEET_CI_TEST_RUNNER_NAME=default`, + and a dedicated CI step: `ginkgo -v -p --procs=4 --race --cover -coverprofile=work-applier-v1-it-coverage.out + ./pkg/v1/controllers/workapplier/`, following the `stagedupdaterun` / old work applier pattern. **Implemented + (October 22, 2026)**: `TestMain` in `suite_test.go`, and the step `Run work applier (v1) unit and integration + tests with Ginkgo CLI & generate coverage` in `.github/workflows/ci.yml`, after the staged update run step. + +## Shared test utilities ([utils_integration_test.go](../../../pkg/v1/controllers/workapplier/utils_integration_test.go)) + +All helpers go in this file (Requirement 5), including the ones listed below; do not define helpers in the scenario +files. + +- Fixtures: `secret`, `job`, `clusterRole`, `service`, `statefulSet`, `daemonSet`, `pdb`, `testResource` + (`test.kubernetes-fleet.io/v1alpha1`, preinstalled on the member), and a CRD builder with random group/plural. +- `createPreExistingObject(obj, fieldOwner)`: creates an object on the member directly (own field manager, optional + extra owner refs, optional old-API AppliedWork owner ref). +- `ownerRefsActual(obj, wantRefs)`, `objectUntouchedActual(obj, wantResourceVersion)` (stays as it was, for + `Consistently`), generic `memberObjectActual` comparisons for the new kinds. +- `manifestStatusActual(workNS, workName, ordinal, want)` for single-manifest assertions (incl. `DiffDetails`), and + `retrieveFirstDiffedTimestamp(...)`. +- `appliedWorkBeingDeletedWithOrphanActual`, `appliedWorkRemovedWithOrphanActual` (simulate orphan GC: strip the + AppliedWork owner refs from listed dependents, then the `orphan` finalizer). +- `appliedWorkForcedDeletedAnnotationActual`. +- `buildWorkObjectWithOwnerNS(...)` (non-empty `placement.kubefleet.dev/owner-namespace` label). +- `markStatefulSetAsAvailable`, `markDaemonSetAsAvailable`, `markPDBAsAvailable`, `markLBServiceAsAvailable`. +- Instrumented-reconciler accessors: `reconcileTimestamps(workKey)`, `workStatusUpdateCount(workKey)`, + `appliedWorkStatusUpdateCount(name)`. + +## Test plan + +Priorities: **P0** = core new-API behavior with no coverage; **P1** = important edge/safety behavior; **P2** = nice to +have. Every context is `Ordered` and cleans up in `AfterAll` (delete work → simulate GC → work gone). + +### T. `takeover_integration_test.go`: pre-existing objects (`WhenAlreadyExists`, `WhenOwnedByOthers`) + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| T1 | P0 | default strategy (report error) on pre-existing objects | Pre-create NS-scoped ConfigMap (different data) + matching Deployment; work with `nil` strategy | Both `Applied=False/NotTakenOver`, no `DiffDetails`, no `Available` cond; objects untouched (`Consistently`), no owner ref; work `Applied=False`, `Available=False`; appliedWork lists nothing; on delete, objects stay | +| T2 | P0 | always take over | Pre-create ConfigMap (different data + extra member-only label) with no owners; `AlwaysTakeOver` | Owner ref added; managed fields overwritten; member-only label kept (CSA); `Applied=True`; deletion → foreground on appliedWork | +| T3 | P0 | take over if no diff (no diff) | Pre-create identical NS + Deployment; `TakeOverIfNoDiff` | Taken over (owner refs), `Applied=True`, `Available` after marking | +| T4 | P0 | take over if no diff (diff, partial comparison) → fix on member | Pre-create ConfigMap with `data.foo` diff + Deployment with replicas diff; `TakeOverIfNoDiff`, partial | `FailedToTakeOver` + `DiffDetails` (paths, values, `ObservedInMemberClusterGeneration`), untouched; then revert ConfigMap on member → taken over at next requeue, `DiffDetails` cleared; Deployment still failing | +| T5 | P0 | take over if no diff (unmanaged-only diff), partial vs full | Two contexts: pre-create ConfigMap identical except a member-only label; partial / full | Partial: taken over, no diffs; Full: `FailedToTakeOver` with `/metadata/labels/` (`ValueInMember` only) | +| T6 | P1 | take over if no diff (degraded) | Pre-create Job whose template label differs; `TakeOverIfNoDiff`, partial | `FailedToTakeOver` with the degraded full-comparison diff list (assert with `cmp.Diff`); Job untouched | +| T7 | P0 | co-ownership | Pre-create ConfigMap owned by an arbitrary non-KubeFleet owner; `AlwaysTakeOver` + `WhenOwnedByOthers=ReportError` → switch to `ShareOwnership` → switch back to `ReportError` | `FailedToTakeOver` ("owned by some other source(s)"), untouched → co-owned (both refs), applied → `FailedToApply` ("co-owned … disallowed"), refs unchanged | +| T8 | P1 | owned by old KubeFleet APIs | Pre-create ConfigMap owned by a `placement.kubernetes-fleet.io/v1beta1` AppliedWork; `AlwaysTakeOver` + `ShareOwnership` | `OwnedByOtherKubeFleetAPIObjects`; untouched | +| T9 | P0 | duplicate placement across independent works | Work A (own binding) places ConfigMap X; then work B (different binding) places X with: `nil` → `AlwaysTakeOver` → `AlwaysTakeOver`+`ShareOwnership`; delete work A and strip only the propagation finalizer from A's appliedWork (X keeps a dangling owner ref, as with a GC that has not caught up) | B: `NotTakenOver` → `FailedToTakeOver` (other owners) → `FailedToTakeOver` (another KubeFleet AppliedWork); after work A is gone, B strips the left-behind ref and takes over X (owner refs = B only) | +| T10 | P1 | switch report error → take over if no diff | Pre-existing identical NS + Deployment; `nil` → update strategy to `TakeOverIfNoDiff` | `NotTakenOver` → taken over and applied without spec change on member | +| T11 | P1 | linked works use the primary's strategy | Primary `AlwaysTakeOver` (ConfigMap A); secondary strategy `nil` (pre-existing ConfigMap B) | B taken over and owned by the **primary** appliedWork; secondary status `Applied=True` | +| T12 | P1 | sensitive data in takeover diffs | Pre-create Secret with different `data`; `TakeOverIfNoDiff` | `FailedToTakeOver`; `/data/` values are `(redacted for security reasons)` | + +### D. `drift_integration_test.go`: drift detection (`WhenDrifted`, `ComparisonOption`) + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| D1 | P0 | apply anyway (default), partial | NS + Deployment; change replicas on member; add member-only label | Replicas reverted at the next periodic requeue (no work change); member-only label kept; no `DiffDetails` | +| D2 | P0 | report error, partial, drift appears, changes, then is reverted | NS + Deployment (`WhenDrifted=ReportError`); mark available; change replicas on member (2), then change them again (3) to force a status rewrite while still drifted; revert on member, then re-mark available (the replica edits bump the generation) | `FoundDrifts`, `/spec/replicas` diff, `ObservedInMemberClusterGeneration`; Deployment untouched; manifest `Available` cond **removed**; work `Applied=False`/`Available=False`; `FirstDiffedObservedTimestamp` stable across the rewrite; after revert: `Applied=True`, `DiffDetails=nil`, `Available` back | +| D3 | P1 | report error, partial, unmanaged change only | Add member-only label | No drift; `Applied=True` | +| D4 | P0 | report error, manifest bumped after drift | After drift (D2 setup), update the manifest (e.g., image) | Hash mismatch → apply proceeds and overwrites the drift; `Applied=True`, no diffs | +| D5 | P0 | report error, full comparison | ConfigMap; add member-only label | `FoundDrifts` with `/metadata/labels/` | +| D6 | P0 | apply anyway, full comparison (post-apply diffs) | ConfigMap + NS; add member-only label to ConfigMap | `Applied=True` (reason `Applied`) **with** `DiffDetails` for the label (persists, CSA keeps it); NS reports a diff on its server-populated finalizers (exact path depends on the fixture's `spec`) | +| D7 | P1 | report error, partial, degraded | NS + Job; change the template label on member (immutable field) | `FoundDriftsInDegradedMode`; Job untouched; diff list asserted | +| D8 | P1 | sensitive data in drift details | Secret + ConfigMap, `ReportError`; change both on member | ConfigMap values shown; Secret `/data/*` redacted | +| D9 | P1 | switch report error → apply anyway with drift present | D2 setup, then update strategy | Drift overwritten; `DiffDetails` cleared | +| D10 | P2 | drift on a secondary work's manifest | Primary (`ReportError`) + secondary with ConfigMap; drift on that ConfigMap | Reported on the **secondary** work's status; strategy from primary | + +### A. `apply_integration_test.go`: apply methods and semantics + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| A1 | P0 | CSA manifest updates (built-in, strategic merge) | ConfigMap with labels/annotations/data; third party adds data key + label, changes a managed key; update manifest (change, remove label, add annotation) | Managed values follow manifest; removed label gone; third-party additions kept; `Applied` generation bumps | +| A2 | P1 | CSA on custom resources (JSON merge patch) | `TestResource` (preinstalled CRD); third party adds `items`, changes `foo`; update manifest | Scalars follow manifest, selector replaced; third-party `items` kept; `Available=True/NotTrackable`, work reason `SomeManifestsAreNotTrackable` | +| A3 | P1 | missing last-applied annotation | Strip `placement.kubefleet.dev/last-applied-configuration` on member; update manifest | Still converges; annotation restored | +| A4 | P0 | CSA → SSA fallback for oversized objects | ConfigMap with ~300KB data; then update it | Created with empty last-applied annotation; update applied via SSA; `Applied=True` | +| A5 | P0 | SSA and force conflicts | Pre-create ConfigMap via test field manager (`data.foo=bar`); `ServerSideApply`, `AlwaysTakeOver`, manifest `foo=baz`; then `ForceConflicts=true` | `FailedToApply` (conflict) → applied, `managedFields` show KubeFleet owning `data.foo` | +| A6 | P1 | switch CSA ↔ SSA | NS + Deployment; CSA → SSA → CSA, changing a field each time | Each switch applies (self-conflict avoided via forced SSA); no spurious failures | +| A7 | P0 | namespace auto-creation (default) | ConfigMap in a non-existent namespace, `nil` strategy | Namespace created with no owner refs; ConfigMap applied; on work deletion the namespace stays and is not in appliedWork status | +| A8 | P1 | API server rejects a manifest; replacement fails | NS + ConfigMap + invalid Deployment (e.g., negative replicas) → fix it; then replace all manifests with an invalid one | `FailedToApply` with correct identifier; others applied; fixed → applied; replaced → old objects removed even though the new manifest fails, appliedWork empty | +| A9 | P1 | CRD and its CR in the same work | Random-group CRD + one CR | CRD applied first (wave 1); CR applied at a later reconcile once the REST mapper sees the CRD (first pass `DecodingErred` is allowed); both eventually `Applied`; CRD `Available` | +| A10 | P2 | manifest reorder within a work | Swap the order of two ConfigMaps | No re-creation (UIDs stable); status ordinals follow new order; condition LTTs stable | + +### C. `cleanup_integration_test.go`: deletion policies and leftover safety + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| C1 | P0 | orphan resources on placement deletion (linked works) | Primary (`OrphanResources`) + secondary; delete primary | Primary appliedWork deleting with `orphan` finalizer; secondary appliedWork deleted + finalizer dropped; after `cleanupWaitTime` refs unblocked + forced-deleted annotation; work gone after `2×cleanupWaitTime`; objects remain | +| C2 | P1 | the primary's deletion policy wins | Primary `nil` (cleanup) + secondary `OrphanResources` | Foreground propagation (`foregroundDeletion`) | +| C3 | P0 | leftover object no longer owned by KubeFleet | Apply ConfigMap; strip the AppliedWork owner ref on member (same code path as a user delete + re-create, without racing the periodic requeue that would re-create it); wait for `NotTakenOver`; remove manifest | Object kept | +| C4 | P0 | leftover object never taken over | T1-style pre-existing ConfigMap (`NotTakenOver`); remove manifest | Object kept untouched | +| C5 | P0 | leftover co-owned object | `ShareOwnership` co-owned ConfigMap (T7-style); remove manifest | Only the AppliedWork owner ref removed; object and other owner kept | +| C6 | P1 | co-owned object dropped by a rollout (leftover work path) | Primary + secondary; secondary holds the co-owned ConfigMap; roll out to a new snapshot without it (delete old secondary) | Disowned, not deleted | +| C7 | P1 | remove all manifests | Work with NS + ConfigMap → empty manifests | ConfigMap deleted; work `Applied=True`/`Available=True` (0 manifests); appliedWork status empty | +| C8 | P2 | forced-deletion annotation (extend core `garbage collection`) | — | Assert `placement.kubefleet.dev/applied-work-forced-deleted` on the stuck appliedWork | + +### L. `lifecycle_integration_test.go`: join/leave, work selection, indexing + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| L1 | P0 | leave and re-join (dedicated lifecycle reconciler) | 3 single works (ConfigMaps) applied; `Leave` (twice, idempotent); update all works; `Consistently` not processed; simulate orphan GC; `Join`; `DeferCleanup(Join)` | After `Leave`: no finalizers, appliedWorks deleting with `orphan`; objects intact; updates not applied while left. After GC + `Join`: finalizers back, new appliedWorks; objects (now unowned) → `NotTakenOver` with default strategy (see Behaviors 1); with `AlwaysTakeOver` works, objects re-adopted and updated | +| L2 | P1 | non-primary work objects | A standalone work without the linked-count annotation, plus a secondary-style work owned by a primary work that does not exist | Never processed: no finalizer, no appliedWork, no status, no object or namespace created (`Consistently`); deletable at once | +| L3 | P1 | malformed primary works | Three primaries: no binding label / no snapshot annotation / non-integer count | Never processed (no finalizer, no appliedWork); deletable at once | +| L4 | P0 | owner-namespace isolation in the field index | Two single primaries with the **same** binding name: cluster-scoped (`owner-namespace=""`) and namespaced (`owner-namespace=`), plus a secondary for the namespaced one | Each processed as its own linked set (linked count matches); deleting one does not touch the other | +| L5 | P2 | appliedWork deleted externally | Delete the appliedWork (background); simulate GC by deleting the dependents | appliedWork re-created (new UID); objects re-created and owned by it; appliedWork status lists the new UIDs | + +### V. `availability_integration_test.go`: availability tracking + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| V1 | P1 | immediately available and untrackable kinds | NS, ClusterIP Service, CRD, Secret, ExternalName Service, `TestResource` | Service/CRD/data `Available`; ExternalName + CR `NotTrackable`; work `Available=True/SomeManifestsAreNotTrackable` | +| V2 | P1 | workloads become available | StatefulSet, DaemonSet, PDB, LoadBalancer Service | `Unavailable` → mark each available → `Available`; work reason transitions to `AllManifestsAvailable` | + +Exhaustive per-kind rules belong in unit tests (see below). + +### W. `waves_integration_test.go`: wave ordering (reconciler 2) + +| ID | P | Context | Assert | +| --- | --- | --- | --- | +| W1 | P1 | single wave (NS + PriorityClass) | Created within ~1s of each other | +| W2 | P1 | two consecutive waves (NS → ConfigMap) | Gap ≥ delay, < 2×delay | +| W3 | P1 | two non-consecutive waves (NS → Role) | Gap ≥ delay, < 2×delay (empty wave adds nothing) | +| W4 | P2 | all waves (shuffled manifests incl. EndpointSlice as unknown) | Waves 0–5 + last all present; intra-wave ≤ 2s; adjacent waves ≥ delay | +| W5 | P0 | waves span linked works (runs on reconciler 2, whose hub client also counts status writes) | NS in the **secondary**, ConfigMap in the **primary**, `WhenNamespaceDoesNotExist=ReportError` | ConfigMap applied in the first pass: the primary's work status is written exactly twice (write-ahead + final `Applied=True`). A failed first pass would add a third write. The creation-timestamp gap (NS → ConfigMap: ≥ delay, < 2×delay) also catches a failed first pass: the ConfigMap would then be applied in a later reconciliation, at least one requeue delay (5s) plus wave delays later, beyond the upper bound | + +### R. `requeue_integration_test.go`: requeue and status writes (reconciler 3) + +| ID | P | Context | Assert | +| --- | --- | --- | --- | +| R1 | P0 | status writes are skipped at steady state | NS + ConfigMap single work: exactly 2 work status writes (write-ahead + final) and 1 appliedWork status write; none over ≥2 more reconciles; same per work for a primary + secondary set | +| R2 | P1 | backoff progression (unavailable) + reset on result change | Deployment never available: delays 2s, 2s, 3s, 6s, 8s, 8s; gaps checked from the 2nd reconcile: 2s, 3s, 6s, 8s, 8s. Marking it available → next gap is the fixed delay (2s) | +| R3 | P1 | skip to fast backoff (available) + reset on change | ConfigMap: delays 2s, 2s, 4s, 8s, 8s; gaps checked from the 2nd reconcile: 2s, 4s, 8s, 8s. `nudgeWork` → after skipping the first reconcile at or after the nudge, the next gap is 2s (the fixed delay or the initial slow backoff delay; the skip guards against a periodic reconcile that reads the pre-nudge generation from the cache) | + +Gaps are derived from `When()` with the proposed limiter (fixed 2s ×1; slow ×1.5 from 2s, cap 4s; fast ×2, cap 8s). +They are measured between reconcile starts, so they include the reconcile duration (tolerance −250ms/+1.5s). The +backoff-progression gaps skip the first reconciliation (see +[constraint 10](#test-environment-constraints-and-design-implications)); the fixed delay is checked by the reset +specs instead. R scenarios avoid takeover, so that `Work` `Get` calls map 1:1 to reconciles +(`removeLeftBehindAppliedWorkOwnerRefs` also `Get`s works). Count only successful status writes in R1/W5 to stay +robust to conflicts. R2/R3 are labeled `slow` (Decision 3). + +### Better as unit tests (out of scope here; gaps in the new package) + +The new package has UTs only for `backoff`, `cleanup.shouldCleanUpManifest`, and `eventhandler`. Port/adapt from the +old package: + +- `availability_tracker_test.go` (every kind and status combination). +- `waves_test.go` (`organizeManifestsIntoProcessingWaves`). +- `preprocess_test.go` (decode, identifiers, duplicates, `findLeftOverManifests`, write-ahead status building). +- `apply_test.go` (`sanitizeManifestObject`, `setLastAppliedConfigAnnotation` size limit, + `shouldUseForcedServerSideApply`, `validateOwnerRefs`). +- `drift_detection_takeover_test.go` (`preparePatchDetails`, `discardFieldsIrrelevantInComparisonFrom`, redaction, + `isPlacedByKubeFleetInDuplicate`, `shouldInitiateTakeOverAttempt`). `isOwnedByOlderKubeFleetAPIObjects` is done + (October 26, 2026), in `takeover_drift_detection_test.go` (named after the source file); see [Status](#status). +- `status_test.go` (condition setters, port-back, `FirstDiffedObservedTimestamp`, work-level aggregation). +- Branches envtest cannot reach reliably: `FailedToFindObjInMemberCluster`, `FailedToRunDriftDetection`, + `AppliedWithFailedDriftDetection`, transient decoding errors, context cancellation mid-wave, `Forget` bookkeeping. + +## Old tests → new plan + +| Old area (`pkg/controllers/workapplier`) | New plan | +| --- | --- | +| `apply new manifests (regular)`, migrated single ConfigMap | Covered (core) | +| `garbage collect removed manifests`, migrated "delete manifest removed from work" | Covered (core leftover) | +| Generate name, decoding errors, duplicates, `mixed (pre-processing)` | Covered (failures) | +| GC `blockOwnerDeletion=false` (3 co-ownership variants) | Basic unblock covered (core); co-owned variants → C5/C6. Other-owner valid/invalid variants dropped (no GC in envtest → nothing observable) | +| Takeover `IfNoDiff` (no diff / partial diff / full diff) | → T3 / T4, T5 | +| `never take over` | Behavior changed → T1 (`ReportError` is now the **default**; no diffs are computed) | +| Drift: partial / degraded / full / overwrite (always apply) | → D2 / D7 / D5 / D1 | +| Drift before manifest version bump | → D4 | +| First drifted time preservation | → D2 (`FirstDiffedObservedTimestamp`) | +| Obscure sensitive data (drift / report diff) | → D8 / T12 | +| Report diff (6 contexts), switch report diff ↔ CSA/SSA | Dropped (no `ReportDiff` in `v1alpha1`); diff reporting lives in T4/T5 and D2/D5; strategy switches → T10, D9, A6 | +| Switch never → take over if no diff | → T10 | +| Falling back from CSA to SSA | → A4 | +| Status back-reporting | Dropped (no report-back strategy in `v1alpha1`) | +| Migrated: built-in change / third-party merge | → A1 | +| Migrated: CRD change | → A2 | +| Migrated: missing last-applied annotation | → A3 | +| Migrated: failed manifest identification | Behavior changed (missing namespaces are auto-created by default) → A8 (API validation failure) | +| Migrated: delete even if apply fails | → A8 | +| Migrated: reorder manifests | → A10 | +| Migrated: join and leave | → L1 (now `Leave` orphans appliedWorks; envtest needs simulated GC) | +| `backoff_integration_test.go` | → R2/R3 (via instrumented reconcile timestamps; `ObservationTime` is gone) | +| `status_integration_test.go` (skipped status updates) | → R1 | +| `waves_integration_test.go` | → W1–W4, plus W5 (cross-work waves, new) | +| Priority queue runs (`KUBEFLEET_CI_WORK_APPLIER_RUN_WITH_PRIORITY_QUEUE`), metrics | Dropped (no priority queue or metrics in the new package) | + +## Decisions + +All the decisions below are adopted; the implemented test cases follow them. + +1. **Extra reconcilers share member env 1** (separate hub managers and reserved namespaces) rather than starting extra + member API servers per Ginkgo process, as the old suite did (4 member envs). This keeps `-p` runs light. The + constraint in [Test environment constraints](#test-environment-constraints-and-design-implications) item 9 applies. + Applied to the lifecycle, waves, and instrumented reconcilers (October 21, 2026). +2. **`Join`/`Leave` (L1) runs on a dedicated lifecycle reconciler**, with `DeferCleanup(workApplierLifecycle.Join)`. + No `Serial` decorator is needed (see constraint 8). **Adopted (October 21, 2026)**, replacing the earlier choice of + `Serial` on reconciler 1. L4 does not call `Join`/`Leave`, so it stays on reconciler 1. +3. **Include R2/R3 (backoff timing), labeled `slow`**, on the instrumented reconciler with a fast custom limiter. The + limiter math is already covered by `backoff_test.go`; these specs only check the wiring (counter reset on spec and + result changes, skip-to-fast). **Adopted (October 21, 2026)**; the alternative (drop R2/R3 and keep R1 only) is not + taken. +4. **Wave delay 3s** (old: 5s), with assertions `gap ≥ 3s` and `gap < 6s` (creation timestamps have 1s granularity). +5. **Gate the package in `make local-unit-test` and add a dedicated Ginkgo CI step** (`TestMain` + + `KUBEFLEET_CI_TEST_RUNNER_NAME`), as for `stagedupdaterun`. Previously the package ran under `go test` in + `make local-unit-test` with no `-p`. **Implemented (October 22, 2026)**; see [Suite changes](#suite-changes) item 5. +6. **Unit-test gaps** (previous section) are a follow-up, not part of this IT work. + +## Behaviors to note (documented only; tests assert current behavior) + +1. **Re-join does not re-adopt resources by default.** `Leave` orphans appliedWorks, and the GC strips their owner + refs. On re-join, new appliedWorks get new UIDs and the existing objects are unowned. With the default + `WhenAlreadyExists=ReportError` they become `NotTakenOver`. Product review (October 22, 2026): this is + **expected**. Resources previously applied by a cluster that left and rejoined may need to be taken over again. +2. **Stale-UID owner refs are not cleaned up.** `removeLeftBehindAppliedWorkOwnerRefs` drops AppliedWork owner refs + only when the **work** is gone (name check). A ref to a re-created appliedWork of the same name (old UID) is kept, so + takeover fails as a duplicate placement. +3. `ReportError` + `FullComparison` on objects with server-populated fields (Namespace, Deployment, Job) never + converges: the first apply succeeds with post-apply diffs, and every later reconcile reports `FoundDrifts`. + `TakeOverIfNoDiff` + `FullComparison` never takes such objects over. +4. Post-apply diffs (full comparison) are reported with `Applied=True` (reason `Applied`); there is no dedicated + condition. +5. Not-applied manifests get no `Available` condition (removed, not `False`), and the work reads + `Available=False/SomeManifestsAreNotYetAvailable` even when the cause is an apply failure. +6. Label/annotation-only changes to works (e.g., linked-count or snapshot annotation without a spec change) are + filtered by `GenerationChangedPredicate` and only picked up at the next periodic requeue. +7. Events on secondary works without a valid controller owner ref are dropped (logged); their changes are picked up + only by the primary's periodic requeue. +8. Pre-apply drift detection requires the live manifest-hash annotation to match. Objects just taken over, or whose + manifest just changed, are applied (drifts overwritten) even with `WhenDrifted=ReportError`. +9. A work with zero manifests reports `Applied=True` and `Available=True`. +10. Taking over a pre-existing Deployment bumps its generation (1 → 2) even when its spec is unchanged: the API + server bumps Deployment generations on annotation changes too (annotations are copied to ReplicaSets), and the + apply op adds the KubeFleet annotations. Manifest conditions therefore report generation 2 for taken-over + Deployments (T3), but 1 for Deployments the work applier created or never wrote to. + +## Running the tests + +Always use the Ginkgo CLI in parallel mode, via `go run` so the CLI matches the Ginkgo version pinned in `go.mod` +(v2.33.0): + +```bash +KUBEBUILDER_ASSETS="$(make -s kubebuilder-assets-path | tail -1)" \ + go run github.com/onsi/ginkgo/v2/ginkgo -p --race ./pkg/v1/controllers/workapplier +``` + +Each parallel process runs its own `BeforeSuite` (own envtest hub/member, managers, reconcilers) and runs one spec at +a time; `Ordered` contexts stay within one process and run as a unit. Use +`--focus=""` for a subset (the regex is not anchored at the `Describe` text, so `^takeover` +matches nothing), `--focus-file=_integration_test.go` for one file, and `--label-filter='!slow'` to skip R2/R3. + +## Estimated runtime + +Rough, per context (serial time): most T/D/A/C/L/V contexts 10–25s (one or two periodic requeues plus fast cleanup +via simulated GC); C1/C2 ~15s (`2×cleanupWaitTime`); W1–W3 ~10s each, W4 ~25s, W5 ~10s; R1 ~15s, R2/R3 ~30s each; L1 +~30s. ~55 new contexts ≈ 15–18 min of serial time. With `-p` on ~8 processes this is ~3–4 min wall (vs. 1m13s +today). + +## Status + +- [x] Surveyed the new package source, the existing ITs, and the old ITs (October 21, 2026). +- [x] Baseline run: 155/155 pass with `ginkgo -p --race` (61.7s spec time). +- [x] Decisions adopted (see [Decisions](#decisions)). +- [ ] Suite changes + shared utilities (Suite changes items 0–3 and 5 done; the P1/P2 fixtures pending). +- [x] **T1** (October 21, 2026): `takeover_integration_test.go`, context `pre-existing objects with the default sync + strategy (report error, no takeover)`, 10 specs. Beyond the plan row, the work deletion does **not** simulate GC, so + the work applier takes the forced cleanup path (which visits every manifest with an `Applied` condition) and the test + verifies that path does not write to objects it never owned. Added helper `objectUntouchedActual` (resource version + unchanged) and constant `notTakenOverErrMsg` to `utils_integration_test.go`. Verified: focused run 10/10; mutation + check (`AlwaysTakeOver`) fails as expected; full package run 165/165 with `ginkgo -p --race` (69.2s spec time); + `golangci-lint --fast-only` clean. +- [x] Renamed `utils_test.go` → `utils_integration_test.go` (`git mv`; helpers only, no `Test` functions) (October 21, + 2026). +- [x] **T-section P0s** (October 21, 2026): T2–T5, T7, T9 in `takeover_integration_test.go`. There are 7 new contexts + (T5 is split into partial and full comparison contexts), and the file now has 64 specs: + - T2 `always take over`; T3 `take over if no diff (no diff)`, which asserts the not-yet-available status before + marking the Deployment available, to avoid racing the takeover and apply writes (Behaviors 10 covers its + generation of 2); + - T4 `take over if no diff (diffs in managed fields, partial comparison)`, which also checks that the Deployment's + `FirstDiffedObservedTimestamp` survives a status rewrite; + - T5 `… (diffs in unmanaged fields only, partial comparison)` / `… (full comparison)`; T7 `co-ownership`; T9 + `objects placed by another work object (duplicate placement)`. + - New in `utils_integration_test.go`: helpers `configMapObjectActual` (labels + any owners), + `objectNotRecreatedActual`, `retrieveManifestFirstDiffedTimestamp`; the `externalOwnerRef` fixture; constants + `ownedByOthersErrMsg`, `placedInDuplicateErrMsg`, `coOwnershipDisallowedErrMsg`, and + `memberOnlyLabelKey`/`memberOnlyLabelValue`. + - Verified: the takeover file runs 64/64. Two temporary product mutations were caught (keeping left-behind + AppliedWork owner refs → T9 fails; skipping the co-ownership check in `validateOwnerRefs` → T7 fails). The full + package runs 219/219 with `ginkgo -p --race` (67.0s spec time, 1m15s wall), and `golangci-lint --fast-only`, + `gofmt` and `goimports` are clean. +- [x] **W and R sections** (October 21–22, 2026): W1, W2, W3 (P1) and W5 (P0) in the new `waves_integration_test.go` + (4 contexts, 9 specs, on reconciler 2); R1 (P0, as two contexts: a single work and a linked primary + secondary + set), R2 and R3 (P1, labeled `slow`) in the new `requeue_integration_test.go` (4 contexts, 18 specs, on reconciler + 3). W4 (P2) is not written yet. Suite changes items 1–3 are implemented for these (see + [Suite changes](#suite-changes)). + - W1–W3 compare the creation timestamps of the applied objects (same wave: within 1s; next wave: ≥ delay and + < 2×delay). W5 checks that each linked work status is written exactly twice, i.e., the config map in the primary + work is applied in the same pass as the namespace in the secondary work; it also checks that the config map is + created in the wave right after the namespace. + - R1 checks 2 work status writes and 1 appliedWork status write per work, then no more writes over ≥2 further + reconciles. R2/R3 check the requeue gaps between reconcile starts, then the reset to 2s delays after the + deployment is marked available (R2) or the work is updated (R3; see the review fixes below). + - New in `utils_integration_test.go`: `parallelizerWithFixedDelay`, `recordingClient` (with + `recordingStatusWriter`), `requeueGapsActual`, `retrieveCreationTimestamp`, and the constants + `requeueGapLowerTolerance`/`requeueGapUpperTolerance` (−250ms/+1.5s). + - Mutation checks (October 21). Batch 1 made every manifest wave 0 and disabled the reset on linked-work + generation changes. It failed W2, W3, W5 and R3's reset spec, plus R1's single-work context, which also depends + on wave ordering (its namespace and config map are in different waves). Batch 2 put every manifest in its own + wave, never skipped status writes, and disabled both the reset on processing-result changes and skip-to-fast. + It failed W1, W2, W5, both R1 contexts, R3's backoff spec and R2. + - **R2/R3 first-gap flake (verified and fixed October 22, 2026).** In batch 2, R2 failed at its backoff spec + instead of its reset spec. Re-running showed the mutation check is sound: disabling the processing-result reset + alone fails R2 exactly at the reset spec (gap 8.03s, want 2s), and forced status writes alone leave R2 passing. + The backoff-spec failure was a pre-existing flake. With no mutation, the waves and requeue files failed in 6 of + 14 `ginkgo -p --race` runs (R2 ×4, R3 ×2), and each time only the first gap was off, by ~2–2.5s. Instrumenting + `Reconcile` traced this to the first reconciliation itself (see constraint 10). The fix: `requeueGapsActual` + takes a `skip` count, and the backoff specs count gaps from the second reconciliation (R2: 2s, 3s, 6s, 8s, 8s; + R3: 2s, 4s, 8s, 8s); the specs are renamed `should back off slowly, then back off fast` and + `should skip to fast backoff`. Replaying all 26 recorded R2/R3 runs, every one passes with the first gap + skipped; the worst later overshoot is +84ms. + - Verified after the fix: the waves and requeue files pass in 6 of 6 runs. Three mutations of `backoff.go` were + each caught at the expected spec only: + - no reset on processing-result or work changes fails the R2 and R3 reset specs; + - never skipping to fast backoff fails R3's backoff spec; + - always skipping to fast backoff fails R2's backoff spec. + + The full package runs 368/368 with `ginkgo -p --race` (123.8s spec time, 2m13s wall); lint, gofmt and + goimports are clean. +- [x] **C-section and L-section P0s** (October 21, 2026): C1, C3, C4, C5 in the new `cleanup_integration_test.go` + (4 contexts) and L1, L4 in the new `lifecycle_integration_test.go` (2 contexts), 43 specs in all. + - C1 `orphan resources upon placement deletion (linked work objects)`. Only the primary sets `OrphanResources`; the + secondary has no sync strategy. The test checks that: + - the primary appliedWork is deleted with the `orphan` finalizer; + - after `cleanupWaitTime`, both applied objects' AppliedWork refs are unblocked and the + `applied-work-forced-deleted` annotation is set; + - after `2×cleanupWaitTime`, the secondary finalizer and appliedWork are removed and the primary work is gone, + while the primary appliedWork stays pending deletion for the GC and the applied objects stay in place. + - C3 `leftover objects no longer owned by KubeFleet`: the AppliedWork owner ref is stripped on the member, which + yields `NotTakenOver`. Removing the manifest then leaves the object untouched. + - C4 `leftover objects never taken over`: the pre-existing object is `NotTakenOver`; removing the manifest leaves it + untouched. + - C5 `leftover objects co-owned with other sources`: removing the manifest drops only the AppliedWork owner ref; + the other owner and the data are kept, and the object is not re-created. + - L1 `leaving and re-joining the fleet` (originally `Serial` on reconciler 1; rewired to the dedicated lifecycle + reconciler, see the next entry). Two works are used, one + with the default strategy and one with `AlwaysTakeOver`, instead of three. The test checks that: + - `Leave` (retried on conflicts, then called again to check it is idempotent) removes the finalizers and deletes + the appliedWorks with the `orphan` finalizer; + - work updates are not processed while left; + - after the GC is simulated (the AppliedWork owner refs are stripped from the dependents, then the `orphan` + finalizer) and `Join` is called, the finalizers come back and the appliedWorks are re-created with new UIDs; + - the default-strategy work reports `NotTakenOver` and leaves the orphaned object as is, which confirms + Behaviors 1, while the `AlwaysTakeOver` work re-adopts the object (same UID) and applies the update. + + `DeferCleanup` and the `AfterAll` re-join the fleet if the test fails. + - L4 `owner namespaces of placement bindings (same placement binding name)`: a cluster-scoped binding (work A) and + a namespaced binding of the same name (works B and B2), all linked to a snapshot of the same name. Each set is + processed on its own, and deleting work A leaves B2's finalizer and appliedWork and the B-owned objects alone. + - Changes in `utils_integration_test.go`: `appliedWorkRemovedActual` now also strips the `orphan` finalizer + (simulating orphan GC). New: `appliedWorkBeingDeletedWithOrphanActual`. + - Verified: the C and L files run 43/43. One batch of five temporary product mutations failed exactly the expected + six contexts: + - foreground deletion despite `OrphanResources` fails C1; + - skipping the ownership check on leftovers fails C3 and C4; + - deleting co-owned leftovers fails C5; + - dropping the owner namespace from the field index fails L4; + - reconciling while not joined fails L1. + + The full package runs 328/328 with `ginkgo -p --race` (121.8s spec time, 2m10s wall; the `Serial` L1 tail runs + alone at the end). Lint, gofmt and goimports are clean. +- [x] **Dedicated lifecycle reconciler + L1 rewire** (October 21, 2026). Added `workApplierLifecycle` / `hubMgrLifecycle` + and the `fleet-member-experimental-lifecycle` hub namespace to `suite_test.go` (Suite changes item 0). L1 now uses + them and drops `Serial` (Decision 2 revised; constraint 8 corrected); L4 stays on reconciler 1. Verified: the + lifecycle file runs 18/18; the "reconcile while not joined" mutation still fails L1 at "should not process the work + objects while away from the fleet". The full package runs 328/328 with `ginkgo -p --race` (104.1s spec time, + 1m51s wall, down from 2m10s now that L1 runs in parallel with the other contexts); lint, gofmt and goimports are + clean. +- [x] **Rest of the L section** (October 21, 2026): L2 (P1), L3 (P1), L5 (P2) in `lifecycle_integration_test.go`, all + on reconciler 1 (none of them calls `Join`/`Leave`). The L section is now complete; the file has 5 contexts and 31 + specs. + - L2 `work objects that are not primary work objects`: a standalone work (no linked-count annotation, no owner) + and a secondary-style work whose controller owner (a primary work) does not exist. + - L3 `malformed primary work objects`: three primaries, missing the binding label, missing the snapshot annotation, + and with a non-integer linked count. + - For L2 and L3, the manifests target a namespace that is never created, so an unprocessed work leaves no object + and no auto-created namespace behind. The test checks this with `Consistently`, then checks that each work is + removed at once on deletion (no finalizer). + - The malformed works are safe for the rest of the suite: the field index logs extraction errors and skips the + object. + - L5 `appliedWork objects deleted by an external actor`: the appliedWork is deleted with background propagation, + then GC is simulated by deleting the dependent. At the next requeue, both the appliedWork and the object are + re-created with new UIDs, and the appliedWork status lists the new object UID. If the work applier reconciles + between the two deletions, it briefly reports the object as `NotTakenOver` (it is still owned by the deleted + appliedWork) until the GC catches up. + - New in `utils_integration_test.go`: `workNotProcessedActual`. + - Verified: the lifecycle file runs 31/31. One batch of mutations failed exactly L2, L3 and L5 at the expected + specs, while L1 and L4 passed: + - five product changes: the event handler enqueues ownerless non-primary works; `Reconcile` stops skipping + non-primary works; `retrieval.go` ignores the missing binding label, ignores the missing snapshot annotation, + and treats an unparsable linked count as 1; + - a test-side change: L5 skips the simulated GC of the dependent. + + The full package runs 341/341 with `ginkgo -p --race` (109.8s spec time, 1m59s wall); lint, gofmt and goimports + are clean. +- [x] **A-section P0s** (October 21, 2026): A1, A4, A5, A7 in the new `apply_integration_test.go` (4 contexts, 29 + specs). + - A1 `client-side apply (manifest updates and changes made by third parties)`. A third-party change to a field the + manifest specifies is reverted at the next periodic requeue, while added labels and data keys are kept. The + updated manifest then removes a label, adds an annotation and changes a data value; the third-party additions + survive the three-way merge. + - A4 `client-side apply falling back to server-side apply (oversized objects)`: ~300 KiB of data leaves the + last-applied annotation empty. The update is applied via SSA, which is asserted through `managedFields`: the + changed field is owned only by `kubefleet-member-agent/Apply`. + - A5 `server-side apply (field ownership conflicts)`: the takeover succeeds, but the non-forced SSA reports + `FailedToApply` with `conflict with "integration-test-manager"` and leaves the data alone. With + `ForceConflicts=true` the manifest is applied and KubeFleet owns the field. + - A7 `namespace auto-creation (default)`: the namespace is created without owner refs or KubeFleet annotations, is + not reported in the work or appliedWork status, and is left as is after the work is deleted. + - Changes in `utils_integration_test.go`: `configMapObjectActual` now also compares non-KubeFleet annotations; + `namespaceNotManagedActual` also fails on a deletion timestamp. New: `nonKubeFleetAnnotations`, + `configMapObjectWithDataDigestActual` (digests keep failure output readable for large data), + `objectAnnotationActual`, `fieldManagersActual` (`[MANAGER]/[OPERATION]` owners of a `managedFields` path), and + the constant `thirdPartyFieldManagerName`. + - Verified: the apply file runs 29/29. Four temporary mutations of `apply.go` were each caught at the expected spec: + ignoring the last-applied config in the three-way merge fails A1; disabling the SSA fallback fails A4; always + forcing conflicts fails A5; never creating namespaces fails A7. The full package runs 285/285 with + `ginkgo -p --race` (83.6s spec time, 1m33s wall); lint, gofmt and goimports are clean. +- [x] **D-section P0s** (October 21, 2026): D1, D2, D4, D5, D6 in the new `drift_integration_test.go` (5 contexts, + 37 specs). + - D1 `apply anyway (default), partial comparison`: the drift is overwritten at the next periodic requeue, with no + work change; the label that only the member has is kept; no diff is reported. + - D2 `report error, partial comparison (drift appears, changes, and is reverted)`. Deviation from the plan row: + instead of waiting through ≥2 requeues, the test **changes** the drift (replicas 2 → 3). Unchanged statuses are + never rewritten, so waiting would prove nothing; the change forces a real status rewrite while the object stays + drifted, and the test checks that `FirstDiffedObservedTimestamp` survives it. It also asserts that the + `Available` condition is removed while drifted, and that the manifest is applied again after the revert. + - D4 `report error, partial comparison (manifest updated after a drift)`: a new image in the manifest changes its + hash, so the apply goes ahead and overwrites the drifted replica count. + - D5 `report error, full comparison` (ConfigMap; the namespace is created directly, see Behaviors 3). + - D6 `apply anyway, full comparison (diffs reported after apply ops)`: `Applied=True` with `DiffDetails` for the + label that only the member has, which persists; the Namespace manifest reports `/spec/finalizers` = + `[kubernetes]` (server-populated). + - New in `utils_integration_test.go`: `deploymentObjectActual` (`regularDeploymentObjectAppliedActual` is now a + thin wrapper around it, comparing the same fields), `updateDeploymentInMemberCluster`, + `updateConfigMapInMemberCluster` (retry on conflicts, return the updated object), and `objectLabelsActual`. + - Verified: the drift file runs 37/37. Two batches of temporary product mutations were each caught at the expected + spec: + - (A) ignoring the manifest-hash gate fails D4; disabling post-apply detection fails D6; always resetting the + first-diff timestamp fails D2; + - (B) disabling pre-apply detection fails D2, D4 and D5. + + The full package runs 256/256 with `ginkgo -p --race` (78.8s spec time, 1m27s wall); lint, gofmt and goimports + are clean. +- [x] **Review fixes** (October 22, 2026). Not yet re-run; pending verification. + - T1 `pre-existing objects with the default sync strategy (report error, no takeover)`: the `AfterAll` now deletes + the work itself (ignoring NotFound) and waits for it to be removed. Previously a failed spec skipped the delete + spec, so the `AfterAll` timed out on the appliedWork and left the work behind on reconciler 1. + - T9 `objects placed by another work object (duplicate placement)`: the `AfterAll` deletes both works A and B + ignoring NotFound (work B might not exist yet if the test case fails early). + - R3: the reset spec skips the first reconcile at or after the nudge (`skip: 1`). A periodic reconcile that starts + right after `nudgedAt` might still read the pre-nudge generation from the cache, and the reconcile that the update + triggers would then follow it immediately (a gap of ~0s). The checked gap is 2s either way (the fixed delay or the + initial slow backoff delay), and 8s without the reset. Renamed to + `should requeue after 2 seconds again, as the work object changes`. + - L4: the `Consistently` check compares the UID of work B2's appliedWork (`objectNotRecreatedActual`) instead of + only checking that it exists. Otherwise a cleanup that wrongly touched B2 could be masked: work B's next periodic + reconcile adds the finalizer back and re-creates the appliedWork. + - W1: renamed the spec to `should create the objects in the same wave`; the timestamp check shows that the objects + share a wave, not that they are processed in parallel. +- [x] **T8** (P1) and its unit test (October 26, 2026). Written alongside the + matching safeguard in the older work applier (`pkg/controllers/workapplier`), which now rejects objects owned by + `placement.kubefleet.dev` AppliedWorks. + - `takeover_integration_test.go`, context `objects owned by the older KubeFleet APIs` (12 specs). The pre-existing + config map differs from its manifest and is owned by a `placement.kubernetes-fleet.io/v1beta1` AppliedWork (with + `BlockOwnerDeletion=true`, as the older work applier sets it). Beyond the plan row, the work goes through three + phases: + - `AlwaysTakeOver` + `ShareOwnership` (the planned step); + - the default (`nil`) strategy, which would report `NotTakenOver` without the safeguard, to show that the safeguard + runs regardless of the sync strategy; + - deletion without simulated GC, as in T1, so that the forced cleanup path visits the config map (it has an + `Applied` condition) and must leave the older API's blocking owner reference alone. + + Each phase asserts `Applied=False/OwnedByOtherKubeFleetAPIObjects` (message included in the first phase), work + `Applied=False`/`Available=False`, an empty appliedWork status, and that the config map is untouched (same + resource version, `Consistently` across a periodic requeue). + - New in `utils_integration_test.go`: the `olderAPIAppliedWorkOwnerRef` fixture and the constant + `ownedByOlderKubeFleetAPIObjectsErrMsg`. The planned `createPreExistingObject` helper is not added; like the other + T contexts, T8 creates the object with `memberClient1.Create`. + - Unit test `TestIsOwnedByOlderKubeFleetAPIObjects` in the new `takeover_drift_detection_test.go` (7 cases): + - false: no object, no owners, a non-KubeFleet owner of the `AppliedWork` kind, a `placement.kubefleet.dev` + AppliedWork, an older-API `Work`; + - true: an older-API AppliedWork, alone or alongside a newer-API AppliedWork and a non-KubeFleet owner. + - Verified: gofmt, goimports, `go vet` and `golangci-lint --fast-only` are clean. With `ginkgo -p --race`: + - the T8 context runs 12/12, and the unit test passes (run on its own with `-- -test.run=... -test.v`, as + `ginkgo -p` does not forward the output of plain Go tests that pass); + - the full package runs 380/380 (121.2s spec time, 2m10s wall); + - a temporary mutation (`isOwnedByOlderKubeFleetAPIObjects` always returns false) fails T8 at its first status + spec (the config map is taken over and applied, `Applied=True`), and fails the two `true` cases of the unit + test. + - Follow-up (October 26, 2026), after review: + - The safeguards in both work appliers match owner references by API group and kind only, not by the full API + version, as owner references are not migrated when an API version changes (and the + `placement.kubernetes-fleet.io` group serves AppliedWork in both `v1` and `v1beta1`). + - `TestIsOwnedByOlderKubeFleetAPIObjects` gains 2 cases (9 in total): a `placement.kubernetes-fleet.io/v1` + AppliedWork (true), and an AppliedWork from a group that only shares a prefix with the older API group (false). + The older work applier's `TestIsOwnedByNewerKubeFleetAPIObjects` gains the matching pair. + - The older work applier's IT (`objects owned by the newer KubeFleet APIs`) now sets `BlockOwnerDeletion=true` on + its newer-API owner reference, as the newer work applier does. + - Verified: gofmt, `go vet` and `golangci-lint --fast-only` are clean (goimports is clean on the changed files). + The unit tests pass in both packages, and the new version cases fail against the previous exact-version + guards. With `go test -race` and `-ginkgo.focus`, T8 runs 12/12 and the older work applier's context runs + 13/13. The full packages are not re-run. +- [ ] P1 contexts. +- [ ] P2 contexts. diff --git a/.github/.copilot/breadcrumbs/2026-10-21-2143-autorollout-integration-test-plan.md b/.github/.copilot/breadcrumbs/2026-10-21-2143-autorollout-integration-test-plan.md new file mode 100644 index 000000000..f67b0a595 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-21-2143-autorollout-integration-test-plan.md @@ -0,0 +1,498 @@ +# Auto-Rollout Controller (v1) Integration Test Plan + +**Date**: October 21, 2026 21:43 AEDT +**Task**: Plan and implement the integration tests for `pkg/v1/controllers/autorollout`, following the patterns of the +staged update run controller tests (see [the staged update run plan](./2026-10-20-1723-stagedupdaterun-integration-test-plan.md)). +All planned scenarios are implemented; see [Status](#status) for the progress log. + +## Requirements + +1. Cover the controller end to end against a real API server (envtest), for both `ClusterPlacementPolicy` and + `PlacementPolicy`. +2. Follow repo and v1 conventions: Ginkgo/Gomega, `_integration_test.go`, `Ordered` contexts with step-wise + `It`s, one `Describe` block per scope, whole-struct `cmp.Diff` comparisons, `want` naming, random name suffixes + (`utils.RandStr()`) so that contexts never collide. +3. No production code changes are needed to make the controller testable (see constraint 4). + +## Controller behavior under test + +Reconcile flow (see [controller.go](../../../pkg/v1/controllers/autorollout/controller.go)): + +1. Get the policy from the cache (`ClusterPlacementPolicy` if the request has no namespace). Not found → forget the + request, done. +2. Deleting → relinquish the binding manager role if the cached policy shows it held (no API call otherwise; the event + handler always enqueues a policy whose role this controller holds, so a cache that has yet to show the claim is + followed by another reconcile), done. No finalizer is ever added. +3. `shouldSkipProcessing` → relinquish the role (if held, as in step 2) and stop when: + - auto-rollout is disabled (`kubefleet.dev/auto-rollout-enabled` is not exactly `"true"`) → no requeue; + - no rollout is needed → no requeue. A rollout is needed iff `ResourceCollected=True` **and** `Synchronized=False` + with reason `ResourcesChangedPendingRollout` or `SomeClustersOnOlderResourceRevision`, both with + `observedGeneration == generation`; + - the minimum wait period since `kubefleet.dev/auto-rollout-last-rollout-attempt-timestamp` has not passed → requeue + after the remaining time. The period comes from `kubefleet.dev/auto-rollout-minimum-wait-period` (default 3m; + unparsable → default; clamped to [1m, 1h]). A missing or malformed timestamp counts as never rolled out; one in + the future counts as just rolled out (the remaining time never exceeds the period). +4. Claim the binding manager role (`status.bindingManager = {controllerName: auto-rollout, objectRefs: [the policy + itself]}`). Held by another controller → requeue with the claim rate limiter; nothing is written. +5. Rollout: list bindings by the `spec.placementPolicyName` field index (in the policy's namespace for namespaced + policies), keep only those not being deleted **and** controlled by this policy instance (`IsControlledBy`, i.e. + UID match); `SnapshotResourcesIfStale` (create index 0 / reuse the latest / create the next index; an empty result + is an unexpected error); set `spec.resourceSnapshotName` to the latest primary snapshot on every binding that does + not already point to it (parallelizer, fail fast on the first error). Failure: + - a **user error** (in practice from snapshotting, e.g., a selected resource too large for a snapshot) → a Warning + event regarding the policy (reason `FailedToSnapshotResourcesDuringAutoRollout`, action `SnapshotResources`), + write `last-rollout-attempt-timestamp` (as in step 6), relinquish the role, and requeue after the minimum wait period + with no error, so that the wait period spaces out the retries (C6; behavior 8); + - any other error → return the error **with the role still held**; controller-runtime retries with its default + backoff (5ms doubling, up to ~16m), and no timestamp is written (C8). +6. Success, or a user error (step 5) → write `last-rollout-attempt-timestamp` (RFC3339Nano, UTC) with a JSON merge patch + that carries only that annotation and no resource version (changed on October 21, 2026: this used to be a + resource-version-guarded `Update` made **before** the rollout). +7. On success, requeue after 200ms; that reconcile hits the wait period skip (step 3) and relinquishes the role. + +Watches: `ClusterPlacementPolicy` and `PlacementPolicy` only (see +[eventhandler.go](../../../pkg/v1/controllers/autorollout/eventhandler.go)): + +- Create: enqueue if enabled or holding the role (the latter is only reachable on a controller restart). +- Update: always enqueue if the new object holds the role (the controller's own claim included, which is why a + user-error failure must write the timestamp; behavior 8); otherwise only if old or new is enabled **and** + `autoRolloutRelevantChange` (deletion timestamp, the enabled/min-wait annotations, or — when the new object needs a + rollout — a change in status/reason/observedGeneration of `ResourceCollected` or `Synchronized`). +- Not watched: bindings, snapshots, selected resources (e.g., ConfigMaps), and the last-rollout-attempt annotation. + +## Test environment constraints and design implications + +1. **Only this controller (plus the snapshot manager) runs.** No placement policy controller: tests set + `ResourceCollected`/`Synchronized` on the policy status by hand (stamped with the current generation), and create + bindings by hand with `spec.placementPolicyName` **and a controller owner reference to the policy** (unlike the + staged update run tests, whose bindings carry no owner reference — the auto-rollout controller would skip those). + No GC: tests remove the snapshots the manager creates. +2. **Trigger last.** The controller reads bindings from the shared cache; a binding the cache has yet to see is + silently skipped (the rollout still counts as a success), and is not retried until the next rollout (≥ 1m later). + Set up the policy, bindings, and any + pre-set annotations, `waitForObjectsToBeCached`, and only then write the conditions that call for a rollout (the + trigger). +3. **Only relevant changes trigger a reconcile** (see Watches). Hence: + - backdate `last-rollout-attempt-timestamp` **before** the trigger; changing it afterwards does nothing; + - for a second rollout, flip `Synchronized` to `True/AllClustersSynchronized` and back to `False` (as the + placement policy controller would); rewriting identical conditions does not enqueue; + - a "nothing happens" `Consistently` check after a non-triggering write proves little; where a test needs the + controller's own skip logic to run, pair the write with a min-wait annotation change (alternate `"1m"`/`"2m"`; + harmless when no timestamp exists). +4. **Minimum wait period.** The bounds are package constants (≥ 1m), so tests never wait it out: they write a + backdated timestamp (`time.Now().Add(-d).UTC().Format(time.RFC3339)`) instead. Only D2 relies on the controller's + own `RequeueAfter`, with ~15s remaining. No refresh-period-style override (as in the staged update run suite) is + needed. +5. **Re-rollouts in steady state.** Conditions never change on their own here, so after a rollout the controller + comes back after the wait period (3m by default) and rolls out again if the conditions still call for it. Contexts + finish well within 3m; contexts that continue past a rollout first mark the policy `Synchronized=True`. +6. **The claim is transient on success.** Claim → refresh → timestamp → 200ms → relinquish; the held state is not + reliably observable, so tests assert the end state (role released, i.e. `bindingManager == nil`). Two states are + observable: claim-blocked (no timestamp is written, since the timestamp comes last) and a rollout failing with a + non-user error (the role stays held across retries, C8). A user-error failure is transient too (claim → snapshot + fails → event → timestamp → relinquish), so C6 asserts its end state and the event. +7. **Fast claim retries.** Releasing another controller's claim is not a relevant change; the retry comes from the + claim rate limiter, which the suite sets to 250ms–2s (vs. 5s–180s by default). +8. **Snapshots.** Names are deterministic (`primaryPlacementResourceSnapshotName(policyName, idx)`); the default + revision history limit is 3, so no snapshot is garbage collected with ≤ 3 snapshots per policy. Hand-made + snapshots are not used: the manager relies on annotations (resource hash) that only it writes. To cause a user + error, the suite caps the resource data per snapshot at 16 KiB (the default is ~1.2 MiB), and C6 grows its + ConfigMap past it; the manager checks the cap only when it creates a snapshot, so nothing is created on failure. +9. **Write conflicts.** Test writes to a policy (annotations, status) can race with the controller's claim and + timestamp write; helpers re-read and retry in `Eventually`, and the controller retries on conflict by itself. No + placement policy controller runs here, so the production race in behavior 1 (a status write between the claim + and the timestamp `Update`) is not exercised. + +## Suite changes (`suite_test.go`) + +Mirror [the staged update run suite](../../../pkg/v1/controllers/stagedupdaterun/suite_test.go): + +- envtest with `config/crd/bases`; `placementv1alpha1` in the scheme (`clusterv1beta1` and `rolloutv1alpha1` are not + needed: no member clusters are involved, and the auto-rollout API items are plain annotation keys). +- Manager (metrics off); `hubClient = hubMgr.GetClient()` (cache-backed, shared with the controller); + `fieldindexers.SetupWithHubAgentControllerManager`; the placement resource snapshot manager (dynamic client, empty + informer manager, baseline eligibility checker, and a 16 KiB cap on the resource data per snapshot, see constraint + 8), added to the manager. +- `New(hubMgr.GetClient(), snapshotManager, parallelizer.NewParallelizer(workerCount /* 4 */), + workqueue.NewTypedItemExponentialFailureRateLimiter[ctrl.Request](250ms, 2s), hubMgr.GetEventRecorder(controllerName))`; + `SetupWithManager(hubMgr, maxConcurrentReconciles /* 4 */)`. +- Namespaces `app` and `other-app`, created once per process. +- No `TestMain` CI skip: the package runs in `make test` (decision 2). +- `hubAPIReader = hubMgr.GetAPIReader()` (uncached), used only where a read must reflect a write the test case has + just made, e.g., that the controller released a claim the test wrote (A5); a cached read might predate the claim. + C6 also reads the policy's resource version and the `events.k8s.io/v1` events through it (the cache does not watch + events). + +Implemented with group A ([suite_test.go](../../../pkg/v1/controllers/autorollout/suite_test.go)). + +## Shared test utilities (`utils_integration_test.go`) + +Package-local copies/adaptations of the staged update run helpers (test helpers cannot be imported across packages; +extracting a shared package is out of scope), scope picked by the namespace (`clusterScopedNS == ""`): + +- **Constants**: name templates (`configmap-%s`, `policy-%s`, `binding-%s-%02d`, `cluster-%s-%02d`), + `previousResourceSnapshotName`, `testFinalizer`, the other controller's claim details, timings (`eventuallyDuration` + 20s, `eventuallyInterval` 500ms, `consistentlyDuration` 5s, `consistentlyInterval` 500ms). +- **Stubs**: `newPlacementPolicyObj`, `newPlacementBindingObj`, `newPlacementResourceSnapshotObj`, `newConfigMapObj`. +- **Fixtures**: `createConfigMap`; `createPlacementPolicy(ns, name, configMapName, annotations)`; + `createPlacementBinding(policy, bindingName, clusterName, snapshotName)` (controller owner ref to the policy); + `createPlacementBindingWithOwnerRefs(...)` for the exclusion cases; `createPlacementBindings(policy, randStr, n)` + returning names and objects. +- **Mutations**: `updateConfigMapData`; `updatePlacementPolicyAnnotations(ns, name, func(map[string]string))`; + `setPlacementPolicyConditions(ns, name, conds...)` (stamps the current generation unless the condition already + carries one) with builders `resourceCollectedCond(status, reason)` / `synchronizedCond(status, reason)` and + shorthands `markPlacementPolicyAsPendingRollout(ns, name, reason)` / `markPlacementPolicyAsSynchronized(ns, name)`; + `updatePlacementPolicyStatus(ns, name, func(*PlacementPolicyStatus))` (conditions and claim in one write, for B3/B4); + `updatePlacementPolicyBindingManager`; `bumpPlacementPolicyGeneration` (e.g., set + `resourceRevisionHistoryLimit` to 5); `backdatedTimestamp(d)`; `touchPlacementPolicy(ns, name)` (sets a harmless + test annotation to a fresh value, to enqueue a role-holding policy right away, for C8). +- **Admission (C8)**: `createSnapshotChangeDenyPolicy(randStr)` creates the `ValidatingAdmissionPolicy` and its + binding (selecting the per-context label) and returns both for cleanup; `bindingSnapshotChangeDeniedActual(ns, + name)` / `bindingSnapshotChangeAllowedActual(ns, name)` make a dry-run snapshot change on a binding to tell whether + the rule is in force. The admission types come with client-go's `scheme.Scheme`, which the suite already uses. +- **Lifecycle**: `markObjectForDeletion`, `removeTestFinalizer`, `waitForObjectsToBeCached`, `objectRemovedActual`, + `removeObjects` / `deleteObjectsAndWaitForRemoval` (no approval request ordering needed). +- **Assertions** (`...Actual` closures): `primaryPlacementResourceSnapshotCreatedActual(policy, idx)` (labels and + owner ref, as in the staged update run tests); `placementResourceSnapshotAbsentActual(ns, name)`; + `placementBindingsResourceSnapshotActual`; `placementBindingGenerationActual`; + `placementBindingsUntouchedActual(ns, names)` (previous snapshot and generation 1); `bindingManagerActual`; + `bindingManagerRoleReleasedActual`; `wantAutoRolloutObjectRef(ns, name)` (the policy itself); + `lastRolloutAttemptTimestampActual(ns, name, notBefore)` (present, RFC3339Nano-parsable, UTC, within + `[notBefore, now]`); `lastRolloutAttemptTimestampUnchangedActual(ns, name, wantVal)` (`""` = absent). +- **Composites**: `rolledOutActual(policy, bindingNames, idx, notBefore)` (snapshot `idx` created, bindings point to + it, timestamp ≥ `notBefore`, role released) and `notRolledOutActual(policy, bindingNames, nextIdx, wantTimestamp, + wantBindingManager)` for `Consistently`. + +**Implemented with group A** (only what A uses, to keep the `unused` linter quiet; B–D add their own), with a few +deviations from the list above: + +- `createPlacementBindings(policy, randStr, n)` returns only the names; call sites wait for the cache with + `placementBindingObjs(ns, names)` stubs (returning objects tripped the `prealloc` linter at every call site). +- `allActual(actuals...)` combines checks for `Consistently`, in place of a fixed `notRolledOutActual`; A's + "no rollout" checks are `allActual(placementResourceSnapshotAbsentActual, placementBindingsActual(ns, names, + snapshot, generation), placementPolicyAtRestActual(ns, name, wantTimestamp))`. +- `placementPolicyRolledOutActual(ns, name, notBefore)` checks the timestamp and the released role on **one** read + of the policy: the timestamp is written after the claim, so a read with the timestamp but no binding manager + cannot predate the claim. `placementPolicyAtRestActual` checks no binding manager and an exact timestamp value + (`""` = absent). +- `bindingManagerReleasedOnAPIServerActual(ns, name)` (via `hubAPIReader`), for releases of claims the test wrote. +- `forcePlacementPolicyReconciliation(ns, name)` switches the min-wait annotation between `"1m"` and `"2m"` + (constraint 3); `lastRolloutAttemptTimestamp(ns, name)` captures the current value for exact comparisons. +- `placementPolicySpecAndAnnotationsUntouchedActual` (A1 only) lives in `lifecycle_integration_test.go`. + +**Added with the C6 user-error steps (October 22, 2026):** + +- In `utils_integration_test.go`: `placementPolicyResourceVersionOnAPIServer(ns, name)` and + `placementPolicyUnwrittenOnAPIServerActual(ns, name, wantResourceVersion)` (both via `hubAPIReader`; a retry claims + and releases the role, i.e., writes the policy, so an unchanged resource version shows that no retry happened). + `placementPolicyRolledOutActual` now also covers a user-error failure, which writes the timestamp too. +- In `rollout_integration_test.go`: `oversizedConfigMapData()` (twice the suite's 16 KiB cap); + `snapshotFailedEventActual(policy)` (lists `events.k8s.io/v1` events in the policy's namespace, or `default` for a + cluster-scoped policy, and checks the type, reason, action, reporting controller, and regarding object of the one + regarding the policy in one `cmp.Diff`, plus that its note mentions the too-large resource). + +**Added with group C:** + +- In `utils_integration_test.go`: `controllerOwnerRefTo(policy)`; `newOwnedPlacementBinding(policy, name, + clusterName, snapshotName)` (built but not created, so C3/C4/C8 can change owner refs, namespace, or labels first) + with `createPlacementBinding(binding)`; `createPlacementBindingsPointingTo(policy, nameSeed, n, snapshotName)` + (`createPlacementBindings` now wraps it with the previous snapshot); `touchPlacementPolicy(ns, name)` (sets the + `kubefleet.dev/integration-test-touched` annotation). +- In `rollout_integration_test.go` (C-only, as scenario helpers are in the staged update run tests): + `variantPlacementBindingNames(randStr, variant)`; the C8 admission helpers `newSnapshotChangeDenyPolicyObjs(randStr)` + (name-only stubs, safe for cleanup even if setup fails) and `createSnapshotChangeDenyPolicy(randStr)`; + `placementBindingSnapshotChangeActual(ns, name, wantDenied)` (one dry-run helper for both directions, matching the + denial by its message); `placementPolicyRolloutRetryingActual(ns, name)` (role held by `auto-rollout` and no + timestamp, on one read). + +**Added with group B:** + +- In `utils_integration_test.go`: `updatePlacementPolicyStatus(ns, name, mutate)` (one status write; + `setPlacementPolicyConditions` and `updatePlacementPolicyBindingManager` now build on it); + `setConditionsOnPlacementPolicy(policy, conds...)` (in-memory, with generation stamping); `pendingRolloutConds(reason)` + (used by `markPlacementPolicyAsPendingRollout`); `wantAutoRolloutBindingManager(ns, name)` (now also used by A5 and + C8); `placementPolicyAutoRolloutStateActual(ns, name, wantBindingManager, wantTimestamp)` (binding manager and + timestamp on one read; `placementPolicyAtRestActual` and C8's `placementPolicyRolloutRetryingActual` are now thin + wrappers around it). +- In `lifecycle_integration_test.go`: the other controller's claim (`otherBindingManager(randStr)`, as in the staged + update run tests) and `markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim(ns, name)` (pending-rollout + conditions and an auto-rollout claim in **one** status write, so the controller never observes one without the + other). `gatingBindingCount` was renamed `lifecycleBindingCount`, as groups A, B, and D share the file. +- B2 writes the pending-rollout conditions along with the claim (the plan only asked for the claim), so auto-rollout + being disabled is the only thing that holds the rollout back. + +**Added with group D:** no new helpers; D reuses the existing ones (`backdatedTimestamp`, +`updatePlacementPolicyAnnotations`, `placementPolicyAtRestActual`, `rolledOutActual`). Notes on the assertions: + +- D1, D3 (second step), and D4 (second step) assert the rollout with the usual 20s `Eventually`, which is shorter than + the remaining wait the controller would requeue for (about 58s in D1 and 50s in D3; never, with a 3-hour unclamped + remainder, for D3's upper bound). A pass therefore shows that the annotation change itself triggered the rollout. +- D2 uses the written timestamp plus 1 minute as `rolledOutActual`'s `notBefore`, so it also asserts that the rollout + did not happen before the minimum wait period had passed (measured: the check waits ~9s after the `Consistently`). + +## Test plan + +Two test files: + +- `lifecycle_integration_test.go`: groups A (gating and lifecycle), B (binding manager role), and D (minimum wait + period). +- `rollout_integration_test.go`: group C (binding refresh and snapshots). + +Each group has its own pair of `Describe` blocks, so no two groups share a test node. The namespaced block comes first, +then the cluster-scoped one, and every scenario is its own `Ordered` `Context` with step-wise `It`s. Every group runs +for both scopes unless noted otherwise. Proposed `Describe` texts (`` is "placement policies" or "cluster +placement policies"), each usable with `--focus`: + +| Group | File | `Describe` text | +| --- | --- | --- | +| A | `lifecycle_integration_test.go` | `gating auto-rollouts of ` | +| B | `lifecycle_integration_test.go` | `managing the binding manager role of ` | +| D | `lifecycle_integration_test.go` | `enforcing the minimum wait period between auto-rollouts of ` | +| C | `rollout_integration_test.go` | `rolling out ` | + +### A. Gating and lifecycle (`lifecycle_integration_test.go`) + +- **A1 Happy path.** Enabled at creation, 3 bindings on the previous snapshot; trigger with + `Synchronized=False/ResourcesChangedPendingRollout` → snapshot 0 created (labels, owner ref), all bindings point to + it, timestamp ≥ trigger time, role released; spec untouched (generation 1), the enabled annotation preserved. Then + `Synchronized=True/AllClustersSynchronized` → `Consistently` nothing moves (bindings at generation 2, timestamp + unchanged, no snapshot 1). +- **A2 Disabled.** No annotation; trigger → `Consistently`: no snapshot, bindings untouched, no timestamp, no claim. + Set the annotation to `"True"` (not exactly `"true"`) → `Consistently` the same (not even enqueued, as neither + version counts as enabled). Set `"true"` → rolled out as in A1. +- **A3 Disabled after a rollout.** Roll out; `Synchronized=True`; remove the annotation; change the ConfigMap; + backdate the timestamp; flip `Synchronized=False/ResourcesChangedPendingRollout` → `Consistently` no snapshot 1, + bindings stay on snapshot 0. +- **A4 Rollout-needed matrix** (one context; each step pairs the write with a min-wait toggle, then `Consistently` + no rollout): (a) no conditions; (b) `ResourceCollected=False/FailedToCollectSomeResources`; (c) + `Synchronized=True/AllClustersSynchronized`; (d) `Synchronized=False/FailedToSynchronizeSomeClusters`; (e) the + right conditions at a stale generation (`bumpPlacementPolicyGeneration` first, then write them with + `observedGeneration = generation − 1`; in the other order they would trigger a rollout); (f) + `Synchronized=False/SomeClustersOnOlderResourceRevision` at the current generation → rolled out (covers the second + reason). +- **A5 Deleting policy.** Enabled, with the test finalizer; mark for deletion (cache observes it); trigger → + `Consistently` no rollout, no timestamp. Then write an auto-rollout claim (`wantAutoRolloutObjectRef`) → released. + Remove the finalizer → gone (the NotFound path; nothing further to assert). + +### B. Binding manager role (`lifecycle_integration_test.go`) + +- **B1 Held by another controller.** Write the other claim and wait for the cache to see it, then trigger → + `Consistently`: other claim intact, no timestamp, no snapshot, bindings untouched. + Release it → rolled out within the claim backoff; role released. +- **B2 Stale own claim, disabled policy.** Write an auto-rollout claim on a policy with auto-rollout disabled → + released (update path for role holders; skip without requeue). Bindings untouched, no timestamp. +- **B3 Stale own claim within the wait period.** Timestamp = now; one status write with the rollout-needed conditions + and the claim → released, `Consistently` no rollout (the requeue-after skip also relinquishes). +- **B4 Own claim with a rollout needed.** One status write with the conditions and the claim → rolled out (the + existing claim is reused via the dry-run check), then released. + +### C. Binding refresh and snapshots (`rollout_integration_test.go`) + +- **C1 Many bindings.** 12 bindings (> 4 workers) → all point to snapshot 0. +- **C2 Already up to date.** Some bindings pre-pointed at snapshot 0's (deterministic) name → they stay at generation + 1; the others move (generation 2). +- **C3 Excluded bindings** stay untouched: marked for deletion (test finalizer); no owner ref; controller owner ref to + an older policy instance (same name and kind, different UID); non-controller owner ref to the policy; bound to + another policy in the same scope. One regular binding is refreshed, as a control. +- **C4 Scope isolation.** Namespaced block: a same-named `PlacementPolicy` in `other-app` and a same-named + `ClusterPlacementPolicy`, each with its own bindings → untouched. Also in `other-app`: a binding whose controller + owner reference points at the `app` policy (Kubernetes treats cross-namespace owner references as absent) → + untouched. Only the namespace the controller lists bindings in excludes this one; the others are excluded by the + ownership (UID) check as well, so without it dropping the namespace filter went unnoticed (mutation MC6). + Cluster-scoped block: a same-named `PlacementPolicy` in `app` with its bindings → untouched. +- **C5 No bindings.** → snapshot 0 created, timestamp written, role released. +- **C6 Successive rollouts, with a user error in between.** After rollout 1: `Synchronized=True`; grow the ConfigMap + past the suite's per-snapshot cap (a user error); backdate the timestamp (> 3m); flip + `Synchronized=False/ResourcesChangedPendingRollout` → a Warning event regarding the policy (constraint 8), the + timestamp written (≥ the trigger), and the role released. Then force a reconciliation (min-wait toggle) → + `Consistently`: the policy is not written to (its resource version stays; a retry would claim and release the role), + no snapshot 1, bindings stay on snapshot 0. Shrink the ConfigMap (a changed value that fits), backdate the timestamp, + and force a reconciliation → snapshot 1, bindings move to it, timestamp advances; snapshot 0 is kept (revision + limit 3). +- **C7 Unchanged resources.** Same as C6 without the ConfigMap change (reason `SomeClustersOnOlderResourceRevision`) + → no snapshot 1, binding generations unchanged, timestamp advances, role released. +- **C8 Update failure and retry.** A `ValidatingAdmissionPolicy` plus its binding. The policy matches `UPDATE`s of + `placementbindings` and `clusterplacementbindings`, and its CEL rule + (`object.spec.resourceSnapshotName == oldObject.spec.resourceSnapshotName`) rejects snapshot changes. The binding + limits it to bindings carrying a per-context label (`matchResources.objectSelector`); other updates pass. Before the + trigger, wait until the rule is in force (`Eventually` a dry-run snapshot change on the labeled binding is + rejected). With one such binding among three, trigger → `Consistently`: snapshot 0 exists and no snapshot 1 (retries + reuse it), the denied binding untouched, no timestamp, and the role held by `auto-rollout` across retries (the others + may or may not have moved, due to fail fast). Delete the policy binding, wait until a dry-run change is accepted, + then touch a harmless test annotation on the policy: role holders are always enqueued, which skips the backoff + (about 10s by then). → all bindings on snapshot 0, timestamp written, role released. Both admission objects are + cluster-scoped; each context creates its own (random names) and deletes them in `AfterAll`. + +### D. Minimum wait period (`lifecycle_integration_test.go`) + +- **D1 Recent rollout blocks.** Timestamp = now − 2m (default 3m); trigger → `Consistently` no rollout, timestamp + unchanged (exact string), no claim. Set the min-wait annotation to `"1m"` → rolled out right away. +- **D2 Requeue after the remaining time.** Min wait `"1m"`; write timestamp = now − 45s right before the trigger + (after the cache wait, so that setup time does not eat into the ~15s left) → `Consistently` (5s) no rollout, then + `Eventually` (20s) rolled out with no further writes. +- **D3 Bounds.** Min wait `"1s"`, timestamp = now − 10s → `Consistently` no rollout (clamped up to 1m). Then min wait + `"5h"` with timestamp = now − 2h, in one write → rolled out (clamped down to 1h). +- **D4 Malformed values.** Min wait `"soon"`, timestamp = now − 2m → `Consistently` no rollout (default 3m, not the + 1m lower bound). Then timestamp `"not-a-timestamp"` plus min wait `"later"` (to force a reconcile) → rolled out; + the malformed timestamp is replaced. + +### Better as unit tests (out of scope here; the package has no unit tests yet) + +- `retrieveMinimumWaitPeriod`, `remainingWaitPeriod`, `isRolloutNeeded`, `shouldSkipProcessing` (full matrices). +- `autoRolloutRelevantChange` and the event handler funcs, including the restart path (a create event for a + role-holding, disabled policy), which envtest cannot reach without restarting the manager; `castToPlacementPolicy`. +- `New` argument checks and defaults (including the required event recorder); `refreshBindings` fail fast, error + aggregation, and context cancellation (fake client with `interceptor.Funcs`, as the placement policy controller unit + tests do); rate limiter bookkeeping; the user-error branch's write order (timestamp before release) and its handling + of a failure of either write. + +## Behaviors to note (documented only; no action for now) + +1. **Resolved (October 21, 2026).** The timestamp is now written only after a successful rollout (or, since + October 22, a user-error failure; behavior 8), so other failures are retried promptly (C8). Writing it afterwards + with the old resource-version-guarded `Update` would have opened a + race: in production the placement policy controller owns the bindings, reacts to their generation bumps, and + rewrites the policy's `Synchronized` condition. That changes the policy's resource version, so the `Update` + conflicts. On the retry, if every binding already pointed to the latest snapshot, `Synchronized` would read + `False/FailedToSynchronizeSomeClusters`; no rollout would count as needed, the role would be released, and no + timestamp would ever be written, so the next resource change would roll out regardless of the minimum wait period. + `bookkeepLastRolloutAttemptTimestamp` now uses `client.MergeFrom` (decided with the user over a JSON patch). Verified + against envtest 1.33 with throwaway tests (now removed), writing from a stale copy after concurrent status, spec, + and annotation changes: + - full `Update` (old code): conflict; + - `client.MergeFrom(original)`: the body is only `{"metadata":{"annotations":{"":"..."}}}`, even + when the stale copy had no annotations; the concurrent status, spec, and annotation changes all survive. Checked + again through the real `bookkeepLastRolloutAttemptTimestamp` for both scopes; + - JSON patch `add` on `/metadata/annotations/kubefleet.dev~1auto-rollout-last-rollout-attempt-timestamp`: works when + the annotations map exists, but is rejected when the object has none (the parent must exist), and the `/` in the + key must be escaped as `~1`. + The race is not testable in the integration suite (see constraint 9); a unit test with `interceptor.Funcs` belongs + with the follow-up unit tests. +2. Bindings missing from the cache at rollout time are skipped; the rollout still counts as a success, so they wait + for the next one. +3. Changes to selected resources do not trigger the controller; it relies on the placement policy controller updating + the conditions. Rewriting identical conditions does not trigger it either. +4. While the conditions keep calling for a rollout (e.g., the placement policy controller lags), the controller + re-evaluates every wait period; with unchanged resources only the timestamp moves (C7). +5. A claim under the `auto-rollout` name whose refs do not include the policy itself is never cleaned up: claiming + appends the policy ref, and relinquishing removes only that ref. +6. The timestamp is written as RFC3339Nano and parsed as RFC3339; Go accepts fractional seconds there, so this is + fine, but the asymmetry is worth knowing. +7. While a rollout keeps failing with a non-user error (e.g., a binding update the API server rejects, C8), the + controller holds the binding manager role across retries, so the placement policy controller and staged update + runs cannot manage the bindings meanwhile. Disabling auto-rollout releases it. Retries do not wait for the minimum + wait period, but they back off exponentially and create a new snapshot only if the selected resources changed. +8. **Resolved (October 22, 2026).** A user-error failure (in practice, from snapshotting: e.g., a selected resource + too large for a snapshot, which only snapshot creation checks, so the placement policy controller does not report + it) reports a Warning event, writes the timestamp, releases the role, and requeues after the minimum wait period. + Without the timestamp (the first version of this path), C6 found a hot loop: the claim is a status update, which + the event handler always enqueues; the next reconcile passed the wait period check (nothing recorded the failed + attempt) and claimed, failed, and released again, about 10 times a second per policy; the returned `RequeueAfter` + never came into play. Decided with the user over taking the snapshot before the claim (no write on a user error, + but a snapshot could then be taken while another controller holds the role) and over an in-memory backoff (lost on + restarts). The timestamp is written before the role is released, so the wait period holds retries back even if + the release fails. As a consequence, the annotation records the last rollout attempt that succeeded or failed due + to a user error, hence its name, `last-rollout-attempt-timestamp` (renamed from `last-rolled-out-timestamp`), and + fixing the resource takes effect at the next rollout, i.e., up to one wait period later. + +## Decisions (October 21, 2026) + +1. **Approved.** Run every group for both scopes as two `Describe` blocks (the staged update run precedent). +2. **Approved.** Keep the package in `make test` (estimated ~4–5 min serial); no `TestMain` skip and no dedicated CI + step. +3. **Approved.** Include C8 (failure injection with a `ValidatingAdmissionPolicy`, supported by envtest 1.33), as + redesigned for the timestamp being written after the rollout (see behavior 1). +4. **Approved.** The unit tests listed above are a follow-up, not part of this work. +5. **Approved.** File layout: A, B, and D share `lifecycle_integration_test.go`, and C goes in + `rollout_integration_test.go`. Each group keeps its own `Describe` blocks (see the table under Test plan). +6. **Approved (October 22, 2026).** Cover the user-error path in an existing scenario (C6, both scopes) rather than a + new one; throttle user-error retries by writing the timestamp (behavior 8). + +## Estimated runtime + +About 18 five-second `Consistently` windows per scope, plus short `Eventually` waits (D2 ~10s): ~2–2.5 min per +scope, ~4–5 min serial for both, ~1.5 min wall with `ginkgo -p --procs=4`. + +Measured with group A only (72 specs, both scopes): `ginkgo -p --race` ~40s of spec time, ~49s wall; `go test -race` +(serial, as `make test` runs it) ~135s. Group A holds 10 of the ~18 `Consistently` windows per scope, so the full +suite is likely around ~5 min serial. + +Measured with groups A and C (138 specs, both scopes): `ginkgo -p --race` ~52s of spec time, ~55s wall. + +Measured with groups A, B, and C (162 specs, both scopes): `ginkgo -p --race` ~48s of spec time, ~57s wall. + +Measured with the complete suite, groups A–D (192 specs, both scopes): `ginkgo -p --race` ~52s of spec time, ~62s +wall. + +Measured after the C6 user-error steps (198 specs, both scopes; one more 5-second `Consistently` per scope): +`ginkgo -p --race` ~66s of spec time, ~74s wall. + +## Running the tests + +Always with the Ginkgo CLI in parallel mode (user request, October 22, 2026), never serially with `go test`; this +applies to local runs only (CI keeps the package in `make test`, per decision 2): + +```bash +KUBEBUILDER_ASSETS="$(make -s kubebuilder-assets-path | tail -1)" \ + go run github.com/onsi/ginkgo/v2/ginkgo -p --race ./pkg/v1/controllers/autorollout +``` + +Add `--focus=""` to run a subset (see the `Describe` table under Test plan). + +## Status + +- October 21, 2026: plan drafted; decisions 1, 2, and 4 approved, decision 3 (C8) still open. No test code written. +- October 21, 2026 (later): the controller now writes the timestamp after a successful rollout, and the annotation was + renamed to `last-rolled-out-timestamp`. The plan is updated to match (flow, helpers, C8 redesign, behaviors 1, 2, + and 7). Review finding: the resource-version-guarded `Update` could lose the timestamp to a race with the + placement policy controller (behavior 1). +- October 21, 2026 (later): `bookkeepLastRolledOutTimestamp` in `rollout.go` now writes the annotation with + `client.MergeFrom`. Build, vet, and an envtest check of the function (both scopes, a stale copy, concurrent changes) + pass. `golangci-lint` reports only the missing package comment (ST1000), which predates this change. +- October 21, 2026 (later): decision 3 approved (C8 included); A, B, and D moved into `lifecycle_integration_test.go` + with their own `Describe` blocks (decision 5); C8 admission helpers added to the utilities. All decisions are + settled; the plan is ready for implementation. No test code written yet. +- October 21, 2026 (later): **group A implemented** (A1–A5, both scopes): `suite_test.go`, `utils_integration_test.go`, + and `lifecycle_integration_test.go`. 72 of 72 specs pass with `ginkgo -p --race` and with serial `go test -race`. + `go vet` passes; `golangci-lint` (no issue caps) reports only ST1000 on the controller sources (pre-existing; no + package comment). Mutation check, each mutation reverted afterwards: auto-rollout always enabled → A2 and A3 fail; + `Synchronized` reason ignored → A4(d) fails; condition generations ignored → A4(e) fails; deletion ignored → A5 + fails; no release on the skip path → every rollout check fails (A1–A4); no release on the deletion path → A5's + release check fails. Next: groups B, C, and D. +- October 22, 2026: **group C implemented** (C1–C8, both scopes) in `rollout_integration_test.go`; 138 of 138 specs + (A and C) pass with `ginkgo -p --race`. `go vet` passes; `golangci-lint` reports only the pre-existing ST1000. + Mutation check against the namespaced block, each mutation reverted afterwards: controller ownership ignored (MC1) + → C3 fails; bindings being deleted included (MC2) → C3 fails; never re-snapshot (`SnapshotResourcesIfNoSnapshotExists`, + MC3) → C6 fails; binding update errors ignored (MC4) → C8 fails; timestamp written before the rollout, i.e. the + old order (MC5) → C8 fails; namespace filter dropped (MC6) → initially undetected, now C4 fails after adding the + cross-namespace owner binding. Per the user, tests are now run only with `ginkgo -p` (see Running the tests). + Next: groups B and D. +- October 22, 2026 (later): **group B implemented** (B1–B4, both scopes) in `lifecycle_integration_test.go`, as the + `managing the binding manager role of ` `Describe` blocks; 162 of 162 specs (A, B, and C) pass with + `ginkgo -p --race`. `go vet` passes; `golangci-lint` reports only the pre-existing ST1000. Mutation check against + the namespaced block, each mutation reverted afterwards: claim always succeeds (MB1) → B1's no-rollout check fails; + role holders not enqueued on updates (MB2) → B2's release check fails; no requeue after a claim conflict (MB3) → + B1's rollout-after-release check fails; claim kept when only the wait period holds a rollout back (MB4) → B3's + release check fails, and so do B1's and B4's rollout checks (a completed rollout releases its claim on that path). + Next: group D. +- October 22, 2026 (later): **group D implemented** (D1–D4, both scopes) in `lifecycle_integration_test.go`, as the + `enforcing the minimum wait period between auto-rollouts of ` `Describe` blocks. **All planned scenarios + (A1–A5, B1–B4, C1–C8, D1–D4) are implemented**; 192 of 192 specs pass with `ginkgo -p --race` (~62s wall). + `go vet` passes; `golangci-lint` reports only the pre-existing ST1000 on the controller sources. Mutation check + against the namespaced block, each mutation reverted afterwards: min-wait annotation ignored (MD1) → D1 and D2 fail; + no lower clamp (MD2) → D3's first step fails; no upper clamp (MD3) → D3's second step fails; malformed period falls + back to the lower bound instead of the default (MD4) → D4's first step fails; malformed timestamp blocks instead of + counting as never rolled out (MD5) → D4's second step fails; no requeue while waiting (MD6) → D2 fails; min-wait + changes not treated as relevant by the event handler (MD7) → D1, D3, and D4 fail. Remaining follow-ups: the unit + tests listed under "Better as unit tests" (decision 4), and a package comment for the controller (ST1000). +- October 22, 2026 (later): review fixes in the controller: relinquish only when the cached policy shows the role + held; a user error during the rollout reports a Warning event, releases the role, and requeues after the minimum + wait period (`New` now takes a required `events.EventRecorder`); the remaining wait is capped at the period; an + empty snapshot list is an unexpected error. **C6 now covers the user-error path** (both scopes; decision 6). Its + first run found the hot loop in behavior 8 (the no-retry check failed in both scopes against the code without the + timestamp write, which doubles as the mutation check); with the fix, 198 of 198 specs pass with `ginkgo -p --race`. + `go vet` passes; `golangci-lint` reports no issues on the package. +- October 22, 2026 (later): the timestamp annotation is renamed to `kubefleet.dev/auto-rollout-last-rollout-attempt-timestamp` + (`AutoRolloutLastRolloutAttemptTimestampAnnotationKey`; `bookkeepLastRolloutAttemptTimestamp`), as user-error + failures write it too (behavior 8); the code, the test helpers and step texts, and this plan (except the entries + above) use the new name. diff --git a/.github/.copilot/breadcrumbs/2026-10-22-2138-placementpolicy-e2e-test-plan.md b/.github/.copilot/breadcrumbs/2026-10-22-2138-placementpolicy-e2e-test-plan.md new file mode 100644 index 000000000..bb8a72f97 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-22-2138-placementpolicy-e2e-test-plan.md @@ -0,0 +1,1102 @@ +# Placement Policy APIs (FEP-0001) E2E Test Plan + +**Date**: October 22, 2026 21:38 AEDT (revised October 26, 2026 22:20 AEDT, after the cross-ops and +incomplete-snapshot fixes were merged) +**Task**: Plan E2E coverage in [test/e2e](../../../test/e2e) for the `placement.kubefleet.dev/v1alpha1` and +`rollout.kubefleet.dev/v1alpha1` APIs (placement policies, bindings, snapshots, cluster claims, staged update runs, +approval requests, auto-rollout). Most scenarios are implemented and pass on Kind (see [Status](#status)). Q10 is a +known bug whose scenarios (L1, L2, L6) stay skipped. Q3 and Q11 are fixed, and X2, D2's snapshot part, D5, D6, and +L4 are implemented and pass on Kind. Q16 is open (X2b waits on it); Q17 is decided. + +## Requirements + +1. Cover the user journeys that the controller ITs can't: real member clusters and member agents, the hub GC, + status flowing back (member → Work → binding → policy), and handoffs between controllers (policy ↔ staged update + run ↔ auto-rollout, through the binding manager role). +2. Do **not** port the CRP/RP E2Es 1:1. The new experience differs in ways that change what is worth testing (see + [Experience differences](#experience-differences-vs-crprp)). +3. Follow repo conventions: Ginkgo/Gomega, `Ordered` contexts with step-wise `It`s, whole-struct `cmp.Diff`, + `want` naming, per-process names (`GinkgoParallelProcess()`), and a new `Context` per scenario. +4. Stay inside the CI budget. The new lane also runs through `make e2e-tests`, which has a 70m timeout. + +## Experience differences vs CRP/RP + +These differences drive the scenario selection below. + +| Area | CRP / RP | Placement policies | E2E implication | +| --- | --- | --- | --- | +| Initial placement | `RollingUpdate`: automatic; `External`: waits for an update run | The first snapshot is taken at creation, and new bindings carry the latest snapshot, so resources land with no rollout object | Assert placement without any run | +| Resource changes | Rolled out by the rollout strategy | **Nothing happens.** The policy reports `Synchronized=False/ResourcesChangedPendingRollout` until a staged update run or auto-rollout acts | Pending-rollout state is a first-class scenario | +| Scheduling changes | Bindings go through rollout or delete stages | Bind/unbind apply immediately (no delete stage). New bindings get the latest **existing** snapshot, which may be stale vs the hub | Scale up after a resource change gets the old content | +| Rollout vs scheduling | n/a | Mutually exclusive through `status.bindingManager`. Binding changes wait (5s→5m backoff) while a run or auto-rollout holds the role | Dedicated mutual-exclusion scenarios | +| Picking clusters | PickAll / PickN / PickFixed | A list of `ClusterSelector`s with `count` (int/`All`) and `minCount`. "Fixed" = match on the alias label | Overlapping selectors and count semantics | +| Stickiness | Scheduler semantics | A bound cluster keeps its binding when its labels/properties stop matching, or it is tainted/unhealthy. Only **selector term** changes re-evaluate. Exception, a **known bug (Q10)**: a cluster that starts leaving (deletion timestamp set) has its binding deleted at once. The expected behavior is to delete it only after the cluster has left. A fix will come separately | Sticky relabel and taint scenarios | +| Unfulfillable | Scheduling failure | A `ClusterClaim` is created (unless `KeepSearching`), and withdrawn once eligible clusters satisfy the selector. Nothing provisions clusters for claims in the E2E environment (decision 9) | Claim creation and withdrawal; fulfillment only by clusters the tests bring in | +| Namespace selection | Selecting a namespace takes its contents | A CPP selecting a `Namespace` places **only** the namespace object. Contents need selectors with `namespace` | Explicit scenario | +| Apply config | `ApplyStrategy` (incl. `ReportDiff`), overrides, envelopes | `SyncStrategy` only: no ReportDiff mode, overrides, or envelopes | Those areas are out of scope | +| Status | Per-cluster status on the placement | Per-binding status (counts, `failedResources`, diffs) plus aggregate counts on the policy | Assert both levels | +| Validation | Webhook + CEL | CEL + runtime (`ResourceCollected=False`). VAPs are being added separately and are out of scope here (Q8) | Runtime-error scenarios | +| Approvals | `kubectl fleet approve` | The plugin does not support the new `ApprovalRequest`, so tests patch the status `Approved=True` | Approve helper | + +## Existing FEP-0001 E2E coverage (do not duplicate) + +| File | Covers | +| --- | --- | +| [placement_policy_test.go](../../../test/e2e/placement_policy_test.go) | PP `region=east,count=2`: fulfilled, bindings name a snapshot, CM placed only on east. Unfulfillable PP: claim added, withdrawn on delete. CPP with fulfillable + unfulfillable selectors: claim refs CPP, cluster binding on west | +| [annotation_placement_test.go](../../../test/e2e/annotation_placement_test.go) | Alias label seeded. Generate / update / restore / recreate / invalid-annotation event / delete on annotation removal. API group carried. CPP generated for a Namespace | + +Not covered at all: staged update runs and approvals, auto-rollout, sync strategies, availability, resource +selection errors, snapshots, stickiness, taints, deletion policies, interop with CRP/RP, and join/leave. + +## Prerequisites and blockers + +- **P-1 (done, 00:19)** Register `rollout.kubefleet.dev/v1alpha1` in the E2E scheme + ([setup_test.go](../../../test/e2e/setup_test.go)). Only the placement group is registered today. +- **P-2 (done, 22:02)** Wired `pkg/v1/controllers/autorollout` into + [setup.go](../../../cmd/hubagent/workload/setup.go) behind `enablePlacementPolicyAPIs`. It sits next to the staged + update run controller, shares the placement resource snapshot manager, and uses the `auto-rollout-controller` event + recorder. Its concurrency is a fixed 5 (P-5). The chart RBAC now grants `patch` on (cluster)placementpolicies for + the timestamp merge patch. The existing `events.k8s.io` rule already covers its events. +- **P-3** Add shared v1alpha1 helpers (see [Helpers](#helpers)) in new files, separate from the legacy CRP helpers. +- **P-4 (done, 03:22; decided, Q5)** Give every new-API E2E its own label and lane. Tag all of them with + `Label("placementpolicy")`, including the two existing FEP files and the join/leave scenarios (section L). Add a CI + matrix entry `placementpolicy` running `make e2e-tests LABEL_FILTER="!custom && placementpolicy"` with the same + settings as the other lanes. Add `&& !placementpolicy` to the default lane's filter. The joinleave lane + (`!custom && joinleave`) needs no change, since new-API specs never carry the `joinleave` label. Implemented in + [ci.yml](../../../.github/workflows/ci.yml), [placement_policy_test.go](../../../test/e2e/placement_policy_test.go), + and [annotation_placement_test.go](../../../test/e2e/annotation_placement_test.go). +- Nit: [IMPLEMENTATION-NOTES.md](../../../pkg/v1/controllers/placementpolicy/IMPLEMENTATION-NOTES.md) still says + "Not wired into the hub agent yet". +- **P-5 (done, 23:45; Q12)** The v1 setup no longer reads v1beta1 configuration, and no new flags were added (v1 is + experimental). + - **Hub ([setup.go](../../../cmd/hubagent/workload/setup.go)):** fixed concurrency for the work generator (10), + staged update runs (5), and auto-rollout (5), set by the user at 23:54. These are lower than what + `--max-fleet-size` and `--max-concurrent-cluster-placement` produced at their defaults (100/10/10), so the v1 + controllers now run with less concurrency than before. + - **Member ([main.go](../../../cmd/memberagent/main.go)):** the v1 work applier uses a fixed concurrency of 5, a + fixed cleanup wait of 5m (no longer `--deletion-wait-time`), and its own built-in rate limiters. + - **Kept shared on purpose:** the placeability filters (skipped namespaces, GVK allow/deny lists, + `enableWorkload`) and manager infrastructure. + - **E2E knob:** the member agent reads `TEST_ONLY_PLACEMENT_POLICY_WORK_APPLIER_MAX_REQUEUE_DELAY_SECONDS` only + when it is set. It caps every requeue delay of the v1 applier through the new + `NewDefaultRequeueRateLimiterWithMaxDelay`, which reuses the default backoff shape. + [setup.sh](../../../test/e2e/setup.sh) sets it to 5 with `kubectl set env` after each member-agent install, so the + product chart gains no new value. + - **Docs:** both chart READMEs note that the v1beta1 tuning values don't apply to the placement policy + controllers. + +## Environment facts that shape the design + +- Members: `kind-cluster-1` (region=east, env=prod), `kind-cluster-2` (east, canary), `kind-cluster-3` (west, prod). + Each carries the alias label (= cluster name), so a selector on the alias label is deterministic. Node counts are + 2 / 3 / 4 ([kindconfigs](../../../test/e2e/kindconfigs)), which makes `kubernetes-fleet.io/node-count` usable for + property expressions. CI runs with the Azure property provider; gate on `isAzurePropertyProviderEnabled`. +- Scoring (fewer nodes; more available CPU/memory) depends on live capacity. **Never assert which cluster wins a + ranked pick.** Use alias/label selectors, or assert relative facts (e.g., "the originally picked one is kept"). +- Staged update run `refreshPeriod` is **30s** in the hub agent. Mid-run status can lag up to ~30s, so assert side + effects (binding `spec.resourceSnapshotName`, member contents, approval requests, `bindingManager`) and rest + points. Add `stagedUpdateRunEventuallyDuration` (~90s). `maxWaitTimePerClusterMinutes` is at least 1, so each + timeout scenario costs ≥60s. +- **Staged update runs share only 5 workers (P-5)**, and a reconcile holds its worker for up to the 30s refresh + period while clusters roll out. A cluster that never becomes available holds a worker for its whole + `maxWaitTimePerClusterMinutes`, in 30s slices. The lane runs ~15 ginkgo processes in parallel (16-CPU runner), so + several long-blocking runs at once would starve every other staged update run. Hence U7 runs its timeout runs one + at a time, holding at most one worker. Waiting on an approval or a timed wait holds no worker, and ConfigMap + rollouts finish in seconds, so the other U contexts stay parallel. +- Auto-rollout minimum wait is clamped to [1m, 1h] (default 3m). Any "second auto-rollout" costs ≥1m. +- The policy controller reconciles on **any** policy update (no predicate), so a released role is picked up + promptly. Its periodic requeue is 150s (fulfilled) or 2m (unfulfilled). **Auto-rollout does not react to a role + release**: it is not a relevant change to its event handler, so a blocked auto-rollout retries on its claim + backoff (5s doubling, up to 180s by default). +- The v1 work applier auto-creates namespaces and leaves them behind. Use `cleanupWorkResourcesOnHubAndMembers()`. +- **Cross-experience safeguard (both directions, since the 2026-10-26 merge).** Each work applier refuses an object + that an AppliedWork of the other experience owns: `OwnedByOtherKubeFleetAPIObjects`. The check runs before any + takeover or apply, whatever the strategy; the legacy applier refuses even in `ReportDiff` mode. Owners are matched by + API group and kind, in any version. A namespace that the v1 applier creates has no owner, so the check does not + cover it: a CRP that selects it takes it over (Q16). +- **The v1 work applier's requeue delays are capped at 5s in E2E (P-5).** The v1beta1 flags in setup.sh + (`workApplierRequeueRateLimiterMax*BackoffDelaySeconds=5`) don't reach it; the test-only env var does. Changes + made only on the member side (drift, a namespace created by hand) are therefore noticed within ~5s. Without the + cap (production), a settled Work requeues after 15s growing 1.5× up to 15m. Likewise, setup.sh's + `MaxConcurrentClusterPlacement=200` doesn't affect the v1 controllers, which use fixed concurrency. +- Suite setup creates two **virtual** `MemberCluster`s in every lane: `kind-unhealthy-cluster` (stale heartbeat) and + `kind-left-cluster` (stuck mid-delete). Both are ineligible, so no-selector, `count: All`, NotIn and DoesNotExist + policies never bind them, and `desired` for `count: All` counts only schedulable matches. Never assert on "all + `MemberCluster`s". +- **No claim provisioner (decided, 9).** Nothing in the E2E environment fulfills cluster claims, and nothing will + for now. Tests may assert that a claim is created, its contents and labels, `activeClusterClaims`, and its + withdrawal. They must never wait for a provider: no `Completed` condition and no `provisionedClusterName`. A + selector gets fulfilled only by a cluster the test itself brings in: an existing member relabeled to match (S13), or + a cluster joining the fleet (L4). Specs that leave an unfulfilled selector behind must delete the policy, which + withdraws its claims, so no claims leak between specs. +- **Snapshot ownership (#1012).** Snapshot names hash the policy UID, so a re-created policy never reuses its + predecessor's snapshots. While snapshots of an earlier policy of the same name await garbage collection, the policy + waits: it takes no snapshot, so it creates no binding. Orphaned (ownerless) snapshots, left by a deletion with + `propagationPolicy: Orphan`, stay until a policy of that name comes back. That policy then has them collected, even + if a binding still names them, and starts over at index 0 under a new name. +- **Bindings whose snapshots are gone (work generator, 2026-10-26).** + - **When snapshots are read:** only when a binding needs (re)processing, i.e., a rollout, an unsuspend, or missing + or mismatched Works. An in-sync binding whose snapshot is gone keeps its Works, its member resources, and its + status. + - **When they are missing or incomplete:** the binding reports `Synchronized=False/InvalidSnapshots`. It also + reports `Available=False/IncompleteSetOfResources` if it still targets the snapshot it last processed, or has + processed none. Its Works and member resources stay as they are. It retries on backoff (up to ~1000s), or at its + next spec change. + - **Cache lag:** right after a snapshot switch, a binding can show `InvalidSnapshots` for a moment. Start Consistently + windows only after the binding has settled. + - **Sync-strategy-only changes** (A1, A6, A8, B2) patch the Works without reading the snapshots. The binding's + conditions keep the previous generation until the member re-applies; `Eventually` checks absorb that. The merge + patch needs `patch` on `works` in the hub agent's RBAC, which the 2026-10-26 22:55 entry adds; without it, every + strategy-only change fails with `forbidden` and retries forever. +- Contexts that mutate `MemberCluster` labels or taints must be `Serial`, with `DeferCleanup` restoring them. + Everything else should be parallel-safe: per-process namespaces and policy names, and claim names hash the + policy's namespaced name. The existing FEP Describes are `Serial` without needing to be; consider relaxing that. + +## Test plan + +Priorities: **P0** = core journeys needed for alpha confidence; **P1** = important behaviors and likely regressions; +**P2** = edge cases and nice-to-haves. Scope: PP = `PlacementPolicy`, CPP = `ClusterPlacementPolicy`. Alternate +scopes across scenarios rather than doubling everything. Run both scopes only where code paths differ (staged +runs, claims, cross-namespace selection). + +### S. Scheduling: `placement_policy_scheduling_test.go` (S1–S5), plus `_selectors` (S6–S10) and `_cluster_changes` (S11–S13, all Serial) + +| # | Pri | Scope | Scenario | Key assertions | +| --- | --- | --- | --- | --- | +| S1 | P0 (implemented, passed on Kind) | PP | No `clusterSelectors` | All 3 clusters bound; `desired=scheduled=3`; CM on all | +| S2 | P0 (implemented, passed on Kind) | CPP | `count: All` + `env=prod` | Bindings on clusters 1 and 3 only; CM absent on 2 (Consistently) | +| S3 | P0 (implemented, passed on Kind) | PP | Overlapping selectors: A=`alias=cluster-1,count 1`, B=`env=prod,count 2` | Exactly 2 bindings (1, 3). Binding 1's `spec.clusterSelectors` lists A's and B's terms; binding 3 lists B's. `desired=scheduled=3` (per-selector sums), while `synchronized=available=2` (counted per bound cluster) | +| S4 | P0 (implemented, passed on Kind) | PP | Scale `region=east` count 1→2→1 | 2nd binding added; on scale-down the **newest** is trimmed and the originally picked cluster keeps its binding (same UID) and CM. The scale-up waits out the first binding's creation second (Q14) | +| S5 | P0 (implemented, passed on Kind) | PP, Serial | Sticky: `env=canary,count 1` binds 2, then relabel cluster-2 `env=prod` | Binding and CM kept (Consistently), still counted. Then change the selector **terms** → re-evaluated → unbound, CM removed, claim added | +| S6 | P1 (implemented, passed on Kind) | CPP | `region=east,count 3,minCount 2` | `scheduled=2`, `Scheduled=False/FailedToFindSomeRequiredClusters` (message shows floor met), claim added. **Pins the current behavior as intended (Q7):** reaching `minCount` but not `count` reports `Scheduled=False`. Status pinned in full: `desired=3`, `scheduled=synchronized=available=2`, `activeClusterClaims=1`; the message check looks for "have reached their minimum cluster counts" | +| S7 | P1 (implemented, passed on Kind) | PP | `whenUnfulfilled: KeepSearching` on an unfulfillable selector | No claim and no binding (Consistently), `activeClusterClaims=0`, `Scheduled=False`. Switching the selector to `AddClusterClaim` then adds the claim (`activeClusterClaims=1`), as the contrast | +| S8 | P1 (implemented, passed on Kind) | PP | `matchLabelExpressions` with In / NotIn / Exists / DoesNotExist, all with `count: All` (one Ordered context, swapping terms; each swap is a term change, so the set is re-evaluated) | Bound set follows each change: `env In [canary]` → {2}; `env NotIn [canary]` → {1, 3}; `env Exists` AND ` DoesNotExist` → all; `env DoesNotExist` OR ` Exists` → none (all unbound, CM removed, claim added). Every member cluster carries the same label keys, so Exists/DoesNotExist are each checked against a present and an absent key rather than a labeled subset (which would need a Serial relabel). Without `count: All`, the default count of 1 would bind a single score-ranked cluster | +| S9 | P1 (implemented, passed on Kind) | CPP | `matchClusterPropertyExpressions` `node-count Ge 3`, then `Eq 2`, both with `count: All` (gated on the property provider) | {2, 3}, then {1}. Node counts include the control plane; the legacy pick-all test expects the same {2, 3} for `Ge 3`. Skipped unless `PROPERTY_PROVIDER=azure` (the `setup.sh` default), like the legacy property specs | +| S10 | P1 (implemented, passed on Kind) | PP | Runtime-invalid selector (`Gt` inside `matchLabelExpressions`) | `Scheduled=False/InvalidClusterSelectors`; existing bindings untouched (same UIDs, old terms, CM kept; Consistently); fixing the spec recovers with the same bindings. Also pins the status while invalid: `desired`/`scheduled` are cleared, while `activeClusterClaims`, the resource revision, the synchronized/available counts, and their conditions stay as the last valid round reported them (`observedGeneration` included) | +| S11 | P1 (implemented, passed on Kind) | PP, Serial | Taints: taint cluster-2 NoSchedule; `region=east,count All` | Only 1 bound (cluster-2 neither bound nor given the CM; Consistently); adding a toleration binds 2. The taint is added just before the policy is created, as in the legacy taint specs | +| S12 | P1 (implemented, passed on Kind) | PP, Serial | Taint after bind: `env=prod,count 1` binds X, then taint X | X keeps binding (same UID) and CM, not counted; the other prod cluster Y is bound. While tainted, `desired=scheduled=1` but `synchronized=available=2`: X is still a target of the resource status. Untaint → overpick trimmed: Y unbound, X kept. The taint waits out X's binding creation second (Q14) | +| S13 | P1 (implemented, passed on Kind) | CPP, Serial | An **existing** cluster starts matching: `region=south,count 1` → claim (never fulfilled); relabel cluster-3 `region=south` | Binding on 3, CM placed, claim withdrawn, `activeClusterClaims=0`, `Scheduled=True`. The new-cluster variant is L4 | +| S14 | P2 (implemented, passed on Kind) | PP | Term change on an unfulfilled selector | Claim replaced under the **same name** (new UID) with the new terms | +| S15 | P2 (implemented, passed on Kind) | CPP | Delete a binding out of band | The work generator deletes that binding's Works, so the member's resources are removed (default `CleanUpResources`). The policy then recreates the binding, and the resources come back as new objects (new UIDs). Pin this churn | + +### R. Resource selection and snapshots: `placement_policy_resources_test.go` (R1–R4), plus `_resources_eligibility` (R5) and `_resources_snapshots` (R6, R7) + +| # | Pri | Scope | Scenario | Key assertions | +| --- | --- | --- | --- | --- | +| R1 | P0 (implemented, passed on Kind) | PP | Label-selected CMs; then edit one and add another matching CM on the hub | Initial set placed. After the edits: `Synchronized=False/ResourcesChangedPendingRollout`; `latestResourceRevisionName` unchanged; members keep old data (Consistently) | +| R2 | P0 (implemented, passed on Kind) | PP | Pending change + scale up (count 1→2) | The new cluster gets the **latest existing** (old) snapshot's content, not the hub's current content | +| R3 | P0 (implemented, passed on Kind) | CPP | Select a `Namespace` and a `ClusterRole` by name; then delete the CPP and **re-create** it with a CM selector (with `namespace`) added | Only the namespace (owned by an AppliedWork) and the ClusterRole are placed, not the namespace's CMs (Consistently). Deleting the CPP removes both from the members. After waiting for the old bindings and snapshots to be garbage-collected, the re-created CPP places all three. Adding the selector to the live CPP instead would only report `ResourcesChangedPendingRollout` (a selector change is a resource change), hence the re-create (decision 13). Since #1012, the wait before the re-create no longer guards the snapshots: a quick re-create waits for them on its own (D6). R3 keeps the wait anyway | +| R4 | P0 (implemented, passed on Kind) | PP | Selection errors: `namespace` ≠ the PP's; a named CM that is missing (two contexts) | `ResourceCollected=False/FailedToCollectSomeResources`, `Synchronized=False/FailedToSynchronizeSomeClusters`, `Available=False`, no latest revision, no bindings (Consistently). Fixing the namespace (to the PP's own) or creating the CM → `True` and placed. Once the separate VAPs land, the namespace-mismatch case may be rejected at admission instead; adjust it then | +| R5 | P1 (implemented, passed on Kind) | PP/CPP | Hub eligibility (decision 15), in four contexts: a PP in `kube-public` (a reserved namespace); a CPP naming `kube-root-ca.crt`, then the `default` Namespace; a PP in `default`; a label selector that also matches a controller-owned CM | The `kube-public` PP and both by-name cases → `ResourceCollected=False` with no binding (Consistently); since they share one status, each also checks the `ResourceCollected` message for its cause. The PP in `default` places its CM normally. The controller-owned CM (a `controller: true` owner reference) is silently skipped: binding `selectedResources=1`, and neither it nor its unlabeled owner reaches the members (Consistently) | +| R6 | P1 (implemented, passed on Kind) | CPP | 60 CMs (> 50 per snapshot) | Primary + secondary snapshots (sub-index/count labels): 50 + 10 resources, both index 0, both controlled by the CPP. Per cluster: primary + secondary Works (`linked-work-count=2` on the primary, which controls the secondary): 50 + 10 manifests. All 60 on the members; binding `selectedResources=60` | +| R7 | P1 (implemented, passed on Kind) | CPP | `resourceRevisionHistoryLimit: 1` + 3 staged update runs whose strategy leaves cluster-3 out (U9's setup) | Old snapshots GC'd, **except** the one cluster-3's binding still references: after each run, exactly {initial, latest} remain. Auto-rollout can't produce this state, since it moves every binding. A snapshot that the policy owns counts as in use while any binding of the policy names it in `spec.resourceSnapshotName`. Since #1012, an ownerless snapshot never counts as in use (D2) | +| R8 | P2 (implemented, passed on Kind) | PP | Delete a label-selected CM on the hub, then roll out | Removed from the members (leftover-manifest cleanup). Rolled out by auto-rollout; the other CM keeps its UID (Consistently) | + +### A. Sync strategy, status, and availability: `placement_policy_sync_strategy_test.go` (A1–A4), plus `_sync_strategy_failures` (A5, A7) and `_sync_strategy_takeover` (A6, A8) + +The goal is real-member behavior plus the status contract (binding `failedResources`/counts → policy conditions). +The [v1 work applier ITs](./2026-10-21-2045-workapplier-v1-integration-test-plan.md) already cover the full matrix, +so this section picks a representative subset. + +| # | Pri | Scenario | Key assertions | +| --- | --- | --- | --- | +| A1 | P0 (implemented as CPP, passed on Kind) | Pre-existing unmanaged CM on cluster-1, default `whenAlreadyExists=ReportError` | Binding `failedResources` has the CM with `Applied=False` (NotTakenOver); policy `Synchronized=False/FailedToSynchronizeSomeClusters`; `synchronizedClusters` excludes 1. Then set `AlwaysTakeOver` on the policy → binding `syncStrategy` updated **without a rollout** → taken over → `Synchronized=True` | +| A2 | P0 (implemented, passed on Kind) | `whenDrifted: ReportError`; edit the CM on the member | `FoundDrifts` with `diffDetails` on the binding. Revert → recovers. Contrast with the default `ApplyAnyway` (drift overwritten). Detection waits for the applier's periodic requeue, which the E2E cap keeps within ~5s (P-5) | +| A3 | P0 (implemented, passed on Kind) | `whenPlacementDeleted: OrphanResources`, then delete the policy | Resources stay on the members with no KubeFleet owner refs; hub bindings, Works, and snapshots are gone | +| A4 | P0 (implemented, passed on Kind) | Deployment placed (default availability) | Binding `Available=True/AllResourcesAvailable`, counts set; policy `resourcesAvailableClusters=N`, `Available=True` | +| A5 | P1 (implemented as cluster-1 alone, passed on Kind) | Deployment with an unpullable image | Binding `Synchronized=True`, `Available=False/SomeResourcesUnavailable`; 2 selected, 2 synchronized, 1 available; `failedResources` has the Deployment with `Available=False/Unavailable` (observed generation = the member Deployment's). Policy `Synchronized=True`, `Available=False/ResourcesUnavailableOnSomeClusters`, `resourcesAvailableClusters=0` | +| A6 | P1 (implemented as cluster-1 alone, passed on Kind) | `TakeOverIfNoDiff` with a differing pre-existing CM, partial vs full comparison | `FailedToTakeOver` + `diffDetails`; a field present only on the member matters only under full comparison. The member CM differs in `data.data` and has a `member-only` key: full comparison reports both diffs, partial only `/data/data`. Once the member's `data.data` matches the hub, partial comparison takes it over, and the `member-only` key is kept | +| A7 | P1 (implemented as cluster-1 alone, passed on Kind) | `whenNamespaceDoesNotExist: ReportError`; select only a CM whose namespace is not on the member | First wait until the member namespace from earlier specs is fully gone: per-process namespace names are reused, and cleanup is async. Then: failed apply reported (`FailedToApply`; neither the namespace nor the CM is created, Consistently); creating the ns on the member → applied (ns unmanaged). D1 pins the default `CreateNamespace` leftover | +| A8 | P1 (implemented as cluster-1 alone, passed on Kind) | `whenOwnedByOthers` with a member CM owned by another object (+ AlwaysTakeOver) | `ReportError` → takeover fails (`FailedToTakeOver`, no diff details; CM untouched, Consistently); `ShareOwnership` → co-owned (both owner references, hub data). Also pins that deleting the policy leaves the CM in place, owned by the other object alone: the GC drops the deleted AppliedWork's reference rather than deleting an object that still has an owner | +| A9 | P2 (implemented as PP on cluster-1 alone, passed on Kind) | `ServerSideApply` (+ `forceConflicts`) on a field owned by another manager | Applied with forced ownership; without force → apply failure (`FailedToApply`). The member CM is created with another field manager; the managed fields show who owns `data.data` before and after | + +### X. Interop and conflicts: `placement_policy_interop_test.go` + +| # | Pri | Scenario | Key assertions | +| --- | --- | --- | --- | +| X1 | P1 (implemented, passed on Kind) | A CRP (PickFixed cluster-1) places the app namespace; a PP selects the same CM to cluster-1, even with `AlwaysTakeOver` + `ShareOwnership` | PP binding reports `OwnedByOtherKubeFleetAPIObjects`; the CRP stays healthy and unchanged. The CM stays owned by the CRP's v1beta1 AppliedWork alone, and also after the PP is deleted (Consistently). Since the cross-ops fix, the v1 safeguard matches the older AppliedWork by API group and kind, in any version; X1's expectations don't change | +| X2 | P1 (implemented, passed on Kind; the optional `ReportDiff` step is not) | Reverse of X1. A PP (alias selector, cluster-1) places the CM first; the v1 applier creates the app namespace on cluster-1 without owning it. Then a CRP (PickFixed cluster-1) selects the app namespace, first with the default apply strategy, then with `allowCoOwnership: true`. Finally, delete the PP | **Both CRP strategies:** the legacy applier refuses the CM before any takeover or apply. The CRP's failed placement for the CM on cluster-1 is `Applied=False/OwnedByOtherKubeFleetAPIObjects`; mirror the CRP status for `NotTakenOver` in [placement_apply_strategy_test.go](../../../test/e2e/placement_apply_strategy_test.go). **Namespace:** the CRP takes it over, since it had no owner (default `whenToTakeOver: Always`). Pin this, as it is what Q16 is about. **Throughout (Consistently):** the PP stays synchronized; its CM keeps its UID and data and is owned by the v1 AppliedWork alone. **Deleting the PP lifts the block:** the v1 applier removes the CM, then the CRP creates it anew (new UID, owned by the CRP's AppliedWork) and reports everything applied. The PP goes before the CRP, so X2 never depends on Q16. **Optional (P2):** `type: ReportDiff` is refused too, since the check runs before diff reporting. The Work reports `DiffReported=False/FailedToReportDiff`, with a message naming `OwnedByOtherKubeFleetAPIObjects`. Confirm the CRP-level shape on Kind before pinning it | +| X3 | P1 (implemented, passed on Kind) | Two PPs (same ns) select the same CM to the same cluster; create the second only after the first is synchronized | The second PP is created with `AlwaysTakeOver` + `ShareOwnership` and reports `FailedToTakeOver` (another v1 AppliedWork owns the CM); switched to the default `ReportError`, it reports `NotTakenOver`. The first stays synchronized, the CM owned by its AppliedWork alone (Consistently). Deleting the first removes the CM, then the second creates it (new UID). The second is back on `ReportError` before the deletion on purpose: with a takeover option, it could take the CM over before the garbage collector removes it | + +### U. Staged update runs: `placement_policy_staged_update_run_test.go` (U1–U3), plus `_stages` (U4, U5, U9), `_interruptions` (U6, U8), and `_failures` (U7, U10) + +Each scenario runs in both scopes where noted (PP + `StagedUpdateRun` + `StagedUpdateStrategy` in the same namespace; +CPP + the `Cluster*` variants). Common strategy: stage `canary` (`env=canary`, after-stage Approval) → stage `prod` +(`env=prod`). The policy is placed first (initial content v0), then the hub CM is edited to v1. + +| # | Pri | Scope | Scenario | Key assertions | +| --- | --- | --- | --- | --- | +| U1 | P0 (implemented, passed on Kind) | both | Happy path, run with no `resourceSnapshotName` | `Initialized=True`; a new snapshot is created and the policy's `latestResourceRevisionName` moves; status `resourceSnapshotNameToRollout` set. Canary gets v1 while prod stays v0 → policy `SomeClustersOnOlderResourceRevision`. Approval request created (name from `status.stages[].afterStageTasks[].approvalRequestName`, owned by the run). Prod untouched while unapproved (Consistently). Approve → `ApprovalAccepted` → prod gets v1 → `Completed=True/Succeeded`; stage/cluster statuses (`cmp.Diff`, ignore times); policy `Synchronized=True`; `bindingManager` nil | +| U2 | P0 (implemented, passed on Kind) | PP | Rollback: a run with `resourceSnapshotName` = the index-0 snapshot (after a forward run took index 1; single-stage strategy with an empty selector) | Members return to v0; policy reports `SomeClustersOnOlderResourceRevision` (latest is still index 1); pin this, along with `synchronized=available=0` and `Available=False/SomeClustersUnavailable` (availability is judged on the latest revision only) | +| U3 | P0 (implemented, passed on Kind) | CPP | Mutual exclusion. Policy selectors `env=canary,count 1` + `env=prod,count 1` (binds 2 and one prod cluster X). While the canary approval is pending, raise the prod selector to `count 2` (no relabel, no Serial) | No binding for the other prod cluster Y while the run holds the role (Consistently). After approval and completion → Y bound with the **new** snapshot → Y gets v1. Also pin the policy status while it is deferred: `desired=scheduled=3` (Y already counts), `synchronized=available=1`, `SomeClustersOnOlderResourceRevision`, `bindingManager` = the run | +| U4 | P1 (implemented, passed on Kind) | PP | After-stage TimedWait (~30s) | Prod does not start before the wait elapses (stage timestamps); then completes | +| U5 | P1 (implemented, passed on Kind) | CPP | Before-stage Approval on `prod` | Canary done, prod blocked until approved | +| U6 | P1 (implemented, passed on Kind) | both | Created `suspended: true` | Initialized and snapshot chosen; `Suspended=True`; nothing rolled out (Consistently). Unsuspend → `Suspended=False/Resumed` → proceeds. Suspend while waiting on approval → approving does not advance until resumed | +| U7 | P1 (implemented, passed on Kind) | both | Failure policy, `maxWaitTimePerClusterMinutes: 1`. (a) `maxAllowedClusterFailures: 0`, v1 = Deployment with a bad image. (b) `maxAllowedClusterFailures: 1` with exactly one bad cluster. The new API has no overrides, so every cluster gets the same manifest; make one cluster fail with a member-side precondition, e.g., pre-create the namespace on that member with a ResourceQuota that blocks pods | (a) First cluster `FailedToUpdate` → `Completed=True/Failed`; later stages untouched; role released. (b) `PartiallySucceeded`. Run the four runs **one at a time** in an Ordered context (not `Serial`), so U7 holds at most one of the 5 staged update run workers and the rest of the lane keeps going (see the environment facts). Allow U7 more time: ~1–1.5 min per run, ~5–6 min in total, running alongside the lane | +| U8 | P1 (implemented, passed on Kind) | PP | Run deleted mid-way (waiting on approval) | Approval requests GC'd; role released; partial state kept. A new run completes the rollout | +| U9 | P1 (implemented, passed on Kind) | CPP | Strategy whose stages miss cluster-3 | Run `Succeeded`; cluster-3 stays on the old revision; policy `SomeClustersOnOlderResourceRevision`. Ties to R7 (referenced snapshot not GC'd) | +| U10 | P1 (implemented, passed on Kind) | PP | Init failures: missing strategy; missing policy; `resourceSnapshotName` from another policy; missing `sortingLabelKey` label | `Initialized=False/Failed` (terminal); nothing rolled out | +| U11 | P2 (implemented, passed on Kind) | CPP | `maxConcurrency` `3` and `70%` on a 3-cluster stage | Resolved `status.stages[].maxConcurrency` (3 and 2); all clusters end updated | +| U12 | P2 (implemented, passed on Kind) | PP | Hand-made approval request with the run's deterministic name, created before the run reaches the stage (name computed with the format in [uniquename.go](../../../pkg/v1/controllers/stagedupdaterun/uniquename.go)) | `ApprovalRequestCreated=False/RequestOwnershipMismatch`. Deleting it → the run creates its own request, which then needs approval as usual. The hand-made request is also approved beforehand: an approval on a request the run does not own does not let it through | +| U13 | P2 (implemented, passed on Kind) | PP | Rolling out an annotation-generated policy (name found via provenance labels) | Shows annotation placement and runs work together. Annotation `env=canary;env=prod,count=All`; the bindings keep the generated selectors | +| U14 | P2 (implemented, passed on Kind) | CPP | `binding.spec.suspended=true` set by hand mid-run (see section B) | That cluster counts as succeeded (the Completed message says the binding is suspended); its resources are removed (work generator). The run still moves the suspended binding to the new snapshot, so resuming it places the new version | + +### AR. Auto-rollout: `placement_policy_auto_rollout_test.go` + +| # | Pri | Scenario | Key assertions | +| --- | --- | --- | --- | +| AR1 | P0 (implemented as PP, passed on Kind) | `auto-rollout-enabled: "true"`, min wait `1m`; edit the selected CM | All bound clusters get v1 with no run; the `last-rollout-attempt-timestamp` annotation is written; `Synchronized=True`; `bindingManager` nil | +| AR2 | P0 (implemented, passed on Kind) | Annotation absent, then `"false"` | Pending rollout persists (Consistently ≥30s) | +| AR3 | P1 (implemented, passed on Kind) | Second edit right after AR1 | Not rolled out before timestamp + 2m; rolled out after (~2m cost). The specs' minimum wait period is 2m, not the 1m floor, because the period starts with AR1's attempt, before AR1's checks settle; the spec fails with a clear message if less than 15s of the hold is left to observe | +| AR4 | P1 (implemented, passed on Kind) | Coexistence, in this order: auto-rollout policy synchronized → create a staged run (it claims the role and waits on the canary approval) → edit the CM | While the run holds the role, auto-rollout does not act: prod keeps the old content (Consistently ~15s; keep this window short so the claim backoff stays small). Approve → the run completes and releases the role → auto-rollout rolls out the edit it was blocked on. Auto-rollout does not react to the role release; it retries on its claim backoff (5s doubling, up to 180s). The retry comes at most one backoff interval after the release, and the earlier steps cap how long the run holds the role (about 160s at most), so the interval never exceeds 160s; allow up to 4 min | +| AR5 | P1 (implemented, passed on Kind) | Annotation-generated policy: the user sets up rollout separately by adding the auto-rollout annotation to it (the annotation controller leaves extra annotations alone) | Edits to the annotated resource roll out. Do **not** pin what a generated policy does without the annotation; that default is undecided (Q4) | +| AR6 | P2 (implemented, passed on Kind) | Invalid / out-of-range `minimum-wait-period` | Pin clamping and defaulting only if cheap. Otherwise leave to the ITs. Made cheap by backdating the last-attempt annotation, as the ITs do: `5h` rolls out at once (clamped to 1h), `1s` waits until the 1m mark, `soon` until the 3m mark (about 2 minutes in all) | + +### L. Member cluster lifecycle: `placement_policy_join_leave_test.go` (`placementpolicy` lane) + +These run in the new lane rather than the joinleave lane (Q5). Every L context is `Serial`, so it runs only after the +lane's parallel specs finish. Each one rejoins every cluster it removed and restarts any agent it stopped in +`DeferCleanup`, so the fleet is whole for whatever runs next. + +| # | Pri | Scenario | Key assertions | +| --- | --- | --- | --- | +| L1 | **Skipped (Q10)** | `env=prod,count All` CPP; cluster-3 leaves | **Expected:** the binding stays while cluster-3 is leaving and is deleted only after it has left. Then Works are gone, counts drop, there is no claim, and the resources are **kept** on cluster-3 (orphaned, no owner refs). Today the binding is deleted as soon as cluster-3 starts leaving, a known bug to be fixed separately. Add L1 with the fix | +| L2 | **Skipped (Q10)** | Rejoin cluster-3 (default `ReportError`), then take the resources over again with `TakeOverIfNoDiff` (unchanged resources) and `AlwaysTakeOver` | **Expected (Q2):** resources left behind by a cluster that left are unowned on rejoin and must be taken over again. Default → `NotTakenOver` on the binding's `failedResources`; policy `Synchronized=False`. Switching the policy's `whenAlreadyExists` (no rollout needed) → re-adopted, owner refs restored, `Synchronized=True`. Needs L1's leftover resources, so it lands with the fix. Until then, B2 covers the same re-takeover mechanics through suspension | +| L3 | P2 (implemented as CPP, Serial, passed on Kind) | Member agent stopped on a bound cluster (heartbeat stale, ~3m) | Binding kept, not counted, gap filled. Agent back → overpick trimmed (newest goes). Expensive: the agent is scaled to 0, and the gap is filled once the 5m heartbeat timeout passes (about 5 minutes on Kind; up to 7.5 with the 150s requeue backstop), about 6 minutes of Serial time in all | +| L4 | P0 (implemented as CPP, Serial, passed on Kind) | A **new** cluster fulfills a waiting selector. With cluster-3 out of the fleet, create a CPP selecting `fleet-e2e/new-cluster=true,count 1` → claim created. Then rejoin cluster-3 via `createMemberCluster` with its usual labels plus `fleet-e2e/new-cluster=true` | **Precondition:** before cluster-3 leaves, assert that no (cluster) placement binding targets it, so the Q10 bug can't affect this test. Before the join: `Scheduled=False`, one claim carrying the selector terms, and the claim stays unfulfilled (Consistently: no `Completed`, no `provisionedClusterName`). After cluster-3 joins and is healthy: binding on 3, CM placed, claim withdrawn, `activeClusterClaims=0`, `Scheduled=True`. Remove the extra label in `DeferCleanup` | +| L5 | P1 (implemented as CPP, Serial, passed on Kind) | A matching cluster that registers but never joins: L4's setup, but the new cluster is a `MemberCluster` with no agent (per-process name, matching label) | No binding; the claim stays active (Consistently), since withdrawal requires an eligible cluster. Before the Consistently check, the claim's `lastObservedMostRecentClusterCreationTimestamp` reaches the new cluster's creation time, which shows that the policy reconciled with the cluster in place. Follow the existing virtual-cluster pattern ([scheduler_watchers_test.go](../../../test/e2e/scheduler_watchers_test.go)): create with `createMemberCluster` and clean up with `ensureMemberClusterAndRelatedResourcesDeletion`, which strips finalizers, so there is no force-delete wait | +| L6 | **Deferred to the Q10 fix** | Hub-side timing of a leave, on a **virtual** cluster: create it, mark it healthy, let a `count: All` policy bind it, then delete its `MemberCluster` | With no agent to report the leave, the hub member-cluster controller keeps the `MemberCluster` in `Terminating` until force delete (`forceDeleteWaitTime` is 1m in the E2E hub). **Expected:** the binding is kept throughout that window (Consistently) and deleted once the cluster is gone. This fails today. It checks binding timing deterministically and needs no leave/rejoin of a real cluster. Only L1 can show that the resources are kept, since a virtual cluster has no member side | + +L4 stands alone in Phase 1, with its own leave and rejoin; its precondition keeps it clear of the Q10 bug. Once the +Q10 fix lands, L1, L2 and L4 can share one Ordered `Serial` context (leave → L1 → create L4's policy → rejoin with +the extra label → L4 → L2), so cluster-3 leaves and rejoins only once. L4's policy must then select a different +ConfigMap than L1's; otherwise the two policies conflict over the same object on cluster-3 (as in X3). + +### B. Binding suspension: `placement_policy_binding_suspension_test.go` + +Users don't normally edit `binding.spec.suspended`; upcoming APIs will build on it (Q6). These tests edit it directly +as a stand-in for those APIs, to cover the mechanism end to end. On suspension the work generator deletes the +binding's Works and clears `status.lastProcessedResourceSnapshotName`. The member applier then removes the resources +according to `whenPlacementDeleted`. The placement policy controller never touches the field. + +| # | Pri | Scope | Scenario | Key assertions | +| --- | --- | --- | --- | --- | +| B1 | P1 (implemented, passed on Kind) | PP | Suspend one binding of a 3-cluster policy (no rollout in flight), then unsuspend it | Suspended: that cluster's Works are gone, its resources are removed, and `lastProcessedResourceSnapshotName` is cleared. The other clusters are untouched, and the binding stays suspended with no replacement binding created (Consistently). Policy status as reported: `desired=scheduled=3`, `synchronized=available=2`, `Synchronized=False/FailedToSynchronizeSomeClusters`, `Available=False` (the suspended binding counts as neither synchronized nor outdated). Unsuspended: Works recreated, resources back, `Synchronized=True`, same binding UIDs | +| B2 | P1 (implemented, passed on Kind) | CPP | B1 with `whenPlacementDeleted: OrphanResources` | Suspending leaves the resources on the member, orphaned with no KubeFleet owner refs. After unsuspending, the binding reports `NotTakenOver` until the policy switches `whenAlreadyExists` to a takeover option (`TakeOverIfNoDiff`, as the CM is unchanged; cf. L2), which takes it over without a rollout | +| B3 | P2 (implemented, passed on Kind) | PP | Suspended binding + a resource change rolled out (auto-rollout) | The suspended binding's `resourceSnapshotName` moves but nothing is applied to that cluster. On unsuspension it gets the latest content | + +### D. Deletion and ownership: `placement_policy_lifecycle_test.go` + +| # | Pri | Scenario | Key assertions | +| --- | --- | --- | --- | +| D1 | P0 (implemented, passed on Kind) | Delete a policy (default `CleanUpResources`) | Explicitly: bindings, Works (`fleet-member-*` ns), and snapshots gone on the hub; AppliedWorks and resources gone on the members; auto-created member ns remains (pinned) | +| D2 | P1 (implemented, passed on Kind, snapshot part included) | Delete with `propagationPolicy: Orphan`, then recreate the same name | **While orphaned:** bindings orphaned, Works kept, and the member CM not churned (implemented). Add: the primary snapshot is orphaned too (no owner reference) and is **not** collected (Consistently), since nothing looks at it until a policy of that name returns. **After the re-create (implement):** (1) the old snapshot is collected even though the orphaned bindings still name it, since an ownerless snapshot is never in use. (2) The policy then takes a new index-0 snapshot under a **different name** (names hash the policy UID), controlled by the new policy. (3) Only then are the bindings adopted, since a round without a snapshot creates no binding. Adopted bindings keep their UIDs and get the new controller reference. Their `spec.resourceSnapshotName` and `status.lastProcessedResourceSnapshotName` name the new snapshot, with `Synchronized=True` and `Available=True` for the new generation. (4) The policy is fully synchronized on the new revision. **Kept from today (re-verify):** Works keep their UIDs, and the member CM keeps its resourceVersion (Consistently). Work names come from the policy's namespaced name, and re-linking them to a snapshot with the same content changes only metadata, so their generation does not change. **Then (new):** delete the re-created policy (default cascade). Its bindings, Works, and new snapshot go from the hub; the CM and AppliedWorks go from the members; no old snapshot is left. **Notes:** start the post-adoption Consistently windows only once the bindings report the new snapshot synchronized (see the cache-lag note in the environment facts). Update the context's stale comments: the re-created policy no longer reuses the old snapshot | +| D3 | P1 (implemented, passed on Kind) | Delete the PP's hub namespace | Policy, bindings, snapshots, Works, and member resources all cleaned up; no leaked claims. The PP also has an unfulfillable selector, so it has a claim to withdraw | +| ~~D4~~ | Dropped | ~~CEL smoke~~ | The chart's CRDs are symlinks to `config/crd/bases`, so an E2E smoke would verify nothing an envtest API-validation IT doesn't. The CEL rules belong in that IT (none exists for `kubefleet.dev` today; see "Left to ITs/UTs") | +| D5 | P2 (implemented, passed on Kind) | D2 with a narrower re-create: the PP first selects every cluster, then is re-created with `region=east,count All`, so cluster-3's orphan is not wanted | **Clusters 1 and 2:** their bindings are adopted as in D2. **Cluster-3's orphan:** neither adopted nor deleted, since the policy only controls what it adopts. It doesn't count in the policy status (`desired=scheduled=synchronized=available=2`). Its old snapshot is collected anyway (D2), yet nothing changes for it (Consistently): still `Synchronized=True`, same Works, the CM on cluster-3 untouched. An in-sync binding never re-reads its snapshots, so the work generator doesn't notice. **Deleting the orphan by hand:** removes its Works and cluster-3's CM (its own `whenPlacementDeleted`, default `CleanUpResources`) without needing the snapshot. Pins this as intended (Q17, decided) | +| D6 | P2 (implemented, passed on Kind) | CPP quick re-create: delete the CPP (default cascade), then re-create it under the same name right away, without waiting for its dependents to be collected | Possible only since #1012; before it, the re-created policy could pick up the old snapshot (decision 13). **The waiting window:** while the old bindings and snapshots are being collected, the new policy waits (it takes no snapshot and binds no cluster). The window may be too short to observe, so don't assert it. **Convergence:** a new index-0 snapshot under a different name, controlled by the new UID; new bindings (new UIDs) controlled by the new policy; no old binding, snapshot, or Work left; the CM on every cluster; the policy fully synchronized. Don't pin member UIDs: the old Works' teardown and the new Works share names, so the CM may be deleted and re-created. A hang or a leftover here is a finding, not a flake | + +## Helpers + +New files `placement_policy_utils_test.go` and `placement_policy_actuals_test.go`. Make them scope-agnostic through the +`*Accessor` interfaces in [interface.go](../../../apis/kubefleet.dev/placement/v1alpha1/interface.go): + +- Builders: `newPolicy(scope, name, ns, selectors, resourceSelectors, syncStrategy)`, `aliasSelector(cluster)`, + `labelSelector(k, v, count)`, `newStrategy` / `newRun` (both scopes). +- Actuals: + - `policyStatusActual(key, wantConds, wantCounts)`: `cmp.Diff`, ignoring `LastTransitionTime`/`Message`. + - `bindingsActual(key, wantClusterToSnapshot)`. + - `bindingFailedResourcesActual`. + - `snapshotsOfPolicy` / `latestPrimarySnapshot`. + - `worksOfBinding`. + - `stagedUpdateRunStatusActual`: ignoring timestamps and messages. + - `appConfigMapDataOnClustersActual(wantData, clusters...)`. + - `orphanedOnClusterActual`: object present, no KubeFleet owner refs. +- Mutators: `updateAppConfigMapData`, `approveApprovalRequest(key)` (status update `Approved=True`), + `setBindingSuspended(key, bool)`, and `setMemberClusterLabels` / `addTaint` with `DeferCleanup` restore. All + mutators retry on conflict. +- Claims: `claimsOfPolicyActual(key, wantTerms)` (asserts the claim stays unfulfilled) and + `noClaimsOfPolicyActual(key)`. +- Fleet changes: reuse `setMemberClusterToLeave` / `checkIfMemberClusterHasLeft` and `createMemberCluster` / + `checkIfMemberClusterHasJoined` from [utils_test.go](../../../test/e2e/utils_test.go). Add + `rejoinMemberClusterWithExtraLabels(cluster, labels)` for L4. +- Reuse `createWorkResources`, `cleanupWorkResourcesOnHubAndMembers`, `claimsOfPolicy`, and + `setClusterSelectorsAnnotation`. +- X2 (2026-10-26): + - A CRP status builder for one failed placement: `Applied=False` with + `workapplier.ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects`. The legacy package now exports the reason, + so X1's local constant can switch to it. + - `configMapPlacedByPlacementPolicyOnlyActual`, the inverse of X1's `configMapPlacedByOlderPlacementAPIsOnlyActual`. + - A check that the CRP's AppliedWork owns the member namespace. +- D2, D5, and D6: + - Reuse `placementPolicySnapshots`, which finds snapshots by the owner annotation, so ownerless ones too. + - Reuse `latestPrimarySnapshotOfPlacementPolicy`, which checks the controller reference. + - Add an actual for "this snapshot name is gone". + - Add one for a binding's controller, spec snapshot, and last processed snapshot. + +## Left to ITs/UTs (not E2E) + +The full sync-strategy matrix, waves and backoff, scoring math, snapshot naming and hashing, claim budget with +provisioner finalizers, CEL rule matrix, run status internals, and auto-rollout requeue arithmetic. Claim +**fulfillment** by a provisioner (the `Completed` condition, `provisionedClusterName`, and replacing a completed +claim) is out of E2E scope until a provider exists (decision 9). + +The 2026-10-26 merges add more to this list: +- **Work generator, invalid snapshots:** the full matrix (not found, duplicate or missing sub-indices, fewer snapshots + than the count), the `InvalidSnapshots` and `IncompleteSetOfResources` conditions, the recounts, and the no-op write + skip. +- **Work generator, strategy-only path:** its internals. +- **Bindings:** the CRD's `minLength: 1` on the binding name fields. +- **Appliers:** owner matching by API group across versions, in both directions. +- **Snapshot manager:** the leftover matrix (a stale UID, an orphan, unexpected owners, pending deletion). + +Why invalid snapshots get no E2E: a binding that a policy controls reaches `InvalidSnapshots` only when a snapshot is +deleted out of band, and the policy status reports such a binding like any other unsynchronized one. + +## Phasing and rough cost + +| Phase | Content | Est. added wall clock | +| --- | --- | --- | +| 0 | P-1, P-3, P-4 (scheme, helpers, label/lane) | none | +| 1 | P0: S1–S5, R1–R4, A1–A4, U1–U3, AR1–AR2, D1, and L4 (Serial; it brings in the leave/rejoin helpers early) | ~18–23 min + ~3 min serial | +| 2 | P1 across sections (U7 runs its four timeout runs one at a time, alongside the lane). L1/L2 are skipped (Q10) | ~20 min | +| 3 | P2 + remaining L (Serial, same lane) | ~10 min + ~6 min serial | +| 4 | After the 2026-10-26 merges: X2 and D2's snapshot part (P1), D5 and D6 (P2), all parallel | ~3–4 min | + +The estimates are for the `placementpolicy` lane only, since it no longer shares the default lane's 70m budget. Serial +contexts (S5, S11–S13, all of L) run one after another at the end of the lane, so keep them small. Measure after +Phase 1 and revisit the estimates. Summed, the phases (~57–62 min) plus the two existing FEP files come close to the +70m ginkgo timeout that `make e2e-tests` hard-codes. Before Phase 3, either make the timeout configurable for this +lane or keep the P2s out. + +## Open questions + +1. ~~**Auto-rollout wiring (P-2):** wire it now, or park AR?~~ **Resolved:** wire it now (done; see P-2). The AR + P0s move into Phase 1. +2. ~~**Rejoin:** is `NotTakenOver` on rejoin (default `ReportError`) intended, or a bug?~~ **Resolved:** expected. + Resources previously applied by a cluster that left and rejoined may need to be taken over again; L2 pins this + once the Q10 fix lands (B2 covers the re-takeover mechanics until then). +3. ~~**X2:** should the legacy applier refuse objects owned by the new AppliedWork even with `allowCoOwnership`?~~ + **Resolved:** yes, it should be blocked. That is left alone for now; the fix and X2 come in a separate PR. + **Fixed (2026-10-26, `feat/block-cross-ops-between-v0-v1`):** the legacy applier now refuses objects that a + `placement.kubefleet.dev` AppliedWork owns (`OwnedByOtherKubeFleetAPIObjects`). It refuses before any takeover or + apply, whatever the apply strategy, `ReportDiff` included. Both appliers now match the other side's owner by API + group and kind, in any version. X2 is back in this plan. +4. ~~**Annotation placement:** should generated policies enable auto-rollout by default?~~ **Deferred:** leave it + alone for now and decide later. Tests assume rollout is set up separately (AR5: the auto-rollout annotation on + the generated policy; U13: a staged update run) and do not pin the default for generated policies. +5. ~~**CI:** a dedicated `placementpolicy` lane, or keep the default lane?~~ **Resolved:** a dedicated lane and label + for every new-API E2E, including the existing FEP files and section L (see P-4). +6. ~~**Binding suspension:** is `binding.spec.suspended` user-facing (U14), or internal only?~~ **Resolved:** users + don't typically edit it directly (upcoming APIs build on it), but the E2Es may edit it. Section B was added and U14 + is kept. +7. ~~**`minCount`:** reaching the floor reports `Scheduled=False` (notes item 4). Pin it in S6, or wait for the FEP + decision?~~ **Resolved:** the current behavior (`Scheduled=False`) is the way to go for now; S6 pins it. +8. ~~**VAPs:** the API comment says some validations moved to VAPs, but none ship. Planned?~~ **Resolved:** VAPs are + added separately; these E2Es don't touch them. +9. **Cluster claims (raised in review):** **Decided:** treat claims as never fulfilled. No provider resolves them now. + Tests may cover claim creation but must not rely on fulfillment. They must still cover new clusters fulfilling a + specific selector: L4 (a cluster joins), S13 (an existing cluster starts matching), and L5 (a cluster that never + joins does not count). +10. **Leave race (from the 22:47 review):** **Resolved, known bug.** The expected behavior is that a leaving + cluster's binding is deleted only **after the cluster has left**. Today the placement policy controller deletes it + as soon as the `MemberCluster` gets a deletion timestamp, before the agent has left. This is unexpected and will + be fixed separately. Until then, L1 and L2 are skipped, and L6 (a deterministic hub-side check on a virtual + cluster) is deferred to the fix PR. L4 asserts a precondition that keeps it clear of the bug. Details, for the fix: + - **Why:** the eligibility checker reports a deleting cluster as ineligible, and `evaluateSelectors` keeps it in + neither the schedulable nor the sticky set ("A cluster on its way out is left out, so that its binding is + released", [scheduling.go](../../../pkg/v1/controllers/placementpolicy/scheduling.go)). The member-cluster + watch fires on the deletion timestamp. + - **Probe (23:09):** an envtest probe confirmed this. A bound cluster held in `Terminating` by a finalizer lost + its binding within seconds. The existing IT ("unbinds a cluster that leaves the fleet") deletes a + finalizer-less `MemberCluster`, so it never covered the leaving window. + - **Race:** the hub member-cluster controller keeps the cluster's namespace and Works until the agent reports it + has left. The binding deletion bypasses that: the work generator deletes the Works right away. If the member + applier handles that deletion before the member agent calls `Leave()`, it deletes the resources (default + `CleanUpResources`). `Leave()` sets the applier not ready and orphans AppliedWorks of Works that still carry its + finalizer. Which happens first is a race between two chains of watches. + - **Legacy contrast:** the legacy scheduler doesn't delete such a binding; it marks it `Unscheduled`. Deletion + then goes through the rollout controller (gated by the unavailability budget, so legacy can race too), or + under the External strategy through the update run's delete stage. That is what makes the staged-run E2E's + "resources are kept" deterministic. +11. ~~**Orphan-cascade recreate (from the 22:47 review; affects D2):** **Open, fix in progress (by the user).** Not + intended. Leave this open until the user confirms the fix has landed. Deleting a policy with + `propagationPolicy: Orphan` also orphans its snapshots. The recreated policy finds them by label and reuses them + without an owner ref, since `retrieveLatestSnapshot` and `isSnapshotUpToDate` don't check ownership. Paths that + do check, such as `IsSnapshotSynchronized` and the GC's in-use check, expect one. Until the fix lands, D2's + snapshot assertions stay pending; its binding-adoption part is unaffected.~~ **Fixed (2026-10-26)** by #1012 + (snapshot manager), plus `feat/handle-incomplete-snapshots-in-work-generator` (work generator). + - **Names:** snapshot names hash the policy UID. + - **Owner checks:** `retrieveLatestSnapshot` checks the owner of each primary. + - Orphaned (ownerless): queued for GC, and the call fails as Transient until the snapshot is collected. + - Owned by an earlier policy of the same name: Transient until the garbage collector removes it. + - Any other owner: a User error. + - **GC:** collects orphaned groups even while a binding still names them. + - **Result:** the re-created policy starts over at index 0 under a new name, and adoption moves the bindings to that + snapshot. + - **Work generator:** a binding whose snapshots are missing or incomplete now reports `InvalidSnapshots`. Before, + the work generator only retried, with nothing on the status. + - **Plan impact:** D2's snapshot part is unblocked and rewritten, and D5 and D6 are added. +12. ~~**v1 applier backoff in E2E (from the 22:47 review):** wire the `workApplierRequeueRateLimiter*` flags into the + v1 work applier, so E2E can cap its backoff?~~ **Resolved (done, see P-5):** don't share configuration. The v1 + controllers in both agents get fixed static values and no new flags, since v1 is experimental. E2E caps the v1 + applier's requeue delays through a test-only env var that `setup.sh` sets. +13. ~~**R3 contents (raised while implementing, 01:09):** adding the CM selector to the live CPP is a resource change, + so the policy only reports `ResourcesChangedPendingRollout` and places nothing until a rollout.~~ **Decided:** + delete and re-create the CPP with the selector added. Before re-creating, R3 waits for the deleted CPP's bindings + and snapshots to be garbage-collected. Binding names are deterministic, and `retrieveLatestSnapshot` doesn't check + ownership (the Q11 root cause), so a quick re-create could pick up the old snapshot. **Update (2026-10-26):** since + #1012, the snapshots no longer need the wait: a quick re-create waits for them on its own and never reuses them + (D6 covers that path). R3 keeps its wait anyway, to stay clear of the teardown of the old bindings and Works, which + share names with the new ones. +14. ~~**Scale-down tie-break (found on Kind, 01:50):** `chooseClusters` trims the newest bindings by + `creationTimestamp`, which has one-second resolution, and breaks ties by cluster name; accept that, or record a + finer bind time?~~ **Resolved (2026-10-24): accepted as a limitation.** Two bindings created within the same + second (seen on Kind: 14:40:00.53 and 14:40:00.90) tie, so a trim can drop the earlier pick when its cluster name + sorts later. The code comment ("oldest and name breaking ties") documents the tie-break, but "a scale-down undoes + the latest picks" holds only at one-second granularity. This affects any trim that follows a quick replacement, not + just a scale-down: S12 hit it too, after a taint was lifted (bind, taint, and replacement all landed within one + second on Kind, so the replacement was kept and the original dropped). No finer bind time will be recorded. The + E2Es keep the workaround instead: any spec that relies on which of two bindings is newer calls the shared + `waitPastCreationSecond` helper before causing the second binding (S4 before scaling up, S12 before tainting). +15. ~~**`default` namespace (raised while implementing R5, 2026-10-24):** the plan had a PP in `default` failing + with `ResourceCollected=False`, but the v1 eligibility checker denies only the `kube-` and `fleet-` prefixes, and + its comment (`resourceeligibility.Baseline`) says resources in `default` may be placed; only the `default` + Namespace object itself is refused. (The hub agent's `--skipped-propagating-namespaces` help text says `default` + is blocked, which describes the v0 path.) Intended, or a bug?~~ **Decided: intended.** R5 pins it: a PP in + `default` places its CM; naming the `default` Namespace fails; `kube-public` stands in for the reserved-namespace + case. +16. **A CRP taking over a namespace that the v1 applier created (raised while re-planning X2, 2026-10-26):** **Open.** + - **What happens:** + - The v1 applier creates a missing namespace without owning it (D1). + - A CRP that selects that namespace takes it over, under the legacy default `whenToTakeOver: Always`. The + cross-ops check doesn't stop it, since it looks only at the namespace's own owners, and there are none. + - Deleting the CRP deletes its AppliedWork with foreground propagation, so the namespace goes too. The PP's + ConfigMap inside it goes with it. + - The v1 applier re-creates both (new UIDs) once the namespace is gone. + - **Workarounds:** a user can avoid this with `whenToTakeOver: Never`, but the CRP then reports the namespace as not + taken over. The CRP's `deleteStrategy.propagationPolicy: Abandon` is no workaround: the API declares it, but no + controller implements it. + - **Question:** is this intended (legacy takeover semantics), or should one side guard against it? + - **Effect on the plan:** X2 deletes the PP before the CRP, so it doesn't depend on the answer. If this is intended, + add a short X2b that pins the cascade; if not, X2b comes with the fix. +17. ~~**The orphan left behind by a narrower re-create (D5, 2026-10-26):** **Open; the default is to pin it as + intended.**~~ **Decided (2026-10-26): pin as intended.** A re-created policy adopts only the orphans for clusters it + wants. An orphan for any other cluster is left alone for good. It keeps its Works and the member resources. Its + snapshot is collected (#1012), which has no effect while the binding stays in sync. This mirrors Kubernetes, where a + ReplicaSet adopts only the orphans that match it. D5 pins it. + +## Status + +- 21:38 — Plan drafted from the API types, the existing FEP E2Es, the controller code and notes, and the IT plans for + the staged update run, auto-rollout, v1 work applier, and snapshot manager. Waiting for review and answers to the + open questions before implementation. +- 22:02 — Q1 answered: wire auto-rollout now. P-2 is implemented in + [setup.go](../../../cmd/hubagent/workload/setup.go) and the hub-agent chart + [rbac.yaml](../../../charts/hub-agent/templates/rbac.yaml). Validation: build, vet, `make lint` (0 issues), and + `helm template` (`patch` rendered with the flag on). The controller name `auto-rollout` is unique in the manager. + The auto-rollout IT suite builds the controller the same way (`New` + `SetupWithManager` + + `mgr.GetEventRecorder`) and passes (325s). Not verified live on Kind, since Docker is unavailable in this + environment; AR1–AR2 will cover it. Questions 2–8 are still open. +- 22:08 — Q2 answered: `NotTakenOver` on rejoin is expected, since previously applied resources may need to be taken + over again. L2 now pins it and adds the re-takeover path. Questions 3–8 are still open. +- 22:09 — Q3 answered: the legacy applier should block objects owned by the new APIs even with `allowCoOwnership`. + That will be addressed in a separate PR, so X2 is deferred to it and X1/X3 stay as planned. Questions 4–8 are + still open. +- 22:10 — Q4 deferred: no change to annotation placement for now. Tests assume rollout is set up separately, and + AR5 no longer pins the default for generated policies. Questions 5–8 are still open. +- 22:12 — Q5 answered: every new-API E2E gets the `placementpolicy` label and its own CI lane. P-4 is updated. + Section L moves from the joinleave lane into the new lane as `Serial` contexts that restore the fleet. Questions + 6–8 are still open. +- 22:13 — Q6 answered: `suspended` is not normally edited by users (upcoming APIs build on it), but the E2Es may edit + it. Added section B (B1–B3) after confirming the work generator and placement policy controller behavior. U14 no + longer depends on Q6. Questions 7–8 are still open. +- 22:17 — Q7 answered: keep the current `minCount` behavior (`Scheduled=False` when only the floor is met); S6 pins + it. Question 8 is still open. +- 22:17 — Q8 answered: VAPs are added separately and stay out of these E2Es. R4 notes that its namespace-mismatch case + may move to admission once they land. **All open questions are now resolved or deferred; the plan is ready for + implementation approval.** +- 22:19 — Decision 9: no claim provisioner. Claims may be created and withdrawn in tests but are never fulfilled. + Hub agent check: there is no switch that turns claims on or off, so this is purely a test assumption. Added an + environment fact and L4 (P0: a cluster joins and fulfills a waiting selector; reuses the existing leave/join + helpers). Also added L5 (P1: a registered but never-joined cluster keeps the claim) and reworded S13 as the + existing-cluster variant. Claim fulfillment moves to out of scope, and L4 moves into Phase 1. +- 22:47 — Review pass for logic issues and inconsistencies, with each claim checked against the code. Fixed: + - S8/S9: added `count: All` (the default count of 1 made the planned sets impossible). + - S15: now expects resource churn; deleting a binding deletes its Works. + - R7: switched to CPP to match U9, and now builds on U9's staged runs (auto-rollout can't leave a snapshot + referenced). + - A2: timing note. + - A7: added a precondition; its duplicate pin now lives in D1. + - X3: correct failure reasons, plus a forced creation order. + - U3: defined the policy whose count it raises. + - U7: added a way to make exactly one cluster fail. + - U12: precise expectations. + - AR4: an explicit order, and timing that reflects auto-rollout's claim backoff. + - B2: the takeover step is now explicit. + - L4: decoupled from L1/L2 (L4 is in Phase 1; they aren't), and the shared context now uses distinct ConfigMaps. + - D4: dropped, since the chart CRDs are symlinks to the generated ones. + + Added environment facts on the v1 applier backoff, auto-rollout ignoring role releases, and the virtual fixture + clusters. Added a stickiness exception for leaving clusters and a 70m-timeout risk note. Opened Q10 (leave race; + L1/L2 expectations pending), Q11 (orphaned snapshots on recreate; D2), and Q12 (v1 applier E2E backoff flags). +- 23:09 — Re-checked Q10 after the user recalled that leaving clusters keep their bindings. A throwaway envtest + probe (written, run, and deleted) bound a cluster, held its `MemberCluster` in `Terminating` with a finalizer, and + watched the binding get deleted within seconds. So the binding release is confirmed, and the resource outcome is + still a race. Corrected the plan's description of legacy behavior: the legacy scheduler marks the binding + `Unscheduled` instead of keeping it bound, and deletion goes through rollout or the update run's delete stage. + Q10 is still open. +- 23:27 — Q10 answered: the binding should be deleted only after the cluster has left; deleting it at the deletion + timestamp is a bug to be fixed separately. L1 and L2 are marked skipped. Added L6 (a virtual-cluster check of + binding timing during the 1m force-delete window), deferred to the fix PR. Added a precondition to L4 (no binding + targets cluster-3 when it leaves). Questions 11–12 are still open. +- 23:30 — Q11: not intended; the user is preparing a fix. Kept open until the user confirms it has landed. D2 keeps its + binding-adoption assertions; its snapshot-ownership assertions are pending the fix. Q12 is still open. +- 23:45 — Q12 answered and implemented (P-5). Decisions: + - The v1 setup must not share configuration with v1beta1. + - No new flags; use fixed static values in both agents. + - Keep the placeability filters shared. + - E2E caps the v1 applier's backoff through a test-only env var. + + Code changes: + - Hub: three static concurrency constants. + - Member: static concurrency and cleanup wait, plus the env var helper. + - v1 work applier package: named default backoff constants and `NewDefaultRequeueRateLimiterWithMaxDelay`. + - `setup.sh`: sets the env var on each member agent and waits for the rollout. + - Chart READMEs updated. + + Validation: + - Build and vet. + - `make lint`: 0 issues, after one gofmt fix. + - New unit tests pass: `TestNewDefaultRequeueRateLimiterWithMaxDelay` (6 cases) and + `TestPlacementPolicyWorkApplierRequeueRateLimiter` (5 cases). + - Existing backoff tests and `cmd/...` tests pass. + - `bash -n setup.sh` passes. + - Not run on Kind (no Docker here). + + Updated A2 (detection within ~5s) and the related environment fact. Only Q11 remains open. + + After this entry was written, the hub constants in `setup.go` were found at 10/3/3 (modified at 12:39 UTC, + apparently by the user) rather than the 100/10/10 set here; P-5 reflects the current code. That file's comment + still says the values match the v1beta1 defaults, which is no longer true. +- 23:51 — Pre-implementation check. Confirmed in `process.go` that a staged update run reconcile holds its worker for + up to `refreshPeriod` (30s). With 3 workers and ~15 parallel ginkgo processes, U7's four long-blocking runs would + starve the lane's other staged update runs, so U7 is now `Serial` (environment fact added; phasing updated). + Remaining items before implementation are listed in the reply: the stale `setup.go` comment, how to land the + uncommitted code changes, no local Kind environment, and the Phase 0 prerequisites. +- 23:54 — User decisions: + - Hub concurrency is now 10/5/5 (work generator / staged update runs / auto-rollout); the `setup.go` comment no + longer claims parity with the v1beta1 defaults (build and vet pass). + - U7 can take longer, so it is no longer `Serial`. Its four runs execute one at a time, holding at most one of the 5 + workers, in parallel with the lane (~5–6 min). + - The user will handle commits and PRs for the code changes. + - The user will enable Docker Desktop WSL integration so the E2Es can be run locally (setup-clusters and focused + specs). +- 00:05 (Oct 23) — Implemented S1 and S2 as requested; the user asked not to run the E2Es yet. + - [placement_policy_scheduling_test.go](../../../test/e2e/placement_policy_scheduling_test.go) carries + `Label("placementpolicy")` and two Ordered contexts. S1: a PP with no selectors. S2: a CPP with `env=prod`, + `count: All`, plus a Consistently check that the canary cluster gets neither a binding nor the ConfigMap. + - P-3 has started with scope-agnostic helpers (key with an empty namespace = cluster scope). + [placement_policy_utils_test.go](../../../test/e2e/placement_policy_utils_test.go) builds, retrieves and removes + policies, lists bindings by policy name, finds the latest primary snapshot through the owner annotation, and has + the all-synchronized status builder and cmp options. + [placement_policy_actuals_test.go](../../../test/e2e/placement_policy_actuals_test.go) compares the whole policy + status and, per cluster, each binding's controller ownership, spec and status in one shot. + - The existing ConfigMap and cleanup helpers in placement_policy_test.go are reused. + - Validation: gofmt, `go vet ./test/e2e/`, a compile of the test binary, and a Ginkgo dry run with + `label-filter=placementpolicy` (selects exactly the 7 new specs) all pass; `make lint` reports 0 issues. + - Not done yet: P-4's CI lane, and the label on the two existing FEP files. Until then the new specs run in the + default lane. +- 00:19 — Implemented R1, A1, U1, and AR1 as requested, without running them. P-1 is done: `kfrolloutv1alpha1` is + registered in the E2E scheme in [setup_test.go](../../../test/e2e/setup_test.go). + - [placement_policy_resources_test.go](../../../test/e2e/placement_policy_resources_test.go) (R1, PP): two labeled + ConfigMaps are placed and the unlabeled app ConfigMap is not. After an edit plus a new labeled ConfigMap, the + policy reports `ResourcesChangedPendingRollout`, with the latest revision, bindings, member data and the absent + ConfigMap all held steady (Consistently). + - [placement_policy_sync_strategy_test.go](../../../test/e2e/placement_policy_sync_strategy_test.go) (A1, CPP): a + pre-existing ConfigMap on cluster-1 gets `NotTakenOver` in the binding's `failedResources`. The policy reports + 2/3 synchronized and available, and the member data stays untouched. Switching to `AlwaysTakeOver` updates every + binding's sync strategy on the same snapshot, takes over the ConfigMap (AppliedWork owner reference, hub data), + and leaves the policy fully synchronized on the same revision. + - [placement_policy_staged_update_run_test.go](../../../test/e2e/placement_policy_staged_update_run_test.go) (U1, + both scopes via a loop): the run is initialized with a new snapshot and the policy's latest revision moves. The + canary gets v1 while prod stays on v0, and the policy reports `SomeClustersOnOlderResourceRevision` with the run + holding the binding manager role. The approval request is checked for spec and controller ownership, prod is held + until approval (Consistently), the approval is accepted, and v1 reaches every cluster. The whole run status + (stages, clusters, tasks, conditions) is compared, and the policy ends fully synchronized with the role released. + - [placement_policy_auto_rollout_test.go](../../../test/e2e/placement_policy_auto_rollout_test.go) (AR1, PP): + there is no rollout attempt for the initial placement. After a hub edit, a new snapshot reaches every binding and + member without a run, the timestamp annotation parses, and the policy ends fully synchronized with the role + released. + - Helpers: + - Generalized bindings actual (per-cluster snapshot and status; S1/S2 use a wrapper). + - Member ConfigMap actuals: data, absence, taken over. + - Run-status and approval-request actuals. + - Mutators: policy update, hub ConfigMap data, unmanaged member ConfigMap, approval. + - Scope-agnostic run, strategy and approval request builders and getters, plus cleanup. + - `placementPolicyRolloutEventuallyDuration` (90s). + - Validation: gofmt, `go vet ./test/e2e/`, a Ginkgo dry run with `label-filter=placementpolicy` (52 specs: 7 for + S1/S2 plus 45 new), and `make lint` (0 issues). +- 00:51 — First trial run on Kind. Setup was `test/e2e/setup.sh 3`, then + `ginkgo -v -p --label-filter=placementpolicy` (only the new specs), then teardown with `stop.sh 3`. + - Run 1: 48 passed, 2 failed, 2 skipped. Both failures were U1's "should complete the run successfully" (PP and + CPP). `PerStageStatus.StartedTimestamp`/`CompletedTimestamp` are `*metav1.Time`, and + `cmpopts.IgnoreTypes(metav1.Time{})` doesn't cover pointers: cmp calls `(*metav1.Time).Equal` first. + - Fix: `compareTimePointersByPresence` in + [placement_policy_actuals_test.go](../../../test/e2e/placement_policy_actuals_test.go). It compares only whether + the pointers are set, so the wanted `&metav1.Time{}` still checks that the stage timestamps are set. + - Re-runs: U1 alone passed 26/26; all new specs passed 52/52 in 1m28s. `make lint` reports 0 issues. + - Assumptions now confirmed on Kind: `ActiveClusterClaims=0` is set, the defaulted sync strategy equals the + binding's copy, `EquateEmpty` handles the prod stage's nil `AfterStageTasks`, and the cluster-scoped run and + approval flow works. + - Local environment notes (not repo changes): + - The stale buildx builder `img-builder` (dead Docker Desktop bind mount) was removed; the Makefile recreates it. + - `~/.docker/config.json` uses `credsStore: desktop.exe`, which can't run without WSL interop, so the run used a + temporary `DOCKER_CONFIG` with an empty config. +- 01:09 — Implemented the remaining S and R P0s (S3, S4, S5, R2, R3, R4) as requested, without running them. Each + expectation was checked against the controller code first (`chooseClusters`, `aggregateCounts`, + `summarizeResourceStatus`, the snapshot manager's selection errors, the claim lifecycle). + - [placement_policy_scheduling_test.go](../../../test/e2e/placement_policy_scheduling_test.go): + - S3: the per-selector `desired=scheduled=3` against the per-cluster `synchronized=available=2`. + - S4: the first pick is learned from its binding (ranking isn't asserted), and its binding UID must survive both + the scale-up and the scale-down. + - S5: `Serial`; `AfterAll` restores cluster-2's `env` label in a defer. The changed terms (`env=canary` plus + `region=east`) match nothing, giving one claim carrying them, `desired=1`, `scheduled=0`, and, with no target + cluster left, `Synchronized=True` and `Available=True`. + - [placement_policy_resources_test.go](../../../test/e2e/placement_policy_resources_test.go): + - R2: both east bindings are on the initial snapshot with v0 data, and the policy is still pending a rollout. + - R3: decision 13. + - R4: two contexts. While a selection error stands, the policy has no snapshot, no bindings, and no latest + revision. + - Helpers added: + - Bindings by cluster; wait for a single binding; the other east cluster. + - Member cluster lookup and label setter; an AppliedWork-ownership check, now shared with the takeover actual. + - Pending-rollout and not-collected status builders (R1 now uses the former). + - A status actual for policies without a revision. + - Actuals: no bindings, leftovers removed, claim, namespace placed/absent, ClusterRole placed/absent. + - Validation: gofmt, `go vet ./test/e2e/`, a Ginkgo dry run with `label-filter=placementpolicy` (88 specs, 36 new), + and `make lint` (0 issues). +- 01:50 — Ran the `placementpolicy` specs on Kind (setup via `setup.sh 3` with a temporary `DOCKER_CONFIG`; + teardown via `stop.sh 3`). + - Run 1: 88/88 passed (2m11s). S5 ran as a `Serial` context after the parallel specs. + - Run 2: 87/88. S4 failed because the first pick (cluster-2) was trimmed on scale-down. The hub logs showed both + bindings created within the same second, so their creation timestamps tied and the name tie-break kept + cluster-1 (Q14). + - Fix: S4 now scales up only after the first binding's creation second has passed (plus 250ms). + - Re-runs: S4 alone with `--repeat=4` passed 5/5, with the hub logs confirming that the newer binding was trimmed + each time. A final full run passed 88/88 (1m57s). `make lint` reports 0 issues. +- 02:37 — Implemented the remaining A, U, and AR P0s (A2, A3, A4, U2, U3, AR2) as requested, without running them. + Expectations were checked against the code: the v1 work generator's binding status (failed resources copy the + applier's `Applied` condition and diff details), pre-apply drift detection, `cleanupWhenBindingDeleted` (orphan + propagation), run initialization (an explicit `resourceSnapshotName` must be a primary snapshot the policy controls; + no policy re-validation mid-run), and auto-rollout's exact `"true"` check. + - [placement_policy_sync_strategy_test.go](../../../test/e2e/placement_policy_sync_strategy_test.go): + - A2: PP on cluster-1 only. The default `ApplyAnyway` overwrites a member edit. After switching to `ReportError` + (no rollout), the next edit shows on the binding as `Applied=False/FoundDrifts` with + `diffDetails{observedInMemberClusterGeneration: 0, /data/data: drifted vs test}`, and the policy is 0/1 + synchronized. The drift is left as is (Consistently); reverting it recovers. + - A3: `OrphanResources`. After the policy is deleted, its bindings, snapshots, and Works are gone from the hub, + and the ConfigMap stays on every member with the same data and no AppliedWork owner (Eventually, then + Consistently). + - A4: the ConfigMap plus `appDeployment()`. The Deployment is available on every member (2m wait for the image + pull), every binding reports 2/2/2, and the policy is fully available. + - [placement_policy_staged_update_run_test.go](../../../test/e2e/placement_policy_staged_update_run_test.go): + - U2: a forward run takes index 1. A rollback run names index 0; every binding returns to it and the members get + v0. The status is pinned as described in the row. + - U3: the held-back state (bindings, no ConfigMap on Y, policy status with the run as binding manager) is checked + with Eventually, then Consistently. After approval the run succeeds and Y binds to the new snapshot. + - [placement_policy_auto_rollout_test.go](../../../test/e2e/placement_policy_auto_rollout_test.go): + - AR2: a pending change, the initial version on every member, and no attempt annotation, held for 30s without + the annotation and for another 30s with it set to `"false"`. + - Helpers: + - `deletePlacementPolicyAndWait`, now used by `ensurePlacementPolicyRemoved`. + - `configMapNotAppliedPlacementBindingStatus`, which `notTakenOverPlacementBindingStatus` now wraps. + - `someClustersNotSynchronizedPlacementPolicyStatus`, now also used by A1. + - `waitForAfterStageApprovalRequest`, now also used by U1. + - `otherProdClusterName`, `updateConfigMapDataOnMemberCluster`, `noTermsSelectorsForAllMemberClusters`. + - Actuals for no Works left, orphaned ConfigMaps, available Deployments, and successful runs. + - The bindings actual now also ignores time values, for `diffDetails.firstDiffedObservedTimestamp`. + - Validation: gofmt, `go vet ./test/e2e/`, a Ginkgo dry run with `label-filter=placementpolicy` (126 specs, 38 new), + and `make lint` (0 issues). +- 02:55 — Ran the `placementpolicy` specs on Kind (setup via `setup.sh 3` with a temporary `DOCKER_CONFIG`; + teardown via `stop.sh 3`). Run 1 passed 126/126 (2m47s), and run 2, a flakiness check, passed 126/126 (2m40s). + No code changes were needed. The longest new spec is AR2's second Consistently (about 70s with `AfterAll`). A4's + Deployment became available well within its 2m wait. The other new contexts end in the usual ~40s cleanup. +- 03:22 — The user picked the U and AR P1s plus the remaining P0s; L4 is skipped for now; P-4 is done as well. + Implemented U4–U10, AR3–AR5, and D1, without running them. Expectations were checked against the code first: + - Staged update runs: timed wait, before-stage approval, suspension (a suspended run gives up the binding manager + role and its cleanup finalizer), and the failure policy (`maxAllowedClusterFailures` defaults to 1 and fails the + run once exceeded; a cluster fails once `maxWaitTimePerClusterMinutes` passes without it becoming synchronized + and available). + - Initialization failures: a missing policy goes through the terminal-state path, but still ends at + `Initialized=False/Failed`. + - The v1 Deployment availability rule (updated = available = desired, none unavailable). + - Auto-rollout: requeues on the remaining wait period; on a failed claim, exponential backoff with no reaction to a + role release. + - The annotation controller leaves extra annotations alone. + + New files: + - [placement_policy_staged_update_run_stages_test.go](../../../test/e2e/placement_policy_staged_update_run_stages_test.go): + - U4: compares the whole run status, including the TimedWait task, and checks that the prod stage starts no + earlier than the wait start + 30s - 1s. The start is recorded to the second, so the wait can end up to 1s + early. + - U5: the approval request is found in `beforeStageTasks`. + - U9: the stages are `canary` then `east`, so cluster-3 is in no stage. The run succeeds with results for + clusters 2 and 1 only, and the policy reports `SomeClustersOnOlderResourceRevision` 2/3. + - [placement_policy_staged_update_run_interruptions_test.go](../../../test/e2e/placement_policy_staged_update_run_interruptions_test.go): + - U6, both scopes: created suspended (initialized, new snapshot, role released, nothing rolled out, no `Started`), + then resumed, then suspended at the approval. The approval is neither accepted nor acted on while suspended, + and is accepted after resuming. + - U8: the deleted run's approval request is garbage-collected and the role released. The partial state is kept + (policy 1/3 synchronized). A follow-up run rolls out the same snapshot and completes. + - [placement_policy_staged_update_run_failures_test.go](../../../test/e2e/placement_policy_staged_update_run_failures_test.go): + - U7: four runs in one Ordered context, each with its own Deployment and policy: + - (a), PP then CPP: an unpullable image with `maxAllowedClusterFailures: 0`. The run fails at canary, prod + stays on the initial snapshot, and the role is released. + - A `pods: 0` quota is then added on cluster-2. + - (b), PP then CPP: a pod-template change with 1 failure allowed ends `PartiallySucceeded`; only canary fails. + - U10: missing strategy, missing policy, another policy's snapshot, and a missing sorting label. Each run's whole + status is just `Initialized=False/Failed`, and the change stays pending (no new snapshot, role released, + members on v0). + - [placement_policy_lifecycle_test.go](../../../test/e2e/placement_policy_lifecycle_test.go): + - D1: the Works and AppliedWorks are recorded first. Deletion removes the hub's bindings, snapshots, and Works, + and the members' AppliedWorks and ConfigMap. The applier-created member namespace stays, unmanaged + (Consistently). + + Changes to existing files: + - [placement_policy_auto_rollout_test.go](../../../test/e2e/placement_policy_auto_rollout_test.go): + - AR3 extends AR1: the second edit is held until the last attempt + 1m (Consistently until 5s before), then + rolled out, with the second attempt ≥ 1m after the first. + - AR4: the run holds the role (rolling out the unchanged initial snapshot) while the edit stays pending with no + attempt recorded. After approval and completion, auto-rollout rolls out on its backoff (3m wait). + - AR5: auto-rollout annotations added to an annotation-generated PP (`env=canary`) roll out a hub edit to + cluster-2, and the annotations survive. + - AR1 now uses the shared `autoRolloutAnnotations()`. + - Helpers: + - Generalized `waitForStageApprovalRequest` (before/after). + - Run-status builders moved to package level (U1 uses them). + - `updateStagedUpdateRun`, `waitForPlacementPolicySynchronized`, `placementPolicyWorks`, and a few more. + - Actuals: run condition, per-cluster run results, binding manager, binding snapshots, approval request gone, + AppliedWorks gone, unmanaged namespace, only-canary-updated. + - `someClustersOnOlderResourceRevisionPlacementPolicyStatus`. + - P-4: the `placementpolicy` CI lane (same settings as the default lane), `&& !placementpolicy` in the default + lane, and `Label("placementpolicy")` on the three Describes of the two older FEP files. + + Validation: + - gofmt, `go vet ./test/e2e/`, and `make lint` (0 issues). + - Ginkgo dry runs: `placementpolicy` selects 230 specs, now including the two older FEP files; the default lane's + new filter selects 951 specs, none of them FEP. + - `ci.yml` parses (checked with `sigs.k8s.io/yaml` in a throwaway test, since removed). + - Expected lane cost: U7 adds about 5–6 minutes on one process, AR4 up to about 3, AR3 about 1. +- 03:56 — Ran the `placementpolicy` lane on Kind (setup via `setup.sh 3` with a temporary `DOCKER_CONFIG`; teardown + via `stop.sh 3`). It now has 230 specs, including the two older FEP files labeled under P-4. + - Run 1: 229/230 passed (9m53s). The one failure was U10's `AfterAll`, not an assertion. It removed the second + policy first through `ensurePlacementPolicyRemoved`, which waits for the app ConfigMap to leave the members, but + the main policy still placed it. + - Fix in [placement_policy_staged_update_run_failures_test.go](../../../test/e2e/placement_policy_staged_update_run_failures_test.go): + the main policy goes first, then the second one through `deletePlacementPolicyAndWait`, plus a wait for its + ConfigMap to leave the members. No other `AfterAll` has this ordering issue: U7's policies place Deployments, so + the app ConfigMap check passes trivially there. + - U10 alone then passed 7/7, and a second full run passed 230/230 (8m22s). `make lint` reports 0 issues. + - Timing: U7's four runs took about 61s, 61s, 96s, and 142s (the last includes `AfterAll`), well within their 3m + waits. AR3 and AR4 passed on both runs. +- 2026-10-24 01:49 — Implemented the S-series P1s (S6–S13); not yet run on Kind. Expectations were checked against + the placement policy controller first (`scheduling.go`, `bindings.go`, `claims.go`, `status.go`, `watch.go`). + New files: + - [placement_policy_scheduling_selectors_test.go](../../../test/e2e/placement_policy_scheduling_selectors_test.go), + parallel: + - S6: the claim carries the east terms. The status is pinned in full, and the Scheduled message is checked for + the minimum count being reached, the only thing that tells it apart from an unmet floor. + - S7: no claim or binding (Consistently), then switching to `AddClusterClaim` adds the claim. + - S8: four term swaps (In, NotIn, presence terms every cluster satisfies, presence terms none satisfies). The + member clusters all carry the same label keys, so Exists/DoesNotExist pair a present key with one that no + cluster carries instead of relabeling a cluster (which would make the context Serial). + - S9: skipped unless the Azure property provider is on, like the legacy property specs. + - S10: while the selector is invalid, the controller returns before touching bindings or the resource status, + so the spec pins the cleared `desired`/`scheduled` and the resource conditions left at the last valid + generation. Reverting the terms recovers with the same binding UIDs. + - [placement_policy_scheduling_cluster_changes_test.go](../../../test/e2e/placement_policy_scheduling_cluster_changes_test.go), + all Serial; each context restores its taint or label in `AfterAll`: + - S11: the taint goes on cluster-2 just before the policy is created, as in the legacy taint specs. A hub agent + cache that has yet to see the taint would bind cluster-2 and keep it (sticky), which would show up as a + failure of the first spec. + - S12: also pins `desired=scheduled=1` with `synchronized=available=2` while the bound cluster is tainted. + - S13: the claim is withdrawn once the relabeled cluster is bound; the status reports no active claim. + + Changes to existing files: + - Helpers: `clusterSelectorWithTerms`, `sameSelectorsForClusters`, `waitForPlacementPolicyPlacedOnlyOn`, + `placementPolicyBindingUIDsByCluster`, `placementPolicyTolerationsFor`, `unfulfilledPlacementPolicyStatus` + (S5's final status now uses it), and the spelled-out `InvalidClusterSelectors` reason. Actuals: + `noPlacementPolicyClaimsActual` and `placementPolicyBindingsKeptActual`. + - `placementPolicyStatusActual` now fills in the observed generation only for wanted conditions that name none, + so S10 can pin conditions at an older generation. No existing caller sets one. + + Validation: gofmt, `go vet ./test/e2e/`, `make lint` (0 issues). The `placementpolicy` dry run selects 267 specs + (36 new; S11–S13 are marked Serial). Expected lane cost: the parallel contexts take about 1–2 minutes each, and + the three Serial contexts add about 2–3 minutes at the end of the lane. +- 2026-10-24 03:21 — Ran S6–S13 on Kind (setup via `setup.sh 3` with a temporary `DOCKER_CONFIG`; teardown via + `stop.sh 3`). The focused run also covered S5, whose final status now uses the shared builder. + - Run 1: 39/40 passed. S9's 3 specs were skipped because ginkgo ran without `PROPERTY_PROVIDER=azure`. CI sets it + on the E2E step, so the lane runs S9 there; local runs must pass it to ginkgo too. + - The failure was S12's last step, from the Q14 tie: X (cluster-3) was bound and, after the taint, Y (cluster-1) + was bound within the same second. Once the taint was lifted, the name tie-break kept Y and dropped X. + - Fix: S12 now waits out X's binding creation second before tainting, through the new shared + `waitPastCreationSecond`. S4's inline wait moved to the same helper. + - Runs 2 and 3, with `PROPERTY_PROVIDER=azure` and S4 added to the focus: 48/48 passed each time (3m29s and + 3m12s). `make lint` reports 0 issues. +- 2026-10-24 03:53 — Q14 resolved: the one-second tie-break is accepted as a limitation. No product change; S4 and + S12 keep their `waitPastCreationSecond` workaround. Only Q11 (the user's fix in progress) remains open. +- 2026-10-24 04:36 — Implemented the R- and A-series P1s (R5–R7, A5–A8); not yet run on Kind. Expectations were + checked against the snapshot manager (`resources.go`, `snapshots.go`, `gc.go`), the eligibility checker + (`resourceeligibility`), the work generator (`works.go`, `status.go`), and the v1 work applier (`process.go`, + `takeover_drift_detection.go`, `apply.go`, `cleanup.go`). Decision 15 (`default` is placeable) came from the user + while R5 was being written. + + New files: + - [placement_policy_resources_eligibility_test.go](../../../test/e2e/placement_policy_resources_eligibility_test.go): + R5, in four contexts. The two `ResourceCollected=False` statuses are identical, so each case also checks the + condition message for its cause. The `kube-root-ca.crt` wait allows for Kubernetes publishing the ConfigMap + after the namespace is created. + - [placement_policy_resources_snapshots_test.go](../../../test/e2e/placement_policy_resources_snapshots_test.go): + - R6: snapshot and Work layouts are compared in one shot, keyed by sub-index and by cluster namespace. + - R7: three runs; after each, the primary snapshots are exactly {initial, latest}. + - [placement_policy_sync_strategy_failures_test.go](../../../test/e2e/placement_policy_sync_strategy_failures_test.go): + A5 (reuses U7's unpullable image) and A7. + - [placement_policy_sync_strategy_takeover_test.go](../../../test/e2e/placement_policy_sync_strategy_takeover_test.go): + - A6: full, then partial comparison, then the member edit that lets the takeover happen. + - A8: also pins the co-owned CM surviving policy deletion. + + Changes to existing files: + - Helpers: `placementPolicySnapshots` (`placementPolicyLeftoversRemovedActual` now uses it), and the actuals + `placementPolicyPrimarySnapshotNamesActual`, `placementPolicyConditionMessageActual`, and + `configMapManagedOnClustersActual`. + + Validation: gofmt, `go vet ./test/e2e/`, `make lint` (0 issues). The `placementpolicy` dry run selects 311 specs + (44 new, none Serial). +- 2026-10-24 04:56 — Ran R5–R7 and A5–A8 on Kind (setup via `setup.sh 3` with a temporary `DOCKER_CONFIG`; + teardown via `stop.sh 3`; ginkgo with `PROPERTY_PROVIDER=azure`). The focus also covered D1, R3, and A3, which use + `placementPolicyLeftoversRemovedActual`, now built on the shared `placementPolicySnapshots` lister: 60 specs in all. + Run 1: 60/60 passed (2m07s). Run 2: 60/60 passed (1m23s). No changes were needed. R7's three runs took about 10s + each, and the garbage collection of the unused snapshots was observed within each spec's wait. +- 2026-10-24 06:10 — The user asked for the remaining P1s: X1, X3, L5, B1, B2, D2, and D3. Plan, after checking the + expectations against the code (`workapplier/process.go` and `takeover_drift_detection.go`, + `workgenerator/controller.go` and `cleanup.go`, `placementpolicy/bindings.go`, `claims.go`, `status.go`, and + `watch.go`, and the snapshot manager's `ops.go`): + - X1 and X3 in a new `placement_policy_interop_test.go`: + - X1: the v1 applier refuses any object that a v1beta1 AppliedWork owns (`isOwnedByOlderKubeFleetAPIObjects`), + before looking at the sync strategy, so the binding reports `OwnedByOtherKubeFleetAPIObjects`. + - X3: the second policy is created with `AlwaysTakeOver` + `ShareOwnership` first (`FailedToTakeOver`, from the + duplicate-placement check), then switched to the default (`NotTakenOver`), and only then is the first policy + deleted. The order differs from the plan's on purpose: with a takeover option, the second policy's applier drops + owner references to AppliedWorks whose Work is gone (`removeLeftBehindAppliedWorkOwnerRefs`), so it could take + the ConfigMap over before the garbage collector removes it. Under `ReportError` it cannot, so the ConfigMap is + removed and then created anew (new UID). + - L5 in a new `placement_policy_join_leave_test.go`, `Serial`: a never-joined virtual cluster with the matching + label. Every policy reconciles on a member cluster create, and the claim's + `lastObservedMostRecentClusterCreationTimestamp` advances to the new cluster's creation time, which shows the + policy saw the cluster before the Consistently check that the claim (same UID) stays and nothing is bound. + - B1 and B2 in a new `placement_policy_binding_suspension_test.go`. While suspended, the binding's + `lastProcessedResourceSnapshotName` is cleared and its (stale) conditions are left alone, so the policy counts + it as neither synchronized nor outdated: `someClustersNotSynchronizedPlacementPolicyStatus(3, 2)`. B2 re-adopts + the orphaned ConfigMap with `TakeOverIfNoDiff` (it is unchanged). + - D2 and D3 in `placement_policy_lifecycle_test.go`: + - D2: Q11 is not fixed (`retrieveLatestSnapshot` still ignores ownership), so the re-created policy reuses the + orphaned snapshot. That doesn't block adoption (`adoptOrReportBindingOccupant` runs once creating the binding + hits `AlreadyExists`), but `latestPrimarySnapshotOfPlacementPolicy` requires a controlled snapshot, so D2 skips + the binding and policy status actuals. It checks binding ownership, Work UIDs, and member ConfigMap + resource versions only. + - D3: the policy also has an unfulfillable selector, so that "no leaked claims" checks a claim that existed. +- 2026-10-24 06:55 — Implemented X1, X3, L5, B1, B2, D2 (binding adoption only, Q11), and D3, then ran them on Kind + (setup via `setup.sh 3` with a temporary `DOCKER_CONFIG`; teardown via `stop.sh 3`; ginkgo with + `PROPERTY_PROVIDER=azure`). + + New files: + - [placement_policy_interop_test.go](../../../test/e2e/placement_policy_interop_test.go): X1 and X3. + - [placement_policy_binding_suspension_test.go](../../../test/e2e/placement_policy_binding_suspension_test.go): + B1 and B2. + - [placement_policy_join_leave_test.go](../../../test/e2e/placement_policy_join_leave_test.go): L5 (`Serial`). + + Changes to existing files: + - [placement_policy_lifecycle_test.go](../../../test/e2e/placement_policy_lifecycle_test.go): D2 and D3. D1 now + uses the shared `placementPolicyWorkNamesByCluster`. + - Helpers: `deletePlacementPolicyAndWait` takes delete options; `placementPolicyWorkNamesByCluster`, + `setPlacementPolicyBindingSuspended`, `placementPolicyWorkUIDs`, and `configMapResourceVersionsByCluster`. + - Actuals: `placementPolicyWorkClustersActual`, `suspendedPlacementPolicyBindingActual`, + `placementPolicyBindingOwnershipActual`, `placementPolicyWorksKeptActual`, `configMapUnchangedOnClustersActual`, + `configMapPlacedByPlacementPolicyActual`, `placementPolicyClaimKeptActual`, and + `placementPolicyClaimObservedClusterCreationActual`. + + Validation: + - gofmt, `go vet ./test/e2e/`, and `make lint` (0 issues). The `placementpolicy` dry run selects 353 specs (42 + new; L5's 4 are Serial). + - Run 1, the new contexts plus D1: 38/40. B1 and B2 failed on a test bug: the cluster count was read while the spec + tree was built, before the suite set up the member cluster list, so it was 0. Every other field of the status + matched the plan (3 desired and scheduled, 2 synchronized and available). The count is now read inside the specs. + - B1 and B2 alone: 15/15 (1m23s). The full `placementpolicy` lane: 353/353 (11m29s). A flakiness rerun of the new + contexts plus D1: 47/47 (1m55s). + - Remaining from this plan: the P2s (S14, S15, R8, A9, U11–U14, AR6, L3, B3), L4 (skipped for now, per the + user), L1, L2, and L6 (Q10), X2 (separate PR), and D2's snapshot assertions (Q11). +- 2026-10-24 07:05 — The user asked for the P2s: S14, S15, R8, A9, U11–U14, AR6, L3, and B3. Plan, after checking the + expectations against the code: + - S14 (`_selectors`): a claim's terms are compared on every round; a term change withdraws it (UID-pinned delete), + and the claim watch reissues it under the same deterministic name (`claimName` hashes the policy and selector + index). + - S15 (`placement_policy_scheduling_test.go`): the binding watch admits deletes (`GenerationChangedPredicate` passes + them), so the policy re-creates the binding once the old one is gone. The applier deletes the AppliedWork with + foreground propagation, so the old ConfigMap goes before the re-created binding's Work can apply: new UIDs for + the binding, the Works, and the ConfigMap. + - R8 (`placement_policy_resources_test.go`): rolled out by auto-rollout; the applier's leftover-manifest cleanup + (`findLeftOverManifests`) removes the deleted ConfigMap. + - A9 (`_sync_strategy_failures`): a member ConfigMap applied by another field manager. Without `forceConflicts`, the + server-side apply conflicts (`FailedToApply`); the applier forces on its own only when no foreign manager exists + (`shouldUseForcedServerSideApply`). With it, KubeFleet's manager takes the field over. + - U11 (`_stages`): `initialize.go` resolves the stage's `maxConcurrency` with + `GetScaledValueFromIntOrPercent(..., roundUp=false)`, so 70% of 3 is 2. + - U12 and U14 (`_interruptions`): a request that the run does not control yields + `ApprovalRequestCreated=False/RequestOwnershipMismatch`; deleting it enqueues the run through + `spec.stagedUpdateRunName`. U12 also pre-approves the hand-made request, to show that a request the run does not + own cannot unblock it. U14: the run still moves a suspended binding to the new snapshot, then marks the cluster + succeeded right away; only the message (`binding is suspended`) tells that path apart. + - U13 (`placement_policy_staged_update_run_test.go`): annotation `env=canary;env=prod,count=All`, rolled out by the + usual canary-then-prod strategy. + - AR6 (`placement_policy_auto_rollout_test.go`): cheap enough by backdating the last-rollout-attempt annotation, as + the auto-rollout ITs do: `5h` with the last attempt 61m ago rolls out at once (clamped down to 1h); `1s` with the + last attempt 30s ago waits until the 1m mark (clamped up); `soon` with the last attempt 2m30s ago waits until the + 3m mark (default). The backdated timestamp goes in the same update as the wait period, since a change to the + timestamp alone does not enqueue the policy. + - L3 (`placement_policy_join_leave_test.go`, Serial): scale the bound cluster's `member-agent` Deployment to 0. The + policy controller's eligibility checker has a fixed 5m heartbeat timeout, with the 150s periodic requeue as the + backstop (resource-usage updates of the other clusters usually wake it sooner), so the gap is filled after about + 5–7.5m. Restarting the agent flips the Healthy condition, an event the policy watches, so the overpick is + trimmed promptly. Expected cost: about 7–9 minutes of Serial time. + - B3 (`_binding_suspension`): auto-rollout refreshes every binding it controls, suspended or not. +- 2026-10-24 07:55 — Implemented the P2s (S14, S15, R8, A9, U11–U14, AR6, L3, B3) and ran them on Kind (setup via + `setup.sh 3` with a temporary `DOCKER_CONFIG`; teardown via `stop.sh 3`; ginkgo with `PROPERTY_PROVIDER=azure`). + + Where they live: + - S14 in `_scheduling_selectors`, S15 in `placement_policy_scheduling_test.go`, R8 in + `placement_policy_resources_test.go`, A9 in `_sync_strategy_failures`. + - U11 in `_staged_update_run_stages`, U12 and U14 in `_staged_update_run_interruptions`, U13 in + `placement_policy_staged_update_run_test.go`. + - AR6 in `placement_policy_auto_rollout_test.go`, L3 in `_join_leave` (Serial), B3 in `_binding_suspension`. + + Helper changes: + - `createConfigMapOnMemberCluster` takes create options (A9 sets a field owner). + - R1's labeled ConfigMap builder and label selector moved to file level, now shared with R8. + - New actuals: `configMapUIDsByCluster`, `placementPolicyLatestSnapshotChangedActual`, and + `stagedUpdateRunApprovalTaskActual`. + - File-local helpers: A9's `configMapDataKeyOwnersActual` (reads the managed fields), and L3's + `stopMemberAgent`/`startMemberAgent` (scale the `member-agent` Deployment, then wait for no pods, or for a fresh + heartbeat and the joined check). + + Validation: + - gofmt, `go vet ./test/e2e/`, `make lint` (0 issues). The `placementpolicy` dry run selects 415 specs (62 new). + - New contexts plus R1, whose helpers moved: 67/67 on the first run (8m32s). L3's wait for the replacement took + about 5 minutes (the heartbeat timeout), and L3 about 6 minutes in all; AR6 about 2 minutes. + - Full `placementpolicy` lane: 415/415 (17m52s, up from 11m29s, mostly L3's Serial time). That is well within the + 70m timeout, so the timeout risk noted under Phasing does not apply. + - Remaining from this plan: L4 (skipped for now, per the user), L1, L2, and L6 (Q10), X2 (separate PR), and D2's + snapshot assertions (Q11). +- 2026-10-24 08:05 — A review comment flagged two possible flakes in the auto-rollout specs. The changes: + - AR4's wait for the held-back rollout is 4m, up from 3m. The comment's premise was off: the retry comes at most + one backoff interval after the run lets go, not after the 180s cap, and the steps before it bound that interval + at 160s. That still left a thin margin in the worst case. The constant's comment now states the real bound. + - AR3: the specs' minimum wait period is 2m, up from 1m, because the period starts with AR1's attempt, before + AR1's checks settle. Only AR3 waits on it; AR4, AR5, R8, and B3 use the same annotations but only roll out once. + The hold is no longer silently skipped when the window is gone: the spec fails with a clear message if less than + 15s of it is left once the change shows as pending. + + Validation: + - gofmt, `go vet ./test/e2e/`, and `make lint` (0 issues). + - On Kind, only the five contexts that use the changed constants (AR1/AR3, AR4, AR5, R8, B3), with `--repeat=1`: + 31/31 on both runs (3m27s and 2m52s). + - AR3's hold lasted about 114s of the 115s possible, so AR1's rollout settled within seconds. AR4's retry spec took + about 50s, well under the 4m bound. The AR1/AR3 context costs about 1 more minute. +- 2026-10-26 22:20 — Re-planned after merging `feat/block-cross-ops-between-v0-v1` and + `feat/handle-incomplete-snapshots-in-work-generator` into `test/new-e2e-tests-3` (on top of `main` with #1012). Only + the plan changed; no E2E code was touched. Each claim was checked against the merged code. + - **Q3 fixed (cross-ops branch):** X2 is back as a P1, with its expectations. Re-planning it raised Q16: a CRP takes + over the namespace that the v1 applier created, so deleting that CRP removes the PP's ConfigMap. X2 deletes the PP + first, which keeps it clear of Q16. + - **Q11 fixed (#1012):** D2's snapshot part is unblocked and rewritten. The work generator branch adds the binding + side: missing snapshots now show as `InvalidSnapshots`. + - The re-created policy no longer reuses the orphaned snapshot. It has it collected and starts over at index 0 + under a new name, and adoption moves the bindings to it. + - The Works and the member ConfigMap should still not churn, since re-linking changes only metadata. + - Added D5: a narrower re-create leaves an orphan behind (Q17). + - Added D6: a quick re-create, possible only since #1012. + - **Tweaked:** + - X1: the v1 safeguard matches owners by API group. + - R3 and decision 13: the wait is no longer needed for the snapshots. + - R7: the in-use semantics. + - Environment facts: the cross-experience safeguard, snapshot ownership, missing snapshots, and the strategy-only + path. + - Helpers, Left to ITs/UTs, and Phasing (a new phase 4). + - **Still blocked:** L1, L2, and L6 (Q10). `scheduling.go` still releases a leaving cluster's binding at once, and + neither merge touches the placement policy controller. L4 stays skipped, per the user. + - **To re-run on Kind,** since the merges changed their code paths: + - X1 and X3: the applier safeguard. + - The specs that switch a live policy's sync strategy (A1, A6, A8, B2): they now take the work generator's + strategy-only path. + - D2: adoption now moves the bindings to a new snapshot. +- 2026-10-26 22:32 — User decisions: Q16 stays open (X2 doesn't depend on it; X2b waits). Q17 takes the default: D5 + pins the orphan left behind by a narrower re-create as intended. + - **Not implemented yet:** + - Ready to implement: X2 and D2's snapshot part (P1); D5 and D6 (P2). + - Skipped per the user: L4. + - Blocked on Q10: L1, L2, and L6. + - Waiting on Q16: X2b. +- 2026-10-26 22:55 — Implemented X2, D2's snapshot part, D5, D6, and L4 as requested, and ran them on Kind together + with the existing specs whose code paths the merges changed. Setup was `setup.sh 3` with a temporary + `DOCKER_CONFIG`; the stale `img-builder` buildx builder had to be removed again first. + - **Where they live:** + - X2 is in [placement_policy_interop_test.go](../../../test/e2e/placement_policy_interop_test.go). Its new helpers + are `configMapPlacedByPlacementPolicyOnlyActual`, `namespaceTakenOverByOlderPlacementAPIsActual`, and + `crpRefusedConfigMapOnClusterActual`. The optional `ReportDiff` step is not implemented. + - D2, D5, and D6 are in [placement_policy_lifecycle_test.go](../../../test/e2e/placement_policy_lifecycle_test.go). + D2's stale comments are updated. + - L4 is in [placement_policy_join_leave_test.go](../../../test/e2e/placement_policy_join_leave_test.go), as a Serial + context. It adds `noPlacementBindingTargetsClusterActual`, `placementPolicyClaimUnfulfilledActual`, + `rejoinMemberClusterWithExtraLabels`, and `restoreMemberClusterInFleet`. The last one re-adds cluster-3 with its + usual labels if a spec fails midway. + - Shared actuals in [placement_policy_actuals_test.go](../../../test/e2e/placement_policy_actuals_test.go): + - `placementPolicyBindingControllersActual` generalizes the binding ownership check, so some bindings can be + adopted while others stay orphaned. + - `placementPolicyBindingsIgnoringOrphansActual` and `allSynchronizedAdoptedPlacementPolicyBindingsActual` + compare bindings while leaving out orphans the policy has not adopted. + - Also new: `orphanedPlacementBindingUnchangedActual`, `placementPolicySnapshotsOrphanedActual`, and + `placementPolicyWorksLinkedToSnapshotActual`. + - **Product bug found and fixed:** the work generator's strategy-only path (from + `feat/handle-incomplete-snapshots-in-work-generator`) merge-patches Works. The hub agent's ClusterRole had no + `patch` on `works`, so every sync-strategy-only change failed with `forbidden` and retried forever. The binding + status stayed at the previous generation, and a takeover never happened. The work generator ITs run on envtest + without RBAC, so they could not catch it. + - Fixed in [rbac.yaml](../../../charts/hub-agent/templates/rbac.yaml) by adding `patch` to the `works` rule, with + a comment, in the style of the other rules. + - Applied to the running hub with `helm upgrade --reuse-values`. The Works patch is the work generator's only + new verb; its other writes are covered by the existing rules. + - This change belongs with the work generator branch. + - **Validation:** + - gofmt, `go vet ./test/e2e/`, and `make lint` (0 issues). + - Dry run: the new specs select 32 specs (X2 7, D2 9, D5 7, D6 3, L4 6). + - **Focused run 1** (the new specs, plus the interop, deletion, sync-strategy, takeover, and suspension contexts): + 105 passed and 7 failed. + - The failures were exactly the specs that switch a live policy's sync strategy (A1, A2, A6, A8, A9, B2, X3), + all because of the RBAC bug above. + - All 32 new specs passed, including L4's leave and re-join. + - D6 went through the intended path: when the policy was re-created, 3 old bindings and 1 old snapshot still + named it. + - **Focused run 2** (after the RBAC fix): 125/125 (2m49s). + - **Full `placementpolicy` lane:** 442/442 (19m32s). That is the earlier 415 plus 27 net new specs (D2 grew from + 5 specs to 9). + - **Expectations confirmed on Kind:** + - D2: adoption moves the bindings to the new snapshot without touching the member ConfigMaps (resource versions + unchanged) or replacing the Works (same UIDs). + - D5: cluster-3's orphan stays at the same resource version after its snapshot is collected, and deleting it + by hand cleans up without the snapshot. + - X2: the CRP takes the namespace over and reports `OwnedByOtherKubeFleetAPIObjects` for the ConfigMap, with or + without `allowCoOwnership`. + - **Remaining:** + - L1, L2, and L6 (Q10). + - X2b (Q16). + - X2's optional `ReportDiff` step. diff --git a/.github/.copilot/breadcrumbs/2026-10-22-2300-admissionpolicymanager-integration-test-plan.md b/.github/.copilot/breadcrumbs/2026-10-22-2300-admissionpolicymanager-integration-test-plan.md new file mode 100644 index 000000000..dfd68b2ee --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-22-2300-admissionpolicymanager-integration-test-plan.md @@ -0,0 +1,233 @@ +# Admission Policy Manager Integration Test Plan + +**Date**: October 22, 2026 23:00 AEDT +**Task**: Add integration tests in `pkg/admissionpolicymanager` that verify the **effects** (not only the existence) of +the validating admission policies (VAPs) from all three built-in generators. See [Status](#status) for progress. + +## Requirements + +1. Verify, against a real API server (envtest), that each VAP denies exactly the requests it is meant to deny and + lets everything else through. +2. Follow repo conventions: Ginkgo/Gomega, `_integration_test.go`, one `Ordered` `Describe` per generator, + `want` naming, random name suffixes (`utils.RandStr()`), whole-struct `cmp.Diff` comparisons. +3. No production code changes, except for decisions 3–5. + +## Policies under test + +| Generator | Matches | Allowed when | Denial message | +|-----------|---------|--------------|----------------| +| `DenyPodsAndReplicaSetsOutsideReservedNamespaces` | CREATE `pods`, `replicasets` | namespace starts with `fleet-` or `kube-` (no exemptions for any caller) | `creating pods and replicas is disallowed in the fleet hub cluster` | +| `DenyServiceAccountsAndTokenRequestsInReservedNamespaces` | CREATE/UPDATE/DELETE `serviceaccounts`; CREATE `serviceaccounts/token` | namespace is not `fleet-*`/`kube-*`, **or** caller is `system:kube-scheduler`, `system:kube-controller-manager`, or in `system:nodes`, `system:masters`, `kubeadm:cluster-admins`, `system:serviceaccounts` (+ configured whitelists) | `writing service accounts in reserved namespaces or requesting tokens from such service accounts is disallowed` | +| `DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess` | CREATE/UPDATE on all resources and subresources (`*/*`); match conditions: the new object has `kubefleet.dev/cluster-selectors`, and the annotation is added or its value changed | caller can `create` **and** `update` `placementpolicies` in the object's namespace (namespaced objects) or `clusterplacementpolicies` (cluster-scoped objects) | `adding or changing the kubefleet.dev/cluster-selectors annotation on a namespaced/cluster-scoped object requires ...` | + +## Test environment findings and design implications + +Verified on October 22, 2026 with envtest 1.30.0, 1.33.0 (the CI version), and 1.37.0: all policies are enforced. + +1. **Enforcement lags policy creation by ~1s.** The VAP admission plugin recompiles policies on a timer + (`policyRefreshIntervalDefault = 1 * time.Second` in `k8s.io/apiserver/pkg/admission/plugin/policy/generic`); a + request sent right after `PolicyManager.Start` is not checked. This explains the outdated note at the end of + `manager_integration_test.go`, which will be removed. Each `Describe`'s `BeforeAll` waits (`Eventually`) until a + probe request that the policy must deny is denied. +2. **The envtest client is `system:masters`.** It has all permissions and is exempt from the service account policy. + Permission-dependent cases use impersonated identities (`rest.Config.Impersonate`). +3. **RBAC runs before admission, and an RBAC denial is also `Forbidden`.** Every test identity is granted RBAC for the + target resource, and denial assertions check `apierrors.IsForbidden` **plus** the policy name + (`ValidatingAdmissionPolicy ''`) and the validation message, so that an RBAC denial can never pass as a + policy denial. +4. **Top-level containers run in random order.** Each `Describe` is self-contained (own namespaces, identities, and + enforcement wait). +5. **No namespace controller, GC, or other controllers.** Namespaces cannot be deleted, so names are random and + namespaces are not cleaned up (as in the existing test). `AfterAll` deletes the cluster-scoped objects a `Describe` + creates (ClusterRoles, ClusterRoleBindings). Finalizer removal (D15) is done by hand. +6. **envtest disables the `ServiceAccount` admission plugin**, so pods need no service account; service account + token signing is configured, so TokenRequests work. `NodeRestriction` is not enabled, so impersonating a node + user is fine. +7. **One generator config per suite.** The policy manager garbage-collects policies outside its config. The suite + switches from `DefaultPolicyGeneratorConfigsForV0APIs` to the generators of `DefaultPolicyGeneratorConfigsForV1APIs` + (all three), with one whitelisted username and one whitelisted user group added to the service account generator + (decision 1). The suite builds its own `PolicyGeneratorConfigs` for this rather than mutating the shared default. +8. These tests run in CI through `make local-unit-test` (`go test ./pkg/...`), serially. + +## Changes + +- `suite_test.go`: use the V1 generators, with the service account generator configured with + `WhitelistedUsernames: ["whitelisted-user"]` and `WhitelistedUserGroups: ["whitelisted-group"]`; install the + `TestResource` CRD (`test/manifests/test_testresources_crd.yaml`) through `CRDInstallOptions.Paths` for the custom + resource cases (accessed as `unstructured.Unstructured`, so no scheme change). +- `manager_integration_test.go`: add the third policy (with its match conditions) and binding to the want lists; + remove the outdated note. +- `utils_integration_test.go` (new): helpers — impersonated client builder, RBAC grant helpers (Role/ClusterRole + + binding), `wantDeniedByPolicy(err, policyName, wantMsg) error`, `waitUntilPolicyIsEnforced(probe)`, object builders. +- `podsnreplicasets_integration_test.go`, `svcaccountsntokenreqs_integration_test.go`, + `annotatedforplacementobjects_integration_test.go` (new): one `Describe` each (below). +- `annotatedforplacementobjects.go` (decision 3): match `*/*` instead of `*`. +- `cmd/hubagent/main.go`, `cmd/hubagent/options/webhooks.go` (decision 4): pick the default configuration by + `--enable-placement-policy-apis`; update the `--admission-policy-manager-config` help text. +- `annotatedforplacementobjects.go`, `commons.go` (decision 5): guard the added-or-changed match condition with + `objectHasAnnotation`. +- `annotatedforplacementobjects.go`, `configs.go` (review nits, October 23, 2026): drop the stale TO-DO header; + document that the generator is enabled by setting it to an empty struct (`{}`), and disabled by an empty (null) + value. + +## Test cases + +### A. Policies and bindings (`manager_integration_test.go`, existing `Describe`) + +- A1. All three policies exist with the wanted spec (system defaults filled in), including the two match conditions of + the placement annotation policy, and the whitelisted username and group in the service account policy's expression. +- A2. All three bindings exist with the wanted spec. + +### B. Pods and ReplicaSets (`podsnreplicasets_integration_test.go`) + +Setup: namespaces `work-` (non-reserved), `fleet-` (reserved), `fleetwork-` (prefix boundary). All +requests are made as the envtest admin, which also shows that `system:masters` is not exempt. + +| ID | Request | Want | +|----|---------|------| +| B1 | create a Pod in `work-` | denied | +| B2 | create a ReplicaSet in `work-` | denied | +| B3 | create a Pod in `fleetwork-` (starts with `fleet` but not `fleet-`) | denied | +| B4 | create a Pod and a ReplicaSet in `fleet-` | allowed | +| B5 | create a Pod and a ReplicaSet in `kube-system` | allowed | +| B6 | create a Deployment in `work-` (kind not matched) | allowed | + +### C. Service accounts and token requests (`svcaccountsntokenreqs_integration_test.go`) + +Setup: namespaces `work-`, `fleet-` (plus the existing `kube-system`); a ClusterRole granting +`serviceaccounts` (get/create/update/delete) and `serviceaccounts/token` (create), bound to every identity below; +service accounts seeded by the admin in `fleet-` and `work-`. + +Identities: `plain` (no exemption); exempt — the admin (`system:masters`), `system:kube-scheduler`, +`system:kube-controller-manager`, a user in `system:nodes`, a user in `kubeadm:cluster-admins`, a user in +`system:serviceaccounts`; whitelisted (suite config) — `whitelisted-user`, a user in `whitelisted-group`; and +`whitelisted-user-2` (not whitelisted; checks that usernames are matched exactly, not by prefix). + +| ID | Request | Want | +|----|---------|------| +| C1 | `plain` creates a ServiceAccount in `fleet-` | denied | +| C2 | `plain` creates a ServiceAccount in `kube-system` | denied | +| C3 | `plain` updates the seeded ServiceAccount in `fleet-` | denied | +| C4 | `plain` deletes the seeded ServiceAccount in `fleet-` | denied | +| C5 | `plain` requests a token for the seeded ServiceAccount in `fleet-` | denied | +| C6 | `plain` creates, updates, and deletes a ServiceAccount in `work-` | allowed | +| C7 | `plain` requests a token for the seeded ServiceAccount in `work-` | allowed | +| C8–C13 | each exempt identity creates a ServiceAccount in `fleet-` and requests a token for it | allowed | +| C14 | `whitelisted-user` creates a ServiceAccount in `fleet-` and requests a token for it | allowed | +| C15 | a user in `whitelisted-group` creates a ServiceAccount in `fleet-` and requests a token for it | allowed | +| C16 | `whitelisted-user-2` creates a ServiceAccount in `fleet-` | denied | + +### D. Placement annotation (`annotatedforplacementobjects_integration_test.go`) + +Setup: namespaces `work-`, `other-`; a ClusterRole granting writes to the target objects (ConfigMaps, +TestResources, Namespaces and their `status`/`finalize` subresources, Deployments and their `status`/`scale` +subresources, ClusterRoles), bound to every identity below; objects seeded by the admin as needed. + +| Identity | Placement policy permissions | +|----------|------------------------------| +| `no-perm` | none | +| `pp-ns` | `placementpolicies` create + update in `work-` (Role) | +| `pp-ns-create-only` | `placementpolicies` create only in `work-` (Role) | +| `pp-cluster-wide` | `placementpolicies` create + update in all namespaces (ClusterRole); no `clusterplacementpolicies` | +| `cpp` | `clusterplacementpolicies` create + update; no `placementpolicies` | + +Namespaced objects (ConfigMaps, in `work-` unless noted): + +| ID | Request | Want | +|----|---------|------| +| D1 | `no-perm` creates a ConfigMap without the annotation | allowed | +| D2 | `no-perm` creates an annotated ConfigMap | denied | +| D3 | `pp-ns-create-only` creates an annotated ConfigMap (both verbs are required) | denied | +| D4 | `pp-ns` creates an annotated ConfigMap | allowed | +| D5 | `pp-ns` creates an annotated ConfigMap in `other-` (permissions are per namespace) | denied | +| D6 | `pp-cluster-wide` creates an annotated ConfigMap in `other-` | allowed | +| D7 | `cpp` creates an annotated ConfigMap (cluster permissions do not apply to namespaced objects) | denied | +| D8 | the admin creates an annotated ConfigMap | allowed | +| D9 | `no-perm` adds the annotation to an existing ConfigMap (Update) | denied | +| D10 | `no-perm` adds the annotation with a JSON merge patch (the `kubectl annotate` path) | denied | +| D11 | `no-perm` changes the annotation value | denied | +| D12 | `no-perm` updates data, labels, and other annotations of an annotated ConfigMap, leaving the annotation as is | allowed | +| D13 | `no-perm` removes the annotation | allowed | +| D14 | `pp-ns` adds, then changes the annotation | allowed | +| D15 | `no-perm` deletes an annotated ConfigMap that has a finalizer, then removes the finalizer (the GC scenario) | allowed; the object is gone | + +Namespaced custom resources (`TestResource`, untyped CEL path): + +| ID | Request | Want | +|----|---------|------| +| D16 | `no-perm` creates an annotated TestResource | denied | +| D17 | `pp-ns` creates an annotated TestResource | allowed | + +Cluster-scoped objects: + +| ID | Request | Want | +|----|---------|------| +| D18 | `no-perm` creates an annotated Namespace | denied (cluster-scoped message) | +| D19 | `pp-cluster-wide` creates an annotated Namespace (`request.namespace` is set for Namespaces, but they must still be checked against `clusterplacementpolicies`) | denied | +| D20 | `cpp` creates an annotated Namespace | allowed | +| D21 | `no-perm` adds the annotation to an existing Namespace; `cpp` does the same | denied; allowed | +| D22 | `no-perm` creates an annotated ClusterRole (no rules); `cpp` does the same | denied; allowed | + +Cluster-scoped custom resources need no case of their own: the policy has no logic specific to them, and the two +aspects involved are covered separately — the untyped CEL path by D16/D17, and scope detection +(`object.metadata.namespace`) by D18–D22. + +Subresources (decision 3; some built-in resources keep annotation changes sent to their `status`/`finalize` +subresources): + +| ID | Request | Want | +|----|---------|------| +| D23 | `no-perm` adds the annotation to a Deployment through `deployments/status` | denied | +| D24 | `no-perm` updates the status of an annotated Deployment, leaving the annotation as is | allowed | +| D25 | `no-perm` scales an annotated Deployment through `deployments/scale` (the subresource carries a `Scale` object) | allowed | +| D26 | `no-perm` adds the annotation to a Namespace through `namespaces/status`; `cpp` does the same | denied (cluster-scoped message); allowed | +| D27 | `no-perm` adds the annotation to a Namespace through `namespaces/finalize`; `cpp` does the same | denied (cluster-scoped message); allowed | + +## Not covered + +- Subresources of built-in resources other than Deployments and Namespaces. +- CEL evaluation errors and `failurePolicy: Fail`. + +## Decisions + +1. Whitelist coverage (October 22, 2026): the suite uses the V1 generators plus one whitelisted username and one + whitelisted user group for the service account generator (C14–C16), instead of the plain defaults. +2. File layout (October 22, 2026): one file per generator plus `utils_integration_test.go`. +3. Match subresources (October 23, 2026): review found that a caller without placement policy permissions could add + the annotation through `deployments/status`, `namespaces/status`, and `namespaces/finalize` (the built-in status + strategies of these resources do not reset annotations; custom resources do), which the policy did not match + (`*`). The policy now matches `*/*`; the added-or-changed match condition keeps routine status writes unchecked, + and subresources that carry other kinds of objects (e.g., `Scale`) are skipped by the has-annotation condition. +4. Default configuration wiring (October 23, 2026): when no admission policy manager config file is given, the hub + agent uses `DefaultPolicyGeneratorConfigsForV1APIs` if `--enable-placement-policy-apis` is set, and + `DefaultPolicyGeneratorConfigsForV0APIs` otherwise (`cmd/hubagent/main.go`); the + `--admission-policy-manager-config` help text is updated to match. A given config file still takes precedence. +5. Self-contained added-or-changed match condition (October 23, 2026): on annotation removal, the condition indexed + the missing key in `object.metadata.annotations` and failed with a no-such-key error. This had no visible effect, + as the API server stops at the first match condition that is false (has-annotation, listed first) before it + looks at errors from later ones; with the two conditions swapped, each removal was counted in + `apiserver_admission_match_condition_evaluation_errors_total`. The condition now also checks that the object + carries the annotation (`objectHasAnnotation`); with the conditions swapped, no errors are counted. D13 covers + the behavior; A1 is updated with the new expression. + +## Status + +- [x] Plan drafted; decisions 1 and 2 recorded +- [x] Plan approved (October 22, 2026) +- [x] Suite and existing test updates (A) +- [x] Helpers +- [x] Pods and ReplicaSets (B): B1–B6 implemented as 6 specs +- [x] Service accounts and token requests (C): C1–C16 implemented as 16 specs (C8–C15 as a `DescribeTable`; the + `system:masters` entry impersonates a member of the group rather than using the envtest admin client) +- [x] Placement annotation (D): D1–D22 implemented as 22 specs +- [x] Subresources (D23–D27, decision 3; October 23, 2026): policy switched to `*/*`; D23–D27 implemented as 5 specs + in a new `Context` (D26/D27 as a `DescribeTable`), with the `getAndUpdateSubResource` helper +- [x] `go test -race ./pkg/admissionpolicymanager/...` passes (envtest 1.33.0; 51/51 specs, randomized runs); no + new lint issues. Mutation checks (October 22–23, 2026) — each breaks the policy in one way, and the expected + spec fails: + - D: deciding scope from `request.namespace` (D18), checking only the `create` verb (D3), dropping the + added-or-changed match condition (D12), matching `*` instead of `*/*` (D23, D26, D27, A1; D24/D25 pass either + way); + - B: matching reserved prefixes without the trailing hyphen (B3), not matching `replicasets` (B2); + - C: ignoring whitelisted usernames (C14), matching usernames by prefix (C16), not matching `DELETE` (C4), not + matching `serviceaccounts/token` (C5). diff --git a/.github/.copilot/breadcrumbs/2026-10-26-1748-workgenerator-integration-test-plan.md b/.github/.copilot/breadcrumbs/2026-10-26-1748-workgenerator-integration-test-plan.md new file mode 100644 index 000000000..42fcd2083 --- /dev/null +++ b/.github/.copilot/breadcrumbs/2026-10-26-1748-workgenerator-integration-test-plan.md @@ -0,0 +1,548 @@ +# Work Generator (v1) Integration Test Plan + +**Date**: October 26, 2026 17:48 AEDT (revised 19:33 AEDT for the latest code changes, 21:41 AEDT for the +`Available` condition fix, and October 27, 02:58 AEDT for the `IncompleteSetOfResources` message) +**Task**: Plan (not yet implement) the tests for `pkg/v1/controllers/workgenerator`, based on the current code base, +including the uncommitted changes in [controller.go](../../../pkg/v1/controllers/workgenerator/controller.go), +[retrieval.go](../../../pkg/v1/controllers/workgenerator/retrieval.go), +[status.go](../../../pkg/v1/controllers/workgenerator/status.go), +[works.go](../../../pkg/v1/controllers/workgenerator/works.go), and +[placementbinding_types.go](../../../apis/kubefleet.dev/placement/v1alpha1/placementbinding_types.go): + +- invalid placement resource snapshots are surfaced on the binding (`Synchronized=False/InvalidSnapshots`, plus + `Available=False/IncompleteSetOfResources` when the binding still points to the last processed snapshot or nothing + has been processed; after a rollout to a different snapshot the previous `Available` condition, necessarily from an + older generation, is kept), with the resource counters recomputed from the works already placed; +- sync-strategy-only changes are patched onto the works without reading the snapshots (the new strategy-only path); +- the retrieval checks are reworked (duplicates, gaps, and missing snapshots are all `User` errors); +- the binding spec name fields require a minimum length of 1. + +Cross-reference the existing tests, and identify the scenarios to add, update, and remove. The old work generator ITs +(`pkg/controllers/workgenerator`) are used as a reference only (see [Old tests → new plan](#old-tests--new-plan)). + +**Status**: Implemented (October 26, 2026), apart from the deferred unit tests U1 and U6. All [Decisions](#decisions) +are resolved. Done: X5 and X6 (fixing the six existing specs that the code changes broke; see [Baseline](#baseline)), +the S, L, and R sections, unit tests U2–U5 and U7, the rate limiter option (Decision 8), and the CI wiring +(Decision 5). + +## Requirements + +1. Cover the new invalid-snapshot handling (the `InvalidSnapshots` and `IncompleteSetOfResources` reasons, duplicate, + gap, and missing-snapshot detection, the recount of resources, the no-op write skip) for both scopes. +2. Cover the strategy-only path, and update the existing contexts that the code changes broke. +3. Fill the remaining behavior gaps found in the survey; remove or fold redundant contexts only with approval. +4. Follow repo conventions: Ginkgo/Gomega, `_integration_test.go`, one `Ordered` `Context` per scenario + (`BeforeAll`/`AfterAll`, teardown in `AfterAll`), random names, whole-struct `cmp.Diff`, `want` naming, the existing + `*Actual` helper style, and both scopes through the `scopes` loop and `bindingFixture` + ([lifecycle_integration_test.go](../../../pkg/v1/controllers/workgenerator/lifecycle_integration_test.go)). +5. Always run with `ginkgo -p` (see [Running the tests](#running-the-tests)). +6. Put new IT helpers in [utils_integration_test.go](../../../pkg/v1/controllers/workgenerator/utils_integration_test.go); + scenario files hold only `Describe`/`Context` bodies. + +## Baseline + +```bash +KUBEBUILDER_ASSETS="$(make -s kubebuilder-assets-path | tail -1)" \ + go run github.com/onsi/ginkgo/v2/ginkgo -p --race ./pkg/v1/controllers/workgenerator +``` + +- Before the 19:33 changes (October 26, 2026): 452 of 452 specs pass; 74.6s of spec time, 1m25s wall. +- On the current code (October 26, 2026, after the 19:33 changes): 404 of 452 run; 398 pass, **6 fail**, 48 are + skipped (the rest of the failing `Ordered` contexts); 75.3s of spec time, 1m27s wall. All six failures are expected + consequences of the code changes: + - `handle inconsistent resource snapshots`, both scopes, the first `should not update the … binding status or the + work objects`: a missing secondary is now a `User` error, so the binding is marked `InvalidSnapshots` (X5). + - `changing sync strategies`, 1 and 2 snapshots × both scopes, `should refresh the placement binding status`: the spec + expects `WaitingForSync`, but the strategy-only path leaves the binding status as it was (X6). + +## Controller behavior under test + +Watches ([controller.go](../../../pkg/v1/controllers/workgenerator/controller.go)): `PlacementBinding` and +`ClusterPlacementBinding` with `GenerationChangedPredicate` (status-only updates are not events); `Work` create, delete, +and **status-changing** updates, mapped to the owner binding through the `placement.kubefleet.dev/owner-namespace` label +and the owned-by-binding annotation (events missing either are dropped and logged). Placement resource snapshots are +**not** watched. + +Reconcile flow: + +1. Get the binding (`ClusterPlacementBinding` if the request has no namespace); not found → done. +2. Deleting or suspended → `cleanupWorks` ([cleanup.go](../../../pkg/v1/controllers/workgenerator/cleanup.go)): no + cleanup finalizer → done. Otherwise delete the **primary** work only (secondaries are left to K8s GC through their + controller owner reference to the primary); if suspended and not deleting, unset + `LastProcessedResourceSnapshotName`; remove the finalizer. +3. Add the cleanup finalizer (`placement.kubefleet.dev/work-generator-cleanup`). +4. Sanity check: an empty `clusterName` or `resourceSnapshotName` → `Unexpected` error. Unreachable through the API, as + the CRD requires a minimum length of 1 for `placementPolicyName`, `clusterName`, and `resourceSnapshotName`. +5. List the works in `fleet-member-` through the owner-binding field index + (`/`). +6. `areWorksInSync` ([works.go](../../../pkg/v1/controllers/workgenerator/works.go)) returns one of: + - `ResourcesOutOfSync`: the last processed snapshot differs from the spec snapshot, there are no works, a work is not + linked to the spec snapshot, or the number of works differs from the linked work count on the primary; + - `StrategyOutOfSync`: all of the above hold, but some work's sync strategy differs from the binding's; + - `InSync`: everything matches; + - an `Unexpected` error: multiple works carry the linked work count, or it is invalid. + + Then: + - `InSync` → `refreshPlacementBindingStatus` (step 12) → done. + - `StrategyOutOfSync` → `refreshSyncStrategyOnlyForWorks`: for every work whose strategy differs, a merge patch of + `spec.syncStrategy` only (optimistic lock on the cached copy), in parallel. The snapshots are not read, and the + binding status is not touched: the conditions keep the previous generation and the counters their previous + values. The patch bumps the works' generations, so the member agent re-applies them, and the resulting work status + events lead to the in-sync refresh, which writes the conditions for the new generation. → done. + - `ResourcesOutOfSync` → step 7. +7. Retrieve the snapshots ([retrieval.go](../../../pkg/v1/controllers/workgenerator/retrieval.go)), cached reads: + - Get the primary (not found → an API server error that `IsNotFound`); parse the count label (non-integer or < 1 → + `Unexpected`); count 1 → return the primary alone (no listing). + - Otherwise list by the owned-by-annotation + index field index (missing annotation/label on the primary → + `Unexpected`), and sort by sub-index (malformed label → `Unexpected`). + - Check the sub-indices of the first `count + 1` snapshots: one equal to the previous one → `User` ("found placement + resource snapshots with duplicate sub-indices…"), which also catches a stray right after the counted ones that + duplicates the last of them; a gap among the first `count` → `User` ("found placement resource snapshots with + inconsistent sub-indices…"). Snapshots further down are not checked. + - Fewer snapshots than the count → `User` ("there are fewer placement resource snapshots than the count label + indicates; some snapshots might have been deleted…"). + - A contents-hash mismatch among the first `count` → `Unexpected` (not surfaced on the binding). + - Return the first `count`. +8. On a not-found or `User` error from step 7, `markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots` + ([status.go](../../../pkg/v1/controllers/workgenerator/status.go)): + - `Synchronized=False/InvalidSnapshots` for the current generation, with the message `Failed to synchronize resources + as the given placement resource snapshots are invalid: `; + - if the last processed snapshot is the spec snapshot (or nothing has been processed): + `Available=False/IncompleteSetOfResources` for the current generation, with the message `Cannot check the + availability of resources as the given placement resource snapshots are invalid; some resources might be missing + from the target cluster`; otherwise (a rollout to a different snapshot) `Available` is left as is, which is safe as + it can only carry an older generation (or be absent); + - the counters and failed resources are recomputed from the works linked to the last processed snapshot (none → + 0/0/0 and no failed resources; failed resources capped at 50); + - `LastProcessedResourceSnapshotName` is untouched; the write is skipped if the whole status is unchanged. + + Either way the error is returned, so the controller retries with backoff (the only re-trigger, as snapshots are not + watched). A cache-lag "not found" right after a rollout is accepted; it self-heals on retry. +9. `refreshWorks`: build the primary work and one work per secondary snapshot (names from the policy's namespaced name, + secondaries carry the sub-index; owner labels hashed when over 61 characters or dotted; annotations for the linked + primary snapshot, `placement-resource-snapshot/`, owned-by policy/binding; manifests; sync strategy), set the + linked work count on the primary, delete dangling works first, create/update the primary, set it as the controller + owner of the secondaries, then create/update the secondaries in parallel. Reports whether any work was created or had + its spec changed (metadata-only changes do not count). +10. `reportPlacementBindingProcessingProgress`: `LastProcessedResourceSnapshotName` = primary, + `Synchronized=False/WaitingForSynchronization`, `Available=Unknown/WaitingForAvailabilityCheck`, `SelectedResources` + = manifest total, the other counters and `FailedResources` cleared. +11. No spec change → requeue after 1s (so a same-content rollout refreshes the status without member re-processing); + otherwise wait for work events. +12. `refreshPlacementBindingStatus`: skip (no write) if any work's `Applied`, then `Available`, condition is missing or + stale; `Synchronized` = `True/AllResourcesSynchronized` or `False/FailedToSynchronizeSomeResources`; `Available` = + `True/AllResourcesAvailable` or `False/SomeResourcesUnavailable`; counters from the manifest conditions; failed + resources (`Applied≠True`, or applied with `Available≠True`) carry `DiffDetails` and the false condition, are sorted + by group/version/kind/namespace/name, and are capped at `maxFailedResourcesToReport` (50); the write is skipped + when nothing changed. + +## What the existing tests already cover + +20 contexts in [core_ops_integration_test.go](../../../pkg/v1/controllers/workgenerator/core_ops_integration_test.go) +(four `Describe`s: namespace-/cluster-scoped × single/multiple snapshots, written out per scope) and 22 in +[lifecycle_integration_test.go](../../../pkg/v1/controllers/workgenerator/lifecycle_integration_test.go) (11 per scope, +through `scopes`). + +| File / Context | Behaviors | +| --- | --- | +| core: single, `generate works` | Finalizer, exact primary work, `WaitingForSync` → mark → synced; cleanup on delete | +| core: single, `update works` | Rollout to a new snapshot → work updated; statuses | +| core: single, `refresh binding status upon work status changes` | In-sync path: a synced binding turns `Available=False/SomeResourcesUnavailable` with failed resources | +| core: single & multiple, `update works (no spec changes)` | Same-content rollout: metadata-only re-link in place; status refreshed through the 1s requeue without member re-processing | +| core: single & multiple, `backport status for failed manifests` | Failed-to-apply (with `DiffDetails`) and unavailable manifests → `FailedResources`, counters, `FailedToSynchronizeSomeResources`; multiple works merged | +| core: multiple, `generate works` | Primary + secondaries, owner refs, linked count; status not refreshed until every work is processed | +| core: multiple, `update works` | Scale down (dangling work deleted), in-place updates (same UIDs), scale up (created) | +| core: multiple, `handle inconsistent resource snapshots` | Rewritten (X5, done): missing secondaries → `InvalidSnapshots` ("fewer…"), the pre-rollout `Available` condition kept, counters from the last processed works, works unchanged; a hash mismatch afterwards leaves the status (and message) as is; recovery through retries once the secondaries appear | +| lifecycle: `deleting placement bindings` | Single (another binding on member-2 unaffected); multiple (primary deleted, secondaries left to GC) | +| lifecycle: `suspending placement bindings` | Suspend/resume (works removed, last processed snapshot unset, finalizer removed, no works while suspended, a rollout made while suspended applied on resume); unprocessed works; delete while suspended | +| lifecycle: `changing sync strategies` | Updated (X6, done): set/unset with 1 and 2 snapshots → works patched in place; the binding status left untouched (previous generation) until the works are marked, then synced | +| lifecycle: `handling work object events` | Foreign works (3 kinds) ignored; out-of-band deletion of the primary, a secondary, or the only work → re-created, others untouched | +| unit: [works_test.go](../../../pkg/v1/controllers/workgenerator/works_test.go) | `workOwnerLabelValue` (5 + distinctness); `areWorksInSync` (15 cases, every branch; updated for `ResourcesOutOfSync` / `StrategyOutOfSync`) | +| unit: [uniquename_test.go](../../../pkg/v1/controllers/workgenerator/uniquename_test.go) | Work names (12 cases), uniqueness, the derived-from formatter | + +**Not covered at all**: the invalid-snapshot marking (not found, duplicates, gaps, missing snapshots), the +`IncompleteSetOfResources` reason, the recount of resources in the invalid state, the no-op write skip, duplicates +further down the listing being ignored, invalid snapshots on new bindings / with out-of-band work deletion / with +suspension, the strategy-only path with missing snapshots and combined with a rollout, long and dotted names end to +end, one policy on multiple clusters, secondary works lingering across binding incarnations, a missing reserved +namespace, `Applied`-without-`Available` gating, status-write skipping in the in-sync refresh, work metadata tampering, +the 50-entry cap, and the failed-resource **order** (the status helpers sort before comparing). + +## Test environment constraints and design implications + +1. **No GC controller.** Deleting a binding (or its primary work) leaves the secondary works behind; `teardown` deletes + them explicitly. L3 relies on this to model GC lag. +2. **Snapshots are not watched.** Recovery from an invalid snapshot set (other than through a binding spec change) + relies on error retries. In production these follow controller-runtime's default limiter (per item, 5ms doubling + up to 1000s, reset only on success), so the next retry comes after roughly as long as the binding has been failing. + The suite instead runs the controller with a capped limiter (Decision 8; [Suite changes](#suite-changes) item 2), so + a retry is at most ~1s away and recovery fits in `eventuallyDuration`. Retries also run about once a second during + every `Consistently` window, which makes the "status unchanged while retrying" checks meaningful. +3. **Shared cache.** The tests and the controller share the manager's cached client, so a binding created right after + its snapshot can briefly observe "not found" and get `InvalidSnapshots` before self-healing (B1). `Eventually` + tolerates it; do not start a `Consistently` on a binding's status right after creating it. +4. **Condition messages are ignored** by `placementBindingStatusUpdatedActual` / `placementBindingStatusMatchesActual` + (`ignoreFieldConditionLTTMsg`); the invalid-snapshot message needs its own check. +5. **Mixed generations.** Two cases leave conditions at different generations: the strategy-only path (every condition + keeps the previous generation until the works are re-processed), and an invalid mark after a rollout to a different + snapshot (`Available` kept at the pre-rollout generation). `placementBindingStatusUpdatedActual` stamps every + condition with the current generation, so these need `placementBindingStatusMatchesActual` with explicit + generations. +6. **Strategy patches bump work generations.** `markWorkAsAppliedAndAvailable` stamps the work's current generation, so + marking the works after a strategy patch is what lets the in-sync refresh run (X6, S7). +7. **Snapshot specs are immutable** (CEL); "re-create" means delete, wait until gone, create again. +8. **Label selectors in helpers use raw binding names** (`worksGeneratedActual`, `listWorkUIDs`); they are wrong for + names that get hashed (over 61 characters or dotted). L2 needs them to use `workOwnerLabelValue`. +9. **Namespaces cannot be deleted in envtest.** L4 uses a fresh random cluster name and creates its reserved namespace + mid-test. +10. **The CRD rejects empty strings** for `placementPolicyName`, `clusterName`, and `resourceSnapshotName` + (`minLength: 1`), so step 4's sanity check cannot be reached in envtest. + +## Suite changes + +1. **CI wiring** (Decision 5, implemented October 26, 2026). Previously the package ran under `go test` in + `make local-unit-test` (no `-p`), and the ~30 new contexts make a serial run noticeably slower. It follows the + `stagedupdaterun` / v1 work applier pattern: + - a `TestMain` in [suite_test.go](../../../pkg/v1/controllers/workgenerator/suite_test.go) that exits early when + `KUBEFLEET_CI_TEST_RUNNER_NAME=default` (the `make test` step), and runs the suite otherwise; + - a dedicated step in `.github/workflows/ci.yml`, after the v1 work applier step: + `ginkgo -v -p --procs=4 --race --cover -coverprofile=work-generator-v1-it-coverage.out + ./pkg/v1/controllers/workgenerator/`, with `KUBEFLEET_CI_TEST_RUNNER_NAME: 'ginkgo'` (the Codecov step picks up + the profile, as it searches for all coverage files). + + The unit tests in the package then run in the Ginkgo step as well, as `TestMain` gates the whole test binary. +2. **Rate limiter option** (Decision 8, implemented October 26, 2026; a small production change). + - Product: `New` in [controller.go](../../../pkg/v1/controllers/workgenerator/controller.go) takes an extra + `workqueue.TypedRateLimiter[reconcile.Request]` argument, stored on the `Reconciler` and passed to + `controller.Options.RateLimiter` in `SetupWithManager`. Nil keeps controller-runtime's default (with the default + priority queue, an exponential 5ms → 1000s limiter), following the v1 work applier's constructor, which also takes + optional limiters with nil meaning the default. The only production caller, + [setup.go](../../../cmd/hubagent/workload/setup.go), passes nil, so production behavior does not change; the unit + tests pass nil too. + - Suite: [suite_test.go](../../../pkg/v1/controllers/workgenerator/suite_test.go) passes + `workqueue.NewTypedItemExponentialFailureRateLimiter[reconcile.Request](requeueBaseDelay, requeueMaxDelay)` (5ms, + 1s), with a comment explaining why (no snapshot or namespace watch; recovery depends on retries). + - Existing tests: every `eventuallyDurationForBackoffRetries` (the two `handle inconsistent resource snapshots` + contexts, the S section, and L4) is now `eventuallyDuration`, and the constant is deleted. + +## Shared test utilities + +The helpers live in [utils_integration_test.go](../../../pkg/v1/controllers/workgenerator/utils_integration_test.go). + +- **Done (October 26, 2026):** moved `scope`, `scopes`, `snapshotSet`, and `bindingFixture` here from + `lifecycle_integration_test.go` (Requirement 6; Decision 2). +- **Done (October 26, 2026), used by X5/X6 and the S section:** + - `snapshotSet` now records its index and resources; `newSnapshotSet(index, count)` prepares a set without creating + it, and `createSnapshotsInSet(set, subIdxs...)` creates (or re-creates) some or all of its snapshots in descending + sub-index order; `createSnapshots(index, count)` does both. Together they cover gaps, late snapshots, and missing + snapshots (the plan's former `createSnapshotsWithSubIdxs`). + - `deleteSnapshot(setIdx, subIdx)`; `createStraySnapshot(index, subIdx, count, contentsHash)` / + `deleteStraySnapshot(name)`: extra snapshots under `%s-snapshot-%d-%d-stray-%s`, cleaned up by `teardown`. + - `recordStatus()` (status and resource version), `statusUnchangedActual()`, `resourceVersionUnchangedActual()`, + `recordWorkUIDs()` + `expectWorksNotRecreated()`. `registerBaselineSpecs` records the status once the baseline + is synced. + - `invalidSnapshotsStatusActual(wantStatus, wantMsgSubstr)`: the whole status (conditions without an explicit + observed generation are stamped with the current one; explicit ones are kept) plus the message prefix and details; + `wantPlacementBindingStatusInvalidSnapshots(placed, lastProcessed, previousStatus)` builds the common expected + status: with a previous status (a rollout to a different snapshot), its `Available` condition (generation + included) is kept; with nil, `Available=False/IncompleteSetOfResources` is expected. It is adjusted in place for + the other cases (no last processed snapshot, failed resources). + - `worksMatchBindingStrategyActual(setIdx)`: works carry the binding's sync strategy as written to the API server. + - `updateSpec(mutate)` / `updatePlacementBindingSpec(ns, name, mutate)`: several spec fields in one update (S7b). + - `placementBindingSyncCondMessageActual(ns, name, wantSubstr)`. + - Constants: `invalidSnapshotsMsgPrefix`, `fewerSnapshotsErrMsg`, `duplicateSubIdxErrMsg`, `inconsistentSubIdxErrMsg`, + `snapshotNotFoundErrMsgFmt` (`%q not found`), `straySnapshotNameTemplate`. +- **Done (October 26, 2026), used by the L and R sections:** + - `newBindingFixtureWithNames(sc, clusterName, bindingName, policyName)`; `f.newFixtureOfSamePolicy(clusterName)`: + another binding of the same policy that shares the fixture's snapshot sets (L1); `longDottedName(prefix, length)` + (L2). + - `f.createSnapshotWithResources(index, resources, identifiers)` (R4); `snapshotSet.resourceCount()` now sums the + identifiers instead of assuming three resources per snapshot. + - `f.registerWorkGenerationSpecs(setIdx)`: the first half of the baseline (up to `WaitingForSync`), which + `registerBaselineSpecs` now builds on (R1, R4). + - `markWorkAsApplied` / `f.markWorksAsApplied` (R1); `touchWorkStatus` (R2, R3); `setWorkAnnotation`, + `workAnnotationActual`, `f.workGeneration`, `f.syncCondTransitionedSinceRecordActual` (R3); + `createReservedNamespace` (L4); `placementBindingStatusInOrderActual` (R4: failed resources compared in order). + - `worksGeneratedActual`, `listWorkUIDs`, and `wantWorkForPlacementResourceSnapshot` use + `workOwnerLabelValue(...)` for the owner labels; the snapshot helper's owner label uses `snapshotOwnerLabelValue`, + which mirrors the snapshot manager's truncation (names over 61 characters), so that long policy names are valid + label values. No change for the short names used elsewhere. + - Constants: `paddedConfigMapNameTemplate`, `markedAsAppliedManifestMsg`, `markedAsAppliedWorkMsg`. + +## Test plan + +Priorities: **P0** = the new behavior, and the fixes for the broken existing contexts; **P1** = important edge/safety +behavior; **P2** = nice to have. Every row runs for both scopes unless it says otherwise; every context tears down in +`AfterAll`. + +### S. `invalid_snapshots_integration_test.go`: invalid placement resource snapshots (new) + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| S1 | P0 | roll out to a missing snapshot | Baseline on set 0 (1 snapshot) synced; record status; point the binding to set 1's primary name before set 1 exists; mark the Deployment in set 0's work unavailable; then create set 1 (1 snapshot) | `Synchronized=False/InvalidSnapshots` at the new generation, and the baseline `Available` condition kept (previous generation), message = prefix + the not-found error naming the snapshot; counters 3/3/3 recomputed from set 0's works; last processed still set 0; works unchanged and binding resource version stable over `consistentlyDuration` while the controller retries (no-op skip). After the Deployment turns unavailable: counters 3/3/2 with the Deployment in `FailedResources`, conditions unchanged (the recount follows the works). After set 1 appears: primary work updated in place (same UID), `WaitingForSync(set 1)` → mark → synced | +| S2 | P0 | duplicate of the last counted sub-index; the message follows the cause | Baseline on set 0 (2 snapshots); point to set 1's primary (missing); create set 1 (2 snapshots) plus a stray at sub-index 1 with the **same** contents hash (the case the hash check cannot catch); delete the stray | Not-found message first; then the message changes to the duplicate error (one more status write, then stable); works still from set 0 (`Consistently`); after deleting the stray: works updated in place to set 1, `WaitingForSync` → mark → synced. Variant (P2): the stray has a different hash | +| S3 | P1 | duplicates further down the listing are ignored | Baseline on set 0 (1); create set 1 (count 2) plus two extras at sub-index 2 (sorted after the snapshot right after the counted ones); roll out | No `InvalidSnapshots`; exactly 2 works (no work for sub-index 2); `WaitingForSync` → mark → synced. Pins the `count + 1` limit (B11) | +| S4 | P1 | duplicate primary at the same index | Baseline on set 0 (1); create set 1 (count 2) plus a stray sub-index 0 at index 1; roll out; delete the stray | Duplicate message; recovers to set 1 | +| S5 | P1 | gap in the counted sub-indices | Baseline on set 0 (1); create set 1 with count 3 and sub-indices 0, 2, 3; roll out; then create sub-index 1 | `inconsistent sub-indices` message; after the fix: 3 works (sub-indices 0–2), sub-index 3 ignored as an extra. Gaps do not occur in normal ops; this pins the sanity check | +| S6 | P1 | new binding with a missing snapshot | Create the binding pointing to a nonexistent primary; then create the snapshot | Finalizer added; `Synchronized=False/InvalidSnapshots` and `Available=False/IncompleteSetOfResources` at generation 1; counters 0/0/0, no failed resources, no last processed; no works (`Consistently`); after the snapshot appears: works generated, `WaitingForSync` | +| S7 | P1 | sync strategy change while the processed snapshot is missing (strategy-only path) | Baseline on set 0 (2); delete set 0's primary (premature GC); set a sync strategy; mark the works | Works patched in place with the new strategy (same UIDs, manifests and links unchanged) without reading the snapshots; no `InvalidSnapshots`; the binding status unchanged (conditions at the previous generation, counters 6/6/6; `Consistently`); after marking: synced at the new generation. Pins the resolution of B5 (Decision 7) | +| S7b | P2 | sync strategy change together with a rollout to a missing snapshot | Baseline on set 0 (1); in one update, set a sync strategy and point to set 1 (missing); then create set 1 | Resource path: `InvalidSnapshots` at the new generation, the baseline `Available` condition kept (previous generation); works keep the old (nil) strategy (`Consistently`); after set 1 appears: works updated with set 1 and the new strategy, `WaitingForSync` → mark → synced | +| S8 | P2 | works deleted out of band while the processed snapshot is invalid | Baseline on set 0 (2); delete set 0's primary; delete the secondary work; re-create the primary | `InvalidSnapshots` and `Available=False/IncompleteSetOfResources` at the **same** generation (B4); counters recomputed from the remaining primary work (3/3/3); the secondary work not re-created (`Consistently`); after the re-creation: re-created, `WaitingForSync` → mark → synced | +| S9 | P2 | suspend, resume, and delete while invalid | Binding in S1's invalid state; suspend; resume; delete | Suspend: works removed, last processed unset, finalizer removed, conditions left from before (older generation). Resume: finalizer back, `InvalidSnapshots` + `IncompleteSetOfResources` at the new generation, counters 0/0/0, no works. Delete: binding removed (cleanup does not read snapshots) | + +Missing secondaries ("fewer than count") and a hash mismatch after an invalid mark are covered by the rewritten +`handle inconsistent resource snapshots` contexts (X5). + +### L. `lifecycle_integration_test.go`: topology, naming, and lifecycle gaps + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| L1 | P1 | one policy on two clusters | Two bindings of the same policy (member-1, member-2) on the same set (2 snapshots); mark works on member-1 only; roll out set 1 to binding 1 only; delete binding 1 | Same work names in both reserved namespaces; only binding 1 synced after marking; binding 2's works and status untouched by binding 1's rollout and deletion | +| L2 | P1 | long and dotted names | Policy and binding names over 61 characters with dots; 2 snapshots; baseline; rollout; delete | Work names truncated/dot-free as `uniqueNameForWorkDerivedFromPlacementResourceSnapshot` computes; owner labels hashed; the in-sync path works (listing through the field index, event mapping through the annotation); rollout in place; primary work removed on delete | +| L3 | P2 | binding re-created while old secondary works linger | Baseline on set 0 (3 snapshots); delete the binding (secondaries linger, no GC); re-create it (same name) on set 1 (2 snapshots) | Sub-index 1 work updated in place (same UID) with the new primary as controller owner; sub-index 2 work deleted as dangling; linked count 2; synced after marking | +| L4 | P2 | reserved namespace of the target cluster missing | Binding to `member-` with no reserved namespace; then create `fleet-member-` (one scope is enough) | Finalizer added; no works and no status (`Consistently`); after the namespace appears: works generated (within `eventuallyDuration`, thanks to the suite's capped rate limiter, as the work generator does not watch namespaces), `WaitingForSync` → mark → synced | +| L5 | — | ~~empty `resourceSnapshotName` / `clusterName`~~ | Dropped (October 26, 2026): the CRD now rejects empty values (Decision 7) | — | + +### R. `status_integration_test.go`: status refresh gaps + +| ID | P | Context | Setup → steps | Assert | +| --- | --- | --- | --- | --- | +| R1 | P2 | applied, availability not yet reported | Baseline up to `WaitingForSync`; mark works `Applied` only; then mark them available | Status stays `WaitingForSync` (`Consistently`); synced after the availability update | +| R2 | P2 | no-op refresh skips the write | Synced binding; update a work's status without changing the outcome (e.g., a manifest condition message) | Binding resource version unchanged (`Consistently`). Note: this pins the observable behavior only; the API server also treats an identical status update as a no-op (no resource version bump), so the test cannot tell the client-side skip apart (verified by a mutation run, October 26, 2026). U4 covers the skip itself (its "status unchanged" case fails when the skip is removed) | +| R3 | P2 | work metadata tampered | Synced binding; change the primary work's linked-snapshot annotation; later touch the work's status | Not reconciled while only metadata changes (`Consistently`; B7); after the status touch: re-linked in place, status goes `WaitingForSync` then synced (1s requeue), no spec change | +| R4 | P2 | more than 50 failed resources | One snapshot with 51 ConfigMaps (one scope); mark all failed to apply | `FailedResources` has 50 entries in the documented order (compare **without** `SortSlices`); counters 51/0/0. Could be U4 instead | + +### Existing tests: update, keep, remove, or fold + +| ID | Existing | Proposal | Reason | +| --- | --- | --- | --- | +| X5 | core `handle inconsistent resource snapshots`, both scopes | **Rewrite (P0; done October 26, 2026)** | Index 1 without its secondary: now `InvalidSnapshots` with the "fewer … deleted" message, the pre-rollout `Available` condition kept (its generation included), counters 3/3/3 from index 0's works, works unchanged. Index 1's secondary with a mismatched hash: `Unexpected`, not surfaced, so the status stays exactly as in the previous step (the "fewer" message remains though the cause changed; B2). Index 2 without its secondaries: `InvalidSnapshots` again, at the newer generation. Recovery once the secondaries appear: unchanged; now within `eventuallyDuration` (Decision 8; it took ~15s per scope on the 1-minute budget before) | +| X6 | lifecycle `changing sync strategies`, 1 and 2 snapshots × both scopes | **Update (P0; done October 26, 2026)** | After setting (and after unsetting) the strategy: works patched in place (already asserted); the binding status stays as recorded before the change, with conditions at the previous generation (`Consistently`), instead of `WaitingForSync`; after marking the works: synced at the new generation | +| X1 | core `generate works` (single snapshot), both scopes | Keep (Decision 3) | Overlaps `registerBaselineSpecs` in the single-snapshot lifecycle contexts, but kept | +| X2 | core cluster-scoped `Describe`s (10 contexts, ~2,100 lines) | Keep as is (Decision 4) | They mirror the namespace-scoped ones; folding them into the `scopes` loop is not planned | +| X3 | core `refresh binding status upon work status changes` | Keep | Only test of a synced → unavailable transition through the in-sync path (backport starts from `WaitingForSync`) | +| X4 | core single `update works (no spec changes)` | Keep | Exercises the count-1 fast path; the multiple-snapshot one exercises the listing | + +Apart from X5 and X6, no existing test is obsolete. + +### Unit tests (Decision 6) + +U2–U5 and U7 are implemented (October 26, 2026); U1 and U6 are deferred. The unit tests use the controller-runtime +fake client, with interceptors that count status updates and patches (and inject errors), and share builders in +`status_test.go` (`newCondition`, `newManifestStatus`, `newWorkWithStatus`, `workConds`, `newFailedResource`, +`newBindingWithStatus`, `newUnitTestHubClient`, `countStatusUpdates`, `failedToApplyWorkWithManyManifests`). They also +reuse a few pure helpers from `utils_integration_test.go` (the manifest identifier builders, `objectRefFromManifestIdentifier`, +`paddedConfigMapNameTemplate`), as both are compiled into the same test binary. + +- **U1 `retrievePrimaryAndSecondaryPlacementResourceSnapshots`** (deferred): with a fake client and the field index: + count label invalid / < 1, missing owned-by annotation or index label, malformed sub-index label (sort errors), fewer + than count, duplicates (last counted, right after the counted ones, further down (ignored), primary), gap, hash + mismatch, extras trimmed, ordering. Needs the `fieldindexers` extractors exported (or a test-only index setup). +- **U2 `TestMarkPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots`** (`status_test.go`, 8 cases, done): a + rollout to an invalid snapshot (the previous `Available` condition kept at its generation, counters only from the + works linked to the last processed snapshot); the same with no previous `Available` condition (none added); nothing + processed yet (`Available=False/IncompleteSetOfResources`, 0/0/0, works ignored); the processed snapshot itself + invalid (`Available=False/IncompleteSetOfResources` at the same generation); the 50-entry cap; already marked with the same details (no write) or different details (write); a + failed update (API server error). +- **U3** (`status_test.go`, done): `TestCountResourcesInWorksByProcessingResults` (5 cases: no works, unprocessed + manifests, applied with and without availability results, failures incl. `DiffDetails` and an ignored `Available` + condition on a not-applied manifest, the sort order across works by group/version/kind/namespace/name), + `TestFailedResourceFromManifestStatus` (2), `TestRefreshPlacementBindingSyncCond` (4), and + `TestRefreshPlacementBindingAvailableCond` (4) (missing or stale work conditions → not ready, conditions untouched). +- **U4 `TestRefreshPlacementBindingStatus`** (`status_test.go`, 8 cases, done): not ready (no `Applied`, stale + `Applied`, no `Available`) → no write; all synchronized; some failed (sorted, with counters); **status unchanged → no + write** (the client-side skip that R2 cannot tell apart); the 50-entry cap; a failed update. +- **U5** (`works_test.go`, done): `TestBuildWorkObjectFor` (3 cases: a primary snapshot with a namespace-scoped binding + and a sync strategy (copied, not shared); a secondary snapshot with a cluster-scoped binding with long dotted names + (hashed labels, full names in the annotations); a missing sub-index label → `Unexpected` error) and + `TestUpdateWorkObjectMetadataAndSpec` (2 cases: a bare work object; an existing one, whose owned labels/annotations, + manifests, and sync strategy are overwritten while other labels/annotations are kept). +- **U6 work event handler** (deferred): nil objects, failed casts, missing label/annotation, status-only update + filter. Needs the handler funcs pulled out of `SetupWithManager`. +- **U7 `TestRefreshSyncStrategyOnlyForWorks`** (`works_test.go`, 4 cases, done): only the works whose strategy differs + are patched (set and unset), and only `spec.syncStrategy` changes; nothing patched when all are in sync; a stale + cached copy → conflict error (API server category), thanks to the optimistic lock. +- Unreachable today: the empty-name sanity check in `Reconcile` (the CRD rejects empty names), the + duplicate-work-name check in `refreshWorks` (duplicate sub-indices are rejected first), the `Atoi` error inside the + sub-index loop (the sort has already parsed every label), and non-not-found binding `Get` errors. + +## Old tests → new plan + +| Old area (`pkg/controllers/workgenerator`) | New plan | +| --- | --- | +| Bound binding with a single snapshot; manifests not applied → available | Covered (core single) | +| Multiple resource snapshots | Covered (core multiple) | +| Namespaced / cluster-scoped envelope objects; envelope override snapshots | Dropped (no envelopes in the v1 work generator) | +| Valid / invalid / invalid resource / not found overrides | Dropped (no overrides in the v1 work generator) | +| Should not touch/reset `RolloutStarted` | Dropped (no `RolloutStarted` on v1 bindings) | +| Bound binding with a not found cluster | No direct analog (the v1 work generator does not read `MemberCluster`); nearest is L4 | +| Sync apply strategy changes, valid snapshots | Covered (lifecycle `changing sync strategies`, after X6) | +| Sync apply strategy changes, **missing** snapshots | → S7 (the strategy-only path patches the strategy without the snapshots, as the old work generator did) | +| Apply ↔ `ReportDiff`; status under different apply strategies | Dropped (no `ReportDiff` in `v1alpha1`); diff backport is covered by `backport status for failed manifests` | +| `ResourcePlacement` lifecycle | Covered (namespace-scoped contexts) | + +## Behaviors to note (documented only; tests assert current behavior) + +1. **Cache lag can briefly mark a binding `InvalidSnapshots`** ("not found") right after its snapshot was created; it + self-heals on retry (accepted; noted in the code). +2. **Only contents-hash mismatches are `Unexpected`** and stay off the binding; not found, duplicates, gaps, and missing + snapshots ("fewer than count") are all marked `InvalidSnapshots`. An `Unexpected` error that follows an invalid mark + leaves the earlier message in place (X5). +3. **In the invalid state**, `Synchronized=False/InvalidSnapshots`; `Available=False/IncompleteSetOfResources` when the + binding still points to the last processed snapshot (or none was processed); after a rollout to a different + snapshot, `Available` is left as is, which is safe as it then necessarily carries an older generation (or is + absent); the counters and failed resources are recomputed from the works linked to the last processed snapshot + on every reconcile, so they follow work status changes (S1); `LastProcessedResourceSnapshotName` is untouched. +4. **Same-generation re-read failures** (works deleted out of band after the processed snapshot became invalid) set + `Available=False/IncompleteSetOfResources` at the same generation (S8). Fixed October 26, 2026: `Available` used to + be left as is here, so it could read `True` for the same generation next to `Synchronized=False/InvalidSnapshots`. +5. **Strategy-only changes do not need the snapshots and do not reset the binding status** (resolved October 26, + 2026). Until the works are re-processed, the conditions keep the previous generation and the counters their previous + values (X6, S7); consumers that check the condition generation (staged update runs, the policy status) see the + binding as not synchronized for the new generation meanwhile. A strategy change made together with a resource + change goes through the resource path and needs valid snapshots (S7b). +6. **Recovery without a binding event depends on backoff**: after many failures the next retry can be minutes away (up + to 1000s with the default rate limiter); a binding spec change (e.g., a rollout) is picked up immediately. +7. **Work metadata changes are not events**: tampered labels/annotations are corrected only at the next work status + change or binding event (R3). +8. **Empty `placementPolicyName` / `clusterName` / `resourceSnapshotName` are rejected by the CRD** (`minLength: 1`, + October 26, 2026). No KubeFleet code writes empty values: bindings are created only by the placement policy + controller (a member cluster name and the latest snapshot name; creations are held back while there is no + snapshot), and the snapshot name is changed only by auto-rollout and staged update runs (always the name of a + retrieved snapshot). +9. **Deleting a binding deletes only the primary work**; secondaries rely on K8s GC through their owner reference. A + binding re-created before GC runs adopts or deletes them (L3). +10. **A primary whose count label is 1 is never listed**, so duplicates or extras at its index are never checked. +11. **Only the first `count + 1` snapshots are checked**: duplicates further down the listing are ignored (S3). + +## Decisions + +1. **Resolved (October 26, 2026).** New invalid-snapshot tests go in a new `invalid_snapshots_integration_test.go`, + run for both scopes with `bindingFixture`. Status-refresh gaps go in a new `status_integration_test.go`; L rows join + `lifecycle_integration_test.go`. +2. **Accepted (October 26, 2026).** Move `scope`/`scopes`/`snapshotSet`/`bindingFixture` into + `utils_integration_test.go` (Requirement 6). +3. **Rejected (October 26, 2026).** X1 stays. +4. **Skipped (October 26, 2026).** X2 is not folded into the `scopes` loop. +5. **Accepted (October 26, 2026).** Add a `TestMain` gate and a dedicated `ginkgo -p` CI step for the package (see + [Suite changes](#suite-changes)). +6. **Partially implemented (October 26, 2026).** Unit tests U2–U5 and U7 are added; U1 and U6 (which need small + production refactors first) are left alone for now. +7. **Resolved (October 26, 2026)** by code changes: + - B8, empty spec fields: `+kubebuilder:validation:MinLength=1` on `placementPolicyName`, `clusterName`, and + `resourceSnapshotName` in + [placementbinding_types.go](../../../apis/kubefleet.dev/placement/v1alpha1/placementbinding_types.go); CRDs + regenerated (`make manifests`). No API validation tests for now (by request). L5 is dropped. + - B5, strategy changes blocked by invalid snapshots: the strategy-only path (19:33 changes) patches the strategy + without reading the snapshots. S7 pins it, S7b covers the combined case, and X6 updates the existing contexts. +8. **Resolved (October 26, 2026): option (b).** Background: the 1-minute timeout was the test-side constant + `eventuallyDurationForBackoffRetries`, an `Eventually` budget used only by the two `handle inconsistent resource + snapshots` contexts. The controller does not watch snapshots and only notices a fix at its next error retry, which, + with the default limiter, comes after roughly as long as the binding has been failing. Options considered: + - (a) keep the 1-minute budget with short invalid windows (slow: 30–60s per S context; flake risk under load); + - **(b) add an optional rate limiter to `New` (nil = the default) and use a capped one in the suite** (chosen; see + [Suite changes](#suite-changes) item 2); + - (c) nudge the controller with work status updates (does not apply when a binding has no works; no longer tests + unassisted recovery). + +## Running the tests + +Always use the Ginkgo CLI in parallel mode, via `go run` so the CLI matches the Ginkgo version pinned in `go.mod` +(v2.33.0): + +```bash +KUBEBUILDER_ASSETS="$(make -s kubebuilder-assets-path | tail -1)" \ + go run github.com/onsi/ginkgo/v2/ginkgo -p --race ./pkg/v1/controllers/workgenerator +``` + +Use `--focus=""` for a subset and `--focus-file=_integration_test.go` for one file. +`KUBEBUILDER_ASSETS` must be an absolute path; the make target prints one, which can contain spaces on macOS +(`~/Library/Application Support/...`), so keep the quotes. + +## Estimated runtime + +Rough, per context (serial time), with the suite's capped limiter: S contexts 15–25s each (dominated by baselines and +`Consistently` windows; recovery takes about a second), 20 contexts ≈ 7 min; L1/L2 ~20s each × 2 scopes; L3 ~20s × 2; +L4 ~15s; R1–R3 ~15s × 2; R4 ~15s. About 34 new contexts (S: 10 × 2; L: 3 × 2 + 1; R: 3 × 2 + 1) ≈ 11–13 min of +serial time, mostly waiting; with `-p` this is +~2–3 min wall (vs. 1m25s today). X5 and X6 add a few `Consistently` windows; the X5 contexts also get faster with the +capped limiter. + +## Status + +- [x] Surveyed the package source, the existing ITs and unit tests, and the old ITs (October 26, 2026). +- [x] Baseline run before the 19:33 changes: 452/452 pass with `ginkgo -p --race` (74.6s spec time, 1m25s wall). +- [x] Re-baselined on the 19:33 changes: 398 pass, 6 fail (X5 × 2, X6 × 4), 48 skipped; plan revised accordingly + (October 26, 2026). +- [x] Decisions 1–8 resolved (October 26, 2026). +- [x] `MinLength=1` on the three binding spec name fields; CRDs regenerated (Decision 7). +- [x] Fixed the broken existing contexts (X5, X6) (October 26, 2026). New helpers in `utils_integration_test.go`: + `wantPlacementBindingStatusInvalidSnapshots`, `placementBindingSyncCondMessageActual`, and the constant + `fewerSnapshotsErrMsg`; new `bindingFixture` methods `recordStatus` / `statusUnchangedActual` (the fixture still + lives in `lifecycle_integration_test.go` until Decision 2 moves it). Verified: `ginkgo -p --race` passes 464/464 + (105.4s spec time, 1m56s wall; +12 specs); the X5 recovery takes ~15s per scope, within the 1-minute budget. +- [x] **CI wiring** (Decision 5) (October 26, 2026): `TestMain` gate in `suite_test.go` (verified: with + `KUBEFLEET_CI_TEST_RUNNER_NAME=default`, `go test` exits right away without running the suite) and the step `Run + work generator (v1) unit and integration tests with Ginkgo CLI & generate coverage` in `.github/workflows/ci.yml`, + after the v1 work applier step. +- [x] **Rate limiter option** (Decision 8) (October 26, 2026): `New(hubClient, workerCnt, rateLimiter)`; the hub agent + and the unit tests pass nil; the suite passes a 5ms → 1s exponential limiter; all `eventuallyDurationForBackoffRetries` + uses are now `eventuallyDuration`, and the constant is gone. Verified: `go vet` on the package and on + `cmd/hubagent/workload`; `ginkgo -p --race` passes 930/930 (163.8s spec time, 2m58s wall); `golangci-lint` reports + no new issues. +- [x] Shared utilities for X5/X6 and the S section, including the fixture move (Decision 2) (October 26, 2026); the L/R + helpers are pending (see [Shared test utilities](#shared-test-utilities)). +- [x] **S section, all rows** (October 26, 2026): `invalid_snapshots_integration_test.go`, `Describe("handling invalid + placement resource snapshots")`, 11 contexts per scope (S1; S2 with the same and a different stray hash; S3–S9, + S7b), 304 specs. Steps that depend on a retry after a snapshot change (not a binding or work event) used + `eventuallyDurationForBackoffRetries` at first; they are on `eventuallyDuration` since Decision 8 landed. + - Verified: the S contexts pass on their own and in the full run, 768/768 with `ginkgo -p --race` (133.5s spec time, + 2m26s wall). + - Two temporary product mutations were caught: limiting the sub-index check to the counted snapshots fails S2 (both + scopes), and disabling the strategy-only path fails S7 (both scopes). + - A `--repeat=2` run of the S and X5 contexts passed twice. On the third attempt a Ginkgo worker process died of + SIGPIPE mid-run, with no spec failure reported (a runner issue, not seen otherwise). + - `golangci-lint` reports no new issues. + - Found and fixed a race in X5 while doing so: the second "marked as failed" check could pass against the stale + pre-rollout generation in the cache, as the status before and after that rollout differs only in its generations; + the check now first waits until the binding points to the rolled-out snapshot. +- [x] **L section, all rows** (October 26, 2026), in `lifecycle_integration_test.go`, four new `Describe`s (7 contexts, + L5 dropped): + - L1 `binding a placement policy to multiple clusters`; L2 `placement bindings with long and dotted names` (a + 100-character binding name and a 240-character policy name, both dotted; the primary work name is truncated to + 251 characters); L3 `re-creating placement bindings`; L4 `missing reserved namespaces` (namespace-scoped only). +- [x] **R section, all rows** (October 26, 2026), in the new `status_integration_test.go`, `Describe("refreshing + placement binding status")` (7 contexts): R1–R3 for both scopes, R4 namespace-scoped only (51 ConfigMaps placed in + descending name order). + - R3 checks the reset-then-refresh by the Synchronized condition's last transition time (later than before the + tampering), as the transient `WaitingForSync` state lasts only ~1s (the requeue). +- Verification (L and R): the new contexts pass on their own (162 specs) and in the full run, 930/930 with + `ginkgo -p --race` (146.8s spec time, 2m35s wall). Temporary product mutations: dropping the failed-resource sort + fails R4; listing works by the raw binding name (instead of the hashed label value) fails L2 (both scopes); removing + the client-side no-op skip does **not** fail R2 (see the R2 row). `go vet` and `gofmt` are clean; `golangci-lint` + reports no new issues (two `goconst` findings from the new helpers were fixed with constants). +- [x] **Unit tests U2–U5 and U7** (October 26, 2026; see [Unit tests](#unit-tests-decision-6)): 9 test functions, 39 + table cases, in the new `status_test.go` and in `works_test.go`. Verified: all pass with `go test -race`, and the + package runs 930/930 with `ginkgo -p --race` (167.3s spec time, 2m57s wall). Temporary product mutations, each caught + by the intended case: removing the client-side no-op skip in `refreshPlacementBindingStatus` (U4 "status + unchanged"); always setting `IncompleteSetOfResources` (U2 "the processed snapshot has become invalid"); patching + every work regardless of its strategy (U7, 3 cases); dropping the optimistic lock from the patch (U7 conflict case). + `golangci-lint` reports no new issues (four `goconst` findings on repeated literals in the new tests were fixed with + local constants). +- [ ] **`Available` condition fix** (October 26, 2026, 21:41 AEDT; tests updated, not yet run). In + `markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots`, the `IncompleteSetOfResources` check is inverted: it + now applies when the last processed snapshot is the spec snapshot (or none was processed), and the previous + `Available` condition is kept after a rollout to a different snapshot (B4; Behaviors 3 and 4). Test updates: U2 (the + rollout cases keep the previous `Available`; the processed-snapshot case expects `IncompleteSetOfResources`; a new + case for a rollout with no previous `Available`, so 8 cases and 40 table cases in total); the helpers + (`wantPlacementBindingStatusInvalidSnapshots` takes the previous status, `invalidSnapshotsStatusActual` keeps + explicit generations, `registerBaselineSpecs` records the status); S1, S2, S4, S5, S7b, and S9 (rollout) keep the + baseline `Available`; S8 expects `IncompleteSetOfResources` at the same generation; X5 keeps the pre-rollout + `Available`. The verification figures and mutation results above predate this change. +- [x] **`IncompleteSetOfResources` message reworded** (October 27, 2026, 02:58 AEDT). The `Available=False` message set + by `markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots` now gives the cause and fits the nothing-processed + case too (the old "might have become incomplete" implied a previously complete set): `Cannot check the availability + of resources as the given placement resource snapshots are invalid; some resources might be missing from the target + cluster`. Test updates: U2's `incompleteSetOfResourcesCond` builder only; the ITs ignore condition messages + (`ignoreFieldConditionLTTMsg`), so no IT changes. Verified: U2 passes (8/8) with `go test -race -run + TestMarkPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots`. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 02c3c2f9f..1e6b7e3d8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -74,9 +74,36 @@ jobs: KUBEFLEET_CI_WORK_APPLIER_RUN_WITH_PRIORITY_QUEUE=true ginkgo -v -p --procs=4 --race --cover -coverprofile=work-applier-it-no-pri-q-coverage.out ./pkg/controllers/workapplier/ env: KUBEFLEET_CI_TEST_RUNNER_NAME: 'ginkgo' + + # The staged update run controller (v1) integration tests also use in-memory Kubernetes environment setup (one + # per Ginkgo process), and take too long to run serially (some of them wait for the per-cluster wait time limit + # of at least one minute to elapse); run the suite with parallel Ginkgo processes instead. + - name: Run staged update run controller (v1) unit and integration tests with Ginkgo CLI & generate coverage + run: | + ginkgo -v -p --procs=4 --race --cover -coverprofile=staged-update-run-it-coverage.out ./pkg/v1/controllers/stagedupdaterun/ + env: + KUBEFLEET_CI_TEST_RUNNER_NAME: 'ginkgo' + + # The work applier (v1) integration tests also use in-memory Kubernetes environment setup (one per Ginkgo + # process), and many of them wait for periodic requeues, cleanup timeouts, or wave delays; run the suite with + # parallel Ginkgo processes instead. + - name: Run work applier (v1) unit and integration tests with Ginkgo CLI & generate coverage + run: | + ginkgo -v -p --procs=4 --race --cover -coverprofile=work-applier-v1-it-coverage.out ./pkg/v1/controllers/workapplier/ + env: + KUBEFLEET_CI_TEST_RUNNER_NAME: 'ginkgo' + + # The work generator (v1) integration tests also use in-memory Kubernetes environment setup (one per Ginkgo + # process), and many of them wait for retries, requeues, or consistency checks; run the suite with parallel + # Ginkgo processes instead. + - name: Run work generator (v1) unit and integration tests with Ginkgo CLI & generate coverage + run: | + ginkgo -v -p --procs=4 --race --cover -coverprofile=work-generator-v1-it-coverage.out ./pkg/v1/controllers/workgenerator/ + env: + KUBEFLEET_CI_TEST_RUNNER_NAME: 'ginkgo' - name: Upload Codecov report - uses: codecov/codecov-action@0b35c9ecc4f0529d0eb674914510c22f85b196b4 # v7.1.0 + uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1 with: ## Repository upload token - get it from codecov.io. Required only for private repositories token: ${{ secrets.CODECOV_TOKEN }} @@ -87,7 +114,7 @@ jobs: strategy: fail-fast: false matrix: - customized-settings: [default, resourceplacement, joinleave, custom] + customized-settings: [default, resourceplacement, joinleave, placementpolicy, custom] include: - customized-settings: default # to shorten the test duration, set the resource snapshot creation interval to 0 @@ -101,6 +128,10 @@ jobs: # to shorten the test duration, set the resource snapshot creation interval to 0 resource-snapshot-creation-minimum-interval: 0m resource-changes-collection-duration: 0m + - customized-settings: placementpolicy + # to shorten the test duration, set the resource snapshot creation interval to 0 + resource-snapshot-creation-minimum-interval: 0m + resource-changes-collection-duration: 0m - customized-settings: custom resource-snapshot-creation-minimum-interval: 30s resource-changes-collection-duration: 15s @@ -150,11 +181,13 @@ jobs: - name: Run e2e tests run: | if [ "${{ matrix.customized-settings }}" = "default" ]; then - make e2e-tests LABEL_FILTER="!custom && !joinleave && !resourceplacement" + make e2e-tests LABEL_FILTER="!custom && !joinleave && !resourceplacement && !placementpolicy" elif [ "${{ matrix.customized-settings }}" = "resourceplacement" ]; then make e2e-tests LABEL_FILTER="!custom && resourceplacement" elif [ "${{ matrix.customized-settings }}" = "joinleave" ]; then make e2e-tests LABEL_FILTER="!custom && joinleave" + elif [ "${{ matrix.customized-settings }}" = "placementpolicy" ]; then + make e2e-tests LABEL_FILTER="!custom && placementpolicy" else make e2e-tests-custom fi @@ -184,7 +217,7 @@ jobs: - name: Upload logs if: always() - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7 with: name: e2e-logs-${{ matrix.customized-settings }} path: test/e2e/logs-${{ matrix.customized-settings }}/ diff --git a/.github/workflows/codespell.yml b/.github/workflows/codespell.yml index 7991f3002..4bafc0a80 100644 --- a/.github/workflows/codespell.yml +++ b/.github/workflows/codespell.yml @@ -12,7 +12,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Harden Runner - uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1 + uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2 with: egress-policy: audit diff --git a/.github/workflows/pr-title-lint.yml b/.github/workflows/pr-title-lint.yml index 1178161c0..542872ded 100644 --- a/.github/workflows/pr-title-lint.yml +++ b/.github/workflows/pr-title-lint.yml @@ -17,7 +17,7 @@ jobs: check: runs-on: ubuntu-latest steps: - - uses: thehanimo/pr-title-checker@7fbfe05602bdd86f926d3fb3bccb6f3aed43bc70 # v1.4.3 + - uses: thehanimo/pr-title-checker@b69bc3a841cf4e3cfef2d822625d8c90a588e9da # v2.0.0 with: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} pass_on_octokit_error: true diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 12176fce5..13e1a7526 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -252,7 +252,7 @@ jobs: # follow-up issue). Auditing egress at least records what it reaches out # to until that action is replaced. - name: Harden Runner - uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1 + uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2 with: egress-policy: audit diff --git a/.github/workflows/workflow-lint.yml b/.github/workflows/workflow-lint.yml index 903630348..6224c65cd 100644 --- a/.github/workflows/workflow-lint.yml +++ b/.github/workflows/workflow-lint.yml @@ -24,7 +24,7 @@ jobs: timeout-minutes: 10 steps: - name: Harden Runner - uses: step-security/harden-runner@e14015d583714f6e62063499dc959a02595150a1 # v2.21.1 + uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2 with: egress-policy: audit diff --git a/.gitignore b/.gitignore index 38af08316..c2616f258 100644 --- a/.gitignore +++ b/.gitignore @@ -51,3 +51,8 @@ _crd-package/ # Squad: SubSquad activation file (local to this machine) .squad-workstream .squad/.cache/ + +# Binaries from go build ./cmd/ +/hubagent +/memberagent +/crdinstaller diff --git a/Makefile b/Makefile index 7fd7a9bf4..6f51cef71 100644 --- a/Makefile +++ b/Makefile @@ -364,32 +364,20 @@ SHA256SUM ?= $(shell command -v sha256sum >/dev/null 2>&1 && echo "sha256sum" || crd-package: ## Package the raw CRDs into a release tarball with a SHA-256 checksum rm -rf $(CRD_PACKAGE_DIR) mkdir -p $(CRD_PACKAGE_DIR)/crds/hub $(CRD_PACKAGE_DIR)/crds/member - # Source from the charts' CRD sets (symlinks into config/crd/bases) so the - # bundle mirrors exactly what each agent installs: hub-cluster CRDs under - # crds/hub, member-cluster CRDs under crds/member. cp -L dereferences the - # symlinks so the archive holds the raw CRD YAML, not dangling links. - cp -L charts/hub-agent/templates/crds/*.yaml $(CRD_PACKAGE_DIR)/crds/hub/ - cp -L charts/member-agent/templates/crds/*.yaml $(CRD_PACKAGE_DIR)/crds/member/ + # Source from config/crd/bases and split the CRDs the same way the CRD installer + # (cmd/crdinstaller) does: the AppliedWork CRDs go under crds/member, and every + # other CRD goes under crds/hub. The CRDs of the experimental placement policy + # APIs (kubefleet.dev) are included, as if --enable-placement-policy-apis is set. + @set -e; for f in config/crd/bases/*.yaml; do \ + case "$$f" in \ + *_appliedworks.yaml) cp "$$f" $(CRD_PACKAGE_DIR)/crds/member/ ;; \ + *) cp "$$f" $(CRD_PACKAGE_DIR)/crds/hub/ ;; \ + esac; \ + done tar -czf $(CRD_PACKAGE_DIR)/$(CRD_PACKAGE_NAME).tgz -C $(CRD_PACKAGE_DIR) crds cd $(CRD_PACKAGE_DIR) && $(SHA256SUM) $(CRD_PACKAGE_NAME).tgz > $(CRD_PACKAGE_NAME).tgz.sha256 @echo "Packaged CRDs into $(CRD_PACKAGE_DIR)/$(CRD_PACKAGE_NAME).tgz" -.PHONY: crd-verify -crd-verify: ## Verify the chart CRD directories cover every CRD in config/crd/bases; note (chenyu1): kubefleet.dev CRDs are ignored for now until the implementation is completed. - @bases="$$(mktemp)"; charts="$$(mktemp)"; \ - ls config/crd/bases/ | grep -v '^placement\.kubefleet\.dev' | sort > "$$bases"; \ - { ls charts/hub-agent/templates/crds/; ls charts/member-agent/templates/crds/; } | sort > "$$charts"; \ - missing="$$(comm -3 "$$bases" "$$charts")"; \ - rm -f "$$bases" "$$charts"; \ - if [ -n "$$missing" ]; then \ - echo "ERROR: chart CRD directories are out of sync with config/crd/bases."; \ - echo "Left column = only in config/crd/bases; right column = only in the charts:"; \ - echo "$$missing"; \ - echo "If you added a CRD, symlink it into charts/hub-agent/templates/crds/ or charts/member-agent/templates/crds/."; \ - exit 1; \ - fi; \ - echo "crd-verify: chart CRD directories cover all CRDs in config/crd/bases" - # Register QEMU binfmt handlers so the host can build/run non-native binaries. # This must run for any build that targets a non-native platform - whether a # multi-platform build or a single-platform cross build (e.g. PLATFORMS=linux/arm64 diff --git a/SECURITY.md b/SECURITY.md index 3ee4c6508..64d2fd124 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -58,16 +58,18 @@ being finalized: (see the [CNCF TAG-Security `SECURITY.md` template](https://github.com/cncf/tag-security) for the conventional `cncf--distributors-announce@lists.cncf.io` form). Whether KubeFleet stands one up depends on demonstrated downstream demand. -- **GitHub private vulnerability reporting:** to be enabled on this repository as the - preferred reporting channel; the maintainer mailing list (see below) remains the fallback - until it is. +- **GitHub private vulnerability reporting:** report them via the KubeFleet Security Advisories [Report a vulnerability](https://github.com/kubefleet-dev/kubefleet/security/advisories/new) feature.. This section will be updated as each item is decided. ## Reporting Security Issues -**Please do not report security vulnerabilities through public GitHub issues.** Instead, -report them to the [KubeFleet maintainers](mailto:kubefleet@microsoft.com). +**Please do not report security vulnerabilities through public GitHub issues.** + +Instead, report them via the KubeFleet Security Advisories [Report a vulnerability](https://github.com/kubefleet-dev/kubefleet/security/advisories/new) feature. + +You can also follow up with the [KubeFleet maintainers](mailto:kubefleet@microsoft.com). + We prefer all communications to be in English. You should receive a response as soon as possible. If for some reason you do not, please diff --git a/VERSIONING.md b/VERSIONING.md index 16a4d253f..fb64e2008 100644 --- a/VERSIONING.md +++ b/VERSIONING.md @@ -138,11 +138,11 @@ as a model rather than restating it here. Note that the upstream policy formally governs built-in Kubernetes APIs; KubeFleet adopts it by convention for its CRDs. When upgrading with raw manifests, apply the CRDs shipped with the target release -before rolling the agents, as you would for any Kubernetes operator. Helm-based -installs need no separate step: KubeFleet ships its CRDs under -`charts/hub-agent/templates/crds/` and `charts/member-agent/templates/crds/` -(regular templates, not Helm's special unmanaged `crds/` directory), so -`helm upgrade` applies them — ahead of the Deployments — automatically. +before rolling the agents, as you would for any Kubernetes operator. The Helm +charts do not ship CRDs; for Helm-based installs, enable the CRD installer +(`crdInstaller.enabled=true`), which runs as an init container of the agent and +creates/updates the CRDs of the target release before the agent starts, or apply +the CRDs separately (e.g., from the release's CRD bundle) before `helm upgrade`. ## See also diff --git a/apis/kubefleet.dev/placement/v1alpha1/clusterclaim_types.go b/apis/kubefleet.dev/placement/v1alpha1/clusterclaim_types.go index 732e0aced..bb12622dd 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/clusterclaim_types.go +++ b/apis/kubefleet.dev/placement/v1alpha1/clusterclaim_types.go @@ -22,6 +22,22 @@ import ( const ( ClusterClaimCondTypeCompleted = "Completed" + + // ClusterClaimPlacementPolicyNameLabel and ClusterClaimPlacementPolicyNamespaceLabel record the placement + // policy that added a cluster claim. Cluster claims are cluster-scoped and cannot carry an + // owner reference to a namespaced PlacementPolicy, so these labels are how KubeFleet — and + // anyone querying claims, such as a provisioner watching the claims of a specific + // placement — associates a claim with its policy. The namespace label is empty for claims + // added by a ClusterPlacementPolicy. + // + // Object names may be longer than the 63-byte limit on label values: when a policy's name + // does not fit, the label carries a shortened, collision-resistant form of it instead, and + // selecting on the label with the policy's own name then matches nothing. The authoritative + // identity is always spec.placementPolicyRef, so a consumer that must handle policies with + // names longer than 63 bytes should list cluster claims and match on that field rather than + // select on this label. + ClusterClaimPlacementPolicyNameLabel = "placement.kubefleet.dev/placement-policy-name" + ClusterClaimPlacementPolicyNamespaceLabel = "placement.kubefleet.dev/placement-policy-namespace" ) // ClusterClaim is a KubeFleet API that represents a claim for a new member cluster. @@ -46,6 +62,8 @@ type ClusterClaim struct { Status ClusterClaimStatus `json:"status,omitempty"` } +// ClusterClaimSpec is the specification of a cluster claim. +// // +kubebuilder:validation:XValidation:rule="has(self.clusterSelectorTerms) == has(oldSelf.clusterSelectorTerms)",message="the clusterSelectorTerms field cannot be added or removed after creation" type ClusterClaimSpec struct { // The reference to the placement policy that adds the cluster claim. @@ -75,6 +93,15 @@ type ClusterClaimStatus struct { // The name of the cluster that has been provisioned for this cluster claim, if any. // + // A cluster claim asks for a single cluster: a provisioner fulfills it by provisioning one + // cluster, recording its name here, and marking the Completed condition. A placement that still + // needs more clusters than one claim provides is served one claim at a time -- once the + // provisioned cluster becomes eligible, the completed claim is withdrawn and a new claim is + // created, under the same deterministic name, for the next cluster. A provisioner that retries a + // conflicting status write must therefore key the retry on the claim's identity (UID or resource + // version), not on its name alone, so that a write meant for a completed claim is not applied to + // the different, freshly created claim that has taken its name. + // // +kubebuilder:validation:Optional ProvisionedClusterName *string `json:"provisionedClusterName,omitempty"` diff --git a/apis/kubefleet.dev/placement/v1alpha1/common.go b/apis/kubefleet.dev/placement/v1alpha1/common.go index 9ccad8432..97ee0c0b5 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/common.go +++ b/apis/kubefleet.dev/placement/v1alpha1/common.go @@ -28,6 +28,61 @@ const ( WorkKind = "Work" ) +// The annotation and label keys that KubeFleet reserves for the placement APIs. +// +// Note that these keys carry the KubeFleet domain itself rather than the API group; they are +// set on arbitrary Kubernetes objects, not only on the objects of this group. +const ( + // KubeFleetPrefix is the domain that prefixes every reserved annotation and label key below, + // per the Kubernetes convention that reserves unprefixed keys for end users. + KubeFleetPrefix = "kubefleet.dev/" + + // ClusterSelectorsAnnotation is the annotation that requests annotation-based placement for the + // resource it is set on. Its value is a semicolon-separated list of cluster selectors, each of + // which is a comma-separated list of LABEL_KEY=LABEL_VALUE label matchers with an optional + // count=N|All directive, e.g.: + // + // kubefleet.dev/cluster-selectors: "env=staging,count=All;env=canary,region=eastus,count=1" + // + // KubeFleet keeps a PlacementPolicy (or ClusterPlacementPolicy) object in sync with the + // annotation for as long as it is present. + // + // The key names what the value holds rather than what KubeFleet does with it, matching the + // clusterSelectors field of the generated policy: the annotation is one way to write that field, + // and the two are read together often enough that they should not have to be translated. + // + // Within the annotation, `region` may be used in place of the well-known + // topology.kubernetes.io/region label key, and `alias` in place of the cluster alias label + // (placement/v1beta1's ClusterAliasLabel). + ClusterSelectorsAnnotation = KubeFleetPrefix + "cluster-selectors" +) + +// The labels that record, on a placement policy KubeFleet generated from an annotation, the +// resource whose annotation caused it to exist. +// +// These keys carry the API group rather than the bare KubeFleet domain, since unlike the keys +// above they are set on objects of this group. +// +// The labels exist so that a policy can be found with a List when its generated name is not known; +// the owner reference on the policy remains the authoritative record of where it came from. Note +// that ParentNameLabel is lossy: the name of a resource can run to 253 bytes while a label value +// stops at 63, so for a longer name the label holds a prefix and a hash instead, and selecting on +// it with the resource's own name matches nothing. +const ( + // ParentAPIGroupLabel holds the API group of the resource a policy was generated from. It is + // the empty string for resources in the core API group, and is always present, so that + // core-group resources can be selected as readily as any other. + ParentAPIGroupLabel = "placement.kubefleet.dev/parent-api-group" + + // ParentKindLabel holds the kind of the resource a policy was generated from, spelled as the + // kind itself is (Deployment, not deployment). + ParentKindLabel = "placement.kubefleet.dev/parent-kind" + + // ParentNameLabel holds the name of the resource a policy was generated from, shortened if it + // does not fit in a label value. + ParentNameLabel = "placement.kubefleet.dev/parent-name" +) + type ObjectReference struct { // The namespace of the referenced object. // diff --git a/apis/kubefleet.dev/placement/v1alpha1/gvk_info.go b/apis/kubefleet.dev/placement/v1alpha1/gvk_info.go index 81efe4327..40d7c469f 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/gvk_info.go +++ b/apis/kubefleet.dev/placement/v1alpha1/gvk_info.go @@ -33,3 +33,8 @@ var ( // AddToScheme adds the types in this group-version to the given scheme. AddToScheme = SchemeBuilder.AddToScheme ) + +const ( + // The kind names for all the custom resources in this GV. + AppliedWorkKind = "AppliedWork" +) diff --git a/apis/kubefleet.dev/placement/v1alpha1/placementbinding_types.go b/apis/kubefleet.dev/placement/v1alpha1/placementbinding_types.go index e3976ceef..7ecea1e89 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/placementbinding_types.go +++ b/apis/kubefleet.dev/placement/v1alpha1/placementbinding_types.go @@ -29,10 +29,12 @@ const ( // The reasons for each condition type of the PlacementBinding and ClusterPlacementBinding APIs. const ( PlacementBindingSynchronizedCondReasonAllResourcesSynchronized = "AllResourcesSynchronized" + PlacementBindingSynchronizedCondReasonInvalidSnapshots = "InvalidSnapshots" PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources = "FailedToSynchronizeSomeResources" PlacementBindingSynchronizedCondReasonWaitingForSynchronization = "WaitingForSynchronization" PlacementBindingAvailableCondReasonAllResourcesAvailable = "AllResourcesAvailable" + PlacementBindingAvailableCondReasonIncompleteSetOfResources = "IncompleteSetOfResources" PlacementBindingAvailableCondReasonSomeResourcesUnavailable = "SomeResourcesUnavailable" PlacementBindingAvailableCondReasonWaitingForAvailabilityCheck = "WaitingForAvailabilityCheck" ) @@ -87,6 +89,7 @@ type PlacementBindingSpec struct { // The name of the placement policy that this binding is associated with. // // +kubebuilder:validation:Required + // +kubebuilder:validation:MinLength=1 PlacementPolicyName string `json:"placementPolicyName"` // The cluster selectors associated with (fulfilled by) this binding. @@ -100,6 +103,7 @@ type PlacementBindingSpec struct { // The name of the member cluster that this binding is associated with. // // +kubebuilder:validation:Required + // +kubebuilder:validation:MinLength=1 ClusterName string `json:"clusterName"` // The name of the resource snapshot that this binding is associated with. @@ -108,6 +112,7 @@ type PlacementBindingSpec struct { // the name of the primary resource snapshot that this binding is associated with. // // +kubebuilder:validation:Required + // +kubebuilder:validation:MinLength=1 ResourceSnapshotName string `json:"resourceSnapshotName"` // The strategy to synchronize the resources to the member cluster. diff --git a/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go b/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go index 2d54fc82f..fc6bf15df 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go +++ b/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_types.go @@ -38,8 +38,10 @@ const ( PlacementPolicyScheduledCondReasonFoundAllClusters = "FoundAllRequiredClusters" PlacementPolicyScheduledCondReasonFailedToFindSomeClusters = "FailedToFindSomeRequiredClusters" - PlacementPolicySynchronizedCondReasonAllClustersSynchronized = "AllClustersSynchronized" - PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters = "FailedToSynchronizeSomeClusters" + PlacementPolicySynchronizedCondReasonAllClustersSynchronized = "AllClustersSynchronized" + PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters = "FailedToSynchronizeSomeClusters" + PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout = "ResourcesChangedPendingRollout" + PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision = "SomeClustersOnOlderResourceRevision" PlacementPolicyAvailableCondReasonAllClustersAvailable = "ResourcesAvailableOnAllClusters" PlacementPolicyAvailableCondReasonSomeClustersUnavailable = "ResourcesUnavailableOnSomeClusters" @@ -121,7 +123,7 @@ type PlacementPolicySpec struct { // failure recovery purposes; one can inspect them to see the past state of the selected resources, // or roll back to a previous revision if the latest revision is not working as expected. // - // It is also possible to manually request a new resource revision to be created. + // The limit is a soft constraint; KubeFleet may retain more revisions than the specified limit as necessary. // // The default value is 3. // @@ -144,12 +146,20 @@ type PlacementPolicySpec struct { // // +kubebuilder:validation:Optional // +kubebuilder:validation:MaxItems=10 - // +kubebuilder:validation:XValidation:rule="self.all(t, t.key != '' || (t.operator == 'Exists' && t.value == ''))",message="operator must be Exists and value must be empty when key is empty" - // +kubebuilder:validation:XValidation:rule="self.all(t, t.operator != 'Exists' || t.value == '')",message="value must be empty when operator is Exists" Tolerations []Toleration `json:"tolerations,omitempty"` } -// +kubebuilder:validation:XValidation:rule="!has(self.minCount) || !has(self.count) || (type(self.count) == string && self.count == 'All') || (type(self.count) == int && self.minCount <= self.count) || (type(self.count) == string && self.count.matches('^[0-9]+$') && self.minCount <= int(self.count))",message="minCount must be less than or equal to count when count is not All" +// A string form of count is compared against minCount only when it is a numeric one of at most the +// three digits that the count field's own pattern permits -- the {1,3} bound below must move with +// that pattern's ceiling. Any other string ("All", or junk the pattern rejects) passes this rule +// vacuously, so that the field's own validation reports the real problem alone. The previous, +// unbounded digit guard let a string longer than an int64 reach int(), turning a plain pattern +// violation into an opaque evaluation error beside it. The digit class is spelled [0-9]{1,3} +// rather than mirroring the pattern's [1-9][0-9]{0,2} verbatim because the CEL cost estimator +// prices a regex by its length against an unbounded string -- an int-or-string is opaque to the +// sibling MaxLength -- and the longer spelling does not fit the budget. + +// +kubebuilder:validation:XValidation:rule="!has(self.minCount) || !has(self.count) || (type(self.count) == int && self.minCount <= self.count) || (type(self.count) == string && (!self.count.matches('^[0-9]{1,3}$') || self.minCount <= int(self.count)))",message="minCount must be less than or equal to count" type ClusterSelector struct { // A list of terms that form the selector. The terms are ORed, i.e., a cluster would match the selector // if it matches any of the terms. @@ -160,6 +170,12 @@ type ClusterSelector struct { // +kubebuilder:validation:MaxItems=5 Terms []ClusterLabelAndPropertySelectorTerm `json:"terms,omitempty"` + // The two validation markers below split the work by form: a pattern constrains only the string + // form of an int-or-string ("All", and digits arriving as a quoted string), so the CEL rule is + // what bounds the integer form. Without it, an unquoted count outside 1-999 is accepted while + // the same number in quotes is rejected. This block is deliberately detached from the field's + // doc comment: it is rationale for maintainers, not schema documentation for users. + // The desired number of clusters that KubeFleet should select based on the given terms. // // The default value is 1. To select all clusters that match the given terms, use the value "All". @@ -167,7 +183,9 @@ type ClusterSelector struct { // +kubebuilder:validation:Optional // +kubebuilder:default=1 // +kubebuilder:validation:XIntOrString + // +kubebuilder:validation:MaxLength=3 // +kubebuilder:validation:Pattern="^([1-9][0-9]{0,2}|All)$" + // +kubebuilder:validation:XValidation:rule="type(self) == int ? self >= 1 && self <= 999 : true",message="count must be between 1 and 999, or \"All\"" Count *intstr.IntOrString `json:"count,omitempty"` // The minimum number of clusters that KubeFleet should select based on the given terms, when KubeFleet is not able @@ -486,6 +504,14 @@ const ( WhenNamespaceDoesNotExistOptionReportError WhenNamespaceDoesNotExistOption = "ReportError" ) +// Key and Value are optional with no defaults, so the rules below must guard them with has(); reading +// an absent field in CEL is an evaluation error, which the API server reports as a rejection. +// +// The rules use size() rather than comparing against an empty string literal: they sit in the doc +// comment of the type, where gofmt rewrites two consecutive single quotes into a typographic quote. + +// +kubebuilder:validation:XValidation:rule="(has(self.key) && size(self.key) > 0) || self.operator == 'Exists'",message="operator must be Exists when key is empty" +// +kubebuilder:validation:XValidation:rule="self.operator != 'Exists' || !has(self.value) || size(self.value) == 0",message="value must be empty when operator is Exists" type Toleration struct { // The key of the taint that the toleration applies to. // @@ -547,7 +573,10 @@ type PlacementPolicyStatus struct { // The number of clusters that have resources in the available state, as verified by KubeFleet's availability check. ResourcesAvailableClusters *int32 `json:"resourcesAvailableClusters,omitempty"` - // The number of active cluster claims that have been submitted by this placement. + // The number of active cluster claims that have been submitted by this placement. A claim + // that is being withdrawn still counts as active until KubeFleet observes it gone -- + // whether a provisioner finalizer holds it or the withdrawal simply has not been confirmed + // yet -- as it continues to occupy the placement's claim budget. ActiveClusterClaims *int32 `json:"activeClusterClaims,omitempty"` // The binding manager that is currently managing the bindings for this placement. diff --git a/apis/kubefleet.dev/placement/v1alpha1/placementresourcesnapshot_types.go b/apis/kubefleet.dev/placement/v1alpha1/placementresourcesnapshot_types.go index f98db8ed8..f3c09e04c 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/placementresourcesnapshot_types.go +++ b/apis/kubefleet.dev/placement/v1alpha1/placementresourcesnapshot_types.go @@ -41,7 +41,7 @@ const ( // might truncate the name and add a hash suffix as needed. // // This label is set on all placement resource snapshots. - PlacementResourceSnapshotOwnedByLabelKey = "placement.kubefleet.dev/placement-resource-snapshot-owned-by" + PlacementResourceSnapshotOwnedByLabelKey = "placement.kubefleet.dev/owned-by-placement-policy" // PlacementResourceSnapshotIndexLabelKey is a label key that denotes the index of a placement resource snapshot. // Its value is the index integer formatted as a string. // @@ -64,6 +64,12 @@ const ( // // This annotation is set on all placement resource snapshots. PlacementResourceSnapshotContentsHashAnnotationKey = "placement.kubefleet.dev/placement-resource-snapshot-contents-hash" + // PlacementResourceSnapshotOwnedByAnnotationKey is an annotation key that denotes the owner placement policy of + // a placement resource snapshot. Its value is the name of the owner placement policy as it is (i.e., unlike + // the value of the `PlacementResourceSnapshotOwnedByLabelKey` label, it is never truncated). + // + // This annotation is set on all placement resource snapshots. + PlacementResourceSnapshotOwnedByAnnotationKey = "placement.kubefleet.dev/owned-by-placement-policy" ) // PlacementResourceSnapshot is the KubeFleet API that captures the resources selected by a placement policy diff --git a/apis/kubefleet.dev/placement/v1alpha1/work_types.go b/apis/kubefleet.dev/placement/v1alpha1/work_types.go index eb78ba88c..8aa519ee5 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/work_types.go +++ b/apis/kubefleet.dev/placement/v1alpha1/work_types.go @@ -19,6 +19,7 @@ package v1alpha1 import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" ) const ( @@ -54,6 +55,11 @@ const ( WorkDerivedFromSourceAnnotationKey = "placement.kubefleet.dev/derived-from" ) +const ( + LastAppliedConfigAnnotationKey = "placement.kubefleet.dev/last-applied-configuration" + LastAppliedManifestHashAnnotationKey = "placement.kubefleet.dev/manifest-hash" +) + const ( // The condition types for the Work API. WorkCondTypeApplied = "Applied" @@ -64,13 +70,22 @@ const ( ManifestCondTypeAvailable = "Available" ) +const ( + WorkAppliedCondAllManifestsAppliedReason = "AllManifestsApplied" + WorkAppliedCondNotAllManifestsAppliedReason = "SomeManifestsAreNotYetApplied" + + WorkAvailableCondAllManifestsAvailableReason = "AllManifestsAvailable" + WorkAvailableCondNotAllManifestsAvailableReason = "SomeManifestsAreNotYetAvailable" + WorkAvailableCondNotAllManifestsTrackableReason = "SomeManifestsAreNotTrackable" +) + // Work is the KubeFleet API used for synchronizing resources to place between // the hub cluster and a member cluster in the member cluster reserved namespace. // // +genclient // +kubebuilder:object:root=true // +kubebuilder:subresource:status -// +kubebuilder:resource:scope=Namespaced,categories={kubefleet, kubefleet-placement} +// +kubebuilder:resource:scope=Namespaced,categories={kubefleet, kubefleet-placement},shortName=kfwork // +kubebuilder:storageversion // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object type Work struct { @@ -201,21 +216,24 @@ type ManifestIdentifier struct { Namespace string `json:"namespace,omitempty"` // The name of the manifest. - // +kubebuilder:validation:Required - Name string `json:"name"` + // +kubebuilder:validation:Optional + Name string `json:"name,omitempty"` // The API group, version, kind, and resource of the manifest. + // + // Note that runtime.RawExtension, which is used to store manifests, has additional validation on the manifest + // data (e.g., API Group/Version and Kind must be set; they are not guaranteed to be valid though). // +kubebuilder:validation:Optional APIGroup string `json:"apiGroup,omitempty"` // +kubebuilder:validation:Required - APIVersion string `json:"apiVersion,omitempty"` + APIVersion string `json:"apiVersion"` // +kubebuilder:validation:Required - Kind string `json:"kind,omitempty"` + Kind string `json:"kind"` - // +kubebuilder:validation:Required + // +kubebuilder:validation:Optional Resource string `json:"resource,omitempty"` } @@ -231,7 +249,100 @@ type WorkList struct { Items []Work `json:"items"` } +// AppliedWork is the KubeFleet API that serves as owners for manifests applied to a member cluster via +// a work object. +// +// +genclient +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster,categories={kubefleet, kubefleet-placement},shortName=kfappliedwork +// +kubebuilder:storageversion +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type AppliedWork struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // Spec represents the desired configuration of AppliedWork. + // +kubebuilder:validation:Required + Spec AppliedWorkSpec `json:"spec"` + + // Status represents the current status of AppliedWork. + // +kubebuilder:validation:Optional + Status AppliedWorkStatus `json:"status,omitempty"` +} + +type AppliedWorkSpec struct { + // The name of the corresponding work object. Note that the object is on the KubeFleet hub cluster. + // + // +kubebuilder:validation:Required + WorkName string `json:"workName"` + + // The namespace of the corresponding work object. Note that the object is on the KubeFleet hub cluster. + // + // +kubebuilder:validation:Required + WorkNamespace string `json:"workNamespace"` +} + +type AppliedWorkStatus struct { + // A list of applied manifests. This list is populated for informational purposes only. + // + // +kubebuilder:validation:Optional + AppliedResources []AppliedResource `json:"appliedResources,omitempty"` +} + +type AppliedResource struct { + // The ID of the applied manifest. + AppliedResourceIdentifier `json:",inline"` + + // The UID of the applied manifest. + // + // +kubebuilder:validation:Required + UID types.UID `json:"uid,omitempty"` +} + +type AppliedResourceIdentifier struct { + // The ordinal of the manifest. + // + // +kubebuilder:validation:Required + Ordinal int `json:"ordinal"` + + // The namespace of the manifest. + // + // +kubebuilder:validation:Optional + Namespace string `json:"namespace,omitempty"` + + // The name of the manifest. + // +kubebuilder:validation:Required + Name string `json:"name"` + + // The API group, version, kind, and resource of the manifest. + + // +kubebuilder:validation:Optional + APIGroup string `json:"apiGroup,omitempty"` + + // +kubebuilder:validation:Required + APIVersion string `json:"apiVersion"` + + // +kubebuilder:validation:Required + Kind string `json:"kind"` + + // +kubebuilder:validation:Required + Resource string `json:"resource"` +} + +// AppliedWorkList contains a list of AppliedWork. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Cluster +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type AppliedWorkList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []AppliedWork `json:"items"` +} + // Set up the API types with the scheme builder. func init() { - SchemeBuilder.Register(&Work{}, &WorkList{}) + SchemeBuilder.Register(&Work{}, &WorkList{}, &AppliedWork{}, &AppliedWorkList{}) } diff --git a/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go b/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go index 8f5a15a5d..708652d46 100644 --- a/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go +++ b/apis/kubefleet.dev/placement/v1alpha1/zz_generated.deepcopy.go @@ -26,6 +26,131 @@ import ( "k8s.io/apimachinery/pkg/util/intstr" ) +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedResource) DeepCopyInto(out *AppliedResource) { + *out = *in + out.AppliedResourceIdentifier = in.AppliedResourceIdentifier +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedResource. +func (in *AppliedResource) DeepCopy() *AppliedResource { + if in == nil { + return nil + } + out := new(AppliedResource) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedResourceIdentifier) DeepCopyInto(out *AppliedResourceIdentifier) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedResourceIdentifier. +func (in *AppliedResourceIdentifier) DeepCopy() *AppliedResourceIdentifier { + if in == nil { + return nil + } + out := new(AppliedResourceIdentifier) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedWork) DeepCopyInto(out *AppliedWork) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + out.Spec = in.Spec + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedWork. +func (in *AppliedWork) DeepCopy() *AppliedWork { + if in == nil { + return nil + } + out := new(AppliedWork) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *AppliedWork) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedWorkList) DeepCopyInto(out *AppliedWorkList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]AppliedWork, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedWorkList. +func (in *AppliedWorkList) DeepCopy() *AppliedWorkList { + if in == nil { + return nil + } + out := new(AppliedWorkList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *AppliedWorkList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedWorkSpec) DeepCopyInto(out *AppliedWorkSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedWorkSpec. +func (in *AppliedWorkSpec) DeepCopy() *AppliedWorkSpec { + if in == nil { + return nil + } + out := new(AppliedWorkSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *AppliedWorkStatus) DeepCopyInto(out *AppliedWorkStatus) { + *out = *in + if in.AppliedResources != nil { + in, out := &in.AppliedResources, &out.AppliedResources + *out = make([]AppliedResource, len(*in)) + copy(*out, *in) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AppliedWorkStatus. +func (in *AppliedWorkStatus) DeepCopy() *AppliedWorkStatus { + if in == nil { + return nil + } + out := new(AppliedWorkStatus) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *BindingManager) DeepCopyInto(out *BindingManager) { *out = *in diff --git a/apis/kubefleet.dev/rollout/v1alpha1/approvalrequests_types.go b/apis/kubefleet.dev/rollout/v1alpha1/approvalrequests_types.go new file mode 100644 index 000000000..ed9e3afef --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/approvalrequests_types.go @@ -0,0 +1,136 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +const ( + ApprovalRequestCondTypeApproved = "Approved" + ApprovalRequestCondTypeApprovalAccepted = "ApprovalAccepted" +) + +// The reasons for the respective condition types. +const ( + ApprovalRequestApprovalAcceptedCondReasonAccepted = "ApprovalAccepted" +) + +// ApprovalRequest is the KubeFleet API that enables users to approve a rollout to a stage of clusters +// for a placement policy, when staged update is in use. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Namespaced,categories={kubefleet,kubefleet-rollout},shortName=stageapproval +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type ApprovalRequest struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the ApprovalRequest API object. + // + // +kubebuilder:validation:Required + Spec ApprovalRequestSpec `json:"spec"` + + // The observed status of the ApprovalRequest API object. + // + // +kubebuilder:validation:Optional + Status ApprovalRequestStatus `json:"status,omitempty"` +} + +// ClusterApprovalRequest is the KubeFleet API that enables users to approve a rollout to a stage of clusters +// for a cluster placement policy, when staged update is in use. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster,categories={kubefleet,kubefleet-rollout},shortName=clusterstageapproval +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type ClusterApprovalRequest struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the ClusterApprovalRequest API object. + // + // +kubebuilder:validation:Required + Spec ApprovalRequestSpec `json:"spec"` + + // The observed status of the ClusterApprovalRequest API object. + // + // +kubebuilder:validation:Optional + Status ApprovalRequestStatus `json:"status,omitempty"` +} + +type ApprovalRequestSpec struct { + // The name of the StagedUpdateRun API object that the approval request is associated with. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="stagedUpdateRunName is immutable" + StagedUpdateRunName string `json:"stagedUpdateRunName"` + + // The name of the stage that the approval request is associated with in the staged update run. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MaxLength=63 + // +kubebuilder:validation:Pattern="^[a-z0-9]+$" + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="stageName is immutable" + StageName string `json:"stageName"` +} + +type ApprovalRequestStatus struct { + // A list of observed conditions about the approval request. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` +} + +// The list objects for ApprovalRequest and ClusterApprovalRequest APIs. + +// ApprovalRequestList contains a list of ApprovalRequest API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Namespaced" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ApprovalRequestList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []ApprovalRequest `json:"items"` +} + +// ClusterApprovalRequestList contains a list of ClusterApprovalRequest API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Cluster" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ClusterApprovalRequestList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []ClusterApprovalRequest `json:"items"` +} + +// Set up the API types with the scheme builder. +func init() { + SchemeBuilder.Register(&ApprovalRequest{}, &ApprovalRequestList{}) + SchemeBuilder.Register(&ClusterApprovalRequest{}, &ClusterApprovalRequestList{}) +} diff --git a/apis/kubefleet.dev/rollout/v1alpha1/autorollout.go b/apis/kubefleet.dev/rollout/v1alpha1/autorollout.go new file mode 100644 index 000000000..d2c795d70 --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/autorollout.go @@ -0,0 +1,53 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +const ( + // AutoRolloutEnabledAnnotationKey is the annotation key that, once added to a placement policy, enables + // auto-rollout for the placement policy. Changes to the resources selected by the placement policy will + // be rolled out automatically, at most once per minimum wait period (see + // AutoRolloutMinimumWaitPeriodAnnotationKey). + // + // Important: the auto-rollout feature is best used with experimental/staging/testing placements and/or with + // resources that are generally safe to be rolled out automatically, such as policies, configuration data, and namespaces. + // Each auto-rollout syncs the state of the selected resources between the hub cluster and all selected + // member clusters at once, with no gates/checks, and no baking period. For a safer, more controlled rollout + // experience, consider using KubeFleet's staged update APIs. + AutoRolloutEnabledAnnotationKey = "kubefleet.dev/auto-rollout-enabled" + // AutoRolloutEnabledAnnotationValue is the value that should be set for the AutoRolloutEnabledAnnotationKey to + // enable auto-rollout. If the auto-rollout annotation is set to a different value, auto-rollout will be + // considered disabled. + AutoRolloutEnabledAnnotationValue = "true" + + // AutoRolloutMinimumWaitPeriodAnnotationKey is the annotation key that specifies the minimum wait period between + // consecutive auto-rollouts for the placement policy; it also spaces out the retries of an auto-rollout that failed + // due to a user error. The value should be a duration string, e.g., "5m" for 5 minutes. + // The lower and upper bound for this period is 1 minute and 1 hour, respectively. The default is 3 minutes. + // A value that cannot be parsed is ignored in favor of the default; a value out of the allowed range is clamped + // to the closest bound. + AutoRolloutMinimumWaitPeriodAnnotationKey = "kubefleet.dev/auto-rollout-minimum-wait-period" + + // AutoRolloutLastRolloutAttemptTimestampAnnotationKey is the annotation key that stores the timestamp of the last + // auto-rollout attempt for the placement policy, in the RFC 3339 format. It is updated when an auto-rollout + // succeeds, or fails due to a user error (e.g., a selected resource that is too large to snapshot), so that such an + // auto-rollout is retried at most once per minimum wait period; failures that are retried right away (with backoff) + // do not update it. + // + // KubeFleet manages this annotation. A missing or malformed value is treated as if the placement policy has never been + // rolled out; a value in the future is treated as if the placement policy has just been rolled out. + AutoRolloutLastRolloutAttemptTimestampAnnotationKey = "kubefleet.dev/auto-rollout-last-rollout-attempt-timestamp" +) diff --git a/apis/kubefleet.dev/rollout/v1alpha1/doc.go b/apis/kubefleet.dev/rollout/v1alpha1/doc.go new file mode 100644 index 000000000..db3677d0f --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/doc.go @@ -0,0 +1,20 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// +kubebuilder:object:generate=true +// +k8s:deepcopy-gen=package,register +// +groupName=rollout.kubefleet.dev +package v1alpha1 diff --git a/apis/kubefleet.dev/rollout/v1alpha1/gvk_info.go b/apis/kubefleet.dev/rollout/v1alpha1/gvk_info.go new file mode 100644 index 000000000..18dd87be4 --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/gvk_info.go @@ -0,0 +1,45 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// +kubebuilder:object:generate=true +// +groupName=rollout.kubefleet.dev +package v1alpha1 + +import ( + "k8s.io/apimachinery/pkg/runtime/schema" + "sigs.k8s.io/controller-runtime/pkg/scheme" +) + +var ( + // GroupVersion is group version used to register these objects + GroupVersion = schema.GroupVersion{Group: "rollout.kubefleet.dev", Version: "v1alpha1"} + + // SchemeBuilder is used to add go types to the GroupVersionKind scheme + SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion} + + // AddToScheme adds the types in this group-version to the given scheme. + AddToScheme = SchemeBuilder.AddToScheme +) + +const ( + // The kind names for all the custom resources in this GV. + StagedUpdateRunKind = "StagedUpdateRun" + ClusterStagedUpdateRunKind = "ClusterStagedUpdateRun" + StagedUpdateStrategyKind = "StagedUpdateStrategy" + ClusterStagedUpdateStrategyKind = "ClusterStagedUpdateStrategy" + ApprovalRequestKind = "ApprovalRequest" + ClusterApprovalRequestKind = "ClusterApprovalRequest" +) diff --git a/apis/kubefleet.dev/rollout/v1alpha1/interface.go b/apis/kubefleet.dev/rollout/v1alpha1/interface.go new file mode 100644 index 000000000..3c3bf1d8a --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/interface.go @@ -0,0 +1,170 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + "sigs.k8s.io/controller-runtime/pkg/client" +) + +var _ ApprovalRequestAccessor = &ApprovalRequest{} +var _ ApprovalRequestAccessor = &ClusterApprovalRequest{} + +// ApprovalRequestAccessor is an interface that provides unified access to the Spec and Status of +// ApprovalRequest and ClusterApprovalRequest resources. +// +// +kubebuilder:object:generate=false +type ApprovalRequestAccessor interface { + client.Object + + GetSpec() *ApprovalRequestSpec + GetStatus() *ApprovalRequestStatus + + SetSpec(ApprovalRequestSpec) + SetStatus(ApprovalRequestStatus) +} + +// GetSpec returns the spec of the ApprovalRequest. +func (s *ApprovalRequest) GetSpec() *ApprovalRequestSpec { + return &s.Spec +} + +// GetStatus returns the status of the ApprovalRequest. +func (s *ApprovalRequest) GetStatus() *ApprovalRequestStatus { + return &s.Status +} + +// SetSpec sets the spec of the ApprovalRequest. +func (s *ApprovalRequest) SetSpec(spec ApprovalRequestSpec) { + s.Spec = spec +} + +// SetStatus sets the status of the ApprovalRequest. +func (s *ApprovalRequest) SetStatus(status ApprovalRequestStatus) { + s.Status = status +} + +// GetSpec returns the spec of the ClusterApprovalRequest. +func (s *ClusterApprovalRequest) GetSpec() *ApprovalRequestSpec { + return &s.Spec +} + +// GetStatus returns the status of the ClusterApprovalRequest. +func (s *ClusterApprovalRequest) GetStatus() *ApprovalRequestStatus { + return &s.Status +} + +// SetSpec sets the spec of the ClusterApprovalRequest. +func (s *ClusterApprovalRequest) SetSpec(spec ApprovalRequestSpec) { + s.Spec = spec +} + +// SetStatus sets the status of the ClusterApprovalRequest. +func (s *ClusterApprovalRequest) SetStatus(status ApprovalRequestStatus) { + s.Status = status +} + +var _ StagedUpdateRunAccessor = &StagedUpdateRun{} +var _ StagedUpdateRunAccessor = &ClusterStagedUpdateRun{} + +// StagedUpdateRunAccessor is an interface that provides unified access to the Spec and Status of +// StagedUpdateRun and ClusterStagedUpdateRun resources. +// +// +kubebuilder:object:generate=false +type StagedUpdateRunAccessor interface { + client.Object + + GetSpec() *StagedUpdateRunSpec + GetStatus() *StagedUpdateRunStatus + + SetSpec(StagedUpdateRunSpec) + SetStatus(StagedUpdateRunStatus) +} + +// GetSpec returns the spec of the StagedUpdateRun. +func (s *StagedUpdateRun) GetSpec() *StagedUpdateRunSpec { + return &s.Spec +} + +// GetStatus returns the status of the StagedUpdateRun. +func (s *StagedUpdateRun) GetStatus() *StagedUpdateRunStatus { + return &s.Status +} + +// SetSpec sets the spec of the StagedUpdateRun. +func (s *StagedUpdateRun) SetSpec(spec StagedUpdateRunSpec) { + s.Spec = spec +} + +// SetStatus sets the status of the StagedUpdateRun. +func (s *StagedUpdateRun) SetStatus(status StagedUpdateRunStatus) { + s.Status = status +} + +// GetSpec returns the spec of the ClusterStagedUpdateRun. +func (s *ClusterStagedUpdateRun) GetSpec() *StagedUpdateRunSpec { + return &s.Spec +} + +// GetStatus returns the status of the ClusterStagedUpdateRun. +func (s *ClusterStagedUpdateRun) GetStatus() *StagedUpdateRunStatus { + return &s.Status +} + +// SetSpec sets the spec of the ClusterStagedUpdateRun. +func (s *ClusterStagedUpdateRun) SetSpec(spec StagedUpdateRunSpec) { + s.Spec = spec +} + +// SetStatus sets the status of the ClusterStagedUpdateRun. +func (s *ClusterStagedUpdateRun) SetStatus(status StagedUpdateRunStatus) { + s.Status = status +} + +var _ StagedUpdateStrategyAccessor = &StagedUpdateStrategy{} +var _ StagedUpdateStrategyAccessor = &ClusterStagedUpdateStrategy{} + +// StagedUpdateStrategyAccessor is an interface that provides unified access to the Spec of +// StagedUpdateStrategy and ClusterStagedUpdateStrategy resources. +// +// +kubebuilder:object:generate=false +type StagedUpdateStrategyAccessor interface { + client.Object + + GetSpec() *StagedUpdateStrategySpec + + SetSpec(StagedUpdateStrategySpec) +} + +// GetSpec returns the spec of the StagedUpdateStrategy. +func (s *StagedUpdateStrategy) GetSpec() *StagedUpdateStrategySpec { + return &s.Spec +} + +// SetSpec sets the spec of the StagedUpdateStrategy. +func (s *StagedUpdateStrategy) SetSpec(spec StagedUpdateStrategySpec) { + s.Spec = spec +} + +// GetSpec returns the spec of the ClusterStagedUpdateStrategy. +func (s *ClusterStagedUpdateStrategy) GetSpec() *StagedUpdateStrategySpec { + return &s.Spec +} + +// SetSpec sets the spec of the ClusterStagedUpdateStrategy. +func (s *ClusterStagedUpdateStrategy) SetSpec(spec StagedUpdateStrategySpec) { + s.Spec = spec +} diff --git a/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_types.go b/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_types.go new file mode 100644 index 000000000..fedadfdc6 --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_types.go @@ -0,0 +1,371 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +const ( + // The condition types of StagedUpdateRun and ClusterStagedUpdateRun API objects. + StagedUpdateRunCondTypeInitialized = "Initialized" + StagedUpdateRunCondTypeStarted = "Started" + StagedUpdateRunCondTypeCompleted = "Completed" + StagedUpdateRunCondTypeSuspended = "Suspended" + + // The condition type of stages to run in StagedUpdateRun and ClusterStagedUpdateRun API objects. + StagedUpdateRunPerStageCondTypeStarted = "Started" + StagedUpdateRunPerStageCondTypeCompleted = "Completed" + + // The condition type of clusters to roll out to in StagedUpdateRun and ClusterStagedUpdateRun API objects. + StagedUpdateRunPerClusterCondTypeStarted = "Started" + StagedUpdateRunPerClusterCondTypeCompleted = "Completed" + + // The condition type of stage tasks to execute in StagedUpdateRun and ClusterStagedUpdateRun API objects. + StagedUpdateRunTaskCondTypeApprovalRequestCreated = "ApprovalRequestCreated" + StagedUpdateRunTaskCondTypeApprovalRequestApproved = "ApprovalRequestApproved" + StagedUpdateRunTaskCondTypeTimedWaitStarted = "TimedWaitStarted" + StagedUpdateRunTaskCondTypeWaitTimeElapsed = "WaitTimeElapsed" +) + +// The reasons for the respective condition types. +const ( + // The reasons for the run-level condition types. + StagedUpdateRunInitializedCondReasonPreppedResourceSnapshotAndAllStages = "PreppedResourceSnapshotAndAllStages" + StagedUpdateRunInitializedCondReasonFailed = "Failed" + + StagedUpdateRunStartedCondReasonUpdateStarted = "UpdateStarted" + + StagedUpdateRunCompletedCondReasonSucceeded = "Succeeded" + StagedUpdateRunCompletedCondReasonPartiallySucceeded = "PartiallySucceeded" + StagedUpdateRunCompletedCondReasonFailed = "Failed" + + StagedUpdateRunSuspendedCondReasonSuspended = "Suspended" + StagedUpdateRunSuspendedCondReasonResumed = "Resumed" + + // The reasons for the per-stage condition types. + StagedUpdateRunPerStageStartedCondReasonUpdateStarted = "UpdateStarted" + + StagedUpdateRunPerStageCompletedCondReasonSucceeded = "Succeeded" + StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded = "PartiallySucceeded" + StagedUpdateRunPerStageCompletedCondReasonFailed = "Failed" + + // The reasons for the per-cluster condition types. + StagedUpdateRunPerClusterStartedCondReasonUpdateStarted = "UpdateStarted" + + StagedUpdateRunPerClusterCompletedCondReasonSucceeded = "UpdateSucceeded" + StagedUpdateRunPerClusterCompletedCondReasonFailed = "FailedToUpdate" + StagedUpdateRunPerClusterCompletedCondReasonCanceled = "UpdateCanceled" + + // The reasons for the per-task condition types. + StagedUpdateRunTaskApprovalRequestCreatedCondReasonCreated = "RequestCreated" + StagedUpdateRunTaskApprovalRequestCreatedCondReasonOwnershipMismatch = "RequestOwnershipMismatch" + StagedUpdateRunTaskApprovalRequestApprovedCondReasonApproved = "RequestApproved" + + StagedUpdateRunTaskTimedWaitStartedCondReasonTimerStarted = "TimerStarted" + StagedUpdateRunTaskWaitTimeElapsedCondReasonTimerElapsed = "TimerElapsed" +) + +// StagedUpdateRun is the KubeFleet API that enables users to roll out resource changes for a placement policy +// in a staged manner, where clusters are grouped into stages and KubeFleet applies resource changes to each stage +// sequentially. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Namespaced,categories={kubefleet,kubefleet-rollout},shortName=updaterun +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type StagedUpdateRun struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the StagedUpdateRun API object. + // + // +kubebuilder:validation:Required + Spec StagedUpdateRunSpec `json:"spec"` + + // The observed status of the StagedUpdateRun API object. + // + // +kubebuilder:validation:Optional + Status StagedUpdateRunStatus `json:"status,omitempty"` +} + +// ClusterStagedUpdateRun is the KubeFleet API that enables users to roll out resource changes for a cluster +// placement policy in a staged manner, where clusters are grouped into stages and KubeFleet applies resource changes to each stage +// sequentially. +// +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster,categories={kubefleet,kubefleet-rollout},shortName=clusterupdaterun +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type ClusterStagedUpdateRun struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the ClusterStagedUpdateRun API object. + // + // +kubebuilder:validation:Required + Spec StagedUpdateRunSpec `json:"spec"` + + // The observed status of the ClusterStagedUpdateRun API object. + // + // +kubebuilder:validation:Optional + Status StagedUpdateRunStatus `json:"status,omitempty"` +} + +// StagedUpdateRunSpec is the spec of the StagedUpdateRun API object. +// +// +kubebuilder:validation:XValidation:rule="has(self.resourceSnapshotName) == has(oldSelf.resourceSnapshotName)",message="resourceSnapshotName cannot be added or removed after creation" +// +kubebuilder:validation:XValidation:rule="has(self.failurePolicy) == has(oldSelf.failurePolicy)",message="failurePolicy cannot be added or removed after creation" +type StagedUpdateRunSpec struct { + // The name of the placement policy that the StagedUpdateRun API object is associated with, i.e., + // the name of the placement policy where resource changes are rolled out. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="placementPolicyName is immutable" + PlacementPolicyName string `json:"placementPolicyName"` + + // The name of the resource snapshot that the StagedUpdateRun API object is associated with, i.e., + // the name of the resource snapshot where resource changes are introduced. + // + // If left empty, KubeFleet will request a new resource snapshot to be created based on the current state + // of selected resources in the placement policy (if applicable). + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="resourceSnapshotName is immutable" + // +kubebuilder:validation:MaxLength=253 + ResourceSnapshotName string `json:"resourceSnapshotName,omitempty"` + + // The name of the staged update strategy that the StagedUpdateRun API object is associated with, i.e., + // the name of the staged update strategy that organizes clusters into stages and defines how the rollout is executed. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="stagedUpdateStrategyName is immutable" + StagedUpdateStrategyName string `json:"stagedUpdateStrategyName"` + + // Whether the staged update run is suspended. If set to true, KubeFleet will pause the rollout. + // + // If the staged update run is created with this field set to true, KubeFleet will initialize the staged update run, + // i.e., decide on the resource snapshot to roll out and determine the stages and the clusters in each stage, + // but will not start rolling out resource changes until the field is set back to false. + // + // When a staged update run is suspended after the rollout starts, KubeFleet will pause the rollout as soon as + // possible. If a cluster has started updating but has not yet completed, KubeFleet will not consider + // its update a failure even if the maximum wait time for the cluster has been exceeded during the suspension. + // The wait timer will be reset once the suspension is lifted. However, the time spent in suspension will + // still count towards any timed wait tasks you have configured; those tasks will proceed or even complete, + // accounting for the time spent in suspension, once the suspension is lifted. + // + // +kubebuilder:validation:Optional + // +kubebuilder:default=false + Suspended bool `json:"suspended,omitempty"` + + // The failure policy of the staged update run. It helps KubeFleet determine when to stop the staged update run + // when there are too many failures, so as to keep the impact radius under control. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="failurePolicy is immutable" + FailurePolicy *StagedUpdateRunFailurePolicy `json:"failurePolicy,omitempty"` +} + +type StagedUpdateRunFailurePolicy struct { + // The maximum number of clusters that can fail the rollout before KubeFleet stops the staged + // update run. The count is tracked across all stages and clusters. A cluster is considered to have + // failed the rollout if the new (updated) placement does not become available within the maximum + // wait time. + // + // The default value is 1; the value must be between 0 and 1000. + // + // +kubebuilder:validation:Optional + // +kubebuilder:default=1 + // +kubebuilder:validation:Minimum=0 + // +kubebuilder:validation:Maximum=1000 + MaxAllowedClusterFailures *int32 `json:"maxAllowedClusterFailures,omitempty"` + + // The maximum time to wait for a rollout to a cluster to complete before KubeFleet considers the rollout to have + // failed for the cluster. + // + // The default value is 5 minutes; the value must be between 1 and 10080 (7 days). + // + // +kubebuilder:validation:Optional + // +kubebuilder:default=5 + // +kubebuilder:validation:Minimum=1 + // +kubebuilder:validation:Maximum=10080 + MaxWaitTimePerClusterMinutes int32 `json:"maxWaitTimePerClusterMinutes,omitempty"` +} + +type StagedUpdateRunStatus struct { + // A list of observed conditions of the staged update run. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` + + // The name of the resource snapshot that is being rolled out in the staged update run. + // + // If the staged update run is created with the resourceSnapshotName field in the spec set to a specific value, + // this field will be set to the same value; otherwise, KubeFleet will request a new resource snapshot to be created (if applicable) + // and populate this field with the name of the newly created resource snapshot. + // + // +kubebuilder:validation:Optional + ResourceSnapshotNameToRollout string `json:"resourceSnapshotNameToRollout,omitempty"` + + // The status of each stage in the staged update run, including the clusters in each stage and their rollout status. + // + // +kubebuilder:validation:Optional + Stages []PerStageStatus `json:"stages,omitempty"` +} + +type PerStageStatus struct { + // The name of the stage. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MaxLength=63 + // +kubebuilder:validation:Pattern="^[a-z0-9]+$" + StageName string `json:"stageName"` + + // The list of clusters featured in the stage, and their individual rollout status. + // + // +kubebuilder:validation:Optional + Clusters []PerClusterStatus `json:"clusters,omitempty"` + + // The list of tasks to be executed before starting the rollout in the stage, and their execution status. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=1 + BeforeStageTasks []PerStageTaskStatus `json:"beforeStageTasks,omitempty"` + + // The list of tasks to be executed after completing the rollout in the stage, and their execution status. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=2 + AfterStageTasks []PerStageTaskStatus `json:"afterStageTasks,omitempty"` + + // A list of observed conditions about the rollout progress in the stage. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` + + // The timestamp when the rollout in the stage started. If unset, the rollout has not started in the stage yet. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:Type=string + // +kubebuilder:validation:Format=date-time + StartedTimestamp *metav1.Time `json:"startedTimestamp,omitempty"` + + // The timestamp when the rollout in the stage completed. If unset, the rollout has not completed in the stage yet. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:Type=string + // +kubebuilder:validation:Format=date-time + CompletedTimestamp *metav1.Time `json:"completedTimestamp,omitempty"` + + // The (resolved) maximum number of clusters that can be updated concurrently within this stage. + // + // +kubebuilder:validation:Optional + MaxConcurrency *int32 `json:"maxConcurrency,omitempty"` +} + +type PerClusterStatus struct { + // The name of the cluster. + // + // +kubebuilder:validation:Required + ClusterName string `json:"clusterName"` + + // The name of the placement binding. + // + // +kubebuilder:validation:Required + PlacementBindingName string `json:"placementBindingName"` + + // A list of observed conditions about the rollout progress of the cluster. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` +} + +type PerStageTaskStatus struct { + // The type of the stage task. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:Enum=TimedWait;Approval + Type StageTaskType `json:"type"` + + // The name of the approval request that is created for the stage task. + // + // This field is only set when the task type is Approval. + // + // +kubebuilder:validation:Optional + ApprovalRequestName string `json:"approvalRequestName,omitempty"` + + // The time to wait in a TimedWait task. + // + // +kubebuilder:validation:Optional + WaitTime *metav1.Duration `json:"waitTime,omitempty"` + + // A list of observed conditions about the progress of the stage task. + // + // +kubebuilder:validation:Optional + // +patchMergeKey=type + // +patchStrategy=merge + // +listType=map + // +listMapKey=type + Conditions []metav1.Condition `json:"conditions,omitempty"` +} + +// The list objects for StagedUpdateRun and ClusterStagedUpdateRun APIs. + +// StagedUpdateRunList contains a list of StagedUpdateRun API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Namespaced" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type StagedUpdateRunList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []StagedUpdateRun `json:"items"` +} + +// ClusterStagedUpdateRunList contains a list of ClusterStagedUpdateRun API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Cluster" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ClusterStagedUpdateRunList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []ClusterStagedUpdateRun `json:"items"` +} + +// Set up the API types with the scheme builder. +func init() { + SchemeBuilder.Register(&StagedUpdateRun{}, &StagedUpdateRunList{}) + SchemeBuilder.Register(&ClusterStagedUpdateRun{}, &ClusterStagedUpdateRunList{}) +} diff --git a/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterunstrategy_types.go b/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterunstrategy_types.go new file mode 100644 index 000000000..af38e01b1 --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterunstrategy_types.go @@ -0,0 +1,196 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" +) + +type StageTaskType string + +const ( + StageTaskTypeTimedWait StageTaskType = "TimedWait" + StageTaskTypeApproval StageTaskType = "Approval" +) + +// StagedUpdateStrategy is the KubeFleet API that defines how to perform a staged rollout of resource changes +// for a placement policy; the API dictates how clusters are grouped into stages, how the rollout is performed +// within each stage, and which tasks to execute before and after each stage. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Namespaced,categories={kubefleet,kubefleet-rollout},shortName=updatestrategy +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type StagedUpdateStrategy struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the StagedUpdateStrategy API object. + // + // +kubebuilder:validation:Required + Spec StagedUpdateStrategySpec `json:"spec"` +} + +// ClusterStagedUpdateStrategy is the KubeFleet API that defines how to perform a staged rollout of resource changes +// for a cluster placement policy; the API dictates how clusters are grouped into stages, how the rollout is performed +// within each stage, and which tasks to execute before and after each stage. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Cluster,categories={kubefleet,kubefleet-rollout},shortName=clusterupdatestrategy +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +// +kubebuilder:storageversion +type ClusterStagedUpdateStrategy struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + // The specification of the ClusterStagedUpdateStrategy API object. + // + // +kubebuilder:validation:Required + Spec StagedUpdateStrategySpec `json:"spec"` +} + +type StagedUpdateStrategySpec struct { + // The stages that KubeFleet rolls out resource changes to, in the order they are listed. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MinItems=1 + // +kubebuilder:validation:MaxItems=31 + // +kubebuilder:validation:XValidation:rule="self.all(s, self.exists_one(t, t.name == s.name))",message="stage names must be unique" + Stages []Stage `json:"stages"` +} + +type Stage struct { + // The name of the stage. It must be unique within the staged update strategy and may only consist of + // lowercase alphanumeric characters. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:MaxLength=63 + // +kubebuilder:validation:Pattern="^[a-z0-9]+$" + Name string `json:"name"` + + // The label selector that selects clusters to be included in the stage. + // + // Note that if a cluster is selected by multiple stages, it will only be included in the first stage that selects it. + // + // If set to nil, the stage includes no clusters at all. If set to an empty selector, the stage includes all applicable clusters. + // + // +kubebuilder:validation:Optional + LabelSelector *metav1.LabelSelector `json:"labelSelector,omitempty"` + + // The label key used to sort clusters in the stage. KubeFleet rolls out resource changes to clusters in the stage + // based on the sorted order. + // + // Specifically, KubeFleet will read the value of the label key for each cluster in the stage, and sort clusters in ascending + // order based on the label value, interpreted as an integer. If the label is missing on any cluster, an error will be raised. + // + // If this field is not set, KubeFleet will sort clusters in the stage based on the ascending lexical order of their names. + // + // +kubebuilder:validation:Optional + SortingLabelKey *string `json:"sortingLabelKey,omitempty"` + + // The maximum number of clusters that can be rolled out concurrently within this stage. + // + // This field accepts either an integer value (e.g., 5), or a percentage value (e.g., "50%"). For percentage values, + // the concurrency number is calculated based on the total number of clusters in the stage, with fractional results rounded down. + // + // A minimum concurrency of 1 is enforced. Note that KubeFleet might further cap the concurrency as needed. + // + // Integer values must be between 1 and 1000. + // + // Defaults to 1. + // + // +kubebuilder:validation:Optional + // +kubebuilder:default=1 + // +kubebuilder:validation:XIntOrString + // +kubebuilder:validation:Pattern="^(100|[1-9][0-9]?)%$" + // +kubebuilder:validation:XValidation:rule="self == null || type(self) != int || self >= 1",message="maxConcurrency must be at least 1" + // +kubebuilder:validation:XValidation:rule="self == null || type(self) != int || self <= 1000",message="maxConcurrency must be at most 1000" + MaxConcurrency *intstr.IntOrString `json:"maxConcurrency,omitempty"` + + // A list of tasks to execute after the stage rollout is completed. The tasks run in parallel; a staged update will only + // proceed to the next stage after all tasks are completed. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=2 + // +kubebuilder:validation:XValidation:rule="self.filter(e, e.type == 'Approval').size() <= 1",message="afterStageTasks cannot have more than one Approval task" + // +kubebuilder:validation:XValidation:rule="self.filter(e, e.type == 'TimedWait').size() <= 1",message="afterStageTasks cannot have more than one TimedWait task" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="waitTime does not apply to an Approval task in afterStageTasks" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait' && !has(e.waitTime))",message="waitTime is required for a TimedWait task in afterStageTasks" + AfterStageTasks []StageTask `json:"afterStageTasks,omitempty"` + + // A list of tasks to execute before the stage rollout starts. The tasks run in parallel; the current stage will only start + // after all tasks are completed. + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=1 + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="waitTime does not apply to an Approval task in beforeStageTasks" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait')",message="beforeStageTasks cannot include a TimedWait task" + BeforeStageTasks []StageTask `json:"beforeStageTasks,omitempty"` +} + +type StageTask struct { + // The type of the stage task. Currently, the supported task types are TimedWait and Approval. + // + // +kubebuilder:validation:Required + // +kubebuilder:validation:Enum=TimedWait;Approval + Type StageTaskType `json:"type"` + + // The time to wait in a TimedWait task. + // + // Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + // The maximum is 720h (30 days). + // + // +kubebuilder:validation:Optional + // +kubebuilder:validation:Type=string + // +kubebuilder:validation:MaxLength=32 + // +kubebuilder:validation:Pattern="^(?:(?:0|[1-9][0-9]*)(\\.[0-9]+)?(?:s|m|h))+$" + // +kubebuilder:validation:XValidation:rule="duration(self) <= duration('720h')",message="waitTime must be at most 720h (30 days)" + WaitTime *metav1.Duration `json:"waitTime,omitempty"` +} + +// The list objects for StagedUpdateStrategy and ClusterStagedUpdateStrategy APIs. + +// StagedUpdateStrategyList contains a list of StagedUpdateStrategy API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Namespaced" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type StagedUpdateStrategyList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []StagedUpdateStrategy `json:"items"` +} + +// ClusterStagedUpdateStrategyList contains a list of ClusterStagedUpdateStrategy API objects. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope="Cluster" +// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object +type ClusterStagedUpdateStrategyList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + + Items []ClusterStagedUpdateStrategy `json:"items"` +} + +// Set up the API types with the scheme builder. +func init() { + SchemeBuilder.Register(&StagedUpdateStrategy{}, &StagedUpdateStrategyList{}) + SchemeBuilder.Register(&ClusterStagedUpdateStrategy{}, &ClusterStagedUpdateStrategyList{}) +} diff --git a/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go b/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go new file mode 100644 index 000000000..6d24d80f6 --- /dev/null +++ b/apis/kubefleet.dev/rollout/v1alpha1/zz_generated.deepcopy.go @@ -0,0 +1,676 @@ +//go:build !ignore_autogenerated + +/* +Copyright 2025 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Code generated by controller-gen. DO NOT EDIT. + +package v1alpha1 + +import ( + "k8s.io/apimachinery/pkg/apis/meta/v1" + runtime "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/util/intstr" +) + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalRequest) DeepCopyInto(out *ApprovalRequest) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + out.Spec = in.Spec + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalRequest. +func (in *ApprovalRequest) DeepCopy() *ApprovalRequest { + if in == nil { + return nil + } + out := new(ApprovalRequest) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ApprovalRequest) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalRequestList) DeepCopyInto(out *ApprovalRequestList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]ApprovalRequest, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalRequestList. +func (in *ApprovalRequestList) DeepCopy() *ApprovalRequestList { + if in == nil { + return nil + } + out := new(ApprovalRequestList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ApprovalRequestList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalRequestSpec) DeepCopyInto(out *ApprovalRequestSpec) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalRequestSpec. +func (in *ApprovalRequestSpec) DeepCopy() *ApprovalRequestSpec { + if in == nil { + return nil + } + out := new(ApprovalRequestSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ApprovalRequestStatus) DeepCopyInto(out *ApprovalRequestStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ApprovalRequestStatus. +func (in *ApprovalRequestStatus) DeepCopy() *ApprovalRequestStatus { + if in == nil { + return nil + } + out := new(ApprovalRequestStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterApprovalRequest) DeepCopyInto(out *ClusterApprovalRequest) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + out.Spec = in.Spec + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterApprovalRequest. +func (in *ClusterApprovalRequest) DeepCopy() *ClusterApprovalRequest { + if in == nil { + return nil + } + out := new(ClusterApprovalRequest) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterApprovalRequest) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterApprovalRequestList) DeepCopyInto(out *ClusterApprovalRequestList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]ClusterApprovalRequest, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterApprovalRequestList. +func (in *ClusterApprovalRequestList) DeepCopy() *ClusterApprovalRequestList { + if in == nil { + return nil + } + out := new(ClusterApprovalRequestList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterApprovalRequestList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterStagedUpdateRun) DeepCopyInto(out *ClusterStagedUpdateRun) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterStagedUpdateRun. +func (in *ClusterStagedUpdateRun) DeepCopy() *ClusterStagedUpdateRun { + if in == nil { + return nil + } + out := new(ClusterStagedUpdateRun) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterStagedUpdateRun) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterStagedUpdateRunList) DeepCopyInto(out *ClusterStagedUpdateRunList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]ClusterStagedUpdateRun, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterStagedUpdateRunList. +func (in *ClusterStagedUpdateRunList) DeepCopy() *ClusterStagedUpdateRunList { + if in == nil { + return nil + } + out := new(ClusterStagedUpdateRunList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterStagedUpdateRunList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterStagedUpdateStrategy) DeepCopyInto(out *ClusterStagedUpdateStrategy) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterStagedUpdateStrategy. +func (in *ClusterStagedUpdateStrategy) DeepCopy() *ClusterStagedUpdateStrategy { + if in == nil { + return nil + } + out := new(ClusterStagedUpdateStrategy) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterStagedUpdateStrategy) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ClusterStagedUpdateStrategyList) DeepCopyInto(out *ClusterStagedUpdateStrategyList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]ClusterStagedUpdateStrategy, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ClusterStagedUpdateStrategyList. +func (in *ClusterStagedUpdateStrategyList) DeepCopy() *ClusterStagedUpdateStrategyList { + if in == nil { + return nil + } + out := new(ClusterStagedUpdateStrategyList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *ClusterStagedUpdateStrategyList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PerClusterStatus) DeepCopyInto(out *PerClusterStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PerClusterStatus. +func (in *PerClusterStatus) DeepCopy() *PerClusterStatus { + if in == nil { + return nil + } + out := new(PerClusterStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PerStageStatus) DeepCopyInto(out *PerStageStatus) { + *out = *in + if in.Clusters != nil { + in, out := &in.Clusters, &out.Clusters + *out = make([]PerClusterStatus, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.BeforeStageTasks != nil { + in, out := &in.BeforeStageTasks, &out.BeforeStageTasks + *out = make([]PerStageTaskStatus, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.AfterStageTasks != nil { + in, out := &in.AfterStageTasks, &out.AfterStageTasks + *out = make([]PerStageTaskStatus, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.StartedTimestamp != nil { + in, out := &in.StartedTimestamp, &out.StartedTimestamp + *out = (*in).DeepCopy() + } + if in.CompletedTimestamp != nil { + in, out := &in.CompletedTimestamp, &out.CompletedTimestamp + *out = (*in).DeepCopy() + } + if in.MaxConcurrency != nil { + in, out := &in.MaxConcurrency, &out.MaxConcurrency + *out = new(int32) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PerStageStatus. +func (in *PerStageStatus) DeepCopy() *PerStageStatus { + if in == nil { + return nil + } + out := new(PerStageStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *PerStageTaskStatus) DeepCopyInto(out *PerStageTaskStatus) { + *out = *in + if in.WaitTime != nil { + in, out := &in.WaitTime, &out.WaitTime + *out = new(v1.Duration) + **out = **in + } + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PerStageTaskStatus. +func (in *PerStageTaskStatus) DeepCopy() *PerStageTaskStatus { + if in == nil { + return nil + } + out := new(PerStageTaskStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *Stage) DeepCopyInto(out *Stage) { + *out = *in + if in.LabelSelector != nil { + in, out := &in.LabelSelector, &out.LabelSelector + *out = new(v1.LabelSelector) + (*in).DeepCopyInto(*out) + } + if in.SortingLabelKey != nil { + in, out := &in.SortingLabelKey, &out.SortingLabelKey + *out = new(string) + **out = **in + } + if in.MaxConcurrency != nil { + in, out := &in.MaxConcurrency, &out.MaxConcurrency + *out = new(intstr.IntOrString) + **out = **in + } + if in.AfterStageTasks != nil { + in, out := &in.AfterStageTasks, &out.AfterStageTasks + *out = make([]StageTask, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.BeforeStageTasks != nil { + in, out := &in.BeforeStageTasks, &out.BeforeStageTasks + *out = make([]StageTask, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Stage. +func (in *Stage) DeepCopy() *Stage { + if in == nil { + return nil + } + out := new(Stage) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StageTask) DeepCopyInto(out *StageTask) { + *out = *in + if in.WaitTime != nil { + in, out := &in.WaitTime, &out.WaitTime + *out = new(v1.Duration) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StageTask. +func (in *StageTask) DeepCopy() *StageTask { + if in == nil { + return nil + } + out := new(StageTask) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRun) DeepCopyInto(out *StagedUpdateRun) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRun. +func (in *StagedUpdateRun) DeepCopy() *StagedUpdateRun { + if in == nil { + return nil + } + out := new(StagedUpdateRun) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *StagedUpdateRun) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRunFailurePolicy) DeepCopyInto(out *StagedUpdateRunFailurePolicy) { + *out = *in + if in.MaxAllowedClusterFailures != nil { + in, out := &in.MaxAllowedClusterFailures, &out.MaxAllowedClusterFailures + *out = new(int32) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRunFailurePolicy. +func (in *StagedUpdateRunFailurePolicy) DeepCopy() *StagedUpdateRunFailurePolicy { + if in == nil { + return nil + } + out := new(StagedUpdateRunFailurePolicy) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRunList) DeepCopyInto(out *StagedUpdateRunList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]StagedUpdateRun, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRunList. +func (in *StagedUpdateRunList) DeepCopy() *StagedUpdateRunList { + if in == nil { + return nil + } + out := new(StagedUpdateRunList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *StagedUpdateRunList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRunSpec) DeepCopyInto(out *StagedUpdateRunSpec) { + *out = *in + if in.FailurePolicy != nil { + in, out := &in.FailurePolicy, &out.FailurePolicy + *out = new(StagedUpdateRunFailurePolicy) + (*in).DeepCopyInto(*out) + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRunSpec. +func (in *StagedUpdateRunSpec) DeepCopy() *StagedUpdateRunSpec { + if in == nil { + return nil + } + out := new(StagedUpdateRunSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateRunStatus) DeepCopyInto(out *StagedUpdateRunStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } + if in.Stages != nil { + in, out := &in.Stages, &out.Stages + *out = make([]PerStageStatus, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateRunStatus. +func (in *StagedUpdateRunStatus) DeepCopy() *StagedUpdateRunStatus { + if in == nil { + return nil + } + out := new(StagedUpdateRunStatus) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateStrategy) DeepCopyInto(out *StagedUpdateStrategy) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateStrategy. +func (in *StagedUpdateStrategy) DeepCopy() *StagedUpdateStrategy { + if in == nil { + return nil + } + out := new(StagedUpdateStrategy) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *StagedUpdateStrategy) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateStrategyList) DeepCopyInto(out *StagedUpdateStrategyList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]StagedUpdateStrategy, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateStrategyList. +func (in *StagedUpdateStrategyList) DeepCopy() *StagedUpdateStrategyList { + if in == nil { + return nil + } + out := new(StagedUpdateStrategyList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *StagedUpdateStrategyList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *StagedUpdateStrategySpec) DeepCopyInto(out *StagedUpdateStrategySpec) { + *out = *in + if in.Stages != nil { + in, out := &in.Stages, &out.Stages + *out = make([]Stage, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new StagedUpdateStrategySpec. +func (in *StagedUpdateStrategySpec) DeepCopy() *StagedUpdateStrategySpec { + if in == nil { + return nil + } + out := new(StagedUpdateStrategySpec) + in.DeepCopyInto(out) + return out +} diff --git a/apis/placement/v1alpha1/zz_generated.deepcopy.go b/apis/placement/v1alpha1/zz_generated.deepcopy.go index ec4b45d86..086e1ff64 100644 --- a/apis/placement/v1alpha1/zz_generated.deepcopy.go +++ b/apis/placement/v1alpha1/zz_generated.deepcopy.go @@ -21,11 +21,10 @@ limitations under the License. package v1alpha1 import ( - v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "go.goms.io/fleet/apis/placement/v1beta1" + "k8s.io/apimachinery/pkg/apis/meta/v1" runtime "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/intstr" - - "go.goms.io/fleet/apis/placement/v1beta1" ) // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. diff --git a/apis/placement/v1beta1/commons.go b/apis/placement/v1beta1/commons.go index 6631cb7f0..f4781025f 100644 --- a/apis/placement/v1beta1/commons.go +++ b/apis/placement/v1beta1/commons.go @@ -186,6 +186,11 @@ const ( // UpdateRunDeleteStageName is the name of delete stage in the staged update run. UpdateRunDeleteStageName = FleetPrefix + "deleteStage" + // UpdateRunDeleteStageTaskName refers to the delete stage in the names and the labels of the objects created + // for its tasks, where UpdateRunDeleteStageName is not a valid value. It cannot collide with the name of an + // update stage, as that cannot have a hyphen. + UpdateRunDeleteStageTaskName = "delete-stage" + // IsLatestUpdateRunApprovalLabel indicates if the approval is the latest approval on a staged run. IsLatestUpdateRunApprovalLabel = FleetPrefix + "isLatestUpdateRunApproval" diff --git a/apis/placement/v1beta1/stageupdate_types.go b/apis/placement/v1beta1/stageupdate_types.go index 7c6fe3842..2300b3b84 100644 --- a/apis/placement/v1beta1/stageupdate_types.go +++ b/apis/placement/v1beta1/stageupdate_types.go @@ -262,10 +262,29 @@ func (c *ClusterStagedUpdateStrategy) SetUpdateStrategySpec(spec UpdateStrategyS // UpdateStrategySpec defines the desired state of the StagedUpdateStrategy. type UpdateStrategySpec struct { - // Stage specifies the configuration for each update stage. + // Stages specifies the configuration for each update stage. // +kubebuilder:validation:MaxItems=31 // +kubebuilder:validation:Required Stages []StageConfig `json:"stages"` + + // DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + // complete and removes the resources from the clusters that are no longer selected by the placement. + // If not specified, the delete stage starts as soon as the last update stage completes. + // +kubebuilder:validation:Optional + DeleteStage *DeleteStageConfig `json:"deleteStage,omitempty"` +} + +// DeleteStageConfig describes the delete stage. +type DeleteStageConfig struct { + // The collection of tasks that needs to be completed successfully before the delete stage starts removing + // the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + // before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + // The tasks are skipped if there is no cluster to remove the resources from. + // +kubebuilder:validation:Optional + // +kubebuilder:validation:MaxItems=1 + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="BeforeStageTaskType is Approval, waitTime is not allowed" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait')",message="BeforeStageTaskType cannot be TimedWait" + BeforeStageTasks []StageTask `json:"beforeStageTasks,omitempty"` } // ClusterStagedUpdateStrategyList contains a list of StagedUpdateStrategy. @@ -330,11 +349,11 @@ type StageConfig struct { // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait' && !has(e.waitTime))",message="AfterStageTaskType is TimedWait, waitTime is required" AfterStageTasks []StageTask `json:"afterStageTasks,omitempty"` - // The collection of tasks that needs to completed successfully by each stage before starting the stage. + // The collection of tasks that needs to be completed successfully by each stage before starting the stage. // Each task is executed in parallel and there cannot be more than one task of the same type. // +kubebuilder:validation:Optional // +kubebuilder:validation:MaxItems=1 - // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="AfterStageTaskType is Approval, waitTime is not allowed" + // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'Approval' && has(e.waitTime))",message="BeforeStageTaskType is Approval, waitTime is not allowed" // +kubebuilder:validation:XValidation:rule="!self.exists(e, e.type == 'TimedWait')",message="BeforeStageTaskType cannot be TimedWait" BeforeStageTasks []StageTask `json:"beforeStageTasks,omitempty"` } @@ -373,7 +392,7 @@ type UpdateRunStatus struct { // ResourceSnapshotIndexUsed records the resource snapshot index that the update run is based on. // The index represents the same resource snapshots as specified in the spec field, or the latest. - // +kubbebuilder:validation:Optional + // +kubebuilder:validation:Optional ResourceSnapshotIndexUsed string `json:"resourceSnapshotIndexUsed,omitempty"` // ApplyStrategy is the apply strategy that the stagedUpdateRun is using. @@ -548,6 +567,7 @@ const ( ClusterUpdatingConditionSucceeded ClusterUpdatingStatusConditionType = "Succeeded" ) +// StageTaskStatus defines the status of a pre or post update task of a stage. type StageTaskStatus struct { // The type of the pre or post update task. // +kubebuilder:validation:Enum=TimedWait;Approval @@ -555,7 +575,7 @@ type StageTaskStatus struct { Type StageTaskType `json:"type"` // The name of the approval request object that is created for this stage. - // Only valid if the AfterStageTaskType is Approval. + // Only valid if the type of the task is Approval. // +kubebuilder:validation:Optional ApprovalRequestName string `json:"approvalRequestName,omitempty"` diff --git a/apis/placement/v1beta1/zz_generated.deepcopy.go b/apis/placement/v1beta1/zz_generated.deepcopy.go index 73d66c8fa..3561518d1 100644 --- a/apis/placement/v1beta1/zz_generated.deepcopy.go +++ b/apis/placement/v1beta1/zz_generated.deepcopy.go @@ -1266,6 +1266,28 @@ func (in *ClusterUpdatingStatus) DeepCopy() *ClusterUpdatingStatus { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *DeleteStageConfig) DeepCopyInto(out *DeleteStageConfig) { + *out = *in + if in.BeforeStageTasks != nil { + in, out := &in.BeforeStageTasks, &out.BeforeStageTasks + *out = make([]StageTask, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DeleteStageConfig. +func (in *DeleteStageConfig) DeepCopy() *DeleteStageConfig { + if in == nil { + return nil + } + out := new(DeleteStageConfig) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *DeleteStrategy) DeepCopyInto(out *DeleteStrategy) { *out = *in @@ -3044,6 +3066,11 @@ func (in *UpdateStrategySpec) DeepCopyInto(out *UpdateStrategySpec) { (*in)[i].DeepCopyInto(&(*out)[i]) } } + if in.DeleteStage != nil { + in, out := &in.DeleteStage, &out.DeleteStage + *out = new(DeleteStageConfig) + (*in).DeepCopyInto(*out) + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new UpdateStrategySpec. diff --git a/apis/protos/azure/compute/v1/vmsizerecommender.pb.go b/apis/protos/azure/compute/v1/vmsizerecommender.pb.go index 243b11763..543b75155 100644 --- a/apis/protos/azure/compute/v1/vmsizerecommender.pb.go +++ b/apis/protos/azure/compute/v1/vmsizerecommender.pb.go @@ -11,12 +11,11 @@ package v1 import ( + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" reflect "reflect" sync "sync" unsafe "unsafe" - - protoreflect "google.golang.org/protobuf/reflect/protoreflect" - protoimpl "google.golang.org/protobuf/runtime/protoimpl" ) const ( diff --git a/apis/protos/azure/compute/v1/vmsizerecommender_grpc.pb.go b/apis/protos/azure/compute/v1/vmsizerecommender_grpc.pb.go index b3205246b..9d00f1924 100644 --- a/apis/protos/azure/compute/v1/vmsizerecommender_grpc.pb.go +++ b/apis/protos/azure/compute/v1/vmsizerecommender_grpc.pb.go @@ -12,7 +12,6 @@ package v1 import ( context "context" - grpc "google.golang.org/grpc" codes "google.golang.org/grpc/codes" status "google.golang.org/grpc/status" diff --git a/charts/README.md b/charts/README.md index e125e2cf7..1067b2f04 100644 --- a/charts/README.md +++ b/charts/README.md @@ -9,11 +9,11 @@ This directory contains Helm charts for deploying KubeFleet components. ## Chart Versioning -**Important:** Chart versions match the KubeFleet release versions. When a KubeFleet release is tagged (e.g., `v0.3.0`), the Helm charts are published with the same version (`0.3.0`). +**Important:** Chart versions match the KubeFleet release versions. When a KubeFleet release is tagged (e.g., `v0.3.2`), the Helm charts are published with the same version (`0.3.2`). -**Example:** To install KubeFleet v0.3.0, use: +**Example:** To install KubeFleet v0.3.2, use: ```bash -helm install hub-agent oci://ghcr.io/kubefleet-dev/kubefleet/charts/hub-agent --version 0.3.0 --namespace fleet-system --create-namespace +helm install hub-agent oci://ghcr.io/kubefleet-dev/kubefleet/charts/hub-agent --version 0.3.2 --namespace fleet-system --create-namespace ``` This ensures consistency between the application version and the chart version, making it easy to know which chart version to use with each KubeFleet release. @@ -79,9 +79,9 @@ helm install member-agent kubefleet/member-agent \ #### OCI Registry ```bash -# Install a specific version from OCI registry (e.g., v0.3.0 release) +# Install a specific version from OCI registry (e.g., v0.3.2 release) helm install hub-agent oci://ghcr.io/kubefleet-dev/kubefleet/charts/hub-agent \ - --version 0.3.0 \ + --version 0.3.2 \ --namespace fleet-system \ --create-namespace ``` @@ -92,9 +92,9 @@ helm install hub-agent oci://ghcr.io/kubefleet-dev/kubefleet/charts/hub-agent \ # List available versions helm search repo kubefleet --versions -# Install a specific version (e.g., v0.3.0 release) +# Install a specific version (e.g., v0.3.2 release) helm install hub-agent kubefleet/hub-agent \ - --version 0.3.0 \ + --version 0.3.2 \ --namespace fleet-system \ --create-namespace ``` @@ -104,13 +104,13 @@ helm install hub-agent kubefleet/hub-agent \ #### OCI Registry ```bash -# Upgrade to a specific version (e.g., v0.3.0) +# Upgrade to a specific version (e.g., v0.3.2) helm upgrade hub-agent oci://ghcr.io/kubefleet-dev/kubefleet/charts/hub-agent \ - --version 0.3.0 \ + --version 0.3.2 \ --namespace fleet-system helm upgrade member-agent oci://ghcr.io/kubefleet-dev/kubefleet/charts/member-agent \ - --version 0.3.0 \ + --version 0.3.2 \ --namespace fleet-system ``` diff --git a/charts/hub-agent/README.md b/charts/hub-agent/README.md index 7cc93551b..2761cc182 100644 --- a/charts/hub-agent/README.md +++ b/charts/hub-agent/README.md @@ -2,7 +2,7 @@ ## Chart Versioning -Chart versions match the KubeFleet release versions. For example, to install KubeFleet v0.3.0, use chart version `0.3.0`. +Chart versions match the KubeFleet release versions. For example, to install KubeFleet v0.3.2, use chart version `0.3.2`. ## Install Chart @@ -108,8 +108,10 @@ _See [helm install](https://helm.sh/docs/helm/helm_install/) for command documen | `webhookCertSecretName` | Name of the Secret where cert-manager stores the certificate (required when enabled) | `unset` | | `enableClusterInventoryAPI` | Enable cluster inventory APIs | `true` | | `enableStagedUpdateRunAPIs` | Enable staged update run APIs | `true` | +| `enablePlacementPolicyAPIs` | Enable the experimental FEP-0001 placement policy APIs (`placement.kubefleet.dev/v1alpha1`). Their controllers use fixed concurrency; `MaxConcurrentClusterPlacement` and `MaxFleetSizeSupported` don't apply to them. If `crdInstaller.enabled` is `true`, the CRD installer also installs the CRDs of these APIs (`placement.kubefleet.dev` and `rollout.kubefleet.dev`); otherwise, these CRDs must be installed separately | `false` | +| `enableAnnotationBasedPlacement` | Enable the experimental annotation-based placement experience (the `kubefleet.dev/cluster-selectors` annotation); requires `enablePlacementPolicyAPIs` to be `true`. Only enable it if KubeFleet runs in a fully trusted environment, or if the hub cluster has the workload controllers of kube-controller-manager (e.g., the Deployment, StatefulSet, DaemonSet, and CronJob controllers) disabled and the protection validating admission policy (`DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess`) installed; setting `enableWorkload` to `false` does not disable these controllers | `false` | | `enableEvictionAPIs` | Enable eviction APIs | `true` | -| `enablePprof` | Enable pprof endpoint | `true` | +| `enablePprof` | Enable the pprof endpoint (listens on loopback only; use `kubectl port-forward`) | `false` | | `pprofPort` | pprof server port | `6065` | | `hubAPIQPS` | QPS for fleet-apiserver (not including events/node heartbeat) | `250` | | `hubAPIBurst` | Burst for fleet-apiserver (not including events/node heartbeat) | `1000` | diff --git a/charts/hub-agent/templates/crds/crps.yaml b/charts/hub-agent/templates/crds/crps.yaml deleted file mode 100644 index 5701e114a..000000000 --- a/charts/hub-agent/templates/crds/crps.yaml +++ /dev/null @@ -1,2 +0,0 @@ -{{- $files := .Files }} -{{ $files.Get "crdbases/placement.kubernetes-fleet.io_clusterresourceplacements.yaml" }} diff --git a/charts/hub-agent/templates/crds/internalmemberclusters.yaml b/charts/hub-agent/templates/crds/internalmemberclusters.yaml deleted file mode 100644 index 321ae5430..000000000 --- a/charts/hub-agent/templates/crds/internalmemberclusters.yaml +++ /dev/null @@ -1,2 +0,0 @@ -{{- $files := .Files }} -{{ $files.Get "crdbases/cluster.kubernetes-fleet.io_internalmemberclusters.yaml" }} diff --git a/charts/hub-agent/templates/crds/memberclusters.yaml b/charts/hub-agent/templates/crds/memberclusters.yaml deleted file mode 100644 index 79e4692ff..000000000 --- a/charts/hub-agent/templates/crds/memberclusters.yaml +++ /dev/null @@ -1,2 +0,0 @@ -{{- $files := .Files }} -{{ $files.Get "crdbases/cluster.kubernetes-fleet.io_memberclusters.yaml" }} diff --git a/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_approvalrequests.yaml b/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_approvalrequests.yaml deleted file mode 120000 index 044efe183..000000000 --- a/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_approvalrequests.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../../config/crd/bases/placement.kubernetes-fleet.io_approvalrequests.yaml \ No newline at end of file diff --git a/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_stagedupdateruns.yaml b/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_stagedupdateruns.yaml deleted file mode 120000 index 1d44518b1..000000000 --- a/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_stagedupdateruns.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../../config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml \ No newline at end of file diff --git a/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml b/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml deleted file mode 120000 index 211d5ddda..000000000 --- a/charts/hub-agent/templates/crds/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../../config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml \ No newline at end of file diff --git a/charts/hub-agent/templates/crds/works.yaml b/charts/hub-agent/templates/crds/works.yaml deleted file mode 100644 index 5ed5889f1..000000000 --- a/charts/hub-agent/templates/crds/works.yaml +++ /dev/null @@ -1,2 +0,0 @@ -{{- $files := .Files }} -{{ $files.Get "crdbases/placement.kubernetes-fleet.io_works.yaml" }} diff --git a/charts/hub-agent/templates/deployment.yaml b/charts/hub-agent/templates/deployment.yaml index d6c39fbc1..a84bffbfc 100644 --- a/charts/hub-agent/templates/deployment.yaml +++ b/charts/hub-agent/templates/deployment.yaml @@ -34,6 +34,9 @@ spec: args: - --mode=hub - --v={{ .Values.crdInstaller.logVerbosity }} + {{- if .Values.enablePlacementPolicyAPIs }} + - --enable-placement-policy-apis=true + {{- end }} {{- end }} containers: - name: {{ .Chart.Name }} @@ -58,6 +61,8 @@ spec: - -add_dir_header - --enable-cluster-inventory-apis={{ .Values.enableClusterInventoryAPI }} - --enable-staged-update-run-apis={{ .Values.enableStagedUpdateRunAPIs }} + - --enable-placement-policy-apis={{ .Values.enablePlacementPolicyAPIs }} + - --enable-annotation-based-placement={{ .Values.enableAnnotationBasedPlacement }} - --enable-eviction-apis={{ .Values.enableEvictionAPIs}} - --enable-pprof={{ .Values.enablePprof }} - --pprof-port={{ .Values.pprofPort }} diff --git a/charts/hub-agent/values.yaml b/charts/hub-agent/values.yaml index 04bca34a7..7b4316572 100644 --- a/charts/hub-agent/values.yaml +++ b/charts/hub-agent/values.yaml @@ -56,9 +56,19 @@ affinity: {} enableClusterInventoryAPI: true enableStagedUpdateRunAPIs: true +# The experimental FEP-0001 placement policy APIs (placement.kubefleet.dev/v1alpha1). +# If crdInstaller is enabled, this also has the CRD installer install the CRDs of these APIs +# ([placement|rollout].kubefleet.dev); otherwise, these CRDs must be installed separately. +enablePlacementPolicyAPIs: false +# The experimental annotation-based placement experience; requires enablePlacementPolicyAPIs to be true. +# Only enable it if KubeFleet runs in a fully trusted environment, or if the hub cluster has the workload controllers +# of kube-controller-manager (e.g., the Deployment, StatefulSet, DaemonSet, and CronJob controllers) disabled and the +# protection validating admission policy (DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess) installed. Note that +# setting enableWorkload to false does not disable these controllers. +enableAnnotationBasedPlacement: false enableEvictionAPIs: true -enablePprof: true +enablePprof: false pprofPort: 6065 hubAPIQPS: 250 diff --git a/charts/member-agent/README.md b/charts/member-agent/README.md index b5429039b..04a4afb72 100644 --- a/charts/member-agent/README.md +++ b/charts/member-agent/README.md @@ -2,7 +2,7 @@ ## Chart Versioning -Chart versions match the KubeFleet release versions. For example, to install KubeFleet v0.3.0, use chart version `0.3.0`. +Chart versions match the KubeFleet release versions. For example, to install KubeFleet v0.3.2, use chart version `0.3.2`. ## Install Chart @@ -103,8 +103,14 @@ helm upgrade member-agent kubefleet/member-agent --namespace fleet-system | logVerbosity | Log level. Uses V logs (klog) | `3` | | tlsClientInsecure | Skip TLS server certificate verification when the member agent connects to the hub cluster. Leave this `false` unless you explicitly trust the endpoint and understand the risk. | `false` | | useCAAuth | Use certificate-based authentication for the hub connection instead of the token-based path. | `false` | +| useKubeconfig | Authenticate to the hub cluster via a full kubeconfig instead of the refresh-token sidecar or `useCAAuth`. The vendor-neutral way to authenticate via a federated identity. Mutually exclusive with `useCAAuth`. See [Hub connection via kubeconfig](#hub-connection-via-kubeconfig-federated-identity-any-vendor). | `false` | +| config.hubKubeconfigSecretName | The name of a Secret (key `kubeconfig`) containing a full kubeconfig for the hub connection, mounted and passed to the process via the `KUBE_CONFIG_PATH` environment variable. Required (the chart fails to render otherwise) when `useKubeconfig` is `true`. | `` | +| podLabels | Extra labels applied to the member-agent pod template — e.g. workload-identity labels a `hubKubeconfigSecretName` exec plugin needs. | `{}` | +| serviceAccountAnnotations | Extra annotations applied to the member-agent ServiceAccount — e.g. workload-identity annotations a `hubKubeconfigSecretName` exec plugin needs. | `{}` | +| extraVolumes / extraVolumeMounts / extraInitContainers | Standard passthroughs (rendered via `toYaml`) for getting an exec plugin binary, or a workload-API socket, into the pod without a custom image. `extraVolumeMounts` applies to the member-agent container. | `[]` | | propertyProvider | The property provider to use with the member agent; if none is specified, the Fleet member agent will start with no property provider (i.e., the agent will expose no cluster properties, and collect only limited resource usage information) | `` | | region | The region where the member cluster resides | `` | +| enablePlacementPolicyAPIs | Apply the Works of the experimental FEP-0001 placement policy APIs (`placement.kubefleet.dev/v1alpha1`); the hub agent must enable them too. Their work applier uses fixed settings; the `workApplier*` and `priorityQueue` values don't apply to it. If `crdInstaller.enabled` is `true`, the CRD installer also installs the AppliedWork CRD of these APIs; otherwise, the CRD must be installed separately | `false` | | enableNamespaceCollectionInPropertyProvider | Enable namespace collection in the property provider; when enabled, the member agent will collect and report the list of namespaces present in the member cluster to the hub cluster for use in scheduling decisions | `false` | | workApplierRequeueRateLimiterAttemptsWithFixedDelay | This parameter is a set of values to control how frequent KubeFleet should reconcile (processed) manifests; it specifies then number of attempts to requeue with fixed delay before switching to exponential backoff | `1` | | workApplierRequeueRateLimiterFixedDelaySeconds | This parameter is a set of values to control how frequent KubeFleet should reconcile (process) manifests; it specifies the fixed delay in seconds for initial requeue attempts | `5` | @@ -123,6 +129,357 @@ By default, the chart keeps TLS server certificate verification enabled for the Set `tlsClientInsecure=true` only for explicitly trusted test environments where certificate verification cannot be configured. +## Hub connection via kubeconfig (federated identity, any vendor) + +As an alternative to the refresh-token sidecar, the member agent can load a full kubeconfig for the +hub connection (`useKubeconfig` + `config.hubKubeconfigSecretName`, i.e. `--use-kubeconfig` plus the +`KUBE_CONFIG_PATH` environment variable), the same way `kubectl` does. +This mode requires no sidecar container and no shared token volume — `client-go` handles token +caching and refresh itself, whatever credential mechanism the kubeconfig's `users[].user.exec` +stanza invokes. + +This chart, and KubeFleet's own code, have **no vendor-specific knowledge** of any cloud's identity +system — the kubeconfig you supply can invoke whatever exec-credential plugin your platform +provides (`client.authentication.k8s.io`, the same mechanism `kubectl` itself uses). Everything +vendor-specific — the plugin binary, its arguments, and any workload-identity metadata — lives +entirely in the kubeconfig you write and the `podLabels`/`serviceAccountAnnotations` values you set; +none of it is baked into this chart. + +**Kubeconfig mode is fully authoritative for the hub connection.** Once `useKubeconfig` is set, the +loaded kubeconfig's own server URL, TLS configuration, and authentication are used as-is; none of +the following are consulted, even if also set: `config.hubURL`/`HUB_SERVER_URL`, `useCAAuth`, +`tlsClientInsecure`/`--tls-insecure`, `config.CABundle`/`CA_BUNDLE`, `config.hubCA`/ +`HUB_CERTIFICATE_AUTHORITY`, and the refresh-token sidecar's `CONFIG_PATH`/`IDENTITY_KEY`/ +`IDENTITY_CERT`. If you need TLS verification disabled for testing, set +`insecure-skip-tls-verify: true` on the kubeconfig's own `cluster` entry rather than +`tlsClientInsecure` — the latter has no effect in this mode. + +**Security considerations**: the hub kubeconfig can carry — directly or via its exec plugin's +own credentials — everything needed to authenticate to the hub cluster, so treat the Secret +holding it like any other high-value credential (restrict who can `get`/`list` it via RBAC, as +you would `hub-kubeconfig-secret` or `hub-identity` for the other modes). Prefer mounting the +kubeconfig and any exec-plugin binary via Secrets/`extraVolumes` rather than embedding secret +material in Helm values or `--set` flags — values and `--set` arguments can end up in shell +history, `helm get values` output, or process listings, none of which apply to a Secret mounted +as a file. + +Three worked examples: + +### Azure / AKS (kubelogin) + +`kubelogin` covers three Azure identity mechanisms — federated (workload) identity, managed +identity, and a service principal's client secret or certificate — selected by the `--login` mode +in its exec stanza. All three use the same `--use-kubeconfig`/`KUBE_CONFIG_PATH` mechanism and +differ only in the kubeconfig content and prerequisites shown below. + +However you build the kubeconfig for any of the three, package it into a Secret the same way +(`kubectl create secret generic hub-kubeconfig --from-file=kubeconfig=./kubeconfig.yaml`), shown in +each example below. + +`kubelogin` also needs to be present in the member-agent image for all three — either build a small +custom image on top of the official one: +```dockerfile +FROM ghcr.io/kubefleet-dev/kubefleet/member-agent: +COPY --from=ghcr.io/azure/kubelogin: /usr/local/bin/kubelogin /usr/local/bin/kubelogin +``` +or use `extraInitContainers`/`extraVolumes`/`extraVolumeMounts` to place it on a shared volume at +runtime instead — the official `ghcr.io/azure/kubelogin` image is `scratch`-based (no shell), so the +init container has to be a different, shell-capable image that fetches the release binary itself: + +```yaml +# values-kubelogin-init.yaml +extraVolumes: +- name: kubelogin-bin + emptyDir: {} +extraVolumeMounts: +- name: kubelogin-bin + mountPath: /usr/local/bin +extraInitContainers: +- name: install-kubelogin + image: busybox:1.36 + # Match the release asset to the node architecture (kubelogin-linux-arm64.zip on arm64 nodes). + command: + - sh + - -c + - | + wget -qO /tmp/kubelogin.zip https://github.com/Azure/kubelogin/releases/download/v0.2.19/kubelogin-linux-amd64.zip + unzip -oq /tmp/kubelogin.zip -d /tmp + cp /tmp/bin/linux_amd64/kubelogin /kubelogin-bin/kubelogin + chmod +x /kubelogin-bin/kubelogin + volumeMounts: + - name: kubelogin-bin + mountPath: /kubelogin-bin +``` +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set-string podLabels."azure\.workload\.identity/use"=true \ + --set serviceAccountAnnotations."azure\.workload\.identity/client-id"= +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +`kubelogin-bin` is mounted at `/usr/local/bin` on the member-agent container too, so it works with +the bare `command: kubelogin` shown in every example below — no image build required, at the cost +of a network fetch on every pod start. + +#### Federated identity (`--login workloadidentity`) + +Set up [AKS workload identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) on the member cluster, and grant the federated managed identity the appropriate RBAC permissions on the hub cluster (an AKS cluster with Azure AD authentication enabled). Then: + +```yaml +# kubeconfig.yaml +apiVersion: v1 +kind: Config +clusters: +- cluster: + certificate-authority-data: + server: https:// + name: hub +contexts: +- context: + cluster: hub + user: fed + name: hub +current-context: hub +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --environment + - AzurePublicCloud + - --server-id + - + - --client-id + - + - --tenant-id + - + - --login + - workloadidentity + provideClusterInfo: false + interactiveMode: Never +``` + +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set-string podLabels."azure\.workload\.identity/use"=true \ + --set serviceAccountAnnotations."azure\.workload\.identity/client-id"= +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +#### Managed identity (`--login msi`) + +An alternative to federated identity: uses the IMDS-based managed identity available on any Azure +VM/node — so it needs no AKS workload identity setup, no `podLabels`, and no +`serviceAccountAnnotations`. Grant the managed identity the appropriate RBAC permissions on the hub +cluster, then: + +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --server-id + - + - --login + - msi + - --client-id # omit this flag entirely to use the system-assigned identity instead + - + provideClusterInfo: false + interactiveMode: Never +``` + +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +#### Client secret / client certificate (`--login spn`) + +Authenticates as an Azure AD app registration (service principal) using a client secret or a client +certificate, via `--client-id`/`--client-secret`/`--client-certificate`/`--tenant-id` — the same +flag-based style as the federated-identity and managed-identity variants above. + +**Client secret flow:** +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --server-id + - + - --login + - spn + - --client-id + - + - --client-secret + - + - --tenant-id + - + provideClusterInfo: false + interactiveMode: Never +``` +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +**Client certificate flow:** `--client-certificate` must point to a file on disk, so the +certificate needs its own mount via `extraVolumes`/`extraVolumeMounts` rather than living inline in +the kubeconfig: +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --server-id + - + - --login + - spn + - --client-id + - + - --client-certificate + - /etc/kubefleet/spn-cert/tls.crt + - --tenant-id + - + provideClusterInfo: false + interactiveMode: Never +``` +This flow also needs its own Secret for the certificate: +```console +kubectl create secret tls spn-cert --namespace fleet-system --cert=/path/to/spn.crt --key=/path/to/spn.key +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + -f values-kubelogin-init.yaml \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set 'extraVolumes[1].name=spn-cert' \ + --set 'extraVolumes[1].secret.secretName=spn-cert' \ + --set 'extraVolumeMounts[1].name=spn-cert' \ + --set 'extraVolumeMounts[1].mountPath=/etc/kubefleet/spn-cert' \ + --set 'extraVolumeMounts[1].readOnly=true' +``` +`values-kubelogin-init.yaml` already occupies index `0` of `extraVolumes`/`extraVolumeMounts` (the +`kubelogin-bin` `emptyDir`), so the certificate Secret's volume/mount use index `1` here to avoid +overwriting it. +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +### AWS / EKS (IRSA) + +Set up [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) on the member cluster, and grant the IAM role access to the hub EKS cluster (an EKS Access Entry or the `aws-auth` ConfigMap). The kubeconfig's exec stanza runs the small, purpose-built `aws-iam-authenticator` binary (lighter to bundle than the full `aws` CLI): + +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: aws-iam-authenticator + args: + - token + - -i + - + provideClusterInfo: false + interactiveMode: Never +``` + +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set serviceAccountAnnotations."eks\.amazonaws\.com/role-arn"= +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +No `podLabels` needed — IRSA only requires the ServiceAccount annotation. + +### GCP / GKE (Workload Identity) + +Set up [GKE Workload Identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) on the member cluster (binds the pod's ServiceAccount to a Google service account), and grant that service account the appropriate IAM role on the hub GKE cluster. The kubeconfig's exec stanza runs `gke-gcloud-auth-plugin`: + +```yaml +users: +- name: fed + user: + exec: + apiVersion: client.authentication.k8s.io/v1beta1 + command: gke-gcloud-auth-plugin + args: + - --use_application_default_credentials + provideClusterInfo: true +``` + +```console +kubectl create secret generic hub-kubeconfig --namespace fleet-system --from-file=kubeconfig=./kubeconfig.yaml + +helm install member-agent ./charts/member-agent/ \ + --namespace fleet-system \ + --create-namespace \ + --set config.memberClusterName= \ + --set useKubeconfig=true \ + --set config.hubKubeconfigSecretName=hub-kubeconfig \ + --set serviceAccountAnnotations."iam\.gke\.io/gcp-service-account"= +``` +Equivalent `member-agent` invocation: `KUBE_CONFIG_PATH=/etc/kubefleet/hub-kubeconfig/kubeconfig --use-kubeconfig=true` + +No `podLabels` needed here either — Workload Identity is resolved automatically for any pod running +under the annotated ServiceAccount. + ## Override Azure cloud config **If PropertyProvider feature is set to azure, then a cloud configuration is required.** diff --git a/charts/member-agent/templates/deployment.yaml b/charts/member-agent/templates/deployment.yaml index 9aa9ab267..c6d43051a 100644 --- a/charts/member-agent/templates/deployment.yaml +++ b/charts/member-agent/templates/deployment.yaml @@ -1,3 +1,12 @@ +{{- $useSidecar := and (not .Values.useCAAuth) (not .Values.useKubeconfig) }} +{{- $mountIdentitySecret := and .Values.useCAAuth .Values.config.identitySecretName }} +{{- $mountHubKubeconfig := and (not .Values.useCAAuth) .Values.useKubeconfig }} +{{- if and .Values.useCAAuth .Values.useKubeconfig }} +{{- fail "useCAAuth and useKubeconfig must not both be true: with both set, useCAAuth silently wins (only --use-ca-auth is passed to the member agent) and the requested kubeconfig mode is never applied. Set exactly one." }} +{{- end }} +{{- if and .Values.useKubeconfig (not .Values.config.hubKubeconfigSecretName) }} +{{- fail "config.hubKubeconfigSecretName must be set when useKubeconfig is true (it names the Secret, key \"kubeconfig\", to mount as the hub kubeconfig)" }} +{{- end }} apiVersion: apps/v1 kind: Deployment metadata: @@ -13,6 +22,9 @@ spec: metadata: labels: {{- include "member-agent.selectorLabels" . | nindent 8 }} + {{- with .Values.podLabels }} + {{- toYaml . | nindent 8 }} + {{- end }} spec: restartPolicy: Always serviceAccountName: {{ include "member-agent.fullname" . }}-sa @@ -23,6 +35,7 @@ spec: fsGroup: 65532 seccompProfile: type: RuntimeDefault + {{- if or .Values.crdInstaller.enabled .Values.extraInitContainers }} initContainers: {{- if .Values.crdInstaller.enabled }} - name: crd-installer @@ -31,7 +44,14 @@ spec: args: - --mode=member - --v={{ .Values.crdInstaller.logVerbosity }} + {{- if .Values.enablePlacementPolicyAPIs }} + - --enable-placement-policy-apis=true + {{- end }} {{- end }} + {{- with .Values.extraInitContainers }} + {{- toYaml . | nindent 8 }} + {{- end }} + {{- end }} containers: - name: {{ include "member-agent.fullname" . }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" @@ -49,6 +69,8 @@ spec: - --leader-elect=true {{- if .Values.useCAAuth }} - --use-ca-auth={{ .Values.useCAAuth }} + {{- else if $mountHubKubeconfig }} + - --use-kubeconfig=true {{- else }} - --tls-insecure={{ .Values.tlsClientInsecure }} {{- end }} @@ -98,20 +120,31 @@ spec: {{- if .Values.enableNamespaceCollectionInPropertyProvider }} - --enable-namespace-collection-in-property-provider={{ .Values.enableNamespaceCollectionInPropertyProvider }} {{- end }} + - --enable-placement-policy-apis={{ .Values.enablePlacementPolicyAPIs }} env: + {{- if not $mountHubKubeconfig }} - name: HUB_SERVER_URL value: "{{ .Values.config.hubURL }}" + {{- end }} + {{- if $mountHubKubeconfig }} + - name: KUBE_CONFIG_PATH + value: "/etc/kubefleet/hub-kubeconfig/kubeconfig" + {{- end }} + {{- if $useSidecar }} - name: CONFIG_PATH value: "/config/token" + {{- end }} - name: MEMBER_CLUSTER_NAME value: "{{ .Values.config.memberClusterName }}" + {{- if not $mountHubKubeconfig }} - name: HUB_CERTIFICATE_AUTHORITY value: "{{ .Values.config.hubCA }}" + {{- end }} {{- if .Values.useCAAuth }} - name: IDENTITY_KEY - value: "{{ .Values.config.identityKey }}" + value: "{{ if .Values.config.identitySecretName }}/etc/kubefleet/hub-identity/tls.key{{ else }}{{ .Values.config.identityKey }}{{ end }}" - name: IDENTITY_CERT - value: "{{ .Values.config.identityCert }}" + value: "{{ if .Values.config.identitySecretName }}/etc/kubefleet/hub-identity/tls.crt{{ else }}{{ .Values.config.identityCert }}{{ end }}" - name: CA_BUNDLE value: "{{ .Values.config.CABundle }}" {{- end }} @@ -146,10 +179,10 @@ spec: httpGet: path: /readyz port: hubhealthz - {{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }} + {{- if or ($useSidecar) (eq .Values.propertyProvider "azure") ($mountIdentitySecret) ($mountHubKubeconfig) (.Values.extraVolumeMounts) }} volumeMounts: - {{- if not .Values.useCAAuth }} - - name: provider-token + {{- if $useSidecar }} + - name: provider-token mountPath: /config {{- end }} {{- if eq .Values.propertyProvider "azure" }} @@ -157,8 +190,21 @@ spec: mountPath: /etc/kubernetes/provider readOnly: true {{- end }} + {{- if $mountIdentitySecret }} + - name: hub-identity + mountPath: /etc/kubefleet/hub-identity + readOnly: true + {{- end }} + {{- if $mountHubKubeconfig }} + - name: hub-kubeconfig + mountPath: /etc/kubefleet/hub-kubeconfig + readOnly: true + {{- end }} + {{- with .Values.extraVolumeMounts }} + {{- toYaml . | nindent 10 }} + {{- end }} {{- end }} - {{- if not .Values.useCAAuth }} + {{- if $useSidecar }} - name: refresh-token image: "{{ .Values.refreshtoken.repository }}:{{ .Values.refreshtoken.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.refreshtoken.pullPolicy }} @@ -184,9 +230,9 @@ spec: - name: provider-token mountPath: /config {{- end }} - {{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }} + {{- if or ($useSidecar) (eq .Values.propertyProvider "azure") ($mountIdentitySecret) ($mountHubKubeconfig) (.Values.extraVolumes) }} volumes: - {{- if not .Values.useCAAuth }} + {{- if $useSidecar }} - name: provider-token emptyDir: {} {{- end }} @@ -195,6 +241,19 @@ spec: secret: secretName: cloud-config {{- end }} + {{- if $mountIdentitySecret }} + - name: hub-identity + secret: + secretName: {{ .Values.config.identitySecretName }} + {{- end }} + {{- if $mountHubKubeconfig }} + - name: hub-kubeconfig + secret: + secretName: {{ .Values.config.hubKubeconfigSecretName }} + {{- end }} + {{- with .Values.extraVolumes }} + {{- toYaml . | nindent 6 }} + {{- end }} {{- end }} {{- with .Values.nodeSelector }} nodeSelector: diff --git a/charts/member-agent/templates/serviceaccount.yaml b/charts/member-agent/templates/serviceaccount.yaml index e312541d3..213f0cd00 100644 --- a/charts/member-agent/templates/serviceaccount.yaml +++ b/charts/member-agent/templates/serviceaccount.yaml @@ -5,3 +5,7 @@ metadata: namespace: {{ .Values.namespace }} labels: {{- include "member-agent.labels" . | nindent 4 }} + {{- with .Values.serviceAccountAnnotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} diff --git a/charts/member-agent/values.yaml b/charts/member-agent/values.yaml index 686f36d6b..4f91b94e4 100644 --- a/charts/member-agent/values.yaml +++ b/charts/member-agent/values.yaml @@ -44,6 +44,19 @@ config: identityKey: "identity-key-path" identityCert: "identity-cert-path" CABundle: "ca-bundle-path" + # identitySecretName, when set, mounts a kubernetes.io/tls Secret (keys tls.crt/tls.key) of + # this name and uses it for IDENTITY_KEY/IDENTITY_CERT instead of the identityKey/identityCert + # paths above. Only used when useCAAuth is true. Leave empty ("") to keep bringing your own + # cert/key files by another means (identityKey/identityCert must then point at that mount). + identitySecretName: "" + # hubKubeconfigSecretName names a Secret (key "kubeconfig") containing a full kubeconfig for + # the hub connection, mounted and passed to the process via the KUBE_CONFIG_PATH environment + # variable. Only used when useKubeconfig is true. This is the vendor-neutral way to + # authenticate via a federated identity (AWS IRSA, GCP Workload Identity, Azure AD workload + # identity, SPIFFE, etc.): the kubeconfig's exec stanza invokes whatever credential plugin + # your platform provides — this chart has no vendor-specific knowledge of any of them. See + # the README for worked examples. + hubKubeconfigSecretName: "" azureCloudConfig: cloud: "" tenantId: "" @@ -68,7 +81,29 @@ azure: tlsClientInsecure: false useCAAuth: false -enablePprof: true +# useKubeconfig authenticates to the hub cluster via a full kubeconfig instead of the +# refresh-token sidecar or useCAAuth — see config.hubKubeconfigSecretName above for the +# companion value naming the Secret that holds it. Mutually exclusive with useCAAuth. +useKubeconfig: false + +# podLabels and serviceAccountAnnotations are passed straight through to the pod template and the +# member agent's ServiceAccount, respectively. Use them to attach whatever platform-specific +# workload-identity metadata your hub-kubeconfig's exec plugin needs — e.g. for AKS workload +# identity: podLabels: {azure.workload.identity/use: "true"}, serviceAccountAnnotations: +# {azure.workload.identity/client-id: }; for AWS IRSA or GCP Workload Identity, +# only a serviceAccountAnnotations entry is needed. See the README for worked examples. +podLabels: {} +serviceAccountAnnotations: {} + +# extraVolumes, extraVolumeMounts (on the member-agent container), and extraInitContainers are +# escape hatches for getting a hub-kubeconfig exec plugin's binary (or, for SPIFFE, a workload-API +# socket) into the pod, without this chart needing any vendor-specific knowledge of how. The +# alternative is building a custom image layered on top of the official one. +extraVolumes: [] +extraVolumeMounts: [] +extraInitContainers: [] + +enablePprof: false pprofPort: 6065 hubPprofPort: 6066 @@ -78,3 +113,8 @@ priorityQueue: priorityLinearEquationCoeffB: 100 enableNamespaceCollectionInPropertyProvider: false + +# Apply the Works of the experimental FEP-0001 placement policy APIs (placement.kubefleet.dev/v1alpha1). +# The hub agent must run with enablePlacementPolicyAPIs too. If crdInstaller is enabled, this also has the +# CRD installer install the AppliedWork CRD of these APIs; otherwise, the CRD must be installed separately. +enablePlacementPolicyAPIs: false diff --git a/cmd/crdinstaller/main.go b/cmd/crdinstaller/main.go index 1f6639fb4..5b4600d69 100644 --- a/cmd/crdinstaller/main.go +++ b/cmd/crdinstaller/main.go @@ -24,6 +24,9 @@ import ( var ( mode = flag.String("mode", "", "Mode to run in: 'hub', 'member', 'arcMember', (required)") + // Note (chenyu1): adding a temporary flag for enabling placement policy APIs. + enablePlacementPolicyAPIs = flag.Bool("enable-placement-policy-apis", false, + "Install the CRDs of the placement policy APIs ([placement|rollout].kubefleet.dev/v1alpha1) or not. Hub and member agents must have the APIs enabled together.") ) func main() { @@ -68,7 +71,7 @@ func main() { // Install CRDs from the fixed location. const crdPath = "/workspace/config/crd/bases" - if err := installCRDs(ctx, client, crdPath, *mode); err != nil { + if err := installCRDs(ctx, client, crdPath, *mode, *enablePlacementPolicyAPIs); err != nil { klog.Fatalf("Failed to install CRDs: %v", err) } @@ -76,9 +79,9 @@ func main() { } // installCRDs installs the CRDs from the specified directory based on the mode. -func installCRDs(ctx context.Context, client client.Client, crdPath, mode string) error { +func installCRDs(ctx context.Context, client client.Client, crdPath, mode string, enablePlacementPolicyAPIs bool) error { // List of CRDs to install based on mode. - crdsToInstall, err := utils.CollectCRDs(crdPath, mode, client.Scheme()) + crdsToInstall, err := utils.CollectCRDs(crdPath, mode, enablePlacementPolicyAPIs, client.Scheme()) if err != nil { return err } diff --git a/cmd/crdinstaller/main_test.go b/cmd/crdinstaller/main_test.go new file mode 100644 index 000000000..dde568058 --- /dev/null +++ b/cmd/crdinstaller/main_test.go @@ -0,0 +1,140 @@ +/* +Copyright (c) Microsoft Corporation. +Licensed under the MIT license. +*/ + +package main + +import ( + "context" + "os" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + admv1 "k8s.io/api/admissionregistration/v1" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/runtime" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + "go.goms.io/fleet/cmd/crdinstaller/utils" +) + +const ( + actualCRDPath = "../../config/crd/bases" +) + +// TestInstallCRDs verifies that installCRDs installs exactly the CRDs collected for each mode, with the +// expected labels. The exact set of CRDs collected for each mode is verified in the utils package +// (TestCollectCRDFileNamesWithActualPath). +func TestInstallCRDs(t *testing.T) { + if _, err := os.Stat(actualCRDPath); os.IsNotExist(err) { + t.Skipf("Skipping test: %s does not exist", actualCRDPath) + } + + scheme := runtime.NewScheme() + if err := apiextensionsv1.AddToScheme(scheme); err != nil { + t.Fatalf("Failed to add apiextensions scheme: %v", err) + } + if err := admv1.AddToScheme(scheme); err != nil { + t.Fatalf("Failed to add admissionregistration scheme: %v", err) + } + + tests := []struct { + name string + crdPath string + mode string + enablePlacementPolicyAPIs bool + wantAzureManagedLabel string + wantErr bool + }{ + { + name: "hub mode", + crdPath: actualCRDPath, + mode: utils.ModeHub, + wantAzureManagedLabel: utils.FleetLabelValue, + }, + { + name: "hub mode with placement policy APIs enabled", + crdPath: actualCRDPath, + mode: utils.ModeHub, + enablePlacementPolicyAPIs: true, + wantAzureManagedLabel: utils.FleetLabelValue, + }, + { + name: "member mode", + crdPath: actualCRDPath, + mode: utils.ModeMember, + wantAzureManagedLabel: utils.FleetLabelValue, + }, + { + name: "member mode with placement policy APIs enabled", + crdPath: actualCRDPath, + mode: utils.ModeMember, + enablePlacementPolicyAPIs: true, + wantAzureManagedLabel: utils.FleetLabelValue, + }, + { + name: "arcMember mode", + crdPath: actualCRDPath, + mode: utils.ModeArcMember, + wantAzureManagedLabel: utils.ModeArcMember, + }, + { + name: "arcMember mode with placement policy APIs enabled", + crdPath: actualCRDPath, + mode: utils.ModeArcMember, + enablePlacementPolicyAPIs: true, + wantAzureManagedLabel: utils.ModeArcMember, + }, + { + name: "no CRDs found", + crdPath: t.TempDir(), + mode: utils.ModeHub, + enablePlacementPolicyAPIs: true, + wantErr: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + fakeClient := fake.NewClientBuilder().WithScheme(scheme).Build() + + collectedCRDs, err := utils.CollectCRDs(tt.crdPath, tt.mode, tt.enablePlacementPolicyAPIs, scheme) + if err != nil { + t.Fatalf("CollectCRDs() = %v, want no error", err) + } + wantCRDSpecs := make(map[string]apiextensionsv1.CustomResourceDefinitionSpec, len(collectedCRDs)) + for i := range collectedCRDs { + wantCRDSpecs[collectedCRDs[i].Name] = collectedCRDs[i].Spec + } + + err = installCRDs(ctx, fakeClient, tt.crdPath, tt.mode, tt.enablePlacementPolicyAPIs) + if (err != nil) != tt.wantErr { + t.Fatalf("installCRDs() = %v, wantErr %t", err, tt.wantErr) + } + + crdList := &apiextensionsv1.CustomResourceDefinitionList{} + if err := fakeClient.List(ctx, crdList); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + + gotCRDSpecs := make(map[string]apiextensionsv1.CustomResourceDefinitionSpec, len(crdList.Items)) + wantLabels := map[string]string{ + utils.CRDInstallerLabelKey: "true", + utils.AzureManagedLabelKey: tt.wantAzureManagedLabel, + } + for i := range crdList.Items { + crd := &crdList.Items[i] + gotCRDSpecs[crd.Name] = crd.Spec + if diff := cmp.Diff(wantLabels, crd.Labels); diff != "" { + t.Errorf("CRD %s labels mismatch (-want +got):\n%s", crd.Name, diff) + } + } + if diff := cmp.Diff(wantCRDSpecs, gotCRDSpecs, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("installed CRDs mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/cmd/crdinstaller/utils/util.go b/cmd/crdinstaller/utils/util.go index 65611a943..57b5aaa58 100644 --- a/cmd/crdinstaller/utils/util.go +++ b/cmd/crdinstaller/utils/util.go @@ -53,6 +53,9 @@ var ( memberCRD = map[string]bool{ "placement.kubernetes-fleet.io_appliedworks.yaml": true, } + placementPolicyAPIMemberCRD = map[string]bool{ + "placement.kubefleet.dev_appliedworks.yaml": true, + } ) // InstallCRD creates/updates a Custom Resource Definition (CRD) from the provided CRD object. @@ -106,8 +109,9 @@ func install(ctx context.Context, client client.Client, obj client.Object, mut f return nil } -// CollectCRDs collects CRDs from the specified path based on the mode either hub/member. -func CollectCRDs(crdDirectoryPath, mode string, scheme *runtime.Scheme) ([]apiextensionsv1.CustomResourceDefinition, error) { +// CollectCRDs collects CRDs from the specified path based on the mode either hub/member. The CRDs of the +// experimental placement policy APIs are collected only if enablePlacementPolicyAPIs is true. +func CollectCRDs(crdDirectoryPath, mode string, enablePlacementPolicyAPIs bool, scheme *runtime.Scheme) ([]apiextensionsv1.CustomResourceDefinition, error) { // Set of CRDs to install based on mode. crdsToInstall := []apiextensionsv1.CustomResourceDefinition{} @@ -129,11 +133,16 @@ func CollectCRDs(crdDirectoryPath, mode string, scheme *runtime.Scheme) ([]apiex var shouldInstall bool switch mode { case ModeMember, ModeArcMember: - shouldInstall = memberCRD[crdFileName] + shouldInstall = memberCRD[crdFileName] || + (enablePlacementPolicyAPIs && placementPolicyAPIMemberCRD[crdFileName]) case ModeHub: // Install multicluster CRD or CRDs with kubernetes-fleet.io in the filename (excluding member-only CRDs). + // If the placement policy APIs are enabled, also install CRDs with kubefleet.dev in the filename + // (placement.kubefleet.dev and rollout.kubefleet.dev), excluding member-only CRDs. // CRD filenames follow the pattern _.yaml, so we can check the filename. - shouldInstall = multiclusterCRD[crdFileName] || (strings.Contains(crdFileName, "kubernetes-fleet.io") && !memberCRD[crdFileName]) + shouldInstall = multiclusterCRD[crdFileName] || + (strings.Contains(crdFileName, "kubernetes-fleet.io") && !memberCRD[crdFileName]) || + (enablePlacementPolicyAPIs && strings.Contains(crdFileName, "kubefleet.dev") && !placementPolicyAPIMemberCRD[crdFileName]) } if shouldInstall { diff --git a/cmd/crdinstaller/utils/util_test.go b/cmd/crdinstaller/utils/util_test.go index ca4562a5f..55c62d24d 100644 --- a/cmd/crdinstaller/utils/util_test.go +++ b/cmd/crdinstaller/utils/util_test.go @@ -9,6 +9,7 @@ package utils import ( "context" "os" + "slices" "testing" "github.com/google/go-cmp/cmp" @@ -48,53 +49,103 @@ func TestCollectCRDFileNamesWithActualPath(t *testing.T) { // runTest contains the actual test logic, separated to allow running with different paths. func runTest(t *testing.T, crdPath string) { + wantHubCRDNames := []string{ + "memberclusters.cluster.kubernetes-fleet.io", + "internalmemberclusters.cluster.kubernetes-fleet.io", + "approvalrequests.placement.kubernetes-fleet.io", + "clusterapprovalrequests.placement.kubernetes-fleet.io", + "clusterresourcebindings.placement.kubernetes-fleet.io", + "clusterresourceenvelopes.placement.kubernetes-fleet.io", + "clusterresourceplacements.placement.kubernetes-fleet.io", + "clusterresourceplacementstatuses.placement.kubernetes-fleet.io", + "clusterresourceoverrides.placement.kubernetes-fleet.io", + "clusterresourceoverridesnapshots.placement.kubernetes-fleet.io", + "clusterresourceplacementdisruptionbudgets.placement.kubernetes-fleet.io", + "clusterresourceplacementevictions.placement.kubernetes-fleet.io", + "clusterresourcesnapshots.placement.kubernetes-fleet.io", + "clusterschedulingpolicysnapshots.placement.kubernetes-fleet.io", + "clusterstagedupdateruns.placement.kubernetes-fleet.io", + "clusterstagedupdatestrategies.placement.kubernetes-fleet.io", + "resourcebindings.placement.kubernetes-fleet.io", + "resourceenvelopes.placement.kubernetes-fleet.io", + "resourceoverrides.placement.kubernetes-fleet.io", + "resourceoverridesnapshots.placement.kubernetes-fleet.io", + "resourceplacements.placement.kubernetes-fleet.io", + "resourcesnapshots.placement.kubernetes-fleet.io", + "schedulingpolicysnapshots.placement.kubernetes-fleet.io", + "stagedupdateruns.placement.kubernetes-fleet.io", + "stagedupdatestrategies.placement.kubernetes-fleet.io", + "works.placement.kubernetes-fleet.io", + "clusterprofiles.multicluster.x-k8s.io", + } + wantHubPlacementPolicyAPICRDNames := []string{ + "clusterclaims.placement.kubefleet.dev", + "clusterplacementbindings.placement.kubefleet.dev", + "clusterplacementpolicies.placement.kubefleet.dev", + "clusterplacementresourcesnapshots.placement.kubefleet.dev", + "placementbindings.placement.kubefleet.dev", + "placementpolicies.placement.kubefleet.dev", + "placementresourcesnapshots.placement.kubefleet.dev", + "works.placement.kubefleet.dev", + "approvalrequests.rollout.kubefleet.dev", + "clusterapprovalrequests.rollout.kubefleet.dev", + "clusterstagedupdateruns.rollout.kubefleet.dev", + "clusterstagedupdatestrategies.rollout.kubefleet.dev", + "stagedupdateruns.rollout.kubefleet.dev", + "stagedupdatestrategies.rollout.kubefleet.dev", + } + wantMemberCRDNames := []string{ + "appliedworks.placement.kubernetes-fleet.io", + } + wantMemberPlacementPolicyAPICRDNames := []string{ + "appliedworks.placement.kubefleet.dev", + } + tests := []struct { - name string - mode string - wantedCRDNames []string - wantError bool + name string + mode string + enablePlacementPolicyAPIs bool + wantedCRDNames []string + wantError bool }{ { - name: "hub mode v1beta1 with actual directory", - mode: ModeHub, - wantedCRDNames: []string{ - "memberclusters.cluster.kubernetes-fleet.io", - "internalmemberclusters.cluster.kubernetes-fleet.io", - "approvalrequests.placement.kubernetes-fleet.io", - "clusterapprovalrequests.placement.kubernetes-fleet.io", - "clusterresourcebindings.placement.kubernetes-fleet.io", - "clusterresourceenvelopes.placement.kubernetes-fleet.io", - "clusterresourceplacements.placement.kubernetes-fleet.io", - "clusterresourceplacementstatuses.placement.kubernetes-fleet.io", - "clusterresourceoverrides.placement.kubernetes-fleet.io", - "clusterresourceoverridesnapshots.placement.kubernetes-fleet.io", - "clusterresourceplacementdisruptionbudgets.placement.kubernetes-fleet.io", - "clusterresourceplacementevictions.placement.kubernetes-fleet.io", - "clusterresourcesnapshots.placement.kubernetes-fleet.io", - "clusterschedulingpolicysnapshots.placement.kubernetes-fleet.io", - "clusterstagedupdateruns.placement.kubernetes-fleet.io", - "clusterstagedupdatestrategies.placement.kubernetes-fleet.io", - "resourcebindings.placement.kubernetes-fleet.io", - "resourceenvelopes.placement.kubernetes-fleet.io", - "resourceoverrides.placement.kubernetes-fleet.io", - "resourceoverridesnapshots.placement.kubernetes-fleet.io", - "resourceplacements.placement.kubernetes-fleet.io", - "resourcesnapshots.placement.kubernetes-fleet.io", - "schedulingpolicysnapshots.placement.kubernetes-fleet.io", - "stagedupdateruns.placement.kubernetes-fleet.io", - "stagedupdatestrategies.placement.kubernetes-fleet.io", - "works.placement.kubernetes-fleet.io", - "clusterprofiles.multicluster.x-k8s.io", - }, - wantError: false, + name: "hub mode v1beta1 with actual directory", + mode: ModeHub, + wantedCRDNames: wantHubCRDNames, + wantError: false, }, { - name: "member mode v1beta1 with actual directory", - mode: ModeMember, - wantedCRDNames: []string{ - "appliedworks.placement.kubernetes-fleet.io", - }, - wantError: false, + name: "hub mode with placement policy APIs enabled with actual directory", + mode: ModeHub, + enablePlacementPolicyAPIs: true, + wantedCRDNames: slices.Concat(wantHubCRDNames, wantHubPlacementPolicyAPICRDNames), + wantError: false, + }, + { + name: "member mode v1beta1 with actual directory", + mode: ModeMember, + wantedCRDNames: wantMemberCRDNames, + wantError: false, + }, + { + name: "member mode with placement policy APIs enabled with actual directory", + mode: ModeMember, + enablePlacementPolicyAPIs: true, + wantedCRDNames: slices.Concat(wantMemberCRDNames, wantMemberPlacementPolicyAPICRDNames), + wantError: false, + }, + { + name: "arcMember mode v1beta1 with actual directory", + mode: ModeArcMember, + wantedCRDNames: wantMemberCRDNames, + wantError: false, + }, + { + name: "arcMember mode with placement policy APIs enabled with actual directory", + mode: ModeArcMember, + enablePlacementPolicyAPIs: true, + wantedCRDNames: slices.Concat(wantMemberCRDNames, wantMemberPlacementPolicyAPICRDNames), + wantError: false, }, } @@ -105,7 +156,7 @@ func runTest(t *testing.T, crdPath string) { } t.Run(tt.name, func(t *testing.T) { // Call the function. - gotCRDs, err := CollectCRDs(crdPath, tt.mode, scheme) + gotCRDs, err := CollectCRDs(crdPath, tt.mode, tt.enablePlacementPolicyAPIs, scheme) if (err != nil) != tt.wantError { t.Errorf("collectCRDs() error = %v, wantError %v", err, tt.wantError) } diff --git a/cmd/hubagent/main.go b/cmd/hubagent/main.go index 97fc44dfb..2ada6701d 100644 --- a/cmd/hubagent/main.go +++ b/cmd/hubagent/main.go @@ -45,6 +45,8 @@ import ( "go.goms.io/fleet/pkg/webhook/managedresource" clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" placementv1alpha1 "go.goms.io/fleet/apis/placement/v1alpha1" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/cmd/hubagent/options" @@ -80,6 +82,8 @@ func init() { utilruntime.Must(apiextensionsv1.AddToScheme(scheme)) utilruntime.Must(fleetnetworkingv1alpha1.AddToScheme(scheme)) utilruntime.Must(placementv1alpha1.AddToScheme(scheme)) + utilruntime.Must(kfplacementv1alpha1.AddToScheme(scheme)) + utilruntime.Must(kfrolloutv1alpha1.AddToScheme(scheme)) utilruntime.Must(clusterinventory.AddToScheme(scheme)) // +kubebuilder:scaffold:scheme klog.InitFlags(nil) @@ -156,7 +160,7 @@ func main() { }), } if opts.CtrlMgrOpts.EnablePprof { - mgrOpts.PprofBindAddress = fmt.Sprintf(":%d", opts.CtrlMgrOpts.PprofPort) + mgrOpts.PprofBindAddress = fmt.Sprintf("127.0.0.1:%d", opts.CtrlMgrOpts.PprofPort) } mgr, err := ctrl.NewManager(defaultCfg, mgrOpts) if err != nil { @@ -168,10 +172,11 @@ func main() { if opts.FeatureFlags.EnableV1Beta1APIs { klog.Info("Setting up memberCluster v1beta1 controller") if err = (&mcv1beta1.Reconciler{ - Client: mgr.GetClient(), - NetworkingAgentsEnabled: opts.ClusterMgmtOpts.NetworkingAgentsEnabled, - MaxConcurrentReconciles: int(math.Ceil(float64(opts.PlacementMgmtOpts.MaxFleetSize) / 100)), //one member cluster reconciler routine per 100 member clusters - ForceDeleteWaitTime: opts.ClusterMgmtOpts.ForceDeleteWaitTime.Duration, + Client: mgr.GetClient(), + NetworkingAgentsEnabled: opts.ClusterMgmtOpts.NetworkingAgentsEnabled, + MaxConcurrentReconciles: int(math.Ceil(float64(opts.PlacementMgmtOpts.MaxFleetSize) / 100)), //one member cluster reconciler routine per 100 member clusters + ForceDeleteWaitTime: opts.ClusterMgmtOpts.ForceDeleteWaitTime.Duration, + PlacementPolicyAPIsEnabled: opts.FeatureFlags.EnablePlacementPolicyAPIs, }).SetupWithManager(mgr, "membercluster-controller"); err != nil { klog.ErrorS(err, "unable to create v1beta1 controller", "controller", "MemberCluster") exitWithErrorFunc() @@ -226,7 +231,8 @@ func main() { ctx := ctrl.SetupSignalHandler() if opts.WebhookAndAdmissionPolicyOpts.EnableAdmissionPolicyManager { - policyManagerCfgs := admissionpolicymanager.DefaultPolicyGeneratorConfigs + policyManagerCfgs := admissionpolicymanager.DefaultPolicyGeneratorConfigsForAzure( + opts.FeatureFlags.EnablePlacementPolicyAPIs && opts.FeatureFlags.EnableAnnotationBasedPlacement) if len(opts.WebhookAndAdmissionPolicyOpts.AdmissionPolicyManagerConfig) != 0 { // Read user-provided admission policy manager config from given path. cfgData, err := os.ReadFile(opts.WebhookAndAdmissionPolicyOpts.AdmissionPolicyManagerConfig) diff --git a/cmd/hubagent/options/featureflags.go b/cmd/hubagent/options/featureflags.go index 00b0d6e6b..2fc045a94 100644 --- a/cmd/hubagent/options/featureflags.go +++ b/cmd/hubagent/options/featureflags.go @@ -46,6 +46,23 @@ type FeatureFlags struct { // with minimal disruptions. EnableEvictionAPIs bool + // Enable the experimental FEP-0001 placement policy APIs (placement.kubefleet.dev/v1alpha1) in + // the KubeFleet hub agent or not. + EnablePlacementPolicyAPIs bool + + // Enable the experimental annotation-based placement experience in the KubeFleet hub agent or not. + // + // The experience places resources that carry the kubefleet.dev/cluster-selectors annotation through + // generated placement policies. It requires the placement policy APIs to be enabled. + // + // Important: only enable this experience if KubeFleet runs in a fully trusted environment, or if the hub + // cluster has the workload controllers of kube-controller-manager (e.g., the Deployment, StatefulSet, + // DaemonSet, and CronJob controllers) disabled and the protection validating admission policy + // (DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess) installed; otherwise, any user who can write a + // resource in the hub cluster can have it placed across the fleet. Note that setting the EnableWorkload + // option to false does not disable these controllers. + EnableAnnotationBasedPlacement bool + // Enable the ResourcePlacement API support in the KubeFleet hub agent or not. // // ResourcePlacement APIs are a set of KubeFleet APIs for processing namespace scoped resource placements. @@ -68,6 +85,22 @@ func (o *FeatureFlags) AddFlags(flags *flag.FlagSet) { "Enable the ClusterInventory API support in the KubeFleet hub agent or not.", ) + // Note (chenyu1): backport only: placement policy APIs are disabled by default in the backport for now. + flags.BoolVar( + &o.EnablePlacementPolicyAPIs, + "enable-placement-policy-apis", + false, + "Enable the experimental FEP-0001 placement policy APIs (placement.kubefleet.dev/v1alpha1) in the KubeFleet hub agent or not.", + ) + + // Note (chenyu1): backport only: placement policy APIs are disabled by default in the backport for now. + flags.BoolVar( + &o.EnableAnnotationBasedPlacement, + "enable-annotation-based-placement", + false, + "Enable the experimental annotation-based placement experience in the KubeFleet hub agent or not. The experience places resources that carry the kubefleet.dev/cluster-selectors annotation through generated placement policies, and requires the placement policy APIs to be enabled. Only enable it if KubeFleet runs in a fully trusted environment, or if the hub cluster has the workload controllers of kube-controller-manager (e.g., the Deployment, StatefulSet, DaemonSet, and CronJob controllers) disabled and the protection validating admission policy (DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess) installed; otherwise, any user who can write a resource in the hub cluster can have it placed across the fleet. Note that setting --enable-workload to false does not disable these controllers.", + ) + flags.BoolVar( &o.EnableStagedUpdateRunAPIs, "enable-staged-update-run-apis", diff --git a/cmd/hubagent/options/options_test.go b/cmd/hubagent/options/options_test.go index 73d1339f7..b1793d239 100644 --- a/cmd/hubagent/options/options_test.go +++ b/cmd/hubagent/options/options_test.go @@ -346,13 +346,15 @@ func TestFeatureFlags(t *testing.T) { "--enable-staged-update-run-apis=false", "--enable-eviction-apis=false", "--enable-resource-placement=false", + "--enable-annotation-based-placement=true", }, wantFeatureFlags: FeatureFlags{ - EnableV1Beta1APIs: true, - EnableClusterInventoryAPIs: false, - EnableStagedUpdateRunAPIs: false, - EnableEvictionAPIs: false, - EnableResourcePlacementAPIs: false, + EnableV1Beta1APIs: true, + EnableClusterInventoryAPIs: false, + EnableStagedUpdateRunAPIs: false, + EnableEvictionAPIs: false, + EnableResourcePlacementAPIs: false, + EnableAnnotationBasedPlacement: true, }, }, { @@ -585,6 +587,13 @@ func TestPlacementManagementOptions(t *testing.T) { wantErred: true, wantErrMsgSubStr: "invalid list of allowed for propagation APIs", }, + { + name: "allowed propagating APIs with no entries", + flagSetName: "allowedPropagatingAPIsEmpty", + args: []string{"--allowed-propagating-apis= "}, + wantErred: true, + wantErrMsgSubStr: "the list cannot be empty", + }, { name: "concurrent resource change syncs parse error", flagSetName: "concurrentResourceChangeSyncsParseError", diff --git a/cmd/hubagent/options/placementmgmt.go b/cmd/hubagent/options/placementmgmt.go index 7608485a5..4cfef4a39 100644 --- a/cmd/hubagent/options/placementmgmt.go +++ b/cmd/hubagent/options/placementmgmt.go @@ -25,7 +25,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/klog/v2" - "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/resourceeligibility" ) // PlacementManagementOptions is a set of options the KubeFleet hub agent exposes for @@ -36,22 +36,24 @@ type PlacementManagementOptions struct { // This option is no longer in use and is only kept for compatibility reasons. WorkPendingGracePeriod metav1.Duration - // A list of APIs that are block-listed for resource placement. Any resources under such APIs will be ignored - // by the KubeFleet hub agent and will not be selected for resource placement. + // A list of APIs that are deny-listed for resource placement. Any resources under such APIs will be blocked/denied + // for placement by the KubeFleet hub agent. // - // The list is a collection of GVKs separated by semicolons. A GVK can be of the format GROUP, - // GROUP/VERSION, or GROUP/VERSION/KINDS, where KINDS is a comma separated array of Kind values. If you would - // like to skip specific versions and/or kinds in the core API group, use the format VERSION, or - // VERSION/KINDS instead. Below are some examples: + // The list is a collection of GVKs separated by semicolons. A GVK can be of the format [API-GROUP], + // [API-GROUP]/[API-VERSION], or [API-GROUP]/[API-VERSION]/[KINDS], where [KINDS] is a comma separated array of + // Kind values. If you would like to skip specific versions and/or kinds in the core API group, use the format + // [API-VERSION], or [API-VERSION]/[KINDS] instead. Below are some examples: // // * networking.k8s.io: skip all resources in the networking.k8s.io API group for placement; // * networking.k8s.io/v1beta1: skip all resources in the networking.k8s.io/v1beta1 group version for placement; // * networking.k8s.io/v1beta1/Ingress,IngressClass: skip the Ingress and IngressClass resources // in the networking.k8s.io/v1beta1 group version for placement; - // * v1beta1: skip all resources of version v1beta1 in the core API group for placement; + // * v1: skip all resources of version v1 in the core API group for placement; // * v1/ConfigMap: skip ConfigMap resources of version v1 in the core API group for placement; - // * networking.k8s.io/v1beta1/Ingress; v1beta1: skip the Ingress resource in the networking.k8s.io/v1beta1 group version - // and all resources of version v1beta1 in the core API group for placement. + // * networking.k8s.io/v1beta1/Ingress;v1/ConfigMap: skip the Ingress resource in the networking.k8s.io/v1beta1 + // group version and the ConfigMap resource of version v1 in the core API group for placement. + // + // You can leave the [API-VERSION] and/or [KINDS] parts empty to deny all versions and/or kinds in an API group. // // This option is mutually exclusive with the AllowedPropagatingAPIs option. KubeFleet comes with a built-in // block list of APIs for resource placement that covers most KubeFleet APIs and a select few of critical Kubernetes @@ -60,19 +62,21 @@ type PlacementManagementOptions struct { // A list of APIs that are allow-listed for resource placement. If specified, only resources under such APIs // will be selected for resource placement by the KubeFleet hub agent. // - // The list is a collection of GVKs separated by semicolons. A GVK can be of the format GROUP, - // GROUP/VERSION, or GROUP/VERSION/KINDS, where KINDS is a comma separated array of Kind values. If you would - // like to skip specific versions and/or kinds in the core API group, use the format VERSION, or - // VERSION/KINDS instead. Below are some examples: + // The list is a collection of GVKs separated by semicolons. A GVK can be of the format [API-GROUP], + // [API-GROUP]/[API-VERSION], or [API-GROUP]/[API-VERSION]/[KINDS], where [KINDS] is a comma separated array of + // Kind values. If you would like to allow specific versions and/or kinds in the core API group, use the format + // [API-VERSION], or [API-VERSION]/[KINDS] instead. Below are some examples: // // * networking.k8s.io: allow all resources in the networking.k8s.io API group for placement only; // * networking.k8s.io/v1beta1: allow all resources in the networking.k8s.io/v1beta1 group version for placement only; // * networking.k8s.io/v1beta1/Ingress,IngressClass: allow the Ingress and IngressClass resources // in the networking.k8s.io/v1beta1 group version for placement only; - // * v1beta1: allow all resources of version v1beta1 in the core API group for placement only; + // * v1: allow all resources of version v1 in the core API group for placement only; // * v1/ConfigMap: allow ConfigMap resources of version v1 in the core API group for placement only; - // * networking.k8s.io/v1beta1/Ingress; v1beta1: only allow the Ingress resource in the networking.k8s.io/v1beta1 - // group version and all resources of version v1beta1 in the core API group for placement. + // * networking.k8s.io/v1beta1/Ingress;v1: only allow the Ingress resource in the networking.k8s.io/v1beta1 + // group version and all resources of version v1 in the core API group for placement. + // + // You can leave the [API-VERSION] and/or [KINDS] parts empty to allow all versions and/or kinds in an API group. // // This option is mutually exclusive with the SkippedPropagatingAPIs option. AllowedPropagatingAPIs string @@ -80,9 +84,9 @@ type PlacementManagementOptions struct { // A list of namespace names that are block-listed for resource placement. The KubeFleet hub agent // will ignore the namespaces and any resources within them when selecting resources for placement. // - // This list is a collection of names separated by commas, such as `internals,monitoring`. KubeFleet + // This list is a collection of names separated by semicolons, such as `internals;monitoring`. KubeFleet // also blocks a number of reserved namespace names for placement by default; such namespaces include - // those that are prefixed with `kube-`, and `fleet-system`. + // those that are prefixed with `kube-` or `fleet-`. SkippedPropagatingNamespaces string // The number of concurrent workers that help process resource changes for the placement APIs. @@ -141,7 +145,7 @@ func (o *PlacementManagementOptions) AddFlags(flags *flag.FlagSet) { &o.SkippedPropagatingNamespaces, "skipped-propagating-namespaces", "", - "A list of comma-separated namespace names that are block-listed for resource placement. The KubeFleet hub agent will ignore the namespaces and any resources within them when selecting resources for placement.", + "A list of semicolon-separated namespace names that are block-listed for resource placement, such as `internals;monitoring`. The KubeFleet hub agent will ignore the namespaces and any resources within them when selecting resources for placement. KubeFleet also blocks the `default` namespace and any namespace prefixed with `kube-` or `fleet-` by default.", ) flags.Var( @@ -203,8 +207,7 @@ func (v *SkippedPropagatingAPIsValueWithValidation) String() string { } func (v *SkippedPropagatingAPIsValueWithValidation) Set(s string) error { - rc := utils.NewResourceConfig(false) - if err := rc.Parse(s); err != nil { + if _, err := resourceeligibility.ParseGVKs(s); err != nil { return fmt.Errorf("invalid list of skipped for propagation APIs: %w", err) } *v = SkippedPropagatingAPIsValueWithValidation(s) @@ -223,10 +226,14 @@ func (v *AllowedPropagatingAPIsValueWithValidation) String() string { } func (v *AllowedPropagatingAPIsValueWithValidation) Set(s string) error { - rc := utils.NewResourceConfig(true) - if err := rc.Parse(s); err != nil { + gvks, err := resourceeligibility.ParseGVKs(s) + if err != nil { return fmt.Errorf("invalid list of allowed for propagation APIs: %w", err) } + if len(gvks) == 0 { + // An empty allow list would block every resource from placement. + return fmt.Errorf("invalid list of allowed for propagation APIs: the list cannot be empty") + } *v = AllowedPropagatingAPIsValueWithValidation(s) return nil } diff --git a/cmd/hubagent/options/validation.go b/cmd/hubagent/options/validation.go index 251df15c2..18149b4c4 100644 --- a/cmd/hubagent/options/validation.go +++ b/cmd/hubagent/options/validation.go @@ -71,6 +71,17 @@ func (o *Options) Validate() field.ErrorList { errs = append(errs, field.Invalid(newPath.Child("PlacementControllerWorkQueueRateLimiterOpts").Child("RateLimiterQPS"), o.PlacementMgmtOpts.PlacementControllerWorkQueueRateLimiterOpts.RateLimiterQPS, "the QPS for the placement controller set rate limiter must be less than its bucket size")) } + // The placement policy controllers are set up with the v1beta1 controllers; refuse a flag that would do nothing. + if o.FeatureFlags.EnablePlacementPolicyAPIs && !o.FeatureFlags.EnableV1Beta1APIs { + errs = append(errs, field.Invalid(newPath.Child("EnablePlacementPolicyAPIs"), true, "the placement policy APIs require the v1beta1 APIs to be enabled")) + } + + // The annotation-based placement controller is set up with the placement policy controllers; refuse a flag + // that would do nothing. + if o.FeatureFlags.EnableAnnotationBasedPlacement && !o.FeatureFlags.EnablePlacementPolicyAPIs { + errs = append(errs, field.Invalid(newPath.Child("EnableAnnotationBasedPlacement"), true, "annotation-based placement requires the placement policy APIs to be enabled")) + } + // Validate admission policy manager setup (if enabled). if err := o.validateAdmissionPolicyManagerConfig(newPath); err != nil { errs = append(errs, err) diff --git a/cmd/hubagent/options/validation_test.go b/cmd/hubagent/options/validation_test.go index edd964900..ee6e509d2 100644 --- a/cmd/hubagent/options/validation_test.go +++ b/cmd/hubagent/options/validation_test.go @@ -132,6 +132,28 @@ func TestValidation(t *testing.T) { }), want: field.ErrorList{field.Invalid(newPath.Child("PlacementControllerWorkQueueRateLimiterOpts").Child("RateLimiterBaseDelay"), 60*time.Second, "the base delay for the placement controller set rate limiter must be less than its max delay")}, }, + "placement policy APIs without the v1beta1 APIs": { + opt: newTestOptions(func(option *Options) { + option.FeatureFlags.EnablePlacementPolicyAPIs = true + option.FeatureFlags.EnableV1Beta1APIs = false + }), + want: field.ErrorList{field.Invalid(newPath.Child("EnablePlacementPolicyAPIs"), true, "the placement policy APIs require the v1beta1 APIs to be enabled")}, + }, + "annotation-based placement without the placement policy APIs": { + opt: newTestOptions(func(option *Options) { + option.FeatureFlags.EnableAnnotationBasedPlacement = true + option.FeatureFlags.EnablePlacementPolicyAPIs = false + }), + want: field.ErrorList{field.Invalid(newPath.Child("EnableAnnotationBasedPlacement"), true, "annotation-based placement requires the placement policy APIs to be enabled")}, + }, + "annotation-based placement with the placement policy APIs": { + opt: newTestOptions(func(option *Options) { + option.FeatureFlags.EnableAnnotationBasedPlacement = true + option.FeatureFlags.EnablePlacementPolicyAPIs = true + option.FeatureFlags.EnableV1Beta1APIs = true + }), + want: field.ErrorList{}, + }, "rate limiter qps must be less than bucket size": { opt: newTestOptions(func(option *Options) { option.PlacementMgmtOpts.PlacementControllerWorkQueueRateLimiterOpts.RateLimiterQPS = 100 diff --git a/cmd/hubagent/options/webhooks.go b/cmd/hubagent/options/webhooks.go index 510e0ae7a..e8c6cc0ad 100644 --- a/cmd/hubagent/options/webhooks.go +++ b/cmd/hubagent/options/webhooks.go @@ -82,7 +82,8 @@ type WebhookAndAdmissionPolicyOptions struct { // is a YAML file that specifies configuration for each policy generator available // in the admission policy manager. See the KubeFleet source code for more information. // If not specified, the default configuration will be used, which enables all available - // policy generators. This option only applies if the admission policy manager is enabled. + // policy generators but the placement annotation one (DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess). + // This option only applies if the admission policy manager is enabled. AdmissionPolicyManagerConfig string } @@ -164,7 +165,7 @@ func (o *WebhookAndAdmissionPolicyOptions) AddFlags(flags *flag.FlagSet) { &o.AdmissionPolicyManagerConfig, "admission-policy-manager-config", "", - "A file path to the configuration file for the KubeFleet admission policy manager. The file is a JSON or YAML file that specifies configuration for each policy generator available in the admission policy manager. See the KubeFleet source code for more information. If not specified, the default configuration will be used, which enables all available policy generators. This option only applies if the admission policy manager is enabled.", + "A file path to the configuration file for the KubeFleet admission policy manager. The file is a JSON or YAML file that specifies configuration for each policy generator available in the admission policy manager. See the KubeFleet source code for more information. If not specified, the default configuration will be used, which enables all available policy generators but the placement annotation one (DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess). This option only applies if the admission policy manager is enabled.", ) } diff --git a/cmd/hubagent/workload/setup.go b/cmd/hubagent/workload/setup.go index 2f4fcfbad..dc81d21f0 100644 --- a/cmd/hubagent/workload/setup.go +++ b/cmd/hubagent/workload/setup.go @@ -21,6 +21,7 @@ import ( "math" "strings" "sync" + "time" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/client-go/discovery" @@ -31,6 +32,8 @@ import ( ctrl "sigs.k8s.io/controller-runtime" clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/cmd/hubagent/options" "go.goms.io/fleet/pkg/clients/azure/compute" @@ -62,7 +65,16 @@ import ( "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/controller" "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/parallelizer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" "go.goms.io/fleet/pkg/utils/validator" + "go.goms.io/fleet/pkg/v1/controllers/annotationplacement" + "go.goms.io/fleet/pkg/v1/controllers/autorollout" + "go.goms.io/fleet/pkg/v1/controllers/placementpolicy" + "go.goms.io/fleet/pkg/v1/controllers/stagedupdaterun" + v1workgenerator "go.goms.io/fleet/pkg/v1/controllers/workgenerator" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" ) const ( @@ -76,6 +88,18 @@ const ( schedulerQueueName = "scheduler-queue" ) +// placementResourceSnapshotSlots is the snapshot manager's lock striping; its minimum. +const placementResourceSnapshotSlots int32 = 256 + +// Fixed concurrency for the experimental placement policy controllers. These are deliberately not derived from +// the placement management flags (--max-fleet-size, --max-concurrent-cluster-placement), which tune the +// v1beta1 controllers only. +const ( + placementPolicyWorkGeneratorConcurrentReconciles = 10 + placementPolicyStagedUpdateRunConcurrentReconciles = 5 + placementPolicyAutoRolloutConcurrentReconciles = 5 +) + var ( v1Beta1RequiredGVKs = []schema.GroupVersionKind{ clusterv1beta1.GroupVersion.WithKind(clusterv1beta1.MemberClusterKind), @@ -100,6 +124,25 @@ var ( placementv1beta1.GroupVersion.WithKind(placementv1beta1.SchedulingPolicySnapshotKind), } + placementPolicyGVKs = []schema.GroupVersionKind{ + kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.PlacementPolicyKind), + kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.ClusterPlacementPolicyKind), + kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.ClusterClaimKind), + kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.PlacementBindingKind), + kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.ClusterPlacementBindingKind), + kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.PlacementResourceSnapshotKind), + kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.ClusterPlacementResourceSnapshotKind), + // The hub field indexes and the work generator cover Works too, which starts an informer for them. + kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.WorkKind), + // The staged update run controller. + kfrolloutv1alpha1.GroupVersion.WithKind(kfrolloutv1alpha1.StagedUpdateRunKind), + kfrolloutv1alpha1.GroupVersion.WithKind(kfrolloutv1alpha1.ClusterStagedUpdateRunKind), + kfrolloutv1alpha1.GroupVersion.WithKind(kfrolloutv1alpha1.StagedUpdateStrategyKind), + kfrolloutv1alpha1.GroupVersion.WithKind(kfrolloutv1alpha1.ClusterStagedUpdateStrategyKind), + kfrolloutv1alpha1.GroupVersion.WithKind(kfrolloutv1alpha1.ApprovalRequestKind), + kfrolloutv1alpha1.GroupVersion.WithKind(kfrolloutv1alpha1.ClusterApprovalRequestKind), + } + clusterStagedUpdateRunGVKs = []schema.GroupVersionKind{ placementv1beta1.GroupVersion.WithKind(placementv1beta1.ClusterStagedUpdateRunKind), placementv1beta1.GroupVersion.WithKind(placementv1beta1.ClusterStagedUpdateStrategyKind), @@ -132,29 +175,56 @@ func SetupControllers(ctx context.Context, wg *sync.WaitGroup, mgr ctrl.Manager, } discoverClient := discovery.NewDiscoveryClientForConfigOrDie(config) - // AllowedPropagatingAPIs and SkippedPropagatingAPIs are mutually exclusive. - // If none of them are set, the resourceConfig by default stores a list of skipped propagation APIs. - resourceConfig := utils.NewResourceConfig(opts.PlacementMgmtOpts.AllowedPropagatingAPIs != "") - if err = resourceConfig.Parse(opts.PlacementMgmtOpts.AllowedPropagatingAPIs); err != nil { - // The program will never go here because the parameters have been checked. - return err - } - if err = resourceConfig.Parse(opts.PlacementMgmtOpts.SkippedPropagatingAPIs); err != nil { - // The program will never go here because the parameters have been checked - return err + + var gvkEligibilityMode resourceeligibility.GVKEligibilityMode + var parsedGVKs []schema.GroupVersionKind + if opts.PlacementMgmtOpts.AllowedPropagatingAPIs == "" { + // Prepare a deny list for placement eligible GVKs. + gvkEligibilityMode = resourceeligibility.GVKEligibilityModeDenyList + parsedGVKs, err = resourceeligibility.ParseGVKs(opts.PlacementMgmtOpts.SkippedPropagatingAPIs) + if err != nil { + // Normally this branch would never run as the input has been validated beforehand. + return err + } + } else { + // Prepare an allow list for placement eligible GVKs. + gvkEligibilityMode = resourceeligibility.GVKEligibilityModeAllowList + parsedGVKs, err = resourceeligibility.ParseGVKs(opts.PlacementMgmtOpts.AllowedPropagatingAPIs) + if err != nil { + // Normally this branch would never run as the input has been validated beforehand. + return err + } } - // setup namespaces we skip propagation - skippedNamespaces := make(map[string]bool) - skippedNamespaces["default"] = true - optionalSkipNS := strings.Split(opts.PlacementMgmtOpts.SkippedPropagatingNamespaces, ";") - for _, ns := range optionalSkipNS { - if len(ns) > 0 { - klog.InfoS("user specified a namespace to skip", "namespace", ns) - skippedNamespaces[ns] = true + // Prepare a namespace deny list for namespaces. + var userDefinedSkippedNSNames []string + for _, ns := range strings.Split(opts.PlacementMgmtOpts.SkippedPropagatingNamespaces, ";") { + if ns != "" { + userDefinedSkippedNSNames = append(userDefinedSkippedNSNames, ns) } } + // Set up the resource eligibility checker (V0 APIs only). + customResourceEligibilityChecker := resourceeligibility.New(resourceeligibility.CheckerOptions{ + RestMapper: mgr.GetRESTMapper(), + SkipNSCheck: len(userDefinedSkippedNSNames) == 0, + SkipGVKRCheck: len(parsedGVKs) == 0, + SkipObjectCheck: true, + }) + customResourceEligibilityChecker.SetNamespaceCheckList(resourceeligibility.NamespaceEligibilityModeDenyList, userDefinedSkippedNSNames, nil) + if err := customResourceEligibilityChecker.SetGVKCheckList(gvkEligibilityMode, parsedGVKs); err != nil { + klog.ErrorS(err, "Failed to set GVK check list for the custom resource eligibility checker") + return err + } + + // Build the composite resource eligibility checker for v0 APIs. + v0ResourceEligibilityChecker := customResourceEligibilityChecker.DeepCopy() + v0ResourceEligibilityChecker.Wraps(resourceeligibility.DefaultForV0APIs(mgr.GetRESTMapper())) + + // The same user-supplied filters for the v1 APIs (placement policies); the v1 built-in ones deny a few more kinds than v0. + v1ResourceEligibilityChecker := customResourceEligibilityChecker.DeepCopy() + v1ResourceEligibilityChecker.Wraps(resourceeligibility.DefaultForV1APIs(mgr.GetRESTMapper())) + // the manager for all the dynamically created informers dynamicInformerManager := informer.NewInformerManager(dynamicClient, opts.CtrlMgrOpts.ResyncPeriod.Duration, ctx.Done()) validator.ResourceInformer = dynamicInformerManager // webhook needs this to check resource scope @@ -162,11 +232,10 @@ func SetupControllers(ctx context.Context, wg *sync.WaitGroup, mgr ctrl.Manager, // Set up a custom controller to reconcile placement objects resourceSelectorResolver := controller.ResourceSelectorResolver{ - RestMapper: mgr.GetRESTMapper(), - InformerManager: dynamicInformerManager, - ResourceConfig: resourceConfig, - SkippedNamespaces: skippedNamespaces, - EnableWorkload: opts.WebhookAndAdmissionPolicyOpts.EnableWorkload, + RestMapper: mgr.GetRESTMapper(), + InformerManager: dynamicInformerManager, + ResourceEligibilityChecker: v0ResourceEligibilityChecker, + EnableWorkload: opts.WebhookAndAdmissionPolicyOpts.EnableWorkload, } resourceSnapshotResolver := controller.NewResourceSnapshotResolver(mgr.GetClient(), mgr.GetScheme()) resourceSnapshotResolver.Config = controller.NewResourceSnapshotConfig(opts.PlacementMgmtOpts.ResourceSnapshotCreationMinimumInterval, opts.PlacementMgmtOpts.ResourceChangesCollectionDuration) @@ -182,6 +251,8 @@ func SetupControllers(ctx context.Context, wg *sync.WaitGroup, mgr ctrl.Manager, rateLimiter := options.DefaultControllerRateLimiter(opts.PlacementMgmtOpts.PlacementControllerWorkQueueRateLimiterOpts) var clusterResourcePlacementControllerV1Beta1 controller.Controller var resourcePlacementController controller.Controller + var placementPolicies *placementpolicy.Reconciler + var annotatedSources *annotationplacement.SourceQueue if opts.FeatureFlags.EnableV1Beta1APIs { for _, gvk := range v1Beta1RequiredGVKs { @@ -287,6 +358,90 @@ func SetupControllers(ctx context.Context, wg *sync.WaitGroup, mgr ctrl.Manager, } } + if opts.FeatureFlags.EnablePlacementPolicyAPIs { + for _, gvk := range placementPolicyGVKs { + if err = utils.CheckCRDInstalled(discoverClient, gvk); err != nil { + klog.ErrorS(err, "Unable to find the required CRD", "GVK", gvk) + return err + } + } + klog.Info("Setting up the placement policy controllers") + if err := fieldindexers.SetupWithHubAgentControllerManager(ctx, mgr); err != nil { + klog.ErrorS(err, "Unable to set up the field indexes for the placement policy controllers") + return err + } + snapshots, snapshotErr := placementresourcesnapshot.New(mgr, + dynamicClient, dynamicInformerManager, + v1ResourceEligibilityChecker, mgr.GetRESTMapper(), + placementResourceSnapshotSlots, + placementresourcesnapshot.DefaultMaxPerSnapshotResourceDataSizeBytes, + placementresourcesnapshot.DefaultMaxPerSnapshotResourceCnt) + if snapshotErr != nil { + klog.ErrorS(snapshotErr, "Unable to set up the placement resource snapshot manager") + return snapshotErr + } + // Added to the manager so its snapshot GC runs (leader only). + if err := mgr.Add(snapshots); err != nil { + klog.ErrorS(err, "Unable to add the placement resource snapshot manager") + return err + } + placementPolicies = placementpolicy.NewReconciler(mgr.GetClient(), mgr.GetAPIReader(), snapshots) + if err := placementPolicies.SetupWithManagerForPlacementPolicy(mgr); err != nil { + klog.ErrorS(err, "Unable to set up the placement policy controller") + return err + } + if err := placementPolicies.SetupWithManagerForClusterPlacementPolicy(mgr); err != nil { + klog.ErrorS(err, "Unable to set up the cluster placement policy controller") + return err + } + klog.Info("Setting up the placement policy work generator") + if err := v1workgenerator.New(mgr.GetClient(), parallelizer.DefaultNumOfWorkers, nil). + SetupWithManager(mgr, placementPolicyWorkGeneratorConcurrentReconciles); err != nil { + klog.ErrorS(err, "Unable to set up the placement policy work generator") + return err + } + klog.Info("Setting up the placement policy staged update run controller") + // Nil rate limiters pick the controller defaults; 30s is the shortest refresh period it accepts. + stagedUpdateRuns, surErr := stagedupdaterun.New(mgr.GetClient(), mgr.GetAPIReader(), snapshots, + parallelizer.NewParallelizer(parallelizer.DefaultNumOfWorkers), nil, nil, 30*time.Second) + if surErr != nil { + klog.ErrorS(surErr, "Unable to create the placement policy staged update run controller") + return surErr + } + if err := stagedUpdateRuns.SetupWithManager(mgr, placementPolicyStagedUpdateRunConcurrentReconciles); err != nil { + klog.ErrorS(err, "Unable to set up the placement policy staged update run controller") + return err + } + klog.Info("Setting up the placement policy auto-rollout controller") + // A nil parallelizer and rate limiter pick the controller defaults. + autoRollouts, arErr := autorollout.New(mgr.GetClient(), snapshots, nil, nil, mgr.GetEventRecorder("auto-rollout-controller")) + if arErr != nil { + klog.ErrorS(arErr, "Unable to create the placement policy auto-rollout controller") + return arErr + } + if err := autoRollouts.SetupWithManager(mgr, placementPolicyAutoRolloutConcurrentReconciles); err != nil { + klog.ErrorS(err, "Unable to set up the placement policy auto-rollout controller") + return err + } + // Annotation-based placement generates the policies above; the resource watcher feeds it. The source + // queue is created only along with the controller, as nothing would drain it otherwise; the resource + // watcher does not report annotated resources if the queue is nil. + if opts.FeatureFlags.EnableAnnotationBasedPlacement { + klog.Info("Setting up the annotation placement controller") + annotatedSources = &annotationplacement.SourceQueue{} + if err := (&annotationplacement.Reconciler{ + Client: mgr.GetClient(), + APIReader: mgr.GetAPIReader(), + RESTMapper: mgr.GetRESTMapper(), + Recorder: mgr.GetEventRecorder("annotation-placement-controller"), + ResourceEligibilityChecker: v1ResourceEligibilityChecker, + }).SetupWithManager(mgr, annotatedSources); err != nil { + klog.ErrorS(err, "Unable to set up the annotation placement controller") + return err + } + } + } + if opts.FeatureFlags.EnableEvictionAPIs { for _, gvk := range evictionGVKs { if err = utils.CheckCRDInstalled(discoverClient, gvk); err != nil { @@ -520,16 +675,24 @@ func SetupControllers(ctx context.Context, wg *sync.WaitGroup, mgr ctrl.Manager, PlacementControllerV1Beta1: clusterResourcePlacementControllerV1Beta1, ResourcePlacementController: resourcePlacementController, } + if placementPolicies != nil { + rcr.EnqueuePlacementPolicies = placementPolicies.EnqueuePoliciesSelecting + } resourceChangeController := controller.NewController(resourceChangeControllerName, controller.ClusterWideKeyFunc, rcr.Reconcile, rateLimiter) // Set up the InformerPopulator that runs on ALL pods (leader and followers) // This ensures all pods have synced informer caches for webhook validation klog.Info("Setting up informer populator") + checkerForV1APIsInInformerPopulator := v1ResourceEligibilityChecker + if !opts.FeatureFlags.EnablePlacementPolicyAPIs { + checkerForV1APIsInInformerPopulator = nil + } informerPopulator := &resourcewatcher.InformerPopulator{ - DiscoveryClient: discoverClient, - RESTMapper: mgr.GetRESTMapper(), - InformerManager: dynamicInformerManager, - ResourceConfig: resourceConfig, + DiscoveryClient: discoverClient, + RESTMapper: mgr.GetRESTMapper(), + InformerManager: dynamicInformerManager, + ResourceEligibilityCheckerForV0APIs: v0ResourceEligibilityChecker, + ResourceEligibilityCheckerForV1APIs: checkerForV1APIsInInformerPopulator, } if err := mgr.Add(informerPopulator); err != nil { @@ -539,15 +702,20 @@ func SetupControllers(ctx context.Context, wg *sync.WaitGroup, mgr ctrl.Manager, // Set up a runner that starts all the custom controllers we created above // This runs ONLY on the leader and adds event handlers to the informers created by InformerPopulator + checkerForV1APIsInChangeDetector := v1ResourceEligibilityChecker + if !opts.FeatureFlags.EnablePlacementPolicyAPIs { + checkerForV1APIsInChangeDetector = nil + } resourceChangeDetector := &resourcewatcher.ChangeDetector{ DiscoveryClient: discoverClient, RESTMapper: mgr.GetRESTMapper(), ClusterResourcePlacementControllerV1Beta1: clusterResourcePlacementControllerV1Beta1, ResourcePlacementController: resourcePlacementController, ResourceChangeController: resourceChangeController, + AnnotatedSources: annotatedSources, InformerManager: dynamicInformerManager, - ResourceConfig: resourceConfig, - SkippedNamespaces: skippedNamespaces, + ResourceEligibilityCheckerForV0APIs: v0ResourceEligibilityChecker, + ResourceEligibilityCheckerForV1APIs: checkerForV1APIsInChangeDetector, ConcurrentPlacementWorker: int(math.Ceil(float64(opts.PlacementMgmtOpts.MaxConcurrentClusterPlacement) / 10)), ConcurrentResourceChangeWorker: opts.PlacementMgmtOpts.ConcurrentResourceChangeSyncs, EnableWorkload: opts.WebhookAndAdmissionPolicyOpts.EnableWorkload, diff --git a/cmd/memberagent/main.go b/cmd/memberagent/main.go index 9581d7686..c0692f4b8 100644 --- a/cmd/memberagent/main.go +++ b/cmd/memberagent/main.go @@ -28,9 +28,11 @@ import ( "net/http" "net/textproto" "os" + "strconv" "strings" "time" + "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/runtime" utilrand "k8s.io/apimachinery/pkg/util/rand" utilruntime "k8s.io/apimachinery/pkg/util/runtime" @@ -38,6 +40,7 @@ import ( "k8s.io/client-go/dynamic" clientgoscheme "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/rest" + "k8s.io/client-go/tools/clientcmd" "k8s.io/client-go/util/retry" "k8s.io/klog/v2" ctrl "sigs.k8s.io/controller-runtime" @@ -49,6 +52,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/webhook" clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/cmd/memberagent/options" imcv1beta1 "go.goms.io/fleet/pkg/controllers/internalmembercluster/v1beta1" @@ -56,8 +60,11 @@ import ( "go.goms.io/fleet/pkg/propertyprovider" "go.goms.io/fleet/pkg/propertyprovider/azure" "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/controller" "go.goms.io/fleet/pkg/utils/httpclient" "go.goms.io/fleet/pkg/utils/parallelizer" + v1workapplier "go.goms.io/fleet/pkg/v1/controllers/workapplier" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" //+kubebuilder:scaffold:imports ) @@ -66,6 +73,19 @@ const ( azurePropertyProvider = "azure" ) +// Fixed settings for the experimental placement policy work applier. These are deliberately not taken from the +// applier flags (--deletion-wait-time, --work-applier-requeue-rate-limiter-*, and the like), which tune the v1beta1 +// work applier only; the placement policy work applier also uses its own built-in requeue rate limiters. +const ( + placementPolicyWorkApplierConcurrentReconciles = 5 + placementPolicyWorkApplierCleanupWaitTime = 5 * time.Minute +) + +// testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName names a test-only environment variable that caps, +// in seconds, every requeue delay of the placement policy work applier. The E2E test environment sets it so that +// changes made only on the member cluster side are picked up quickly; it is not a supported setting. +const testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName = "TEST_ONLY_PLACEMENT_POLICY_WORK_APPLIER_MAX_REQUEUE_DELAY_SECONDS" + var ( scheme = runtime.NewScheme() ) @@ -76,6 +96,7 @@ func init() { utilruntime.Must(clientgoscheme.AddToScheme(scheme)) utilruntime.Must(clusterv1beta1.AddToScheme(scheme)) utilruntime.Must(placementv1beta1.AddToScheme(scheme)) + utilruntime.Must(kfplacementv1alpha1.AddToScheme(scheme)) //+kubebuilder:scaffold:scheme } @@ -99,13 +120,15 @@ func main() { // Set up controller-runtime logger ctrl.SetLogger(zap.New(zap.UseDevMode(true))) - hubURL := os.Getenv("HUB_SERVER_URL") - - if hubURL == "" { - klog.ErrorS(errors.New("hub server api cannot be empty"), "Failed to read URL for the hub cluster") - klog.FlushAndExit(klog.ExitFlushTimeout, 1) + var hubURL string + if !opts.HubConnectivityOpts.UseKubeConfig { + hubURL = os.Getenv("HUB_SERVER_URL") + if hubURL == "" { + klog.ErrorS(errors.New("hub server api cannot be empty"), "Failed to read URL for the hub cluster") + klog.FlushAndExit(klog.ExitFlushTimeout, 1) + } } - hubConfig, err := buildHubConfig(hubURL, opts.HubConnectivityOpts.UseCertificateAuth, opts.HubConnectivityOpts.UseInsecureTLSClient) + hubConfig, err := buildHubConfig(hubURL, opts.HubConnectivityOpts) if err != nil { klog.ErrorS(err, "Failed to build Kubernetes client configuration for the hub cluster") klog.FlushAndExit(klog.ExitFlushTimeout, 1) @@ -172,8 +195,8 @@ func main() { } //+kubebuilder:scaffold:builder if opts.CtrlManagerOptions.EnablePprof { - memberOpts.PprofBindAddress = fmt.Sprintf(":%d", opts.CtrlManagerOptions.MemberManagerOpts.PprofPort) - hubOpts.PprofBindAddress = fmt.Sprintf(":%d", opts.CtrlManagerOptions.HubManagerOpts.PprofPort) + memberOpts.PprofBindAddress = fmt.Sprintf("127.0.0.1:%d", opts.CtrlManagerOptions.MemberManagerOpts.PprofPort) + hubOpts.PprofBindAddress = fmt.Sprintf("127.0.0.1:%d", opts.CtrlManagerOptions.HubManagerOpts.PprofPort) } if err := Start(ctrl.SetupSignalHandler(), hubConfig, memberConfig, hubOpts, memberOpts, *opts); err != nil { @@ -182,11 +205,32 @@ func main() { } } -func buildHubConfig(hubURL string, useCertificateAuth bool, tlsClientInsecure bool) (*rest.Config, error) { +func buildHubConfig(hubURL string, opts options.HubConnectivityOptions) (*rest.Config, error) { var hubConfig = &rest.Config{ Host: hubURL, } - if useCertificateAuth { + + if opts.UseKubeConfig { + // A full kubeconfig is authoritative for the hub connection: it already carries the + // server URL, TLS configuration, and authentication (a bearer token, a client + // certificate, or a standard Kubernetes exec credential plugin for a federated + // identity), so none of the env-var/flag-based configuration below applies. + kubeConfigPath := os.Getenv("KUBE_CONFIG_PATH") + if kubeConfigPath == "" { + err := errors.New("hub kubeconfig file path cannot be empty if use-kubeconfig is set") + klog.ErrorS(err, "Failed to retrieve hub kubeconfig path") + return nil, err + } + clientConfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig( + &clientcmd.ClientConfigLoadingRules{ExplicitPath: kubeConfigPath}, + &clientcmd.ConfigOverrides{}) + loadedConfig, err := clientConfig.ClientConfig() + if err != nil { + klog.ErrorS(err, "Failed to load hub kubeconfig", "path", kubeConfigPath) + return nil, err + } + return applyHubKubeHeader(loadedConfig) + } else if opts.UseCertificateAuth { keyFilePath := os.Getenv("IDENTITY_KEY") certFilePath := os.Getenv("IDENTITY_CERT") if keyFilePath == "" { @@ -224,11 +268,19 @@ func buildHubConfig(hubURL string, useCertificateAuth bool, tlsClientInsecure bo hubConfig.BearerTokenFile = tokenFilePath } - hubConfig.TLSClientConfig.Insecure = tlsClientInsecure - if tlsClientInsecure { + return applyHubTLSConfig(hubConfig, opts) +} + +// applyHubTLSConfig applies the TLS insecure/CA settings shared by the certificate-auth and +// token-auth hub connection modes, then wraps the config with the HUB_KUBE_HEADER transport if +// one is configured. The kubeconfig mode does not go through this path: a loaded kubeconfig is +// already authoritative for TLS and skips straight to the header wrapping. +func applyHubTLSConfig(hubConfig *rest.Config, opts options.HubConnectivityOptions) (*rest.Config, error) { + hubConfig.TLSClientConfig.Insecure = opts.UseInsecureTLSClient + if opts.UseInsecureTLSClient { klog.Warning("TLS verification is disabled for hub cluster connection. This is insecure and should not be used in production.") } - if !tlsClientInsecure { + if !opts.UseInsecureTLSClient { caBundle, ok := os.LookupEnv("CA_BUNDLE") if ok && caBundle == "" { err := errors.New("environment variable CA_BUNDLE should not be empty") @@ -259,8 +311,14 @@ func buildHubConfig(hubURL string, useCertificateAuth bool, tlsClientInsecure bo } } - // Sometime the hub cluster need additional http header for authentication or authorization. - // the "HUB_KUBE_HEADER" to allow sending custom header to hub's API Server for authentication and authorization. + return applyHubKubeHeader(hubConfig) +} + +// applyHubKubeHeader wires in the "HUB_KUBE_HEADER" custom header, if set, to the given hub +// rest.Config. Sometimes the hub cluster needs an additional HTTP header for authentication or +// authorization; this applies regardless of how the rest of hubConfig was built, since it is an +// independent, orthogonal transport-level concern (e.g. a front-door header requirement). +func applyHubKubeHeader(hubConfig *rest.Config) (*rest.Config, error) { if header, ok := os.LookupEnv("HUB_KUBE_HEADER"); ok { r := textproto.NewReader(bufio.NewReader(strings.NewReader(header))) h, err := r.ReadMIMEHeader() @@ -404,6 +462,16 @@ func Start(ctx context.Context, hubCfg, memberConfig *rest.Config, hubOpts, memb klog.ErrorS(err, "Failed to create v1beta1 controller", "controller", "work") return err } + workAppliers := memberControllers{workApplier} + + if globalOpts.EnablePlacementPolicyAPIs { + klog.Info("Setting up the placement policy work applier") + v1WorkApplier, v1Err := setUpPlacementPolicyWorkApplier(ctx, hubCfg, hubMgr, memberMgr, discoverClient, spokeDynamicClient, restMapper, targetNS) + if v1Err != nil { + return v1Err + } + workAppliers = append(workAppliers, v1WorkApplier) + } klog.Info("Setting up the internalMemberCluster v1beta1 controller") // Set up a provider provider (if applicable). @@ -432,7 +500,7 @@ func Start(ctx context.Context, hubCfg, memberConfig *rest.Config, hubOpts, memb ctx, hubMgr.GetClient(), memberMgr.GetConfig(), memberMgr.GetClient(), - workApplier, + workAppliers, pp) if err != nil { klog.ErrorS(err, "Failed to create InternalMemberCluster v1beta1 reconciler") @@ -461,3 +529,102 @@ func Start(ctx context.Context, hubCfg, memberConfig *rest.Config, hubOpts, memb return nil } + +// setUpPlacementPolicyWorkApplier sets up the work applier for the Works of the experimental placement policy APIs. +func setUpPlacementPolicyWorkApplier( + ctx context.Context, + hubCfg *rest.Config, + hubMgr, memberMgr ctrl.Manager, + memberDiscoveryClient discovery.DiscoveryInterface, + spokeDynamicClient dynamic.Interface, + restMapper meta.RESTMapper, + workNS string, +) (*v1workapplier.Reconciler, error) { + // Check both sides up front; a missing CRD would otherwise only show up as a cache sync timeout. + appliedWorkGVK := kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.AppliedWorkKind) + if err := utils.CheckCRDInstalled(memberDiscoveryClient, appliedWorkGVK); err != nil { + klog.ErrorS(err, "Unable to find the required CRD in the member cluster", "GVK", appliedWorkGVK) + return nil, err + } + hubDiscoveryClient, err := discovery.NewDiscoveryClientForConfig(hubCfg) + if err != nil { + klog.ErrorS(err, "Failed to create the hub discovery client") + return nil, err + } + workGVK := kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.WorkKind) + if err := utils.CheckCRDInstalled(hubDiscoveryClient, workGVK); err != nil { + klog.ErrorS(err, "Unable to find the required CRD in the hub cluster; the hub agent might predate the placement policy APIs", "GVK", workGVK) + return nil, err + } + + // The work applier finds the Works of a placement binding via these indexes. + if err := fieldindexers.SetupWithMemberAgentControllerManager(ctx, hubMgr); err != nil { + klog.ErrorS(err, "Failed to set up the field indexes for the placement policy work applier") + return nil, err + } + + periodicRequeueRateLimiter, err := placementPolicyWorkApplierRequeueRateLimiter() + if err != nil { + klog.ErrorS(err, "Failed to set up the requeue rate limiter for the placement policy work applier") + return nil, err + } + + workApplier := v1workapplier.New( + workNS, + hubMgr.GetClient(), hubMgr.GetAPIReader(), + memberMgr.GetClient(), spokeDynamicClient, + restMapper, + parallelizer.NewParallelizer(parallelizer.DefaultNumOfWorkers), + placementPolicyWorkApplierConcurrentReconciles, + // Nil picks the package's own default cleanup rate limiter; a nil periodic requeue rate limiter + // picks its default one too. + nil, periodicRequeueRateLimiter, + placementPolicyWorkApplierCleanupWaitTime, + ) + if err := workApplier.SetupWithManager(hubMgr); err != nil { + klog.ErrorS(err, "Failed to set up the placement policy work applier") + return nil, err + } + return workApplier, nil +} + +// placementPolicyWorkApplierRequeueRateLimiter returns the periodic requeue rate limiter for the placement policy +// work applier: nil (the package default) unless the test-only environment variable caps the requeue delays. +func placementPolicyWorkApplierRequeueRateLimiter() (*v1workapplier.MultiStageRequeueWithExponentialBackoffRateLimiter, error) { + val, found := os.LookupEnv(testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName) + if !found || val == "" { + return nil, nil + } + maxDelaySeconds, err := strconv.Atoi(val) + if err != nil { + return nil, fmt.Errorf("failed to parse %s=%q as an integer: %w", testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName, val, err) + } + rateLimiter, err := v1workapplier.NewDefaultRequeueRateLimiterWithMaxDelay(maxDelaySeconds) + if err != nil { + return nil, fmt.Errorf("invalid %s=%q: %w", testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName, val, err) + } + klog.InfoS("Capping the requeue delays of the placement policy work applier; this is a test-only setting", + "envVar", testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName, "maxDelaySeconds", maxDelaySeconds) + return rateLimiter, nil +} + +// memberControllers lets the InternalMemberCluster controller join and leave the fleet for +// several member controllers at once. Every controller gets the call even if an earlier one +// fails, so a single failure does not leave the others in the wrong state. +type memberControllers []controller.MemberController + +func (m memberControllers) Join(ctx context.Context) error { + errs := make([]error, 0, len(m)) + for _, c := range m { + errs = append(errs, c.Join(ctx)) + } + return errors.Join(errs...) +} + +func (m memberControllers) Leave(ctx context.Context) error { + errs := make([]error, 0, len(m)) + for _, c := range m { + errs = append(errs, c.Leave(ctx)) + } + return errors.Join(errs...) +} diff --git a/cmd/memberagent/main_test.go b/cmd/memberagent/main_test.go index d729a6fba..f92b8f0d1 100644 --- a/cmd/memberagent/main_test.go +++ b/cmd/memberagent/main_test.go @@ -1,31 +1,110 @@ package main import ( + "context" + "errors" + "os" "testing" + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" "github.com/stretchr/testify/assert" "k8s.io/client-go/rest" + "k8s.io/utils/ptr" + + "go.goms.io/fleet/cmd/memberagent/options" + v1workapplier "go.goms.io/fleet/pkg/v1/controllers/workapplier" ) +// TestPlacementPolicyWorkApplierRequeueRateLimiter tests that the placement policy work applier uses its default +// requeue rate limiter unless the test-only environment variable caps the requeue delays. +func TestPlacementPolicyWorkApplierRequeueRateLimiter(t *testing.T) { + wantCapped, err := v1workapplier.NewDefaultRequeueRateLimiterWithMaxDelay(5) + if err != nil { + t.Fatalf("NewDefaultRequeueRateLimiterWithMaxDelay(5) = %v, want no error", err) + } + + testCases := []struct { + name string + envVal *string + want *v1workapplier.MultiStageRequeueWithExponentialBackoffRateLimiter + wantErred bool + }{ + { + name: "not set", + }, + { + name: "set to empty", + envVal: ptr.To(""), + }, + { + name: "set to a valid cap", + envVal: ptr.To("5"), + want: wantCapped, + }, + { + name: "set to a non-integer", + envVal: ptr.To("5s"), + wantErred: true, + }, + { + name: "set below the minimum", + envVal: ptr.To("1"), + wantErred: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + // Setenv restores the original value after the test, even when the variable is then unset. + t.Setenv(testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName, "") + if tc.envVal == nil { + if err := os.Unsetenv(testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName); err != nil { + t.Fatalf("os.Unsetenv() = %v, want no error", err) + } + } else { + t.Setenv(testOnlyPlacementPolicyWorkApplierMaxRequeueDelayEnvVarName, *tc.envVal) + } + + got, err := placementPolicyWorkApplierRequeueRateLimiter() + if tc.wantErred { + if err == nil { + t.Fatalf("placementPolicyWorkApplierRequeueRateLimiter() = %v, want an error", got) + } + return + } + if err != nil { + t.Fatalf("placementPolicyWorkApplierRequeueRateLimiter() = %v, want no error", err) + } + if diff := cmp.Diff(got, tc.want, + cmp.AllowUnexported(v1workapplier.MultiStageRequeueWithExponentialBackoffRateLimiter{}), + cmpopts.IgnoreFields(v1workapplier.MultiStageRequeueWithExponentialBackoffRateLimiter{}, "mu"), + ); diff != "" { + t.Errorf("placementPolicyWorkApplierRequeueRateLimiter() mismatches (-got, +want):\n%s", diff) + } + }) + } +} + func Test_buildHubConfig(t *testing.T) { t.Run("use CA auth, no key file - error", func(t *testing.T) { t.Setenv("IDENTITY_KEY", "") t.Setenv("IDENTITY_CERT", "/path/to/cert") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use CA auth, no cert file - error", func(t *testing.T) { t.Setenv("IDENTITY_KEY", "/path/to/key") t.Setenv("IDENTITY_CERT", "") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use CA auth - success", func(t *testing.T) { t.Setenv("IDENTITY_KEY", "/path/to/key") t.Setenv("IDENTITY_CERT", "/path/to/cert") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -40,7 +119,7 @@ func Test_buildHubConfig(t *testing.T) { t.Setenv("IDENTITY_KEY", "/path/to/key") t.Setenv("IDENTITY_CERT", "/path/to/cert") t.Setenv("CA_BUNDLE", "") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) @@ -48,7 +127,7 @@ func Test_buildHubConfig(t *testing.T) { t.Setenv("IDENTITY_KEY", "/path/to/key") t.Setenv("IDENTITY_CERT", "/path/to/cert") t.Setenv("CA_BUNDLE", "/path/to/ca/bundle") - config, err := buildHubConfig("https://hub.domain.com", true, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -63,7 +142,7 @@ func Test_buildHubConfig(t *testing.T) { t.Run("use CA data - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") t.Setenv("HUB_CERTIFICATE_AUTHORITY", "dGhpcyBpcyBhIGZha2UgY2E=") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -77,7 +156,7 @@ func Test_buildHubConfig(t *testing.T) { t.Run("empty CA data - error", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") t.Setenv("HUB_CERTIFICATE_AUTHORITY", "") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) @@ -85,25 +164,25 @@ func Test_buildHubConfig(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") t.Setenv("HUB_CERTIFICATE_AUTHORITY", "dGhpcyBpcyBhIGZha2UgY2E=") t.Setenv("CA_BUNDLE", "/path/to/ca/bundle") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use token auth, no token path - error", func(t *testing.T) { t.Setenv("CONFIG_PATH", "") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use token auth, not exists token path - error", func(t *testing.T) { t.Setenv("CONFIG_PATH", "/hot/exists/token/path") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.Nil(t, config) assert.NotNil(t, err) }) t.Run("use token auth - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -113,7 +192,7 @@ func Test_buildHubConfig(t *testing.T) { }) t.Run("No CA bundle, no Hub CA, not insecure - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") - config, err := buildHubConfig("https://hub.domain.com", false, false) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: false}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -123,7 +202,7 @@ func Test_buildHubConfig(t *testing.T) { }) t.Run("use insecure client - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") - config, err := buildHubConfig("https://hub.domain.com", false, true) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: true}) assert.NotNil(t, config) assert.Nil(t, err) assert.Equal(t, rest.Config{ @@ -137,9 +216,212 @@ func Test_buildHubConfig(t *testing.T) { t.Run("use insecure client and custom header - success", func(t *testing.T) { t.Setenv("CONFIG_PATH", "./testdata/token") t.Setenv("HUB_KUBE_HEADER", "Member-Resource-ID: some-id") - config, err := buildHubConfig("https://hub.domain.com", false, true) + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: false, UseInsecureTLSClient: true}) assert.NotNil(t, config) assert.Nil(t, err) assert.NotNil(t, config.WrapTransport) }) + t.Run("use CA auth and insecure client - success", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: true}) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, rest.Config{ + Host: "https://hub.domain.com", + TLSClientConfig: rest.TLSClientConfig{ + KeyFile: "/path/to/key", + CertFile: "/path/to/cert", + Insecure: true, + }, + }, *config) + }) + t.Run("use CA auth and CA data - success", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + t.Setenv("HUB_CERTIFICATE_AUTHORITY", "dGhpcyBpcyBhIGZha2UgY2E=") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, rest.Config{ + Host: "https://hub.domain.com", + TLSClientConfig: rest.TLSClientConfig{ + KeyFile: "/path/to/key", + CertFile: "/path/to/cert", + CAData: []byte("this is a fake ca"), + }, + }, *config) + }) + t.Run("use CA auth, empty CA data - error", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + t.Setenv("HUB_CERTIFICATE_AUTHORITY", "") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use CA auth, both of CA bundle and CA data present - error", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + t.Setenv("HUB_CERTIFICATE_AUTHORITY", "dGhpcyBpcyBhIGZha2UgY2E=") + t.Setenv("CA_BUNDLE", "/path/to/ca/bundle") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use CA auth with custom header - success", func(t *testing.T) { + t.Setenv("IDENTITY_KEY", "/path/to/key") + t.Setenv("IDENTITY_CERT", "/path/to/cert") + t.Setenv("HUB_KUBE_HEADER", "Member-Resource-ID: some-id") + config, err := buildHubConfig("https://hub.domain.com", options.HubConnectivityOptions{UseCertificateAuth: true, UseInsecureTLSClient: false}) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.NotNil(t, config.WrapTransport) + }) + t.Run("use hub kubeconfig - success", func(t *testing.T) { + // hubURL is deliberately left empty here, mirroring main()'s behavior of skipping the + // HUB_SERVER_URL read entirely when UseKubeConfig is set: the kubeconfig's own + // cluster server URL is authoritative. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, "https://hub.fixture.example.com", config.Host) + assert.Equal(t, "fixture-bearer-token", config.BearerToken) + }) + t.Run("use hub kubeconfig with an exec credential plugin - success", func(t *testing.T) { + // Exercises the users[].user.exec flow this feature exists for (a federated-identity + // exec plugin, e.g. kubelogin/aws-iam-authenticator/gke-gcloud-auth-plugin), as opposed + // to the plain-bearer-token fixture above. clientcmd populates rest.Config.ExecProvider + // from the exec stanza but does not invoke the plugin binary at ClientConfig() time - + // that only happens lazily on the first real HTTP request - so this can assert on the + // parsed exec configuration without needing "kubelogin" to actually be on PATH. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig-exec") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, "https://hub.fixture.example.com", config.Host) + if assert.NotNil(t, config.ExecProvider) { + assert.Equal(t, "kubelogin", config.ExecProvider.Command) + assert.Equal(t, []string{"get-token", "--server-id", "fixture-server-id", "--login", "workloadidentity"}, config.ExecProvider.Args) + assert.Equal(t, "client.authentication.k8s.io/v1", config.ExecProvider.APIVersion) + } + }) + t.Run("use hub kubeconfig, no path - error", func(t *testing.T) { + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use hub kubeconfig, not exists - error", func(t *testing.T) { + t.Setenv("KUBE_CONFIG_PATH", "./testdata/does-not-exist") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use hub kubeconfig with custom header - success", func(t *testing.T) { + t.Setenv("HUB_KUBE_HEADER", "Member-Resource-ID: some-id") + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.NotNil(t, config.WrapTransport) + }) + t.Run("use hub kubeconfig, malformed content - error", func(t *testing.T) { + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig-malformed") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.Nil(t, config) + assert.NotNil(t, err) + }) + t.Run("use hub kubeconfig, hubURL argument is ignored - success", func(t *testing.T) { + // The kubeconfig's own cluster server URL is authoritative; a non-empty hubURL passed + // in (which main() only does when UseKubeConfig is unset) must not leak through. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("https://should-be-ignored.example.com", options.HubConnectivityOptions{ + UseKubeConfig: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, "https://hub.fixture.example.com", config.Host) + }) + t.Run("use hub kubeconfig, TLSClientConfig.Insecure from kubeconfig is not overridden - success", func(t *testing.T) { + // The fixture kubeconfig sets insecure-skip-tls-verify: true; UseInsecureTLSClient is + // left at its false zero-value here to confirm the kubeconfig's own setting wins, + // rather than being reset by the (ignored, per hub.go's doc comment) opts field. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + UseInsecureTLSClient: false, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.True(t, config.TLSClientConfig.Insecure) + }) + t.Run("use hub kubeconfig takes precedence over CA auth - success", func(t *testing.T) { + // options.Validate() rejects UseCertificateAuth+UseKubeConfig together in practice, but + // buildHubConfig itself does not re-validate; this locks in that UseKubeConfig, checked + // first in the if/else chain, wins if it is ever called with both set. + t.Setenv("KUBE_CONFIG_PATH", "./testdata/kubeconfig") + config, err := buildHubConfig("", options.HubConnectivityOptions{ + UseKubeConfig: true, + UseCertificateAuth: true, + }) + assert.NotNil(t, config) + assert.Nil(t, err) + assert.Equal(t, "https://hub.fixture.example.com", config.Host) + assert.Empty(t, config.TLSClientConfig.CertFile) + assert.Empty(t, config.TLSClientConfig.KeyFile) + }) +} + +// fakeMemberController records the calls it gets and fails with err. +type fakeMemberController struct { + err error + joins, leaves int +} + +func (f *fakeMemberController) Join(_ context.Context) error { + f.joins++ + return f.err +} + +func (f *fakeMemberController) Leave(_ context.Context) error { + f.leaves++ + return f.err +} + +func TestMemberControllers(t *testing.T) { + errFirst := errors.New("first") + first := &fakeMemberController{err: errFirst} + second := &fakeMemberController{} + m := memberControllers{first, second} + + // A failure in one controller must not stop the call from reaching the rest. + if err := m.Join(context.Background()); !errors.Is(err, errFirst) { + t.Errorf("Join() = %v, want %v", err, errFirst) + } + if err := m.Leave(context.Background()); !errors.Is(err, errFirst) { + t.Errorf("Leave() = %v, want %v", err, errFirst) + } + for name, c := range map[string]*fakeMemberController{"first": first, "second": second} { + if c.joins != 1 || c.leaves != 1 { + t.Errorf("%s controller got %d joins and %d leaves, want 1 and 1", name, c.joins, c.leaves) + } + } + + second.err, first.err = nil, nil + if err := m.Join(context.Background()); err != nil { + t.Errorf("Join() = %v, want nil", err) + } } diff --git a/cmd/memberagent/options/hub.go b/cmd/memberagent/options/hub.go index aacd89091..648bef8bb 100644 --- a/cmd/memberagent/options/hub.go +++ b/cmd/memberagent/options/hub.go @@ -29,9 +29,9 @@ type HubConnectivityOptions struct { // and certificate to use for authentication via the `IDENTITY_KEY` and `IDENTITY_CERT` // environment variables respectively. // - // Otherwise, the member agent will use token-based authentication when connecting - // to the hub cluster. The agent will read the token from the file path specified - // by the `CONFIG_PATH` environment variable. + // Otherwise, unless UseKubeConfig is set, the member agent will use token-based + // authentication when connecting to the hub cluster. The agent will read the token + // from the file path specified by the `CONFIG_PATH` environment variable. UseCertificateAuth bool // Use an insecure client or not when connecting to the hub cluster. @@ -41,6 +41,26 @@ type HubConnectivityOptions struct { // variable; you can also give the member agent a file path to the CA data instead // via the `HUB_CERTIFICATE_AUTHORITY` environment variable. UseInsecureTLSClient bool + + // Use a kubeconfig file for the hub cluster connection instead of the refresh-token + // sidecar or UseCertificateAuth. When set, provide the member agent the path to a + // kubeconfig file via the `KUBE_CONFIG_PATH` environment variable; it's loaded via + // client-go's clientcmd (the same mechanism kubectl uses). This takes over the hub + // connection entirely: HUB_SERVER_URL, UseCertificateAuth, UseInsecureTLSClient, + // CA_BUNDLE/HUB_CERTIFICATE_AUTHORITY, and CONFIG_PATH/IDENTITY_KEY/IDENTITY_CERT are all + // ignored, since the kubeconfig already carries the server URL, TLS configuration, and + // authentication (a bearer token, a client certificate, or a standard Kubernetes exec + // credential plugin — https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins). + // + // This is the vendor-neutral way to authenticate to the hub via a federated identity + // (AWS IRSA, GCP Workload Identity, Azure AD workload identity, SPIFFE, etc.): point this + // at a kubeconfig whose exec stanza invokes whatever credential plugin your platform + // provides. KubeFleet itself has no vendor-specific knowledge of any of these; the exec + // plugin binary must be present in the member agent image (e.g. via a custom image + // layered on top of the official one, or via the Helm chart's extraInitContainers). + // + // Mutually exclusive with UseCertificateAuth. + UseKubeConfig bool } // AddFlags adds flags for HubConnectivityOptions to the specified FlagSet. @@ -56,4 +76,10 @@ func (o *HubConnectivityOptions) AddFlags(flags *flag.FlagSet) { "tls-insecure", false, "Use an insecure client or not when connecting to the hub cluster.") + + flags.BoolVar( + &o.UseKubeConfig, + "use-kubeconfig", + false, + "Use a kubeconfig file for the hub cluster connection instead of the refresh-token sidecar or --use-ca-auth. Requires the KUBE_CONFIG_PATH environment variable to also be set. Mutually exclusive with --use-ca-auth.") } diff --git a/cmd/memberagent/options/options.go b/cmd/memberagent/options/options.go index 29376f9fe..8d69df792 100644 --- a/cmd/memberagent/options/options.go +++ b/cmd/memberagent/options/options.go @@ -36,6 +36,10 @@ type Options struct { // KubeFleet cluster property provider related options. PropertyProviderOpts PropertyProviderOptions + // Apply the Works of the experimental FEP-0001 placement policy APIs (placement.kubefleet.dev/v1alpha1) + // or not. The hub agent must run with the same APIs enabled. + EnablePlacementPolicyAPIs bool + // The fields below are added only for backwards compatibility reasons. // Their values are never read. UseV1Beta1APIs bool @@ -51,6 +55,13 @@ func (o *Options) AddFlags(flags *flag.FlagSet) { o.ApplierOpts.AddFlags(flags) o.PropertyProviderOpts.AddFlags(flags) + // Note (chenyu1): backport only: placement policy APIs are disabled by default in the backport for now. + flags.BoolVar( + &o.EnablePlacementPolicyAPIs, + "enable-placement-policy-apis", + false, + "Apply the Works of the experimental placement policy APIs (placement.kubefleet.dev/v1alpha1) or not. The hub agent must run with the same APIs enabled.") + // The flags set up below are added only for backwards compatibility reasons. // They are no-op flags and their values are never read. flags.BoolVar( diff --git a/cmd/memberagent/options/options_test.go b/cmd/memberagent/options/options_test.go index dbc1f4c91..42b436b87 100644 --- a/cmd/memberagent/options/options_test.go +++ b/cmd/memberagent/options/options_test.go @@ -58,6 +58,18 @@ func TestHubConnectivityOptions(t *testing.T) { UseInsecureTLSClient: true, }, }, + { + name: "use kubeconfig specified", + flagSetName: "useKubeconfigSpecified", + args: []string{ + "--use-kubeconfig=true", + }, + wantHubConnectOpts: HubConnectivityOptions{ + UseCertificateAuth: false, + UseInsecureTLSClient: false, + UseKubeConfig: true, + }, + }, } for _, tc := range testCases { diff --git a/cmd/memberagent/options/validation.go b/cmd/memberagent/options/validation.go index 5016205d8..947e41171 100644 --- a/cmd/memberagent/options/validation.go +++ b/cmd/memberagent/options/validation.go @@ -28,6 +28,11 @@ func (o *Options) Validate() field.ErrorList { errs := field.ErrorList{} newPath := field.NewPath("Options") + // Cross-field validation for hub connectivity options. + if o.HubConnectivityOpts.UseCertificateAuth && o.HubConnectivityOpts.UseKubeConfig { + errs = append(errs, field.Invalid(newPath.Child("HubConnectivityOpts").Child("UseKubeConfig"), o.HubConnectivityOpts.UseKubeConfig, "UseCertificateAuth and UseKubeConfig must not both be set")) + } + // Cross-field validation for controller manager options. if float64(o.CtrlManagerOptions.HubManagerOpts.Burst) < o.CtrlManagerOptions.HubManagerOpts.QPS { errs = append(errs, field.Invalid(newPath.Child("HubManagerOpts").Child("Burst"), o.CtrlManagerOptions.HubManagerOpts.Burst, "The burst limit for hub cluster client-side throttling must be greater than or equal to its QPS limit")) diff --git a/cmd/memberagent/options/validation_test.go b/cmd/memberagent/options/validation_test.go index a77ef9ad9..5986401af 100644 --- a/cmd/memberagent/options/validation_test.go +++ b/cmd/memberagent/options/validation_test.go @@ -64,6 +64,15 @@ func TestValidate(t *testing.T) { opt: newTestOptions(nil), want: field.ErrorList{}, }, + "use certificate auth and use kubeconfig together": { + opt: newTestOptions(func(option *Options) { + option.HubConnectivityOpts.UseCertificateAuth = true + option.HubConnectivityOpts.UseKubeConfig = true + }), + want: field.ErrorList{ + field.Invalid(newPath.Child("HubConnectivityOpts").Child("UseKubeConfig"), true, "UseCertificateAuth and UseKubeConfig must not both be set"), + }, + }, "hub burst less than hub QPS": { opt: newTestOptions(func(option *Options) { option.CtrlManagerOptions.HubManagerOpts.QPS = 200 diff --git a/cmd/memberagent/testdata/kubeconfig b/cmd/memberagent/testdata/kubeconfig new file mode 100644 index 000000000..43859c4dd --- /dev/null +++ b/cmd/memberagent/testdata/kubeconfig @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: Config +clusters: +- cluster: + server: https://hub.fixture.example.com + insecure-skip-tls-verify: true + name: hub +contexts: +- context: + cluster: hub + user: fixture-user + name: hub +current-context: hub +users: +- name: fixture-user + user: + token: fixture-bearer-token diff --git a/cmd/memberagent/testdata/kubeconfig-exec b/cmd/memberagent/testdata/kubeconfig-exec new file mode 100644 index 000000000..68f688c31 --- /dev/null +++ b/cmd/memberagent/testdata/kubeconfig-exec @@ -0,0 +1,27 @@ +apiVersion: v1 +kind: Config +clusters: +- cluster: + server: https://hub.fixture.example.com + insecure-skip-tls-verify: true + name: hub +contexts: +- context: + cluster: hub + user: fixture-exec-user + name: hub +current-context: hub +users: +- name: fixture-exec-user + user: + exec: + apiVersion: client.authentication.k8s.io/v1 + command: kubelogin + args: + - get-token + - --server-id + - fixture-server-id + - --login + - workloadidentity + provideClusterInfo: false + interactiveMode: Never diff --git a/cmd/memberagent/testdata/kubeconfig-malformed b/cmd/memberagent/testdata/kubeconfig-malformed new file mode 100644 index 000000000..d91a64f78 --- /dev/null +++ b/cmd/memberagent/testdata/kubeconfig-malformed @@ -0,0 +1,2 @@ +this is not: [valid kubeconfig yaml + - broken structure diff --git a/config/crd/bases/placement.kubefleet.dev_appliedworks.yaml b/config/crd/bases/placement.kubefleet.dev_appliedworks.yaml new file mode 100644 index 000000000..c8e9d6c7a --- /dev/null +++ b/config/crd/bases/placement.kubefleet.dev_appliedworks.yaml @@ -0,0 +1,105 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: appliedworks.placement.kubefleet.dev +spec: + group: placement.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-placement + kind: AppliedWork + listKind: AppliedWorkList + plural: appliedworks + shortNames: + - kfappliedwork + singular: appliedwork + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + AppliedWork is the KubeFleet API that serves as owners for manifests applied to a member cluster via + a work object. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: Spec represents the desired configuration of AppliedWork. + properties: + workName: + description: The name of the corresponding work object. Note that + the object is on the KubeFleet hub cluster. + type: string + workNamespace: + description: The namespace of the corresponding work object. Note + that the object is on the KubeFleet hub cluster. + type: string + required: + - workName + - workNamespace + type: object + status: + description: Status represents the current status of AppliedWork. + properties: + appliedResources: + description: A list of applied manifests. This list is populated for + informational purposes only. + items: + properties: + apiGroup: + type: string + apiVersion: + type: string + kind: + type: string + name: + description: The name of the manifest. + type: string + namespace: + description: The namespace of the manifest. + type: string + ordinal: + description: The ordinal of the manifest. + type: integer + resource: + type: string + uid: + description: The UID of the applied manifest. + type: string + required: + - apiVersion + - kind + - name + - ordinal + - resource + - uid + type: object + type: array + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/placement.kubefleet.dev_clusterclaims.yaml b/config/crd/bases/placement.kubefleet.dev_clusterclaims.yaml index bb1fc81e0..c83712d4d 100644 --- a/config/crd/bases/placement.kubefleet.dev_clusterclaims.yaml +++ b/config/crd/bases/placement.kubefleet.dev_clusterclaims.yaml @@ -297,8 +297,17 @@ spec: format: date-time type: string provisionedClusterName: - description: The name of the cluster that has been provisioned for - this cluster claim, if any. + description: |- + The name of the cluster that has been provisioned for this cluster claim, if any. + + A cluster claim asks for a single cluster: a provisioner fulfills it by provisioning one + cluster, recording its name here, and marking the Completed condition. A placement that still + needs more clusters than one claim provides is served one claim at a time -- once the + provisioned cluster becomes eligible, the completed claim is withdrawn and a new claim is + created, under the same deterministic name, for the next cluster. A provisioner that retries a + conflicting status write must therefore key the retry on the claim's identity (UID or resource + version), not on its name alone, so that a write meant for a completed claim is not applied to + the different, freshly created claim that has taken its name. type: string type: object required: diff --git a/config/crd/bases/placement.kubefleet.dev_clusterplacementbindings.yaml b/config/crd/bases/placement.kubefleet.dev_clusterplacementbindings.yaml index 87dfb4ef9..e4d3a9049 100644 --- a/config/crd/bases/placement.kubefleet.dev_clusterplacementbindings.yaml +++ b/config/crd/bases/placement.kubefleet.dev_clusterplacementbindings.yaml @@ -47,6 +47,7 @@ spec: clusterName: description: The name of the member cluster that this binding is associated with. + minLength: 1 type: string clusterSelectors: description: |- @@ -201,6 +202,7 @@ spec: placementPolicyName: description: The name of the placement policy that this binding is associated with. + minLength: 1 type: string resourceSnapshotName: description: |- @@ -208,6 +210,7 @@ spec: If the resources being selected cannot fit within a single resource snapshot, this field tracks the name of the primary resource snapshot that this binding is associated with. + minLength: 1 type: string suspended: default: false diff --git a/config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml b/config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml index 6e18f5a7e..a01bca037 100644 --- a/config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml +++ b/config/crd/bases/placement.kubefleet.dev_clusterplacementpolicies.yaml @@ -63,8 +63,12 @@ spec: The desired number of clusters that KubeFleet should select based on the given terms. The default value is 1. To select all clusters that match the given terms, use the value "All". + maxLength: 3 pattern: ^([1-9][0-9]{0,2}|All)$ x-kubernetes-int-or-string: true + x-kubernetes-validations: + - message: count must be between 1 and 999, or "All" + rule: 'type(self) == int ? self >= 1 && self <= 999 : true' minCount: description: |- The minimum number of clusters that KubeFleet should select based on the given terms, when KubeFleet is not able @@ -238,13 +242,11 @@ spec: type: string type: object x-kubernetes-validations: - - message: minCount must be less than or equal to count when count - is not All + - message: minCount must be less than or equal to count rule: '!has(self.minCount) || !has(self.count) || (type(self.count) - == string && self.count == ''All'') || (type(self.count) == - int && self.minCount <= self.count) || (type(self.count) == - string && self.count.matches(''^[0-9]+$'') && self.minCount - <= int(self.count))' + == int && self.minCount <= self.count) || (type(self.count) + == string && (!self.count.matches(''^[0-9]{1,3}$'') || self.minCount + <= int(self.count)))' maxItems: 10 minItems: 1 type: array @@ -260,7 +262,7 @@ spec: failure recovery purposes; one can inspect them to see the past state of the selected resources, or roll back to a previous revision if the latest revision is not working as expected. - It is also possible to manually request a new resource revision to be created. + The limit is a soft constraint; KubeFleet may retain more revisions than the specified limit as necessary. The default value is 3. format: int32 @@ -534,15 +536,15 @@ spec: If the Operator field is set to Exists, this field must be left empty. type: string type: object + x-kubernetes-validations: + - message: operator must be Exists when key is empty + rule: (has(self.key) && size(self.key) > 0) || self.operator == + 'Exists' + - message: value must be empty when operator is Exists + rule: self.operator != 'Exists' || !has(self.value) || size(self.value) + == 0 maxItems: 10 type: array - x-kubernetes-validations: - - message: operator must be Exists and value must be empty when key - is empty - rule: self.all(t, t.key != '' || (t.operator == 'Exists' && t.value - == '')) - - message: value must be empty when operator is Exists - rule: self.all(t, t.operator != 'Exists' || t.value == '') required: - resourceSelectors type: object @@ -550,8 +552,11 @@ spec: description: The observed status of the cluster placement policy. properties: activeClusterClaims: - description: The number of active cluster claims that have been submitted - by this placement. + description: |- + The number of active cluster claims that have been submitted by this placement. A claim + that is being withdrawn still counts as active until KubeFleet observes it gone -- + whether a provisioner finalizer holds it or the withdrawal simply has not been confirmed + yet -- as it continues to occupy the placement's claim budget. format: int32 type: integer bindingManager: diff --git a/config/crd/bases/placement.kubefleet.dev_placementbindings.yaml b/config/crd/bases/placement.kubefleet.dev_placementbindings.yaml index e6c135e67..8f5568fe8 100644 --- a/config/crd/bases/placement.kubefleet.dev_placementbindings.yaml +++ b/config/crd/bases/placement.kubefleet.dev_placementbindings.yaml @@ -47,6 +47,7 @@ spec: clusterName: description: The name of the member cluster that this binding is associated with. + minLength: 1 type: string clusterSelectors: description: |- @@ -201,6 +202,7 @@ spec: placementPolicyName: description: The name of the placement policy that this binding is associated with. + minLength: 1 type: string resourceSnapshotName: description: |- @@ -208,6 +210,7 @@ spec: If the resources being selected cannot fit within a single resource snapshot, this field tracks the name of the primary resource snapshot that this binding is associated with. + minLength: 1 type: string suspended: default: false diff --git a/config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml b/config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml index 36a9c326e..44a47cb8e 100644 --- a/config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml +++ b/config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml @@ -63,8 +63,12 @@ spec: The desired number of clusters that KubeFleet should select based on the given terms. The default value is 1. To select all clusters that match the given terms, use the value "All". + maxLength: 3 pattern: ^([1-9][0-9]{0,2}|All)$ x-kubernetes-int-or-string: true + x-kubernetes-validations: + - message: count must be between 1 and 999, or "All" + rule: 'type(self) == int ? self >= 1 && self <= 999 : true' minCount: description: |- The minimum number of clusters that KubeFleet should select based on the given terms, when KubeFleet is not able @@ -238,13 +242,11 @@ spec: type: string type: object x-kubernetes-validations: - - message: minCount must be less than or equal to count when count - is not All + - message: minCount must be less than or equal to count rule: '!has(self.minCount) || !has(self.count) || (type(self.count) - == string && self.count == ''All'') || (type(self.count) == - int && self.minCount <= self.count) || (type(self.count) == - string && self.count.matches(''^[0-9]+$'') && self.minCount - <= int(self.count))' + == int && self.minCount <= self.count) || (type(self.count) + == string && (!self.count.matches(''^[0-9]{1,3}$'') || self.minCount + <= int(self.count)))' maxItems: 10 minItems: 1 type: array @@ -260,7 +262,7 @@ spec: failure recovery purposes; one can inspect them to see the past state of the selected resources, or roll back to a previous revision if the latest revision is not working as expected. - It is also possible to manually request a new resource revision to be created. + The limit is a soft constraint; KubeFleet may retain more revisions than the specified limit as necessary. The default value is 3. format: int32 @@ -534,15 +536,15 @@ spec: If the Operator field is set to Exists, this field must be left empty. type: string type: object + x-kubernetes-validations: + - message: operator must be Exists when key is empty + rule: (has(self.key) && size(self.key) > 0) || self.operator == + 'Exists' + - message: value must be empty when operator is Exists + rule: self.operator != 'Exists' || !has(self.value) || size(self.value) + == 0 maxItems: 10 type: array - x-kubernetes-validations: - - message: operator must be Exists and value must be empty when key - is empty - rule: self.all(t, t.key != '' || (t.operator == 'Exists' && t.value - == '')) - - message: value must be empty when operator is Exists - rule: self.all(t, t.operator != 'Exists' || t.value == '') required: - resourceSelectors type: object @@ -550,8 +552,11 @@ spec: description: The observed status of the placement policy. properties: activeClusterClaims: - description: The number of active cluster claims that have been submitted - by this placement. + description: |- + The number of active cluster claims that have been submitted by this placement. A claim + that is being withdrawn still counts as active until KubeFleet observes it gone -- + whether a provisioner finalizer holds it or the withdrawal simply has not been confirmed + yet -- as it continues to occupy the placement's claim budget. format: int32 type: integer bindingManager: diff --git a/config/crd/bases/placement.kubefleet.dev_works.yaml b/config/crd/bases/placement.kubefleet.dev_works.yaml index 08084e435..c9209b1e0 100644 --- a/config/crd/bases/placement.kubefleet.dev_works.yaml +++ b/config/crd/bases/placement.kubefleet.dev_works.yaml @@ -14,6 +14,8 @@ spec: kind: Work listKind: WorkList plural: works + shortNames: + - kfwork singular: work scope: Namespaced versions: @@ -377,9 +379,7 @@ spec: required: - apiVersion - kind - - name - ordinal - - resource type: object required: - identifier diff --git a/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdateruns.yaml b/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdateruns.yaml index 27dc987da..22369d2f3 100644 --- a/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdateruns.yaml +++ b/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdateruns.yaml @@ -2859,11 +2859,13 @@ spec: The status of the post-update tasks associated with the current stage. Empty if the stage has not finished updating all the clusters. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -2944,11 +2946,13 @@ spec: description: The status of the pre-update tasks associated with the current stage. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -3242,8 +3246,49 @@ spec: The update run fails to initialize if the strategy fails to produce a valid list of stages where each selected cluster is included in exactly one stage. properties: + deleteStage: + description: |- + DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + complete and removes the resources from the clusters that are no longer selected by the placement. + If not specified, the delete stage starts as soon as the last update stage completes. + properties: + beforeStageTasks: + description: |- + The collection of tasks that needs to be completed successfully before the delete stage starts removing + the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + The tasks are skipped if there is no cluster to remove the resources from. + items: + description: StageTask is the pre or post stage task that + needs to be completed before starting or moving to the + next stage. + properties: + type: + description: The type of the before or after stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait after all the clusters in the current stage complete the update before moving to the next stage. + Only hours (h), minutes (m), and seconds (s) units are accepted. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: BeforeStageTaskType is Approval, waitTime is not + allowed + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: BeforeStageTaskType cannot be TimedWait + rule: '!self.exists(e, e.type == ''TimedWait'')' + type: object stages: - description: Stage specifies the configuration for each update + description: Stages specifies the configuration for each update stage. items: description: |- @@ -3287,7 +3332,7 @@ spec: rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' beforeStageTasks: description: |- - The collection of tasks that needs to completed successfully by each stage before starting the stage. + The collection of tasks that needs to be completed successfully by each stage before starting the stage. Each task is executed in parallel and there cannot be more than one task of the same type. items: description: StageTask is the pre or post stage task that @@ -3313,8 +3358,8 @@ spec: maxItems: 1 type: array x-kubernetes-validations: - - message: AfterStageTaskType is Approval, waitTime is not - allowed + - message: BeforeStageTaskType is Approval, waitTime is + not allowed rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' - message: BeforeStageTaskType cannot be TimedWait rule: '!self.exists(e, e.type == ''TimedWait'')' @@ -3418,11 +3463,13 @@ spec: The status of the post-update tasks associated with the current stage. Empty if the stage has not finished updating all the clusters. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -3503,11 +3550,13 @@ spec: description: The status of the pre-update tasks associated with the current stage. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- diff --git a/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdatestrategies.yaml b/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdatestrategies.yaml index b134a75b5..a67d7ba7d 100644 --- a/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdatestrategies.yaml +++ b/config/crd/bases/placement.kubernetes-fleet.io_clusterstagedupdatestrategies.yaml @@ -375,8 +375,47 @@ spec: spec: description: The desired state of ClusterStagedUpdateStrategy. properties: + deleteStage: + description: |- + DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + complete and removes the resources from the clusters that are no longer selected by the placement. + If not specified, the delete stage starts as soon as the last update stage completes. + properties: + beforeStageTasks: + description: |- + The collection of tasks that needs to be completed successfully before the delete stage starts removing + the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + The tasks are skipped if there is no cluster to remove the resources from. + items: + description: StageTask is the pre or post stage task that needs + to be completed before starting or moving to the next stage. + properties: + type: + description: The type of the before or after stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait after all the clusters in the current stage complete the update before moving to the next stage. + Only hours (h), minutes (m), and seconds (s) units are accepted. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: BeforeStageTaskType is Approval, waitTime is not allowed + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: BeforeStageTaskType cannot be TimedWait + rule: '!self.exists(e, e.type == ''TimedWait'')' + type: object stages: - description: Stage specifies the configuration for each update stage. + description: Stages specifies the configuration for each update stage. items: description: |- StageConfig describes a single update stage. @@ -416,7 +455,7 @@ spec: rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' beforeStageTasks: description: |- - The collection of tasks that needs to completed successfully by each stage before starting the stage. + The collection of tasks that needs to be completed successfully by each stage before starting the stage. Each task is executed in parallel and there cannot be more than one task of the same type. items: description: StageTask is the pre or post stage task that @@ -441,7 +480,8 @@ spec: maxItems: 1 type: array x-kubernetes-validations: - - message: AfterStageTaskType is Approval, waitTime is not allowed + - message: BeforeStageTaskType is Approval, waitTime is not + allowed rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' - message: BeforeStageTaskType cannot be TimedWait rule: '!self.exists(e, e.type == ''TimedWait'')' diff --git a/config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml b/config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml index 81b506f4a..4622f51d1 100644 --- a/config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml +++ b/config/crd/bases/placement.kubernetes-fleet.io_stagedupdateruns.yaml @@ -1779,11 +1779,13 @@ spec: The status of the post-update tasks associated with the current stage. Empty if the stage has not finished updating all the clusters. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -1864,11 +1866,13 @@ spec: description: The status of the pre-update tasks associated with the current stage. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -2162,8 +2166,49 @@ spec: The update run fails to initialize if the strategy fails to produce a valid list of stages where each selected cluster is included in exactly one stage. properties: + deleteStage: + description: |- + DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + complete and removes the resources from the clusters that are no longer selected by the placement. + If not specified, the delete stage starts as soon as the last update stage completes. + properties: + beforeStageTasks: + description: |- + The collection of tasks that needs to be completed successfully before the delete stage starts removing + the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + The tasks are skipped if there is no cluster to remove the resources from. + items: + description: StageTask is the pre or post stage task that + needs to be completed before starting or moving to the + next stage. + properties: + type: + description: The type of the before or after stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait after all the clusters in the current stage complete the update before moving to the next stage. + Only hours (h), minutes (m), and seconds (s) units are accepted. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: BeforeStageTaskType is Approval, waitTime is not + allowed + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: BeforeStageTaskType cannot be TimedWait + rule: '!self.exists(e, e.type == ''TimedWait'')' + type: object stages: - description: Stage specifies the configuration for each update + description: Stages specifies the configuration for each update stage. items: description: |- @@ -2207,7 +2252,7 @@ spec: rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' beforeStageTasks: description: |- - The collection of tasks that needs to completed successfully by each stage before starting the stage. + The collection of tasks that needs to be completed successfully by each stage before starting the stage. Each task is executed in parallel and there cannot be more than one task of the same type. items: description: StageTask is the pre or post stage task that @@ -2233,8 +2278,8 @@ spec: maxItems: 1 type: array x-kubernetes-validations: - - message: AfterStageTaskType is Approval, waitTime is not - allowed + - message: BeforeStageTaskType is Approval, waitTime is + not allowed rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' - message: BeforeStageTaskType cannot be TimedWait rule: '!self.exists(e, e.type == ''TimedWait'')' @@ -2338,11 +2383,13 @@ spec: The status of the post-update tasks associated with the current stage. Empty if the stage has not finished updating all the clusters. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- @@ -2423,11 +2470,13 @@ spec: description: The status of the pre-update tasks associated with the current stage. items: + description: StageTaskStatus defines the status of a pre or + post update task of a stage. properties: approvalRequestName: description: |- The name of the approval request object that is created for this stage. - Only valid if the AfterStageTaskType is Approval. + Only valid if the type of the task is Approval. type: string conditions: description: |- diff --git a/config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml b/config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml index ccfb44e85..f8e8dabd1 100644 --- a/config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml +++ b/config/crd/bases/placement.kubernetes-fleet.io_stagedupdatestrategies.yaml @@ -235,8 +235,47 @@ spec: spec: description: The desired state of StagedUpdateStrategy. properties: + deleteStage: + description: |- + DeleteStage specifies the configuration for the delete stage, which runs after all the update stages + complete and removes the resources from the clusters that are no longer selected by the placement. + If not specified, the delete stage starts as soon as the last update stage completes. + properties: + beforeStageTasks: + description: |- + The collection of tasks that needs to be completed successfully before the delete stage starts removing + the resources. Same as an update stage, the only task allowed is a single Approval; to wait for some time + before the removal, add a TimedWait task to the afterStageTasks of the last update stage instead. + The tasks are skipped if there is no cluster to remove the resources from. + items: + description: StageTask is the pre or post stage task that needs + to be completed before starting or moving to the next stage. + properties: + type: + description: The type of the before or after stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait after all the clusters in the current stage complete the update before moving to the next stage. + Only hours (h), minutes (m), and seconds (s) units are accepted. + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: BeforeStageTaskType is Approval, waitTime is not allowed + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: BeforeStageTaskType cannot be TimedWait + rule: '!self.exists(e, e.type == ''TimedWait'')' + type: object stages: - description: Stage specifies the configuration for each update stage. + description: Stages specifies the configuration for each update stage. items: description: |- StageConfig describes a single update stage. @@ -276,7 +315,7 @@ spec: rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' beforeStageTasks: description: |- - The collection of tasks that needs to completed successfully by each stage before starting the stage. + The collection of tasks that needs to be completed successfully by each stage before starting the stage. Each task is executed in parallel and there cannot be more than one task of the same type. items: description: StageTask is the pre or post stage task that @@ -301,7 +340,8 @@ spec: maxItems: 1 type: array x-kubernetes-validations: - - message: AfterStageTaskType is Approval, waitTime is not allowed + - message: BeforeStageTaskType is Approval, waitTime is not + allowed rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' - message: BeforeStageTaskType cannot be TimedWait rule: '!self.exists(e, e.type == ''TimedWait'')' diff --git a/config/crd/bases/rollout.kubefleet.dev_approvalrequests.yaml b/config/crd/bases/rollout.kubefleet.dev_approvalrequests.yaml new file mode 100644 index 000000000..1be9602f3 --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_approvalrequests.yaml @@ -0,0 +1,139 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: approvalrequests.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: ApprovalRequest + listKind: ApprovalRequestList + plural: approvalrequests + shortNames: + - stageapproval + singular: approvalrequest + scope: Namespaced + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + ApprovalRequest is the KubeFleet API that enables users to approve a rollout to a stage of clusters + for a placement policy, when staged update is in use. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the ApprovalRequest API object. + properties: + stageName: + description: The name of the stage that the approval request is associated + with in the staged update run. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + x-kubernetes-validations: + - message: stageName is immutable + rule: self == oldSelf + stagedUpdateRunName: + description: The name of the StagedUpdateRun API object that the approval + request is associated with. + type: string + x-kubernetes-validations: + - message: stagedUpdateRunName is immutable + rule: self == oldSelf + required: + - stageName + - stagedUpdateRunName + type: object + status: + description: The observed status of the ApprovalRequest API object. + properties: + conditions: + description: A list of observed conditions about the approval request. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/rollout.kubefleet.dev_clusterapprovalrequests.yaml b/config/crd/bases/rollout.kubefleet.dev_clusterapprovalrequests.yaml new file mode 100644 index 000000000..5a2302379 --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_clusterapprovalrequests.yaml @@ -0,0 +1,139 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: clusterapprovalrequests.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: ClusterApprovalRequest + listKind: ClusterApprovalRequestList + plural: clusterapprovalrequests + shortNames: + - clusterstageapproval + singular: clusterapprovalrequest + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + ClusterApprovalRequest is the KubeFleet API that enables users to approve a rollout to a stage of clusters + for a cluster placement policy, when staged update is in use. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the ClusterApprovalRequest API object. + properties: + stageName: + description: The name of the stage that the approval request is associated + with in the staged update run. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + x-kubernetes-validations: + - message: stageName is immutable + rule: self == oldSelf + stagedUpdateRunName: + description: The name of the StagedUpdateRun API object that the approval + request is associated with. + type: string + x-kubernetes-validations: + - message: stagedUpdateRunName is immutable + rule: self == oldSelf + required: + - stageName + - stagedUpdateRunName + type: object + status: + description: The observed status of the ClusterApprovalRequest API object. + properties: + conditions: + description: A list of observed conditions about the approval request. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdateruns.yaml b/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdateruns.yaml new file mode 100644 index 000000000..44e6d822f --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdateruns.yaml @@ -0,0 +1,560 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: clusterstagedupdateruns.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: ClusterStagedUpdateRun + listKind: ClusterStagedUpdateRunList + plural: clusterstagedupdateruns + shortNames: + - clusterupdaterun + singular: clusterstagedupdaterun + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + ClusterStagedUpdateRun is the KubeFleet API that enables users to roll out resource changes for a cluster + placement policy in a staged manner, where clusters are grouped into stages and KubeFleet applies resource changes to each stage + sequentially. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the ClusterStagedUpdateRun API object. + properties: + failurePolicy: + description: |- + The failure policy of the staged update run. It helps KubeFleet determine when to stop the staged update run + when there are too many failures, so as to keep the impact radius under control. + properties: + maxAllowedClusterFailures: + default: 1 + description: |- + The maximum number of clusters that can fail the rollout before KubeFleet stops the staged + update run. The count is tracked across all stages and clusters. A cluster is considered to have + failed the rollout if the new (updated) placement does not become available within the maximum + wait time. + + The default value is 1; the value must be between 0 and 1000. + format: int32 + maximum: 1000 + minimum: 0 + type: integer + maxWaitTimePerClusterMinutes: + default: 5 + description: |- + The maximum time to wait for a rollout to a cluster to complete before KubeFleet considers the rollout to have + failed for the cluster. + + The default value is 5 minutes; the value must be between 1 and 10080 (7 days). + format: int32 + maximum: 10080 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: failurePolicy is immutable + rule: self == oldSelf + placementPolicyName: + description: |- + The name of the placement policy that the StagedUpdateRun API object is associated with, i.e., + the name of the placement policy where resource changes are rolled out. + type: string + x-kubernetes-validations: + - message: placementPolicyName is immutable + rule: self == oldSelf + resourceSnapshotName: + description: |- + The name of the resource snapshot that the StagedUpdateRun API object is associated with, i.e., + the name of the resource snapshot where resource changes are introduced. + + If left empty, KubeFleet will request a new resource snapshot to be created based on the current state + of selected resources in the placement policy (if applicable). + maxLength: 253 + type: string + x-kubernetes-validations: + - message: resourceSnapshotName is immutable + rule: self == oldSelf + stagedUpdateStrategyName: + description: |- + The name of the staged update strategy that the StagedUpdateRun API object is associated with, i.e., + the name of the staged update strategy that organizes clusters into stages and defines how the rollout is executed. + type: string + x-kubernetes-validations: + - message: stagedUpdateStrategyName is immutable + rule: self == oldSelf + suspended: + default: false + description: |- + Whether the staged update run is suspended. If set to true, KubeFleet will pause the rollout. + + If the staged update run is created with this field set to true, KubeFleet will initialize the staged update run, + i.e., decide on the resource snapshot to roll out and determine the stages and the clusters in each stage, + but will not start rolling out resource changes until the field is set back to false. + + When a staged update run is suspended after the rollout starts, KubeFleet will pause the rollout as soon as + possible. If a cluster has started updating but has not yet completed, KubeFleet will not consider + its update a failure even if the maximum wait time for the cluster has been exceeded during the suspension. + The wait timer will be reset once the suspension is lifted. However, the time spent in suspension will + still count towards any timed wait tasks you have configured; those tasks will proceed or even complete, + accounting for the time spent in suspension, once the suspension is lifted. + type: boolean + required: + - placementPolicyName + - stagedUpdateStrategyName + type: object + x-kubernetes-validations: + - message: resourceSnapshotName cannot be added or removed after creation + rule: has(self.resourceSnapshotName) == has(oldSelf.resourceSnapshotName) + - message: failurePolicy cannot be added or removed after creation + rule: has(self.failurePolicy) == has(oldSelf.failurePolicy) + status: + description: The observed status of the ClusterStagedUpdateRun API object. + properties: + conditions: + description: A list of observed conditions of the staged update run. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + resourceSnapshotNameToRollout: + description: |- + The name of the resource snapshot that is being rolled out in the staged update run. + + If the staged update run is created with the resourceSnapshotName field in the spec set to a specific value, + this field will be set to the same value; otherwise, KubeFleet will request a new resource snapshot to be created (if applicable) + and populate this field with the name of the newly created resource snapshot. + type: string + stages: + description: The status of each stage in the staged update run, including + the clusters in each stage and their rollout status. + items: + properties: + afterStageTasks: + description: The list of tasks to be executed after completing + the rollout in the stage, and their execution status. + items: + properties: + approvalRequestName: + description: |- + The name of the approval request that is created for the stage task. + + This field is only set when the task type is Approval. + type: string + conditions: + description: A list of observed conditions about the progress + of the stage task. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: + description: The type of the stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: The time to wait in a TimedWait task. + type: string + required: + - type + type: object + maxItems: 2 + type: array + beforeStageTasks: + description: The list of tasks to be executed before starting + the rollout in the stage, and their execution status. + items: + properties: + approvalRequestName: + description: |- + The name of the approval request that is created for the stage task. + + This field is only set when the task type is Approval. + type: string + conditions: + description: A list of observed conditions about the progress + of the stage task. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: + description: The type of the stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: The time to wait in a TimedWait task. + type: string + required: + - type + type: object + maxItems: 1 + type: array + clusters: + description: The list of clusters featured in the stage, and + their individual rollout status. + items: + properties: + clusterName: + description: The name of the cluster. + type: string + conditions: + description: A list of observed conditions about the rollout + progress of the cluster. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + placementBindingName: + description: The name of the placement binding. + type: string + required: + - clusterName + - placementBindingName + type: object + type: array + completedTimestamp: + description: The timestamp when the rollout in the stage completed. + If unset, the rollout has not completed in the stage yet. + format: date-time + type: string + conditions: + description: A list of observed conditions about the rollout + progress in the stage. + items: + description: Condition contains details for one aspect of + the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + maxConcurrency: + description: The (resolved) maximum number of clusters that + can be updated concurrently within this stage. + format: int32 + type: integer + stageName: + description: The name of the stage. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + startedTimestamp: + description: The timestamp when the rollout in the stage started. + If unset, the rollout has not started in the stage yet. + format: date-time + type: string + required: + - stageName + type: object + type: array + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdatestrategies.yaml b/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdatestrategies.yaml new file mode 100644 index 000000000..4e894b82c --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_clusterstagedupdatestrategies.yaml @@ -0,0 +1,239 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: clusterstagedupdatestrategies.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: ClusterStagedUpdateStrategy + listKind: ClusterStagedUpdateStrategyList + plural: clusterstagedupdatestrategies + shortNames: + - clusterupdatestrategy + singular: clusterstagedupdatestrategy + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + ClusterStagedUpdateStrategy is the KubeFleet API that defines how to perform a staged rollout of resource changes + for a cluster placement policy; the API dictates how clusters are grouped into stages, how the rollout is performed + within each stage, and which tasks to execute before and after each stage. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the ClusterStagedUpdateStrategy API + object. + properties: + stages: + description: The stages that KubeFleet rolls out resource changes + to, in the order they are listed. + items: + properties: + afterStageTasks: + description: |- + A list of tasks to execute after the stage rollout is completed. The tasks run in parallel; a staged update will only + proceed to the next stage after all tasks are completed. + items: + properties: + type: + description: The type of the stage task. Currently, the + supported task types are TimedWait and Approval. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait in a TimedWait task. + + Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + The maximum is 720h (30 days). + maxLength: 32 + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + x-kubernetes-validations: + - message: waitTime must be at most 720h (30 days) + rule: duration(self) <= duration('720h') + required: + - type + type: object + maxItems: 2 + type: array + x-kubernetes-validations: + - message: afterStageTasks cannot have more than one Approval + task + rule: self.filter(e, e.type == 'Approval').size() <= 1 + - message: afterStageTasks cannot have more than one TimedWait + task + rule: self.filter(e, e.type == 'TimedWait').size() <= 1 + - message: waitTime does not apply to an Approval task in afterStageTasks + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: waitTime is required for a TimedWait task in afterStageTasks + rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' + beforeStageTasks: + description: |- + A list of tasks to execute before the stage rollout starts. The tasks run in parallel; the current stage will only start + after all tasks are completed. + items: + properties: + type: + description: The type of the stage task. Currently, the + supported task types are TimedWait and Approval. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait in a TimedWait task. + + Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + The maximum is 720h (30 days). + maxLength: 32 + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + x-kubernetes-validations: + - message: waitTime must be at most 720h (30 days) + rule: duration(self) <= duration('720h') + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: waitTime does not apply to an Approval task in beforeStageTasks + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: beforeStageTasks cannot include a TimedWait task + rule: '!self.exists(e, e.type == ''TimedWait'')' + labelSelector: + description: |- + The label selector that selects clusters to be included in the stage. + + Note that if a cluster is selected by multiple stages, it will only be included in the first stage that selects it. + + If set to nil, the stage includes no clusters at all. If set to an empty selector, the stage includes all applicable clusters. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + maxConcurrency: + anyOf: + - type: integer + - type: string + default: 1 + description: |- + The maximum number of clusters that can be rolled out concurrently within this stage. + + This field accepts either an integer value (e.g., 5), or a percentage value (e.g., "50%"). For percentage values, + the concurrency number is calculated based on the total number of clusters in the stage, with fractional results rounded down. + + A minimum concurrency of 1 is enforced. Note that KubeFleet might further cap the concurrency as needed. + + Integer values must be between 1 and 1000. + + Defaults to 1. + pattern: ^(100|[1-9][0-9]?)%$ + x-kubernetes-int-or-string: true + x-kubernetes-validations: + - message: maxConcurrency must be at least 1 + rule: self == null || type(self) != int || self >= 1 + - message: maxConcurrency must be at most 1000 + rule: self == null || type(self) != int || self <= 1000 + name: + description: |- + The name of the stage. It must be unique within the staged update strategy and may only consist of + lowercase alphanumeric characters. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + sortingLabelKey: + description: |- + The label key used to sort clusters in the stage. KubeFleet rolls out resource changes to clusters in the stage + based on the sorted order. + + Specifically, KubeFleet will read the value of the label key for each cluster in the stage, and sort clusters in ascending + order based on the label value, interpreted as an integer. If the label is missing on any cluster, an error will be raised. + + If this field is not set, KubeFleet will sort clusters in the stage based on the ascending lexical order of their names. + type: string + required: + - name + type: object + maxItems: 31 + minItems: 1 + type: array + x-kubernetes-validations: + - message: stage names must be unique + rule: self.all(s, self.exists_one(t, t.name == s.name)) + required: + - stages + type: object + required: + - spec + type: object + served: true + storage: true diff --git a/config/crd/bases/rollout.kubefleet.dev_stagedupdateruns.yaml b/config/crd/bases/rollout.kubefleet.dev_stagedupdateruns.yaml new file mode 100644 index 000000000..4a57af81e --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_stagedupdateruns.yaml @@ -0,0 +1,560 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: stagedupdateruns.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: StagedUpdateRun + listKind: StagedUpdateRunList + plural: stagedupdateruns + shortNames: + - updaterun + singular: stagedupdaterun + scope: Namespaced + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + StagedUpdateRun is the KubeFleet API that enables users to roll out resource changes for a placement policy + in a staged manner, where clusters are grouped into stages and KubeFleet applies resource changes to each stage + sequentially. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the StagedUpdateRun API object. + properties: + failurePolicy: + description: |- + The failure policy of the staged update run. It helps KubeFleet determine when to stop the staged update run + when there are too many failures, so as to keep the impact radius under control. + properties: + maxAllowedClusterFailures: + default: 1 + description: |- + The maximum number of clusters that can fail the rollout before KubeFleet stops the staged + update run. The count is tracked across all stages and clusters. A cluster is considered to have + failed the rollout if the new (updated) placement does not become available within the maximum + wait time. + + The default value is 1; the value must be between 0 and 1000. + format: int32 + maximum: 1000 + minimum: 0 + type: integer + maxWaitTimePerClusterMinutes: + default: 5 + description: |- + The maximum time to wait for a rollout to a cluster to complete before KubeFleet considers the rollout to have + failed for the cluster. + + The default value is 5 minutes; the value must be between 1 and 10080 (7 days). + format: int32 + maximum: 10080 + minimum: 1 + type: integer + type: object + x-kubernetes-validations: + - message: failurePolicy is immutable + rule: self == oldSelf + placementPolicyName: + description: |- + The name of the placement policy that the StagedUpdateRun API object is associated with, i.e., + the name of the placement policy where resource changes are rolled out. + type: string + x-kubernetes-validations: + - message: placementPolicyName is immutable + rule: self == oldSelf + resourceSnapshotName: + description: |- + The name of the resource snapshot that the StagedUpdateRun API object is associated with, i.e., + the name of the resource snapshot where resource changes are introduced. + + If left empty, KubeFleet will request a new resource snapshot to be created based on the current state + of selected resources in the placement policy (if applicable). + maxLength: 253 + type: string + x-kubernetes-validations: + - message: resourceSnapshotName is immutable + rule: self == oldSelf + stagedUpdateStrategyName: + description: |- + The name of the staged update strategy that the StagedUpdateRun API object is associated with, i.e., + the name of the staged update strategy that organizes clusters into stages and defines how the rollout is executed. + type: string + x-kubernetes-validations: + - message: stagedUpdateStrategyName is immutable + rule: self == oldSelf + suspended: + default: false + description: |- + Whether the staged update run is suspended. If set to true, KubeFleet will pause the rollout. + + If the staged update run is created with this field set to true, KubeFleet will initialize the staged update run, + i.e., decide on the resource snapshot to roll out and determine the stages and the clusters in each stage, + but will not start rolling out resource changes until the field is set back to false. + + When a staged update run is suspended after the rollout starts, KubeFleet will pause the rollout as soon as + possible. If a cluster has started updating but has not yet completed, KubeFleet will not consider + its update a failure even if the maximum wait time for the cluster has been exceeded during the suspension. + The wait timer will be reset once the suspension is lifted. However, the time spent in suspension will + still count towards any timed wait tasks you have configured; those tasks will proceed or even complete, + accounting for the time spent in suspension, once the suspension is lifted. + type: boolean + required: + - placementPolicyName + - stagedUpdateStrategyName + type: object + x-kubernetes-validations: + - message: resourceSnapshotName cannot be added or removed after creation + rule: has(self.resourceSnapshotName) == has(oldSelf.resourceSnapshotName) + - message: failurePolicy cannot be added or removed after creation + rule: has(self.failurePolicy) == has(oldSelf.failurePolicy) + status: + description: The observed status of the StagedUpdateRun API object. + properties: + conditions: + description: A list of observed conditions of the staged update run. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + resourceSnapshotNameToRollout: + description: |- + The name of the resource snapshot that is being rolled out in the staged update run. + + If the staged update run is created with the resourceSnapshotName field in the spec set to a specific value, + this field will be set to the same value; otherwise, KubeFleet will request a new resource snapshot to be created (if applicable) + and populate this field with the name of the newly created resource snapshot. + type: string + stages: + description: The status of each stage in the staged update run, including + the clusters in each stage and their rollout status. + items: + properties: + afterStageTasks: + description: The list of tasks to be executed after completing + the rollout in the stage, and their execution status. + items: + properties: + approvalRequestName: + description: |- + The name of the approval request that is created for the stage task. + + This field is only set when the task type is Approval. + type: string + conditions: + description: A list of observed conditions about the progress + of the stage task. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: + description: The type of the stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: The time to wait in a TimedWait task. + type: string + required: + - type + type: object + maxItems: 2 + type: array + beforeStageTasks: + description: The list of tasks to be executed before starting + the rollout in the stage, and their execution status. + items: + properties: + approvalRequestName: + description: |- + The name of the approval request that is created for the stage task. + + This field is only set when the task type is Approval. + type: string + conditions: + description: A list of observed conditions about the progress + of the stage task. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + type: + description: The type of the stage task. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: The time to wait in a TimedWait task. + type: string + required: + - type + type: object + maxItems: 1 + type: array + clusters: + description: The list of clusters featured in the stage, and + their individual rollout status. + items: + properties: + clusterName: + description: The name of the cluster. + type: string + conditions: + description: A list of observed conditions about the rollout + progress of the cluster. + items: + description: Condition contains details for one aspect + of the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, + False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in + foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + placementBindingName: + description: The name of the placement binding. + type: string + required: + - clusterName + - placementBindingName + type: object + type: array + completedTimestamp: + description: The timestamp when the rollout in the stage completed. + If unset, the rollout has not completed in the stage yet. + format: date-time + type: string + conditions: + description: A list of observed conditions about the rollout + progress in the stage. + items: + description: Condition contains details for one aspect of + the current state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, + Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + x-kubernetes-list-map-keys: + - type + x-kubernetes-list-type: map + maxConcurrency: + description: The (resolved) maximum number of clusters that + can be updated concurrently within this stage. + format: int32 + type: integer + stageName: + description: The name of the stage. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + startedTimestamp: + description: The timestamp when the rollout in the stage started. + If unset, the rollout has not started in the stage yet. + format: date-time + type: string + required: + - stageName + type: object + type: array + type: object + required: + - spec + type: object + served: true + storage: true + subresources: + status: {} diff --git a/config/crd/bases/rollout.kubefleet.dev_stagedupdatestrategies.yaml b/config/crd/bases/rollout.kubefleet.dev_stagedupdatestrategies.yaml new file mode 100644 index 000000000..29b0daf51 --- /dev/null +++ b/config/crd/bases/rollout.kubefleet.dev_stagedupdatestrategies.yaml @@ -0,0 +1,238 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.20.0 + name: stagedupdatestrategies.rollout.kubefleet.dev +spec: + group: rollout.kubefleet.dev + names: + categories: + - kubefleet + - kubefleet-rollout + kind: StagedUpdateStrategy + listKind: StagedUpdateStrategyList + plural: stagedupdatestrategies + shortNames: + - updatestrategy + singular: stagedupdatestrategy + scope: Namespaced + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + StagedUpdateStrategy is the KubeFleet API that defines how to perform a staged rollout of resource changes + for a placement policy; the API dictates how clusters are grouped into stages, how the rollout is performed + within each stage, and which tasks to execute before and after each stage. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: The specification of the StagedUpdateStrategy API object. + properties: + stages: + description: The stages that KubeFleet rolls out resource changes + to, in the order they are listed. + items: + properties: + afterStageTasks: + description: |- + A list of tasks to execute after the stage rollout is completed. The tasks run in parallel; a staged update will only + proceed to the next stage after all tasks are completed. + items: + properties: + type: + description: The type of the stage task. Currently, the + supported task types are TimedWait and Approval. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait in a TimedWait task. + + Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + The maximum is 720h (30 days). + maxLength: 32 + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + x-kubernetes-validations: + - message: waitTime must be at most 720h (30 days) + rule: duration(self) <= duration('720h') + required: + - type + type: object + maxItems: 2 + type: array + x-kubernetes-validations: + - message: afterStageTasks cannot have more than one Approval + task + rule: self.filter(e, e.type == 'Approval').size() <= 1 + - message: afterStageTasks cannot have more than one TimedWait + task + rule: self.filter(e, e.type == 'TimedWait').size() <= 1 + - message: waitTime does not apply to an Approval task in afterStageTasks + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: waitTime is required for a TimedWait task in afterStageTasks + rule: '!self.exists(e, e.type == ''TimedWait'' && !has(e.waitTime))' + beforeStageTasks: + description: |- + A list of tasks to execute before the stage rollout starts. The tasks run in parallel; the current stage will only start + after all tasks are completed. + items: + properties: + type: + description: The type of the stage task. Currently, the + supported task types are TimedWait and Approval. + enum: + - TimedWait + - Approval + type: string + waitTime: + description: |- + The time to wait in a TimedWait task. + + Specify the time period using hours (h), minutes (m), and seconds (s) units, e.g., "1h30m" for 1 hour and 30 minutes. + The maximum is 720h (30 days). + maxLength: 32 + pattern: ^(?:(?:0|[1-9][0-9]*)(\.[0-9]+)?(?:s|m|h))+$ + type: string + x-kubernetes-validations: + - message: waitTime must be at most 720h (30 days) + rule: duration(self) <= duration('720h') + required: + - type + type: object + maxItems: 1 + type: array + x-kubernetes-validations: + - message: waitTime does not apply to an Approval task in beforeStageTasks + rule: '!self.exists(e, e.type == ''Approval'' && has(e.waitTime))' + - message: beforeStageTasks cannot include a TimedWait task + rule: '!self.exists(e, e.type == ''TimedWait'')' + labelSelector: + description: |- + The label selector that selects clusters to be included in the stage. + + Note that if a cluster is selected by multiple stages, it will only be included in the first stage that selects it. + + If set to nil, the stage includes no clusters at all. If set to an empty selector, the stage includes all applicable clusters. + properties: + matchExpressions: + description: matchExpressions is a list of label selector + requirements. The requirements are ANDed. + items: + description: |- + A label selector requirement is a selector that contains values, a key, and an operator that + relates the key and values. + properties: + key: + description: key is the label key that the selector + applies to. + type: string + operator: + description: |- + operator represents a key's relationship to a set of values. + Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: |- + values is an array of string values. If the operator is In or NotIn, + the values array must be non-empty. If the operator is Exists or DoesNotExist, + the values array must be empty. This array is replaced during a strategic + merge patch. + items: + type: string + type: array + x-kubernetes-list-type: atomic + required: + - key + - operator + type: object + type: array + x-kubernetes-list-type: atomic + matchLabels: + additionalProperties: + type: string + description: |- + matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels + map is equivalent to an element of matchExpressions, whose key field is "key", the + operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + x-kubernetes-map-type: atomic + maxConcurrency: + anyOf: + - type: integer + - type: string + default: 1 + description: |- + The maximum number of clusters that can be rolled out concurrently within this stage. + + This field accepts either an integer value (e.g., 5), or a percentage value (e.g., "50%"). For percentage values, + the concurrency number is calculated based on the total number of clusters in the stage, with fractional results rounded down. + + A minimum concurrency of 1 is enforced. Note that KubeFleet might further cap the concurrency as needed. + + Integer values must be between 1 and 1000. + + Defaults to 1. + pattern: ^(100|[1-9][0-9]?)%$ + x-kubernetes-int-or-string: true + x-kubernetes-validations: + - message: maxConcurrency must be at least 1 + rule: self == null || type(self) != int || self >= 1 + - message: maxConcurrency must be at most 1000 + rule: self == null || type(self) != int || self <= 1000 + name: + description: |- + The name of the stage. It must be unique within the staged update strategy and may only consist of + lowercase alphanumeric characters. + maxLength: 63 + pattern: ^[a-z0-9]+$ + type: string + sortingLabelKey: + description: |- + The label key used to sort clusters in the stage. KubeFleet rolls out resource changes to clusters in the stage + based on the sorted order. + + Specifically, KubeFleet will read the value of the label key for each cluster in the stage, and sort clusters in ascending + order based on the label value, interpreted as an integer. If the label is missing on any cluster, an error will be raised. + + If this field is not set, KubeFleet will sort clusters in the stage based on the ascending lexical order of their names. + type: string + required: + - name + type: object + maxItems: 31 + minItems: 1 + type: array + x-kubernetes-validations: + - message: stage names must be unique + rule: self.all(s, self.exists_one(t, t.name == s.name)) + required: + - stages + type: object + required: + - spec + type: object + served: true + storage: true diff --git a/docker/hub-agent.Dockerfile b/docker/hub-agent.Dockerfile index 2e42a1de5..17d0f2e19 100644 --- a/docker/hub-agent.Dockerfile +++ b/docker/hub-agent.Dockerfile @@ -8,7 +8,7 @@ # Segmentation fault" / "cgo: gcc produced no output" in runtime/cgo and net) # and, when it did not crash, took over an hour per image. Only the final # distroless stage is per-target, and it runs no commands. -FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.8-1 AS builder +FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.10-1 AS builder WORKDIR /workspace # Copy the Go Modules manifests diff --git a/docker/member-agent.Dockerfile b/docker/member-agent.Dockerfile index f095a9adb..c3f1581ea 100644 --- a/docker/member-agent.Dockerfile +++ b/docker/member-agent.Dockerfile @@ -8,7 +8,7 @@ # Segmentation fault" / "cgo: gcc produced no output" in runtime/cgo and net) # and, when it did not crash, took over an hour per image. Only the final # distroless stage is per-target, and it runs no commands. -FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.8-1 AS builder +FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.10-1 AS builder WORKDIR /workspace # Copy the Go Modules manifests diff --git a/docker/refresh-token.Dockerfile b/docker/refresh-token.Dockerfile index 9ee6e5681..d40bb0afa 100644 --- a/docker/refresh-token.Dockerfile +++ b/docker/refresh-token.Dockerfile @@ -8,7 +8,7 @@ # Segmentation fault" / "cgo: gcc produced no output" in runtime/cgo and net) # and, when it did not crash, took over an hour per image. Only the final # distroless stage is per-target, and it runs no commands. -FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.8-1 AS builder +FROM --platform=$BUILDPLATFORM mcr.microsoft.com/oss/go/microsoft/golang:1.26.10-1 AS builder WORKDIR /workspace # Copy the Go Modules manifests diff --git a/go.mod b/go.mod index e427eaf49..b4c91309a 100644 --- a/go.mod +++ b/go.mod @@ -6,14 +6,14 @@ require ( github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.1 github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.1 github.com/Azure/karpenter-provider-azure v1.14.2 - github.com/crossplane/crossplane-runtime/v2 v2.4.0 + github.com/crossplane/crossplane-runtime/v2 v2.4.2 github.com/evanphx/json-patch/v5 v5.9.11 github.com/go-logr/logr v1.4.4 github.com/gofrs/uuid v4.4.0+incompatible github.com/google/go-cmp v0.7.0 github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 - github.com/onsi/ginkgo/v2 v2.32.2 - github.com/onsi/gomega v1.43.0 + github.com/onsi/ginkgo/v2 v2.33.0 + github.com/onsi/gomega v1.44.0 github.com/prometheus/client_golang v1.24.1 github.com/prometheus/client_model v0.6.3 github.com/qri-io/jsonpointer v0.1.1 @@ -22,7 +22,7 @@ require ( github.com/stretchr/testify v1.12.1 github.com/wI2L/jsondiff v0.7.1 go.goms.io/fleet-networking v0.3.44 - go.uber.org/atomic v1.11.0 + go.uber.org/atomic v1.12.0 go.uber.org/zap v1.28.0 golang.org/x/sync v0.23.0 golang.org/x/time v0.16.0 @@ -128,17 +128,17 @@ require ( go.uber.org/multierr v1.11.0 // indirect go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.5 // indirect - golang.org/x/crypto v0.56.0 // indirect + golang.org/x/crypto v0.57.0 // indirect golang.org/x/exp v0.0.0-20260112195511-716be5621a96 // indirect - golang.org/x/mod v0.40.0 // indirect - golang.org/x/net v0.58.0 // indirect + golang.org/x/mod v0.41.0 // indirect + golang.org/x/net v0.60.0 // indirect golang.org/x/oauth2 v0.36.0 // indirect - golang.org/x/sys v0.47.0 // indirect - golang.org/x/term v0.45.0 // indirect - golang.org/x/text v0.41.0 // indirect + golang.org/x/sys v0.48.0 // indirect + golang.org/x/term v0.46.0 // indirect + golang.org/x/text v0.42.0 // indirect golang.org/x/tools v0.49.0 // indirect - google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d // indirect gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect k8s.io/cli-runtime v0.35.8 // indirect diff --git a/go.sum b/go.sum index 3f8d6b4ac..dfc80a940 100644 --- a/go.sum +++ b/go.sum @@ -110,8 +110,8 @@ github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2y github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= -github.com/crossplane/crossplane-runtime/v2 v2.4.0 h1:m5iGVHbtEh9nZYz7w/wUWDsXVlpG2c8v5i0SZScwHh8= -github.com/crossplane/crossplane-runtime/v2 v2.4.0/go.mod h1:Pf+hg5A/46QaGjKi7Pk+HdsrFOA/THvST9qM4El8Rzs= +github.com/crossplane/crossplane-runtime/v2 v2.4.2 h1:MXc37IyL7iyrQIxs0qrAtdOsy0mdQkxOMie4UVWSFQ4= +github.com/crossplane/crossplane-runtime/v2 v2.4.2/go.mod h1:jDwOCwt+LMD6KXsnvvm1mNNiVqKZ2ZWZxuUnkxdkBPE= github.com/crossplane/crossplane/apis/v2 v2.3.4 h1:qB1YqF300msWEkDQTFUhqfeOq3HtVfSZ1+3wNz1JStA= github.com/crossplane/crossplane/apis/v2 v2.3.4/go.mod h1:s0cfMZA+Y/dwQ577PLQn5SBWyUg9k6vKjGAkHEwBShg= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= @@ -277,10 +277,10 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ= github.com/oklog/ulid/v2 v2.1.1 h1:suPZ4ARWLOJLegGFiZZ1dFAkqzhMjL3J1TzI+5wHz8s= github.com/oklog/ulid/v2 v2.1.1/go.mod h1:rcEKHmBBKfef9DhnvX7y1HZBYxjXb0cP5ExxNsTT1QQ= -github.com/onsi/ginkgo/v2 v2.32.2 h1:2o6vyFvR6snrJWgRVztC+OwuqqPEMI1UzYl2s2iU7Cg= -github.com/onsi/ginkgo/v2 v2.32.2/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44= -github.com/onsi/gomega v1.43.0 h1:VlG/1FxqNxhSO+lq/OHBNaaqwiBK/mO8JbVkX9Y+FeU= -github.com/onsi/gomega v1.43.0/go.mod h1:REff/hsDsodHoKlWsP2mAPhu1+5/6hVYNf9rIEBpeSg= +github.com/onsi/ginkgo/v2 v2.33.0 h1:C8gBA6Uc2ZEubiV+SXiu5tZnMTwEmXHgkJwGozKtZf8= +github.com/onsi/ginkgo/v2 v2.33.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44= +github.com/onsi/gomega v1.44.0 h1:eAiGl3Pw5jz5GQdDff0BcxYpAX1JxW8xD7mFUuwNfZQ= +github.com/onsi/gomega v1.44.0/go.mod h1:e/C2HwaZ1DhvjzXXuFhcR7hY7Sh9pl7MmoWKEjzwcdA= github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc= github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ= github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ= @@ -361,8 +361,8 @@ go.opentelemetry.io/otel/sdk/metric v1.46.0 h1:0piZ26EG4RBfebb2jhDH6ERCYHoVWduc3 go.opentelemetry.io/otel/sdk/metric v1.46.0/go.mod h1:I1PbKrdVc8Qu8HYVDNtqVIwLwjNrhsV/uFuxfwg8mO4= go.opentelemetry.io/otel/trace v1.46.0 h1:OULy7ccdJnZtJ0UDYFOIGaCmiWzJ8Vi2G/Rsu60qs1c= go.opentelemetry.io/otel/trace v1.46.0/go.mod h1:J7GAXweO77XSFkB/rmAqk9D6ihszhFjLU+d9WuUxDLI= -go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE= -go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0= +go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE= +go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/mock v0.6.0 h1:hyF9dfmbgIX5EfOdasqLsWD6xqpNZlXblLB/Dbnwv3Y= @@ -380,22 +380,22 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4= -golang.org/x/crypto v0.56.0 h1:GUh5Ii4J5jtcseSMiRqr1jXCNHoxjeV9Fmekc2oLy6Y= -golang.org/x/crypto v0.56.0/go.mod h1:OMW5y6CY9l38uPLmxU6l6pwcXp1obtLo3e6gT7gQR2I= +golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M= +golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA= golang.org/x/exp v0.0.0-20260112195511-716be5621a96 h1:Z/6YuSHTLOHfNFdb8zVZomZr7cqNgTJvA8+Qz75D8gU= golang.org/x/exp v0.0.0-20260112195511-716be5621a96/go.mod h1:nzimsREAkjBCIEFtHiYkrJyT+2uy9YZJB7H1k68CXZU= golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/mod v0.40.0 h1:hUv+3cXcdRHz08UmSiOob7sadHig73uo5bkXxQ/tvUs= -golang.org/x/mod v0.40.0/go.mod h1:0/weTWkPWGBikyTWAX3dkjVztMmBA5hM0DH6BElSupE= +golang.org/x/mod v0.41.0 h1:qJmnOUb4YB+FsEuM3HcWucdZASCPGhsX6uljO6pog0c= +golang.org/x/mod v0.41.0/go.mod h1:Ek9pY8RKWXwsWvd3rQiHYtMqkjSUV+s1Rj7j4H5Ur6o= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= -golang.org/x/net v0.58.0 h1:ynWG7rqYi4ccpTEuPZ2QGWHktVEM9DMCj9yzDE0Q7To= -golang.org/x/net v0.58.0/go.mod h1:YwCddHnFlT7eLQqVprV19OnhLGtc5xOKgE0RyqgfWAU= +golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU= +golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg= golang.org/x/oauth2 v0.36.0 h1:peZ/1z27fi9hUOFCAZaHyrpWG5lwe0RJEEEeH0ThlIs= golang.org/x/oauth2 v0.36.0/go.mod h1:YDBUJMTkDnJS+A4BP4eZBjCqtokkg1hODuPjwiGPO7Q= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -413,15 +413,15 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs= -golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo= +golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo= golang.org/x/term v0.15.0/go.mod h1:BDl952bC7+uMoWR75FIrCDx79TPU9oHkTZ9yRbYOrX0= -golang.org/x/term v0.45.0 h1:NwWyBmoJCbfTHpxrWoZ9C6/VxOf7ic219I8xZZFdrf0= -golang.org/x/term v0.45.0/go.mod h1:9aqxs0blBcrm/n0L9QW0aRVD+ktan8ssZromtqJC43w= +golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE= +golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= @@ -429,8 +429,8 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= -golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8= -golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M= +golang.org/x/text v0.42.0 h1:JbOZXgfeCPU9gacVtYliJqOhD+zhrEqK4LfdpmlUZqI= +golang.org/x/text v0.42.0/go.mod h1:ojzP1Z+2QtioaF8DTtO8K5q7JWVVYwZKenzujK0Zd0E= golang.org/x/time v0.16.0 h1:vMb6ptszcQMkcwiRTAuNNU50gom6++Q/6gY2hDM6VDE= golang.org/x/time v0.16.0/go.mod h1:rVKOqvZeKvrDKTQiAHJ7wmwP0RzleSphoEA9RcdLA0s= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= @@ -444,10 +444,10 @@ gomodules.xyz/jsonpatch/v2 v2.5.0 h1:JELs8RLM12qJGXU4u/TO3V25KW8GreMKl9pdkk14RM0 gomodules.xyz/jsonpatch/v2 v2.5.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY= gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= -google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa h1:Kjn0N0tCrDgiAFW+lGO4JZ3ck44CehvJQMAwj9QF0G8= -google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= -google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d h1:FarXi840EJWSHYTN3ERkADbPWjl307+FGrA22KAVjjc= +google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d/go.mod h1:K/+WGbmBY7aNW1HDw1fJnKYo10i0DkAX6pows00dLig= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d h1:IL4hdHzcUv2l/gcg98/Rj3FbtE6axwqslOW8SW0C+S0= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU= google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8= google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc= diff --git a/hack/perftest/1000placements/README.md b/hack/perftest/1000placements/README.md index ffdfb79d8..045e18449 100644 --- a/hack/perftest/1000placements/README.md +++ b/hack/perftest/1000placements/README.md @@ -13,6 +13,10 @@ Placements created by this utility program will be each assigned an index X, ran and a placement of index X will select all member clusters that are labelled with `placement-group=X%10`. * The program requires the following tools (aside from the Go runtime) to be installed: * `curl` (for retrieving pprof data) +* Install the hub agent chart with `--set enablePprof=true` (the `fleetbuilder` scripts do this). The pprof + server is off by default and listens on loopback only, so forward it first, for example + `kubectl port-forward -n fleet-system deploy/hub-agent 6065`, and set + `PPROF_ENDPOINT=http://localhost:6065/debug/pprof/profile`. > If you have followed the instructions in `../../README.md` and have used the given scripts and utility programs > to run the performance/scalability test, all the steps above should have been done for you already. diff --git a/hack/perftest/fleetbuilder/deploy-setup-hub-cluster.sh b/hack/perftest/fleetbuilder/deploy-setup-hub-cluster.sh index c9fd42966..f1fd5ab92 100644 --- a/hack/perftest/fleetbuilder/deploy-setup-hub-cluster.sh +++ b/hack/perftest/fleetbuilder/deploy-setup-hub-cluster.sh @@ -64,6 +64,7 @@ helm upgrade hub-agent charts/hub-agent/ \ --set namespace=fleet-system \ --set logVerbosity=2 \ --set logFileMaxSize=100000 \ + --set enablePprof=true \ --set enableWebhook=true \ --set enableGuardRail=true \ --set enableWorkload=false \ diff --git a/hack/quickstart/join-member-clusters.ps1 b/hack/quickstart/join-member-clusters.ps1 index 66ea7c319..bc6825441 100644 --- a/hack/quickstart/join-member-clusters.ps1 +++ b/hack/quickstart/join-member-clusters.ps1 @@ -8,7 +8,7 @@ # You can assume port 6443 unless you have explicitly changed the API server port for your hub cluster. Don't use the mapped Docker port (i.e. 50063) # # Example usage: -# ./join-member-clusters.ps1 0.2.2 demo-hub-01 https://172.18.0.2:6443 member-cluster-1 member-cluster-2 +# ./join-member-clusters.ps1 0.3.2 demo-hub-01 https://172.18.0.2:6443 member-cluster-1 member-cluster-2 [CmdletBinding()] param( @@ -33,7 +33,7 @@ Usage: ./join-member-clusters.ps1 [ ...] Example: - ./join-member-clusters.ps1 0.2.2 demo-hub-01 https://172.18.0.2:6443 member-cluster-1 member-cluster-2 + ./join-member-clusters.ps1 0.3.2 demo-hub-01 https://172.18.0.2:6443 member-cluster-1 member-cluster-2 Requirements: - PowerShell 7, kubectl, and helm must be installed diff --git a/hack/quickstart/join-member-clusters.sh b/hack/quickstart/join-member-clusters.sh index 27ddb25fc..67262fe98 100755 --- a/hack/quickstart/join-member-clusters.sh +++ b/hack/quickstart/join-member-clusters.sh @@ -8,7 +8,7 @@ # # You can assume port 6443 unless you have explicitly changed the API server port for your hub cluster. Don't use the mapped Docker port (i.e. 50063) # -# Example usage: ./join-member-clusters.sh 0.2.2 demo-hub-01 https://172.18.0.2:6443 member-cluster-1 [ ...] +# Example usage: ./join-member-clusters.sh 0.3.2 demo-hub-01 https://172.18.0.2:6443 member-cluster-1 [ ...] usage() { cat <<'EOF' @@ -16,7 +16,7 @@ Usage: ./join-member-clusters.sh [ ...] Example: - ./join-member-clusters.sh 0.2.2 demo-hub-01 https://172.18.0.2:6443 member-cluster-1 member-cluster-2 + ./join-member-clusters.sh 0.3.2 demo-hub-01 https://172.18.0.2:6443 member-cluster-1 member-cluster-2 Requirements: - kubectl and helm must be installed diff --git a/pkg/admissionpolicymanager/annotatedforplacementobjects.go b/pkg/admissionpolicymanager/annotatedforplacementobjects.go new file mode 100644 index 000000000..a21b2e5c8 --- /dev/null +++ b/pkg/admissionpolicymanager/annotatedforplacementobjects.go @@ -0,0 +1,286 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package admissionpolicymanager + +import ( + "fmt" + "strings" + + admissionregistrationv1 "k8s.io/api/admissionregistration/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +const ( + AnnotatedForPlacementObjectsVAPGeneratorName = "DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess" +) + +const ( + annotatedForPlacementObjsVAPPolicyName = "fleet-builtin-deny-placement-annotation-if-no-placement-policy" + annotatedForPlacementObjsVAPPolicyBindingName = "fleet-builtin-deny-placement-annotation-if-no-placement-policy-binding" +) + +const ( + annotatedForPlacementObjsHasAnnotationMatchCondName = "has-placement-annotation" + annotatedForPlacementObjsAnnotationAddedOrChangedMatchCondName = "placement-annotation-added-or-changed" + + placementPoliciesResourceName = "placementpolicies" + clusterPlacementPoliciesResourceName = "clusterplacementpolicies" + + endpointsKind = "Endpoints" + + authzVerbCreate = "create" + authzVerbUpdate = "update" +) + +// Verify that AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator implements +// the ValidatingAdmissionPolicyGenerator interface. +var _ ValidatingAdmissionPolicyGenerator = &AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator{} + +// AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator generates a ValidatingAdmissionPolicy +// and its binding that denies the addition of the KubeFleet placement annotation onto objects if the caller +// does not have the permission to create/update a (cluster) placement policy in the cluster/namespace. In +// reserved namespaces, and on Endpoints objects, the annotation cannot be added at all, regardless of the caller. +// +// Important: this policy should only be used in cases where the workload controllers of kube-controller-manager +// (e.g., the Deployment, StatefulSet, DaemonSet, and CronJob controllers) have been disabled in the hub cluster; +// setting the hub agent option --enable-workload to false does not disable them. These controllers copy +// annotations from the objects they manage (or from the templates in them) onto the objects they create (e.g., +// the Deployment controller onto ReplicaSets, and the StatefulSet and DaemonSet controllers onto +// ControllerRevisions); these writes are made by the controllers themselves, which have no access to the +// placement policy APIs, so the policy denies them and blocks the controllers. +// +// TO-DO (chenyu1): add a mutating admission policy (MAP) to complement this policy, so that all clusters, +// including those with workload controllers enabled, can have proper protection to use the annotation-based +// placement. The planned setup: this policy allows the annotation on objects with a controller owner reference +// (which the annotation-based placement never places), and the MAP removes the annotation from an object once +// its controller owner reference is removed (e.g., when the object is orphaned). MAPs are enabled by default +// in Kubernetes 1.36 and later. +type AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator struct { + // ReservedNamespacePrefixes are the prefixes of the reserved namespaces. The annotation can never be + // added to (or changed on) objects in these namespaces, or the namespaces themselves. The field is + // optional; if no prefixes are specified, no namespaces are reserved. + ReservedNamespacePrefixes []string +} + +// Name returns the name of the validating admission policy generator, which is used to determine if +// a specific generator has been enabled or not. +func (g *AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator) Name() string { + return AnnotatedForPlacementObjectsVAPGeneratorName +} + +// Validate validates the configuration of the validating admission policy generator. +func (g *AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator) Validate() error { + // Check if any of the prefixes includes illegal characters. + for _, prefix := range g.ReservedNamespacePrefixes { + if !reservedNamespacePrefixRegexp.MatchString(prefix) { + return errors.NewUserError(nil, "prefix contains illegal characters; only lowercase alphanumeric characters and hyphens are allowed", "prefix", prefix) + } + } + return nil +} + +// PoliciesWithBindings generates a ValidatingAdmissionPolicy and its policy binding that denies the addition +// of the KubeFleet placement annotation (kubefleet.dev/cluster-selectors) onto objects if the caller does not +// have the permission to create/update a (cluster) placement policy in the cluster/namespace, or if the objects +// are in reserved namespaces (or are reserved namespaces themselves), or are Endpoints objects. +// +// The policy covers the creation of objects with the annotation, and the updates of objects (including those +// made through subresources) that add the annotation or change its value; updates that leave the annotation +// as it is (e.g., system controllers removing finalizers or writing status) are not subject to the check, +// so that they will not be blocked. +// +// For simplicity reasons, the code here assumes that the generator has been validated before +// PoliciesWithBindings() is called. +func (g *AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator) PoliciesWithBindings() ([]PolicyWithBindings, error) { + annotationKey := kfplacementv1alpha1.ClusterSelectorsAnnotation + placementAPIGroup := kfplacementv1alpha1.GroupVersion.Group + + // Match conditions are evaluated before the validations; the policy is skipped if any of + // them evaluates to false. They help filter out (cheaply) requests that this policy is not + // concerned with, as the policy matches all resources and their subresources. + hasAnnotationExpr, err := objectHasAnnotation(annotationKey).Build() + if err != nil { + return nil, errors.Wraps(err, "failed to build CEL expression", "matchCondition", annotatedForPlacementObjsHasAnnotationMatchCondName) + } + + // For CREATE requests, oldObject is always null and the condition is always true. + // + // The check on the object keeps the condition from failing with a no-such-key error when the annotation + // is being removed; such requests are filtered out by the has-annotation condition. + isAnnotationAddedOrChanged := LogicalNot( + LogicalAnd( + objectHasAnnotation(annotationKey), + oldObjectHasAnnotation(annotationKey), + annotationValueIsUnchanged(annotationKey), + ), + ) + isAnnotationAddedOrChangedExpr, err := isAnnotationAddedOrChanged.Build() + if err != nil { + return nil, errors.Wraps(err, "failed to build CEL expression", "matchCondition", annotatedForPlacementObjsAnnotationAddedOrChangedMatchCondName) + } + + // The API server reports the first failed validation; the checks that no permission can lift come first. + validations := make([]admissionregistrationv1.Validation, 0, 4) + + // The reserved namespace check is skipped if no prefixes are specified, as no namespaces are reserved. + if len(g.ReservedNamespacePrefixes) > 0 { + // For Namespace objects, the API server sets request.namespace to the name of the namespace, so that + // the reserved namespaces themselves are covered as well. + isInReservedNamespacePrefixes := LogicalOr() + for _, prefix := range g.ReservedNamespacePrefixes { + isInReservedNamespacePrefixes.Add(isInNamespaceWithPrefix(prefix)) + } + isNotInReservedNamespacesExpr, err := LogicalNot(LogicalAnd(requestHasNamespace(), isInReservedNamespacePrefixes)).Build() + if err != nil { + return nil, errors.Wraps(err, "failed to build CEL expression", "reservedNamespacePrefixes", g.ReservedNamespacePrefixes) + } + validations = append(validations, admissionregistrationv1.Validation{ + Expression: isNotInReservedNamespacesExpr, + Message: fmt.Sprintf( + "adding or changing the %s annotation on reserved namespaces or objects in them is disallowed (reserved namespace prefixes: %s)", + annotationKey, strings.Join(g.ReservedNamespacePrefixes, ", ")), + Reason: ptr.To(metav1.StatusReasonForbidden), + }) + } + + // The EndpointSlice mirroring controller copies the annotations of an Endpoints object onto the EndpointSlices + // mirrored from it, and keeps them in sync; as the controller has no access to the placement policy APIs, its + // writes would be denied by this policy. The annotation has no use on Endpoints objects anyway, as they cannot + // be placed with the placement policy APIs. + isNotEndpointsExpr, err := LogicalNot(requestIsForKind(corev1.GroupName, endpointsKind)).Build() + if err != nil { + return nil, errors.Wraps(err, "failed to build CEL expression", "kind", endpointsKind) + } + + // The annotation-based placement controller generates a PlacementPolicy object for namespaced objects, + // and a ClusterPlacementPolicy object for cluster-scoped objects. + isNamespaced := isNamespacedObject() + + canCreateAndUpdatePlacementPoliciesInNamespace := LogicalOr( + LogicalNot(isNamespaced), + LogicalAnd( + callerCanInObjectNamespace(placementAPIGroup, placementPoliciesResourceName, authzVerbCreate), + callerCanInObjectNamespace(placementAPIGroup, placementPoliciesResourceName, authzVerbUpdate), + ), + ) + canCreateAndUpdatePlacementPoliciesInNamespaceExpr, err := canCreateAndUpdatePlacementPoliciesInNamespace.Build() + if err != nil { + return nil, errors.Wraps(err, "failed to build CEL expression", "resource", placementPoliciesResourceName) + } + + canCreateAndUpdateClusterPlacementPolicies := LogicalOr( + isNamespaced, + LogicalAnd( + callerCanClusterWide(placementAPIGroup, clusterPlacementPoliciesResourceName, authzVerbCreate), + callerCanClusterWide(placementAPIGroup, clusterPlacementPoliciesResourceName, authzVerbUpdate), + ), + ) + canCreateAndUpdateClusterPlacementPoliciesExpr, err := canCreateAndUpdateClusterPlacementPolicies.Build() + if err != nil { + return nil, errors.Wraps(err, "failed to build CEL expression", "resource", clusterPlacementPoliciesResourceName) + } + + validations = append(validations, + admissionregistrationv1.Validation{ + Expression: isNotEndpointsExpr, + Message: fmt.Sprintf( + "adding or changing the %s annotation on Endpoints objects is disallowed; Endpoints cannot be placed, and the annotation would be copied onto the EndpointSlices mirrored from them", + annotationKey), + Reason: ptr.To(metav1.StatusReasonForbidden), + }, + admissionregistrationv1.Validation{ + Expression: canCreateAndUpdatePlacementPoliciesInNamespaceExpr, + Message: fmt.Sprintf( + "adding or changing the %s annotation on a namespaced object requires the permissions to create and update %s.%s in the namespace of the object", + annotationKey, placementPoliciesResourceName, placementAPIGroup), + Reason: ptr.To(metav1.StatusReasonForbidden), + }, + admissionregistrationv1.Validation{ + Expression: canCreateAndUpdateClusterPlacementPoliciesExpr, + Message: fmt.Sprintf( + "adding or changing the %s annotation on a cluster-scoped object requires the permissions to create and update %s.%s", + annotationKey, clusterPlacementPoliciesResourceName, placementAPIGroup), + Reason: ptr.To(metav1.StatusReasonForbidden), + }, + ) + + policy := &admissionregistrationv1.ValidatingAdmissionPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: annotatedForPlacementObjsVAPPolicyName, + }, + Spec: admissionregistrationv1.ValidatingAdmissionPolicySpec{ + FailurePolicy: ptr.To(admissionregistrationv1.Fail), + MatchConstraints: &admissionregistrationv1.MatchResources{ + ResourceRules: []admissionregistrationv1.NamedRuleWithOperations{ + { + RuleWithOperations: admissionregistrationv1.RuleWithOperations{ + Operations: []admissionregistrationv1.OperationType{ + admissionregistrationv1.Create, + admissionregistrationv1.Update, + }, + Rule: admissionregistrationv1.Rule{ + APIGroups: []string{"*"}, + APIVersions: []string{"*"}, + // Match all resources and their subresources, as some built-in resources + // (e.g., Deployments and Namespaces) accept annotation changes through + // subresources such as status and finalize. Subresources that carry other + // kinds of objects (e.g., scale and eviction) are filtered out by the + // match conditions unless the object itself carries the annotation. + Resources: []string{"*/*"}, + }, + }, + }, + }, + }, + MatchConditions: []admissionregistrationv1.MatchCondition{ + { + Name: annotatedForPlacementObjsHasAnnotationMatchCondName, + Expression: hasAnnotationExpr, + }, + { + Name: annotatedForPlacementObjsAnnotationAddedOrChangedMatchCondName, + Expression: isAnnotationAddedOrChangedExpr, + }, + }, + Validations: validations, + }, + } + + binding := &admissionregistrationv1.ValidatingAdmissionPolicyBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: annotatedForPlacementObjsVAPPolicyBindingName, + }, + Spec: admissionregistrationv1.ValidatingAdmissionPolicyBindingSpec{ + PolicyName: annotatedForPlacementObjsVAPPolicyName, + ValidationActions: []admissionregistrationv1.ValidationAction{ + admissionregistrationv1.Deny, + }, + }, + } + + return []PolicyWithBindings{ + { + Policy: policy, + Bindings: []*admissionregistrationv1.ValidatingAdmissionPolicyBinding{binding}, + }, + }, nil +} diff --git a/pkg/admissionpolicymanager/annotatedforplacementobjects_integration_test.go b/pkg/admissionpolicymanager/annotatedforplacementobjects_integration_test.go new file mode 100644 index 000000000..ac401d5d1 --- /dev/null +++ b/pkg/admissionpolicymanager/annotatedforplacementobjects_integration_test.go @@ -0,0 +1,565 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package admissionpolicymanager + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + autoscalingv1 "k8s.io/api/autoscaling/v1" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +const ( + placementAnnotationValue = "env=prod" + altPlacementAnnotationValue = "env=staging" + + testDataKey = "foo" + testDataValue = "bar" + altTestDataValue = "baz" +) + +var ( + placementPolicyAPIGroup = kfplacementv1alpha1.GroupVersion.Group + + objectWriterVerbs = []string{"get", authzVerbCreate, authzVerbUpdate, "patch", "delete"} +) + +func placementAnnotation(value string) map[string]string { + return map[string]string{ + kfplacementv1alpha1.ClusterSelectorsAnnotation: value, + } +} + +func newConfigMap(namespace, name string, annotations map[string]string) *corev1.ConfigMap { + return &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + Annotations: annotations, + }, + Data: map[string]string{ + testDataKey: testDataValue, + }, + } +} + +func newTestResource(namespace, name string, annotations map[string]string) *unstructured.Unstructured { + testResource := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "spec": map[string]interface{}{ + testDataKey: testDataValue, + }, + }, + } + testResource.SetAPIVersion("test.kubernetes-fleet.io/v1alpha1") + testResource.SetKind("TestResource") + testResource.SetNamespace(namespace) + testResource.SetName(name) + testResource.SetAnnotations(annotations) + return testResource +} + +func newNamespace(name string, annotations map[string]string) *corev1.Namespace { + return &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Annotations: annotations, + }, + } +} + +// newEndpoints returns an Endpoints object with no subsets. It is built as an unstructured object, as the typed +// Endpoints API is deprecated. +func newEndpoints(namespace, name string, annotations map[string]string) *unstructured.Unstructured { + endpoints := &unstructured.Unstructured{} + endpoints.SetAPIVersion("v1") + endpoints.SetKind("Endpoints") + endpoints.SetNamespace(namespace) + endpoints.SetName(name) + endpoints.SetAnnotations(annotations) + return endpoints +} + +// newClusterRole returns a cluster role with no rules, which any user who can create cluster roles can create +// (i.e., the RBAC privilege escalation prevention does not apply). +func newClusterRole(name string, annotations map[string]string) *rbacv1.ClusterRole { + return &rbacv1.ClusterRole{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Annotations: annotations, + }, + } +} + +var _ = Describe("deny placement annotation if no placement policy API access", Ordered, func() { + suffix := utils.RandStr() + workNSName := fmt.Sprintf("work-%s", suffix) + otherNSName := fmt.Sprintf("other-%s", suffix) + + // The users that the test cases impersonate. All of them can write the objects that the test cases + // annotate, but they differ in their permissions on the placement policy APIs. + // + // The usernames are randomized so that permissions granted in other test containers never apply. + noPermUsername := fmt.Sprintf("no-perm-%s", suffix) + // Can create and update placement policies in the work namespace only. + ppInNSUsername := fmt.Sprintf("pp-ns-%s", suffix) + // Can create (but not update) placement policies in the work namespace only. + ppCreateOnlyInNSUsername := fmt.Sprintf("pp-ns-create-only-%s", suffix) + // Can create and update placement policies in all namespaces, but not cluster placement policies. + ppClusterWideUsername := fmt.Sprintf("pp-cluster-wide-%s", suffix) + // Can create and update cluster placement policies, but not placement policies. + cppUsername := fmt.Sprintf("cpp-%s", suffix) + + objWriterRBACName := fmt.Sprintf("object-writer-%s", suffix) + ppInNSRBACName := fmt.Sprintf("pp-ns-%s", suffix) + ppCreateOnlyInNSRBACName := fmt.Sprintf("pp-ns-create-only-%s", suffix) + ppClusterWideRBACName := fmt.Sprintf("pp-cluster-wide-%s", suffix) + cppRBACName := fmt.Sprintf("cpp-%s", suffix) + + var noPermClient, ppInNSClient, ppCreateOnlyInNSClient, ppClusterWideClient, cppClient client.Client + + // The names of the cluster roles that the test cases create as annotated objects; they are cleaned up + // after all the test cases have run. + var annotatedClusterRoleNames []string + + BeforeAll(func() { + createNamespace(workNSName) + createNamespace(otherNSName) + + grantClusterWidePermissions(objWriterRBACName, []rbacv1.PolicyRule{ + { + APIGroups: []string{""}, + Resources: []string{"configmaps", "namespaces", "namespaces/status", "namespaces/finalize"}, + Verbs: objectWriterVerbs, + }, + { + APIGroups: []string{appsv1.GroupName}, + Resources: []string{"deployments", "deployments/status", "deployments/scale"}, + Verbs: objectWriterVerbs, + }, + { + APIGroups: []string{"test.kubernetes-fleet.io"}, + Resources: []string{"testresources"}, + Verbs: objectWriterVerbs, + }, + { + APIGroups: []string{rbacv1.GroupName}, + Resources: []string{"clusterroles"}, + Verbs: objectWriterVerbs, + }, + }, + userSubject(noPermUsername), + userSubject(ppInNSUsername), + userSubject(ppCreateOnlyInNSUsername), + userSubject(ppClusterWideUsername), + userSubject(cppUsername), + ) + grantPermissionsInNamespace(workNSName, ppInNSRBACName, []rbacv1.PolicyRule{ + { + APIGroups: []string{placementPolicyAPIGroup}, + Resources: []string{placementPoliciesResourceName}, + Verbs: []string{authzVerbCreate, authzVerbUpdate}, + }, + }, userSubject(ppInNSUsername)) + grantPermissionsInNamespace(workNSName, ppCreateOnlyInNSRBACName, []rbacv1.PolicyRule{ + { + APIGroups: []string{placementPolicyAPIGroup}, + Resources: []string{placementPoliciesResourceName}, + Verbs: []string{authzVerbCreate}, + }, + }, userSubject(ppCreateOnlyInNSUsername)) + grantClusterWidePermissions(ppClusterWideRBACName, []rbacv1.PolicyRule{ + { + APIGroups: []string{placementPolicyAPIGroup}, + Resources: []string{placementPoliciesResourceName}, + Verbs: []string{authzVerbCreate, authzVerbUpdate}, + }, + }, userSubject(ppClusterWideUsername)) + grantClusterWidePermissions(cppRBACName, []rbacv1.PolicyRule{ + { + APIGroups: []string{placementPolicyAPIGroup}, + Resources: []string{clusterPlacementPoliciesResourceName}, + Verbs: []string{authzVerbCreate, authzVerbUpdate}, + }, + }, userSubject(cppUsername)) + + noPermClient = impersonatedClient(noPermUsername) + ppInNSClient = impersonatedClient(ppInNSUsername) + ppCreateOnlyInNSClient = impersonatedClient(ppCreateOnlyInNSUsername) + ppClusterWideClient = impersonatedClient(ppClusterWideUsername) + cppClient = impersonatedClient(cppUsername) + + By("waiting for the admission policy to take effect") + waitUntilPolicyIsEnforced(func() error { + probeCM := newConfigMap(workNSName, fmt.Sprintf("probe-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + return noPermClient.Create(ctx, probeCM) + }, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg) + }) + + Context("namespaced objects", func() { + It("should allow creating an object without the annotation", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), nil) + Expect(noPermClient.Create(ctx, cm)).To(Succeed(), "Failed to create a config map without the annotation") + }) + + It("should deny creating an annotated object if the caller cannot create or update placement policies", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := noPermClient.Create(ctx, cm) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should deny creating an annotated object if the caller can create but not update placement policies", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := ppCreateOnlyInNSClient.Create(ctx, cm) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should allow creating an annotated object if the caller can create and update placement policies in the namespace", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + Expect(ppInNSClient.Create(ctx, cm)).To(Succeed(), "Failed to create an annotated config map") + }) + + It("should deny creating an annotated object in a namespace where the caller cannot create or update placement policies", func() { + cm := newConfigMap(otherNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := ppInNSClient.Create(ctx, cm) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should allow creating an annotated object if the caller can create and update placement policies in all namespaces", func() { + cm := newConfigMap(otherNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + Expect(ppClusterWideClient.Create(ctx, cm)).To(Succeed(), "Failed to create an annotated config map") + }) + + It("should deny creating an annotated object if the caller can only create and update cluster placement policies", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := cppClient.Create(ctx, cm) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should allow creating an annotated object as a cluster admin", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create an annotated config map") + }) + + It("should deny adding the annotation to an existing object if the caller cannot create or update placement policies", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), nil) + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create a config map without the annotation") + + err := getAndUpdate(noPermClient, cm, func(cm *corev1.ConfigMap) { + cm.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should deny adding the annotation to an existing object with a patch if the caller cannot create or update placement policies", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), nil) + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create a config map without the annotation") + + // This is how kubectl annotate adds an annotation. + patch := fmt.Sprintf(`{"metadata":{"annotations":{%q:%q}}}`, kfplacementv1alpha1.ClusterSelectorsAnnotation, placementAnnotationValue) + err := noPermClient.Patch(ctx, cm, client.RawPatch(types.MergePatchType, []byte(patch))) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should deny changing the annotation value if the caller cannot create or update placement policies", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create an annotated config map") + + err := getAndUpdate(noPermClient, cm, func(cm *corev1.ConfigMap) { + cm.Annotations[kfplacementv1alpha1.ClusterSelectorsAnnotation] = altPlacementAnnotationValue + }) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should allow updating an annotated object without changing the annotation", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create an annotated config map") + + err := getAndUpdate(noPermClient, cm, func(cm *corev1.ConfigMap) { + cm.Data = map[string]string{testDataKey: altTestDataValue} + cm.Labels = map[string]string{testDataKey: testDataValue} + cm.Annotations[testDataKey] = testDataValue + }) + Expect(err).To(Succeed(), "Failed to update an annotated config map without changing the annotation") + }) + + It("should allow removing the annotation", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create an annotated config map") + + err := getAndUpdate(noPermClient, cm, func(cm *corev1.ConfigMap) { + delete(cm.Annotations, kfplacementv1alpha1.ClusterSelectorsAnnotation) + }) + Expect(err).To(Succeed(), "Failed to remove the annotation") + }) + + It("should allow adding and changing the annotation if the caller can create and update placement policies in the namespace", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), nil) + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create a config map without the annotation") + + err := getAndUpdate(ppInNSClient, cm, func(cm *corev1.ConfigMap) { + cm.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(err).To(Succeed(), "Failed to add the annotation") + + err = getAndUpdate(ppInNSClient, cm, func(cm *corev1.ConfigMap) { + cm.Annotations[kfplacementv1alpha1.ClusterSelectorsAnnotation] = altPlacementAnnotationValue + }) + Expect(err).To(Succeed(), "Failed to change the annotation value") + }) + + It("should allow deleting an annotated object and removing its finalizers (as the garbage collector would)", func() { + cm := newConfigMap(workNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + cm.Finalizers = []string{testFinalizer} + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create an annotated config map with a finalizer") + + Expect(noPermClient.Delete(ctx, cm)).To(Succeed(), "Failed to delete the annotated config map") + + err := getAndUpdate(noPermClient, cm, func(cm *corev1.ConfigMap) { + cm.Finalizers = nil + }) + Expect(err).To(Succeed(), "Failed to remove the finalizer from the annotated config map") + + Eventually(func() error { + err := hubUncachedClient.Get(ctx, client.ObjectKeyFromObject(cm), &corev1.ConfigMap{}) + if !apierrors.IsNotFound(err) { + return fmt.Errorf("config map still exists or an unexpected error has occurred: %w", err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the annotated config map") + }) + }) + + Context("namespaced custom resources", func() { + It("should deny creating an annotated object if the caller cannot create or update placement policies", func() { + testResource := newTestResource(workNSName, fmt.Sprintf("tr-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := noPermClient.Create(ctx, testResource) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should allow creating an annotated object if the caller can create and update placement policies in the namespace", func() { + testResource := newTestResource(workNSName, fmt.Sprintf("tr-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + Expect(ppInNSClient.Create(ctx, testResource)).To(Succeed(), "Failed to create an annotated test resource") + }) + }) + + Context("cluster-scoped objects", func() { + It("should deny creating an annotated namespace if the caller cannot create or update cluster placement policies", func() { + ns := newNamespace(fmt.Sprintf("annotated-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := noPermClient.Create(ctx, ns) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantClusterScopedObjDenialMsg)).To(Succeed()) + }) + + // For namespaces, the API server sets the namespace of the admission request to the name of the namespace; + // the admission policy must still check them as cluster-scoped objects. + It("should deny creating an annotated namespace if the caller can only create and update placement policies (in all namespaces)", func() { + ns := newNamespace(fmt.Sprintf("annotated-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := ppClusterWideClient.Create(ctx, ns) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantClusterScopedObjDenialMsg)).To(Succeed()) + }) + + It("should allow creating an annotated namespace if the caller can create and update cluster placement policies", func() { + ns := newNamespace(fmt.Sprintf("annotated-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + Expect(cppClient.Create(ctx, ns)).To(Succeed(), "Failed to create an annotated namespace") + }) + + It("should deny adding the annotation to an existing namespace if the caller cannot create or update cluster placement policies, and allow it otherwise", func() { + nsName := fmt.Sprintf("plain-%s", utils.RandStr()) + createNamespace(nsName) + + err := getAndUpdate(noPermClient, newNamespace(nsName, nil), func(ns *corev1.Namespace) { + ns.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantClusterScopedObjDenialMsg)).To(Succeed()) + + err = getAndUpdate(cppClient, newNamespace(nsName, nil), func(ns *corev1.Namespace) { + ns.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(err).To(Succeed(), "Failed to add the annotation to the namespace") + }) + + It("should deny creating an annotated cluster role if the caller cannot create or update cluster placement policies, and allow it otherwise", func() { + clusterRoleName := fmt.Sprintf("annotated-%s", utils.RandStr()) + annotatedClusterRoleNames = append(annotatedClusterRoleNames, clusterRoleName) + + err := noPermClient.Create(ctx, newClusterRole(clusterRoleName, placementAnnotation(placementAnnotationValue))) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantClusterScopedObjDenialMsg)).To(Succeed()) + + Expect(cppClient.Create(ctx, newClusterRole(clusterRoleName, placementAnnotation(placementAnnotationValue)))).To(Succeed(), "Failed to create an annotated cluster role") + }) + }) + + // Some built-in resources (e.g., Deployments and Namespaces) keep the annotation changes sent to some of their + // subresources, such as status and finalize. + Context("subresources", func() { + It("should deny adding the annotation through the status subresource if the caller cannot create or update placement policies", func() { + deploy := newDeployment(workNSName, fmt.Sprintf("deploy-%s", utils.RandStr())) + Expect(hubUncachedClient.Create(ctx, deploy)).To(Succeed(), "Failed to create a deployment without the annotation") + + err := getAndUpdateSubResource(noPermClient, deploy, "status", func(deploy *appsv1.Deployment) { + deploy.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantNamespacedObjDenialMsg)).To(Succeed()) + }) + + It("should allow updating the status of an annotated object without changing the annotation", func() { + deploy := newDeployment(workNSName, fmt.Sprintf("deploy-%s", utils.RandStr())) + deploy.Annotations = placementAnnotation(placementAnnotationValue) + Expect(hubUncachedClient.Create(ctx, deploy)).To(Succeed(), "Failed to create an annotated deployment") + + err := getAndUpdateSubResource(noPermClient, deploy, "status", func(deploy *appsv1.Deployment) { + deploy.Status.ObservedGeneration = deploy.Generation + }) + Expect(err).To(Succeed(), "Failed to update the status of the annotated deployment") + }) + + // The scale subresource carries a Scale object rather than the Deployment itself. + It("should allow scaling an annotated object", func() { + deploy := newDeployment(workNSName, fmt.Sprintf("deploy-%s", utils.RandStr())) + deploy.Annotations = placementAnnotation(placementAnnotationValue) + Expect(hubUncachedClient.Create(ctx, deploy)).To(Succeed(), "Failed to create an annotated deployment") + + scale := &autoscalingv1.Scale{} + Expect(noPermClient.SubResource("scale").Get(ctx, deploy, scale)).To(Succeed(), "Failed to get the scale of the annotated deployment") + scale.Spec.Replicas = 3 + Expect(noPermClient.SubResource("scale").Update(ctx, deploy, client.WithSubResourceBody(scale))).To(Succeed(), "Failed to scale the annotated deployment") + }) + + DescribeTable("should deny adding the annotation to a namespace through a subresource if the caller cannot create or update cluster placement policies, and allow it otherwise", + func(subResource string) { + nsName := fmt.Sprintf("plain-%s", utils.RandStr()) + createNamespace(nsName) + + err := getAndUpdateSubResource(noPermClient, newNamespace(nsName, nil), subResource, func(ns *corev1.Namespace) { + ns.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantClusterScopedObjDenialMsg)).To(Succeed()) + + err = getAndUpdateSubResource(cppClient, newNamespace(nsName, nil), subResource, func(ns *corev1.Namespace) { + ns.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(err).To(Succeed(), "Failed to add the annotation to the namespace through the %s subresource", subResource) + }, + Entry("status", "status"), + Entry("finalize", "finalize"), + ) + }) + + // The annotation can never be added in reserved namespaces, regardless of the permissions of the caller. + Context("reserved namespaces", func() { + It("should deny creating an annotated object in a kube- prefixed namespace, even as a cluster admin", func() { + cm := newConfigMap(kubeSystemNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := hubUncachedClient.Create(ctx, cm) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantReservedNSAnnotationDenialMsg)).To(Succeed()) + }) + + It("should deny creating an annotated object in a fleet- prefixed namespace, even as a cluster admin", func() { + fleetNSName := fmt.Sprintf("fleet-%s", utils.RandStr()) + createNamespace(fleetNSName) + + cm := newConfigMap(fleetNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := hubUncachedClient.Create(ctx, cm) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantReservedNSAnnotationDenialMsg)).To(Succeed()) + }) + + It("should deny adding the annotation to an existing object in a reserved namespace, even as a cluster admin", func() { + cm := newConfigMap(kubeSystemNSName, fmt.Sprintf("cm-%s", utils.RandStr()), nil) + Expect(hubUncachedClient.Create(ctx, cm)).To(Succeed(), "Failed to create a config map without the annotation") + DeferCleanup(func() { + Expect(client.IgnoreNotFound(hubUncachedClient.Delete(ctx, cm))).To(Succeed(), "Failed to delete the config map") + }) + + err := getAndUpdate(hubUncachedClient, cm, func(cm *corev1.ConfigMap) { + cm.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantReservedNSAnnotationDenialMsg)).To(Succeed()) + }) + + // The caller fails the permission check as well; the reserved namespace denial is the one reported. + It("should report the reserved namespace denial if the caller cannot create or update placement policies either", func() { + cm := newConfigMap(kubeSystemNSName, fmt.Sprintf("cm-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := noPermClient.Create(ctx, cm) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantReservedNSAnnotationDenialMsg)).To(Succeed()) + }) + + It("should deny annotating a reserved namespace itself, even if the caller can create and update cluster placement policies", func() { + err := cppClient.Create(ctx, newNamespace(fmt.Sprintf("fleet-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue))) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantReservedNSAnnotationDenialMsg)).To(Succeed()) + + nsName := fmt.Sprintf("fleet-%s", utils.RandStr()) + createNamespace(nsName) + err = getAndUpdate(cppClient, newNamespace(nsName, nil), func(ns *corev1.Namespace) { + ns.Annotations = placementAnnotation(placementAnnotationValue) + }) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantReservedNSAnnotationDenialMsg)).To(Succeed()) + }) + + It("should allow writing objects without the annotation in a reserved namespace", func() { + fleetNSName := fmt.Sprintf("fleet-%s", utils.RandStr()) + createNamespace(fleetNSName) + + cm := newConfigMap(fleetNSName, fmt.Sprintf("cm-%s", utils.RandStr()), nil) + Expect(noPermClient.Create(ctx, cm)).To(Succeed(), "Failed to create a config map without the annotation") + + err := getAndUpdate(noPermClient, cm, func(cm *corev1.ConfigMap) { + cm.Data = map[string]string{testDataKey: altTestDataValue} + }) + Expect(err).To(Succeed(), "Failed to update a config map without the annotation") + }) + }) + + // The annotation can never be added to Endpoints objects, regardless of the permissions of the caller, as the + // EndpointSlice mirroring controller would copy it onto the mirrored EndpointSlices. + Context("Endpoints", func() { + It("should deny creating an annotated Endpoints object, even as a cluster admin", func() { + endpoints := newEndpoints(workNSName, fmt.Sprintf("ep-%s", utils.RandStr()), placementAnnotation(placementAnnotationValue)) + err := hubUncachedClient.Create(ctx, endpoints) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantEndpointsAnnotationDenialMsg)).To(Succeed()) + }) + + It("should deny adding the annotation to an existing Endpoints object, even as a cluster admin", func() { + endpoints := newEndpoints(workNSName, fmt.Sprintf("ep-%s", utils.RandStr()), nil) + Expect(hubUncachedClient.Create(ctx, endpoints)).To(Succeed(), "Failed to create an Endpoints object without the annotation") + + err := getAndUpdate(hubUncachedClient, endpoints, func(endpoints *unstructured.Unstructured) { + endpoints.SetAnnotations(placementAnnotation(placementAnnotationValue)) + }) + Expect(wantDeniedByPolicy(err, annotatedForPlacementObjsVAPPolicyName, wantEndpointsAnnotationDenialMsg)).To(Succeed()) + }) + }) + + AfterAll(func() { + for _, name := range annotatedClusterRoleNames { + Expect(client.IgnoreNotFound(hubUncachedClient.Delete(ctx, newClusterRole(name, nil)))).To(Succeed(), "Failed to delete cluster role %s", name) + } + revokeClusterWidePermissions(objWriterRBACName) + revokeClusterWidePermissions(ppClusterWideRBACName) + revokeClusterWidePermissions(cppRBACName) + }) +}) diff --git a/pkg/admissionpolicymanager/azconfigs.go b/pkg/admissionpolicymanager/azconfigs.go new file mode 100644 index 000000000..d230e3115 --- /dev/null +++ b/pkg/admissionpolicymanager/azconfigs.go @@ -0,0 +1,47 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package admissionpolicymanager + +import ( + "go.goms.io/fleet/pkg/utils" +) + +func DefaultPolicyGeneratorConfigsForAzure(annotationBasedPlacementEnabled bool) *PolicyGeneratorConfigs { + if annotationBasedPlacementEnabled { + return &PolicyGeneratorConfigs{ + PodsAndReplicaSetsVAPGeneratorConfig: &PodsAndReplicaSetsValidatingAdmissionPolicyGenerator{ + ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, + }, + SvcAccountsAndTokenRequestsVAPGeneratorConfig: &ServiceAccountsAndTokenRequestsValidatingAdmissionPolicyGenerator{ + ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, + WhitelistedUsernames: []string{aksSupportUser}, + }, + AnnotatedForPlacementObjectsVAPGeneratorConfig: &AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator{ + ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, + }, + } + } + return &PolicyGeneratorConfigs{ + PodsAndReplicaSetsVAPGeneratorConfig: &PodsAndReplicaSetsValidatingAdmissionPolicyGenerator{ + ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, + }, + SvcAccountsAndTokenRequestsVAPGeneratorConfig: &ServiceAccountsAndTokenRequestsValidatingAdmissionPolicyGenerator{ + ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, + WhitelistedUsernames: []string{aksSupportUser}, + }, + } +} diff --git a/pkg/admissionpolicymanager/commons.go b/pkg/admissionpolicymanager/commons.go index 53156301c..cf96afecd 100644 --- a/pkg/admissionpolicymanager/commons.go +++ b/pkg/admissionpolicymanager/commons.go @@ -92,6 +92,17 @@ func isInNamespaceWithPrefix(p string) CELExprTreeNode { return RawCELExpr(fmt.Sprintf(`request.namespace.startsWith("%s")`, p)) } +// requestHasNamespace guards checks on request.namespace in policies that match cluster-scoped objects, as the +// field is absent from requests on cluster-scoped objects (other than Namespace objects). +func requestHasNamespace() CELExprTreeNode { + return RawCELExpr(`has(request.namespace)`) +} + +// requestIsForKind checks the kind of the object in the request; the core API group is an empty string. +func requestIsForKind(group, kind string) CELExprTreeNode { + return RawCELExpr(fmt.Sprintf(`request.kind.group == "%s" && request.kind.kind == "%s"`, group, kind)) +} + func isFromUsername(u string) CELExprTreeNode { return RawCELExpr(fmt.Sprintf(`request.userInfo.username == "%s"`, u)) } @@ -99,3 +110,32 @@ func isFromUsername(u string) CELExprTreeNode { func isFromUserGroup(g string) CELExprTreeNode { return RawCELExpr(fmt.Sprintf(`"%s" in request.userInfo.groups`, g)) } + +func objectHasAnnotation(key string) CELExprTreeNode { + return RawCELExpr(fmt.Sprintf(`has(object.metadata.annotations) && "%s" in object.metadata.annotations`, key)) +} + +func oldObjectHasAnnotation(key string) CELExprTreeNode { + return RawCELExpr(fmt.Sprintf(`oldObject != null && has(oldObject.metadata.annotations) && "%s" in oldObject.metadata.annotations`, key)) +} + +// annotationValueIsUnchanged fails with a no-such-key error if either the object or the old object does not +// carry the annotation; guard it with objectHasAnnotation and oldObjectHasAnnotation in the same expression. +func annotationValueIsUnchanged(key string) CELExprTreeNode { + return RawCELExpr(fmt.Sprintf(`oldObject.metadata.annotations["%[1]s"] == object.metadata.annotations["%[1]s"]`, key)) +} + +// isNamespacedObject checks the namespace in the object metadata rather than request.namespace, as +// the API server sets request.namespace to the name of the namespace for Namespace objects, even +// though they are cluster-scoped. +func isNamespacedObject() CELExprTreeNode { + return RawCELExpr(`has(object.metadata.namespace) && object.metadata.namespace != ""`) +} + +func callerCanInObjectNamespace(group, resource, verb string) CELExprTreeNode { + return RawCELExpr(fmt.Sprintf(`authorizer.group("%s").resource("%s").namespace(object.metadata.namespace).check("%s").allowed()`, group, resource, verb)) +} + +func callerCanClusterWide(group, resource, verb string) CELExprTreeNode { + return RawCELExpr(fmt.Sprintf(`authorizer.group("%s").resource("%s").check("%s").allowed()`, group, resource, verb)) +} diff --git a/pkg/admissionpolicymanager/configs.go b/pkg/admissionpolicymanager/configs.go index f81995387..cf64808b9 100644 --- a/pkg/admissionpolicymanager/configs.go +++ b/pkg/admissionpolicymanager/configs.go @@ -33,10 +33,29 @@ import ( type PolicyGeneratorConfigs struct { PodsAndReplicaSetsVAPGeneratorConfig *PodsAndReplicaSetsValidatingAdmissionPolicyGenerator `json:"denyPodsAndReplicaSetsOutsideReservedNamespaces,omitempty"` SvcAccountsAndTokenRequestsVAPGeneratorConfig *ServiceAccountsAndTokenRequestsValidatingAdmissionPolicyGenerator `json:"denyServiceAccountsAndTokenRequestsInReservedNamespaces,omitempty"` + // The generator is not enabled in any of the default configurations and must be enabled explicitly. It should + // only be enabled if the workload controllers of kube-controller-manager have been disabled in the hub cluster; + // see AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator for more information. + AnnotatedForPlacementObjectsVAPGeneratorConfig *AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator `json:"denyPlacementAnnotationIfNoPlacementPolicyAPIAccess,omitempty"` } -// DefaultPolicyGeneratorConfigs is the default configuration for all available admission policy generators. -var DefaultPolicyGeneratorConfigs = &PolicyGeneratorConfigs{ +// DefaultPolicyGeneratorConfigsForV0APIs is the default configuration of admission policy generators +// for placement with v0 APIs. +var DefaultPolicyGeneratorConfigsForV0APIs = &PolicyGeneratorConfigs{ + PodsAndReplicaSetsVAPGeneratorConfig: &PodsAndReplicaSetsValidatingAdmissionPolicyGenerator{ + ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, + }, + SvcAccountsAndTokenRequestsVAPGeneratorConfig: &ServiceAccountsAndTokenRequestsValidatingAdmissionPolicyGenerator{ + ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, + WhitelistedUsernames: []string{aksSupportUser}, + }, +} + +// DefaultPolicyGeneratorConfigsForV1APIs is the default configuration of admission policy generators +// for placement with v1 APIs (placement policies). It covers all available admission policy generators +// except the placement annotation one (DenyPlacementAnnotationIfNoPlacementPolicyAPIAccess), which must be +// enabled explicitly. +var DefaultPolicyGeneratorConfigsForV1APIs = &PolicyGeneratorConfigs{ PodsAndReplicaSetsVAPGeneratorConfig: &PodsAndReplicaSetsValidatingAdmissionPolicyGenerator{ ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, }, diff --git a/pkg/admissionpolicymanager/manager.go b/pkg/admissionpolicymanager/manager.go index 178645919..630a81e2f 100644 --- a/pkg/admissionpolicymanager/manager.go +++ b/pkg/admissionpolicymanager/manager.go @@ -51,14 +51,16 @@ var ( ) func init() { - // Add all available generators to the set. - v := reflect.ValueOf(DefaultPolicyGeneratorConfigs).Elem() - for i := range v.NumField() { - field := v.Field(i) - if field.IsNil() { + // Add all available generators to the set; each field of PolicyGeneratorConfigs holds the configuration + // of one generator. The default configurations do not cover all generators, as some generators must be + // enabled explicitly. + t := reflect.TypeFor[PolicyGeneratorConfigs]() + for i := range t.NumField() { + fieldType := t.Field(i).Type + if fieldType.Kind() != reflect.Pointer { continue } - gen, ok := field.Interface().(ValidatingAdmissionPolicyGenerator) + gen, ok := reflect.New(fieldType.Elem()).Interface().(ValidatingAdmissionPolicyGenerator) if !ok { continue } @@ -99,8 +101,8 @@ type PolicyManager struct { func New(client client.Client, policyGeneratorConfigs *PolicyGeneratorConfigs) (*PolicyManager, error) { if policyGeneratorConfigs == nil { - klog.V(2).Info("No admission policy generator configuration provided, falling back to the default configuration") - policyGeneratorConfigs = DefaultPolicyGeneratorConfigs + klog.V(2).Info("No admission policy generator configuration provided, falling back to the default configuration for V0 APIs") + policyGeneratorConfigs = DefaultPolicyGeneratorConfigsForV0APIs } // Prepare a set of generators based on the given configuration. enabledPolicyGenerators, err := preparePolicyGenerators(policyGeneratorConfigs) diff --git a/pkg/admissionpolicymanager/manager_integration_test.go b/pkg/admissionpolicymanager/manager_integration_test.go index 269be078f..d0a8acab2 100644 --- a/pkg/admissionpolicymanager/manager_integration_test.go +++ b/pkg/admissionpolicymanager/manager_integration_test.go @@ -166,13 +166,79 @@ var _ = Describe("Policies, Policy Bindings and their Effects", Ordered, func() }, Validations: []admissionregistrationv1.Validation{ { - Expression: `(!((request.namespace.startsWith("fleet-")) || (request.namespace.startsWith("kube-")))) || ((request.userInfo.username == "system:kube-scheduler") || (request.userInfo.username == "system:kube-controller-manager") || ("system:nodes" in request.userInfo.groups) || ("system:masters" in request.userInfo.groups) || ("kubeadm:cluster-admins" in request.userInfo.groups) || ("system:serviceaccounts" in request.userInfo.groups) || (request.userInfo.username == "aks-support"))`, + Expression: `(!((request.namespace.startsWith("fleet-")) || (request.namespace.startsWith("kube-")))) || ((request.userInfo.username == "system:kube-scheduler") || (request.userInfo.username == "system:kube-controller-manager") || ("system:nodes" in request.userInfo.groups) || ("system:masters" in request.userInfo.groups) || ("kubeadm:cluster-admins" in request.userInfo.groups) || ("system:serviceaccounts" in request.userInfo.groups) || (request.userInfo.username == "whitelisted-user") || ("whitelisted-group" in request.userInfo.groups))`, Message: "writing service accounts in reserved namespaces or requesting tokens from such service accounts is disallowed", Reason: ptr.To(metav1.StatusReasonForbidden), }, }, }, }, + { + ObjectMeta: metav1.ObjectMeta{ + Name: annotatedForPlacementObjsVAPPolicyName, + Labels: map[string]string{ + VAPManagedByKubeFleetLabelKey: VAPManagedByKubeFleetLabelValue, + VAPPartOfKubeFleetLabelKey: VAPPartOfKubeFleetLabelValue, + VAPComponentKubeFleetLabelKey: VAPComponentAdmissionPolicyManagerLabelValue, + }, + }, + Spec: admissionregistrationv1.ValidatingAdmissionPolicySpec{ + FailurePolicy: ptr.To(admissionregistrationv1.Fail), + MatchConstraints: &admissionregistrationv1.MatchResources{ + NamespaceSelector: &metav1.LabelSelector{}, + ObjectSelector: &metav1.LabelSelector{}, + ResourceRules: []admissionregistrationv1.NamedRuleWithOperations{ + { + RuleWithOperations: admissionregistrationv1.RuleWithOperations{ + Operations: []admissionregistrationv1.OperationType{ + admissionregistrationv1.Create, + admissionregistrationv1.Update, + }, + Rule: admissionregistrationv1.Rule{ + APIGroups: []string{"*"}, + APIVersions: []string{"*"}, + Resources: []string{"*/*"}, + Scope: ptr.To(admissionregistrationv1.ScopeType("*")), // The system-enforced default. + }, + }, + }, + }, + MatchPolicy: ptr.To(admissionregistrationv1.Equivalent), // The system-enforced default. + }, + MatchConditions: []admissionregistrationv1.MatchCondition{ + { + Name: "has-placement-annotation", + Expression: `has(object.metadata.annotations) && "kubefleet.dev/cluster-selectors" in object.metadata.annotations`, + }, + { + Name: "placement-annotation-added-or-changed", + Expression: `!((has(object.metadata.annotations) && "kubefleet.dev/cluster-selectors" in object.metadata.annotations) && (oldObject != null && has(oldObject.metadata.annotations) && "kubefleet.dev/cluster-selectors" in oldObject.metadata.annotations) && (oldObject.metadata.annotations["kubefleet.dev/cluster-selectors"] == object.metadata.annotations["kubefleet.dev/cluster-selectors"]))`, + }, + }, + Validations: []admissionregistrationv1.Validation{ + { + Expression: `!((has(request.namespace)) && ((request.namespace.startsWith("fleet-")) || (request.namespace.startsWith("kube-"))))`, + Message: wantReservedNSAnnotationDenialMsg, + Reason: ptr.To(metav1.StatusReasonForbidden), + }, + { + Expression: `!(request.kind.group == "" && request.kind.kind == "Endpoints")`, + Message: wantEndpointsAnnotationDenialMsg, + Reason: ptr.To(metav1.StatusReasonForbidden), + }, + { + Expression: `(!(has(object.metadata.namespace) && object.metadata.namespace != "")) || ((authorizer.group("placement.kubefleet.dev").resource("placementpolicies").namespace(object.metadata.namespace).check("create").allowed()) && (authorizer.group("placement.kubefleet.dev").resource("placementpolicies").namespace(object.metadata.namespace).check("update").allowed()))`, + Message: wantNamespacedObjDenialMsg, + Reason: ptr.To(metav1.StatusReasonForbidden), + }, + { + Expression: `(has(object.metadata.namespace) && object.metadata.namespace != "") || ((authorizer.group("placement.kubefleet.dev").resource("clusterplacementpolicies").check("create").allowed()) && (authorizer.group("placement.kubefleet.dev").resource("clusterplacementpolicies").check("update").allowed()))`, + Message: wantClusterScopedObjDenialMsg, + Reason: ptr.To(metav1.StatusReasonForbidden), + }, + }, + }, + }, } Eventually(ctx, func() error { @@ -237,6 +303,22 @@ var _ = Describe("Policies, Policy Bindings and their Effects", Ordered, func() }, }, }, + { + ObjectMeta: metav1.ObjectMeta{ + Name: annotatedForPlacementObjsVAPPolicyBindingName, + Labels: map[string]string{ + VAPManagedByKubeFleetLabelKey: VAPManagedByKubeFleetLabelValue, + VAPPartOfKubeFleetLabelKey: VAPPartOfKubeFleetLabelValue, + VAPComponentKubeFleetLabelKey: VAPComponentAdmissionPolicyManagerLabelValue, + }, + }, + Spec: admissionregistrationv1.ValidatingAdmissionPolicyBindingSpec{ + PolicyName: annotatedForPlacementObjsVAPPolicyName, + ValidationActions: []admissionregistrationv1.ValidationAction{ + admissionregistrationv1.Deny, + }, + }, + }, } Eventually(ctx, func() error { @@ -266,7 +348,4 @@ var _ = Describe("Policies, Policy Bindings and their Effects", Ordered, func() return nil }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Received unexpected list of policy bindings or an error has occurred") }) - - // Note: in the integration test environment, it appears that validating admission policies have no effect or are - // not being enforced as expected. As a result the effects are validated using E2E tests instead. }) diff --git a/pkg/admissionpolicymanager/podsnreplicasets_integration_test.go b/pkg/admissionpolicymanager/podsnreplicasets_integration_test.go new file mode 100644 index 000000000..01d8acdf2 --- /dev/null +++ b/pkg/admissionpolicymanager/podsnreplicasets_integration_test.go @@ -0,0 +1,162 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package admissionpolicymanager + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + "go.goms.io/fleet/pkg/utils" +) + +const ( + kubeSystemNSName = "kube-system" +) + +func newPodSpec() corev1.PodSpec { + return corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "nginx", + Image: "nginx", + }, + }, + } +} + +func newPod(namespace, name string) *corev1.Pod { + return &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + Spec: newPodSpec(), + } +} + +func newReplicaSet(namespace, name string) *appsv1.ReplicaSet { + podLabels := map[string]string{"app": name} + return &appsv1.ReplicaSet{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + Spec: appsv1.ReplicaSetSpec{ + Selector: &metav1.LabelSelector{ + MatchLabels: podLabels, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: podLabels, + }, + Spec: newPodSpec(), + }, + }, + } +} + +func newDeployment(namespace, name string) *appsv1.Deployment { + podLabels := map[string]string{"app": name} + return &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + Spec: appsv1.DeploymentSpec{ + Selector: &metav1.LabelSelector{ + MatchLabels: podLabels, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: podLabels, + }, + Spec: newPodSpec(), + }, + }, + } +} + +// All the requests in this container are sent as the cluster admin (system:masters), which also verifies that the +// admission policy does not exempt any caller. +var _ = Describe("deny pods and replica sets outside reserved namespaces", Ordered, func() { + suffix := utils.RandStr() + workNSName := fmt.Sprintf("work-%s", suffix) + fleetNSName := fmt.Sprintf("fleet-%s", suffix) + // A namespace whose name starts with a reserved namespace prefix minus the trailing hyphen. + fleetPrefixedNSName := fmt.Sprintf("fleetwork-%s", suffix) + + // The objects that the test cases create in the kube-system namespace, which other test containers share; + // they are cleaned up after all the test cases have run. + var kubeSystemObjs []client.Object + + BeforeAll(func() { + createNamespace(workNSName) + createNamespace(fleetNSName) + createNamespace(fleetPrefixedNSName) + + By("waiting for the admission policy to take effect") + waitUntilPolicyIsEnforced(func() error { + return hubUncachedClient.Create(ctx, newPod(workNSName, fmt.Sprintf("probe-%s", utils.RandStr()))) + }, podsAndReplicaSetsVAPPolicyName, wantPodsAndReplicaSetsDenialMsg) + }) + + It("should deny creating pods in a non-reserved namespace", func() { + err := hubUncachedClient.Create(ctx, newPod(workNSName, fmt.Sprintf("pod-%s", utils.RandStr()))) + Expect(wantDeniedByPolicy(err, podsAndReplicaSetsVAPPolicyName, wantPodsAndReplicaSetsDenialMsg)).To(Succeed()) + }) + + It("should deny creating replica sets in a non-reserved namespace", func() { + err := hubUncachedClient.Create(ctx, newReplicaSet(workNSName, fmt.Sprintf("rs-%s", utils.RandStr()))) + Expect(wantDeniedByPolicy(err, podsAndReplicaSetsVAPPolicyName, wantPodsAndReplicaSetsDenialMsg)).To(Succeed()) + }) + + It("should deny creating pods in a namespace whose name starts with a reserved prefix minus the hyphen", func() { + err := hubUncachedClient.Create(ctx, newPod(fleetPrefixedNSName, fmt.Sprintf("pod-%s", utils.RandStr()))) + Expect(wantDeniedByPolicy(err, podsAndReplicaSetsVAPPolicyName, wantPodsAndReplicaSetsDenialMsg)).To(Succeed()) + }) + + It("should allow creating pods and replica sets in a fleet- prefixed namespace", func() { + Expect(hubUncachedClient.Create(ctx, newPod(fleetNSName, fmt.Sprintf("pod-%s", utils.RandStr())))).To(Succeed(), "Failed to create a pod") + Expect(hubUncachedClient.Create(ctx, newReplicaSet(fleetNSName, fmt.Sprintf("rs-%s", utils.RandStr())))).To(Succeed(), "Failed to create a replica set") + }) + + It("should allow creating pods and replica sets in a kube- prefixed namespace", func() { + pod := newPod(kubeSystemNSName, fmt.Sprintf("pod-%s", utils.RandStr())) + Expect(hubUncachedClient.Create(ctx, pod)).To(Succeed(), "Failed to create a pod") + kubeSystemObjs = append(kubeSystemObjs, pod) + + rs := newReplicaSet(kubeSystemNSName, fmt.Sprintf("rs-%s", utils.RandStr())) + Expect(hubUncachedClient.Create(ctx, rs)).To(Succeed(), "Failed to create a replica set") + kubeSystemObjs = append(kubeSystemObjs, rs) + }) + + It("should allow creating deployments in a non-reserved namespace", func() { + Expect(hubUncachedClient.Create(ctx, newDeployment(workNSName, fmt.Sprintf("deploy-%s", utils.RandStr())))).To(Succeed(), "Failed to create a deployment") + }) + + AfterAll(func() { + for _, obj := range kubeSystemObjs { + Expect(client.IgnoreNotFound(hubUncachedClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %s", client.ObjectKeyFromObject(obj)) + } + }) +}) diff --git a/pkg/admissionpolicymanager/suite_test.go b/pkg/admissionpolicymanager/suite_test.go index 4371ceb30..969a5ec15 100644 --- a/pkg/admissionpolicymanager/suite_test.go +++ b/pkg/admissionpolicymanager/suite_test.go @@ -19,6 +19,7 @@ package admissionpolicymanager import ( "context" "flag" + "path/filepath" "testing" "time" @@ -32,6 +33,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/envtest" "sigs.k8s.io/controller-runtime/pkg/log/zap" "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + "go.goms.io/fleet/pkg/utils" ) // These tests use Ginkgo (BDD-style Go testing framework). Refer to @@ -51,6 +54,31 @@ var ( eventuallyInterval = time.Millisecond * 500 ) +// The username and user group that the suite whitelists in the service accounts and token requests +// admission policy, in addition to the built-in exemptions. +const ( + whitelistedUsername = "whitelisted-user" + whitelistedUserGroup = "whitelisted-group" +) + +// suitePolicyGeneratorConfigs returns the configuration that the policy manager runs with in this suite: all the +// generators in DefaultPolicyGeneratorConfigsForV1APIs, with a whitelisted username and user group added to the +// service accounts and token requests generator, plus the placement annotation generator, which must be enabled +// explicitly. +// +// The shared default configuration is copied rather than modified. +func suitePolicyGeneratorConfigs() *PolicyGeneratorConfigs { + configs := *DefaultPolicyGeneratorConfigsForV1APIs + svcAccountsAndTokenRequestsGen := *configs.SvcAccountsAndTokenRequestsVAPGeneratorConfig + svcAccountsAndTokenRequestsGen.WhitelistedUsernames = []string{whitelistedUsername} + svcAccountsAndTokenRequestsGen.WhitelistedUserGroups = []string{whitelistedUserGroup} + configs.SvcAccountsAndTokenRequestsVAPGeneratorConfig = &svcAccountsAndTokenRequestsGen + configs.AnnotatedForPlacementObjectsVAPGeneratorConfig = &AnnotatedForPlacementObjectsValidatingAdmissionPolicyGenerator{ + ReservedNamespacePrefixes: []string{utils.FleetNSNamePrefix, utils.KubeNSNamePrefix}, + } + return &configs +} + func TestAPIs(t *testing.T) { RegisterFailHandler(Fail) @@ -70,7 +98,14 @@ var _ = BeforeSuite(func() { ctrl.SetLogger(logger) By("Bootstrapping test environment") - testEnv = &envtest.Environment{} + testEnv = &envtest.Environment{ + // The custom resource definition of the test resources, which some test cases use as + // namespaced custom resources. + CRDInstallOptions: envtest.CRDInstallOptions{ + Paths: []string{filepath.Join("../../", "test", "manifests", "test_testresources_crd.yaml")}, + }, + ErrorIfCRDPathMissing: true, + } var err error cfg, err = testEnv.Start() @@ -94,7 +129,7 @@ var _ = BeforeSuite(func() { Expect(hubUncachedClient).ToNot(BeNil()) By("Setting up the policy manager") - policyManager, err := New(hubUncachedClient, DefaultPolicyGeneratorConfigs) + policyManager, err := New(hubUncachedClient, suitePolicyGeneratorConfigs()) Expect(err).ToNot(HaveOccurred()) Expect(policyManager).ToNot(BeNil()) Expect(policyManager.Start(ctx)).To(Succeed()) diff --git a/pkg/admissionpolicymanager/svcaccountsntokenreqs_integration_test.go b/pkg/admissionpolicymanager/svcaccountsntokenreqs_integration_test.go new file mode 100644 index 000000000..b0d0fd922 --- /dev/null +++ b/pkg/admissionpolicymanager/svcaccountsntokenreqs_integration_test.go @@ -0,0 +1,200 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package admissionpolicymanager + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + authenticationv1 "k8s.io/api/authentication/v1" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + "go.goms.io/fleet/pkg/utils" +) + +const ( + // The audience of the service account tokens that the test cases request. + testAudience = "kubefleet-integration-test" +) + +func newServiceAccount(namespace, name string) *corev1.ServiceAccount { + return &corev1.ServiceAccount{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + } +} + +// requestToken requests a token for the given service account with the given client (i.e., via the TokenRequest +// API); it returns an error if the request fails or no token has been issued. +func requestToken(c client.Client, namespace, name string) error { + tokenRequest := &authenticationv1.TokenRequest{ + Spec: authenticationv1.TokenRequestSpec{ + Audiences: []string{testAudience}, + }, + } + if err := c.SubResource("token").Create(ctx, newServiceAccount(namespace, name), tokenRequest); err != nil { + return err + } + if len(tokenRequest.Status.Token) == 0 { + return fmt.Errorf("no token has been issued for service account %s/%s", namespace, name) + } + return nil +} + +var _ = Describe("deny service accounts and token requests in reserved namespaces", Ordered, func() { + suffix := utils.RandStr() + workNSName := fmt.Sprintf("work-%s", suffix) + fleetNSName := fmt.Sprintf("fleet-%s", suffix) + + // The users that the test cases impersonate; all of them can write service accounts and request tokens + // (as far as RBAC is concerned). + // + // A user that is not exempt from the admission policy. + plainUsername := fmt.Sprintf("plain-%s", suffix) + // A user that is not whitelisted, but whose name starts with the whitelisted username; it verifies that + // usernames are matched exactly. + notWhitelistedUsername := fmt.Sprintf("%s-2", whitelistedUsername) + // Users that are exempt from the admission policy by their user groups. + mastersMemberUsername := fmt.Sprintf("masters-member-%s", suffix) + nodeUsername := fmt.Sprintf("system:node:node-%s", suffix) + kubeadmAdminUsername := fmt.Sprintf("kubeadm-admin-%s", suffix) + svcAccountsMemberUsername := fmt.Sprintf("serviceaccounts-member-%s", suffix) + whitelistedGroupMemberUsername := fmt.Sprintf("whitelisted-group-member-%s", suffix) + + svcAccountWriterRBACName := fmt.Sprintf("serviceaccount-writer-%s", suffix) + // The service accounts that the cluster admin creates (in both namespaces) for the test cases to write and + // request tokens for. + seededSvcAccountName := fmt.Sprintf("seeded-%s", suffix) + + var plainClient client.Client + + BeforeAll(func() { + createNamespace(workNSName) + createNamespace(fleetNSName) + + grantClusterWidePermissions(svcAccountWriterRBACName, []rbacv1.PolicyRule{ + { + APIGroups: []string{""}, + Resources: []string{"serviceaccounts"}, + Verbs: []string{"get", authzVerbCreate, authzVerbUpdate, "delete"}, + }, + { + APIGroups: []string{""}, + Resources: []string{"serviceaccounts/token"}, + Verbs: []string{authzVerbCreate}, + }, + }, + userSubject(plainUsername), + userSubject(notWhitelistedUsername), + userSubject("system:kube-scheduler"), + userSubject("system:kube-controller-manager"), + userSubject(whitelistedUsername), + // Members of system:masters can do anything, and need no additional permissions. + groupSubject("system:nodes"), + groupSubject("kubeadm:cluster-admins"), + groupSubject("system:serviceaccounts"), + groupSubject(whitelistedUserGroup), + ) + + plainClient = impersonatedClient(plainUsername) + + // The cluster admin is exempt from the admission policy. + Expect(hubUncachedClient.Create(ctx, newServiceAccount(fleetNSName, seededSvcAccountName))).To(Succeed(), "Failed to create a service account in the reserved namespace") + Expect(hubUncachedClient.Create(ctx, newServiceAccount(workNSName, seededSvcAccountName))).To(Succeed(), "Failed to create a service account in the non-reserved namespace") + + By("waiting for the admission policy to take effect") + waitUntilPolicyIsEnforced(func() error { + return plainClient.Create(ctx, newServiceAccount(fleetNSName, fmt.Sprintf("probe-%s", utils.RandStr()))) + }, svcAccountsAndTokenRequestsVAPPolicyName, wantSvcAccountsAndTokenRequestsDenialMsg) + }) + + It("should deny creating service accounts in a fleet- prefixed namespace", func() { + err := plainClient.Create(ctx, newServiceAccount(fleetNSName, fmt.Sprintf("sa-%s", utils.RandStr()))) + Expect(wantDeniedByPolicy(err, svcAccountsAndTokenRequestsVAPPolicyName, wantSvcAccountsAndTokenRequestsDenialMsg)).To(Succeed()) + }) + + It("should deny creating service accounts in a kube- prefixed namespace", func() { + err := plainClient.Create(ctx, newServiceAccount(kubeSystemNSName, fmt.Sprintf("sa-%s", utils.RandStr()))) + Expect(wantDeniedByPolicy(err, svcAccountsAndTokenRequestsVAPPolicyName, wantSvcAccountsAndTokenRequestsDenialMsg)).To(Succeed()) + }) + + It("should deny updating service accounts in a reserved namespace", func() { + err := getAndUpdate(plainClient, newServiceAccount(fleetNSName, seededSvcAccountName), func(sa *corev1.ServiceAccount) { + sa.Labels = map[string]string{testDataKey: testDataValue} + }) + Expect(wantDeniedByPolicy(err, svcAccountsAndTokenRequestsVAPPolicyName, wantSvcAccountsAndTokenRequestsDenialMsg)).To(Succeed()) + }) + + It("should deny deleting service accounts in a reserved namespace", func() { + err := plainClient.Delete(ctx, newServiceAccount(fleetNSName, seededSvcAccountName)) + Expect(wantDeniedByPolicy(err, svcAccountsAndTokenRequestsVAPPolicyName, wantSvcAccountsAndTokenRequestsDenialMsg)).To(Succeed()) + }) + + It("should deny requesting tokens for service accounts in a reserved namespace", func() { + err := requestToken(plainClient, fleetNSName, seededSvcAccountName) + Expect(wantDeniedByPolicy(err, svcAccountsAndTokenRequestsVAPPolicyName, wantSvcAccountsAndTokenRequestsDenialMsg)).To(Succeed()) + }) + + It("should allow creating, updating, and deleting service accounts in a non-reserved namespace", func() { + sa := newServiceAccount(workNSName, fmt.Sprintf("sa-%s", utils.RandStr())) + Expect(plainClient.Create(ctx, sa)).To(Succeed(), "Failed to create a service account") + + err := getAndUpdate(plainClient, sa, func(sa *corev1.ServiceAccount) { + sa.Labels = map[string]string{testDataKey: testDataValue} + }) + Expect(err).To(Succeed(), "Failed to update the service account") + + Expect(plainClient.Delete(ctx, sa)).To(Succeed(), "Failed to delete the service account") + }) + + It("should allow requesting tokens for service accounts in a non-reserved namespace", func() { + Expect(requestToken(plainClient, workNSName, seededSvcAccountName)).To(Succeed(), "Failed to request a token") + }) + + DescribeTable("should allow exempt callers to write service accounts in a reserved namespace and request tokens for them", + func(username string, userGroups []string) { + c := impersonatedClient(username, userGroups...) + + saName := fmt.Sprintf("sa-%s", utils.RandStr()) + Expect(c.Create(ctx, newServiceAccount(fleetNSName, saName))).To(Succeed(), "Failed to create a service account") + Expect(requestToken(c, fleetNSName, saName)).To(Succeed(), "Failed to request a token") + }, + Entry("members of system:masters (cluster admins)", mastersMemberUsername, []string{"system:masters"}), + Entry("the Kubernetes scheduler", "system:kube-scheduler", []string{}), + Entry("the Kubernetes controller manager", "system:kube-controller-manager", []string{}), + Entry("members of system:nodes (nodes)", nodeUsername, []string{"system:nodes"}), + Entry("members of kubeadm:cluster-admins (kubeadm cluster admins)", kubeadmAdminUsername, []string{"kubeadm:cluster-admins"}), + Entry("members of system:serviceaccounts (service accounts)", svcAccountsMemberUsername, []string{"system:serviceaccounts"}), + Entry("whitelisted users", whitelistedUsername, []string{}), + Entry("members of whitelisted user groups", whitelistedGroupMemberUsername, []string{whitelistedUserGroup}), + ) + + It("should deny creating service accounts in a reserved namespace for users whose names only start with a whitelisted username", func() { + err := impersonatedClient(notWhitelistedUsername).Create(ctx, newServiceAccount(fleetNSName, fmt.Sprintf("sa-%s", utils.RandStr()))) + Expect(wantDeniedByPolicy(err, svcAccountsAndTokenRequestsVAPPolicyName, wantSvcAccountsAndTokenRequestsDenialMsg)).To(Succeed()) + }) + + AfterAll(func() { + revokeClusterWidePermissions(svcAccountWriterRBACName) + }) +}) diff --git a/pkg/admissionpolicymanager/utils_integration_test.go b/pkg/admissionpolicymanager/utils_integration_test.go new file mode 100644 index 000000000..e9f737eda --- /dev/null +++ b/pkg/admissionpolicymanager/utils_integration_test.go @@ -0,0 +1,251 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package admissionpolicymanager + +import ( + "fmt" + "strings" + + . "github.com/onsi/gomega" + authorizationv1 "k8s.io/api/authorization/v1" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +// The messages that the admission policies report when they deny a request. +const ( + wantPodsAndReplicaSetsDenialMsg = "creating pods and replicas is disallowed in the fleet hub cluster" + wantSvcAccountsAndTokenRequestsDenialMsg = "writing service accounts in reserved namespaces or requesting tokens from such service accounts is disallowed" + wantNamespacedObjDenialMsg = "adding or changing the kubefleet.dev/cluster-selectors annotation on a namespaced object requires the permissions to create and update placementpolicies.placement.kubefleet.dev in the namespace of the object" + wantClusterScopedObjDenialMsg = "adding or changing the kubefleet.dev/cluster-selectors annotation on a cluster-scoped object requires the permissions to create and update clusterplacementpolicies.placement.kubefleet.dev" + wantReservedNSAnnotationDenialMsg = "adding or changing the kubefleet.dev/cluster-selectors annotation on reserved namespaces or objects in them is disallowed (reserved namespace prefixes: fleet-, kube-)" + wantEndpointsAnnotationDenialMsg = "adding or changing the kubefleet.dev/cluster-selectors annotation on Endpoints objects is disallowed; Endpoints cannot be placed, and the annotation would be copied onto the EndpointSlices mirrored from them" +) + +// A finalizer that test cases add to objects, so that the objects stay around (with a deletion timestamp set) +// after being deleted. +const testFinalizer = "kubefleet.dev/integration-test" + +// impersonatedClient returns a client that sends requests as the given user, who is a member of the given user +// groups (plus system:authenticated, which the API server adds for impersonated users). +// +// Note that the API server checks if the impersonated user is authorized to make a request (RBAC) before it +// runs any admission policy against the request. +func impersonatedClient(username string, userGroups ...string) client.Client { + impersonatedCfg := rest.CopyConfig(cfg) + impersonatedCfg.Impersonate = rest.ImpersonationConfig{ + UserName: username, + Groups: userGroups, + } + c, err := client.New(impersonatedCfg, client.Options{Scheme: scheme.Scheme}) + Expect(err).ToNot(HaveOccurred(), "Failed to create a client that impersonates user %s", username) + return c +} + +func userSubject(username string) rbacv1.Subject { + return rbacv1.Subject{APIGroup: rbacv1.GroupName, Kind: rbacv1.UserKind, Name: username} +} + +func groupSubject(userGroup string) rbacv1.Subject { + return rbacv1.Subject{APIGroup: rbacv1.GroupName, Kind: rbacv1.GroupKind, Name: userGroup} +} + +// getAndUpdate retrieves the given object with the given client, applies the given mutation to it, and updates +// it with the same client. +func getAndUpdate[T client.Object](c client.Client, obj T, mutate func(T)) error { + if err := c.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil { + return fmt.Errorf("failed to get object %s: %w", client.ObjectKeyFromObject(obj), err) + } + mutate(obj) + return c.Update(ctx, obj) +} + +// getAndUpdateSubResource retrieves the given object with the given client, applies the given mutation to it, and +// sends the whole object to the given subresource (e.g., status) of the object with the same client. +func getAndUpdateSubResource[T client.Object](c client.Client, obj T, subResource string, mutate func(T)) error { + if err := c.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil { + return fmt.Errorf("failed to get object %s: %w", client.ObjectKeyFromObject(obj), err) + } + mutate(obj) + return c.SubResource(subResource).Update(ctx, obj) +} + +// grantClusterWidePermissions creates a cluster role with the given rules, and binds it to the given subjects +// with a cluster role binding of the same name. It returns after the permissions have taken effect. +func grantClusterWidePermissions(name string, rules []rbacv1.PolicyRule, subjects ...rbacv1.Subject) { + clusterRole := &rbacv1.ClusterRole{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + }, + Rules: rules, + } + Expect(hubUncachedClient.Create(ctx, clusterRole)).To(Succeed(), "Failed to create cluster role %s", name) + + clusterRoleBinding := &rbacv1.ClusterRoleBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + }, + RoleRef: rbacv1.RoleRef{ + APIGroup: rbacv1.GroupName, + Kind: "ClusterRole", + Name: name, + }, + Subjects: subjects, + } + Expect(hubUncachedClient.Create(ctx, clusterRoleBinding)).To(Succeed(), "Failed to create cluster role binding %s", name) + + waitUntilPermissionsTakeEffect("", rules, subjects) +} + +// revokeClusterWidePermissions deletes the cluster role and cluster role binding that +// grantClusterWidePermissions creates. +func revokeClusterWidePermissions(name string) { + clusterRoleBinding := &rbacv1.ClusterRoleBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + }, + } + Expect(client.IgnoreNotFound(hubUncachedClient.Delete(ctx, clusterRoleBinding))).To(Succeed(), "Failed to delete cluster role binding %s", name) + + clusterRole := &rbacv1.ClusterRole{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + }, + } + Expect(client.IgnoreNotFound(hubUncachedClient.Delete(ctx, clusterRole))).To(Succeed(), "Failed to delete cluster role %s", name) +} + +// grantPermissionsInNamespace creates a role with the given rules in the given namespace, and binds it to the +// given subjects with a role binding of the same name. It returns after the permissions have taken effect. +func grantPermissionsInNamespace(namespace, name string, rules []rbacv1.PolicyRule, subjects ...rbacv1.Subject) { + role := &rbacv1.Role{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + Rules: rules, + } + Expect(hubUncachedClient.Create(ctx, role)).To(Succeed(), "Failed to create role %s/%s", namespace, name) + + roleBinding := &rbacv1.RoleBinding{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + }, + RoleRef: rbacv1.RoleRef{ + APIGroup: rbacv1.GroupName, + Kind: "Role", + Name: name, + }, + Subjects: subjects, + } + Expect(hubUncachedClient.Create(ctx, roleBinding)).To(Succeed(), "Failed to create role binding %s/%s", namespace, name) + + waitUntilPermissionsTakeEffect(namespace, rules, subjects) +} + +// waitUntilPermissionsTakeEffect waits until the API server authorizes each of the given subjects to perform the +// first verb on the first resource of the first API group of each given rule (in the given namespace, or across +// all namespaces if the namespace is empty). +// +// The RBAC authorizer reads roles and bindings from its informer caches, so that newly created ones might not take +// effect immediately; one check per rule is sufficient, as a role (or cluster role) and its binding become visible +// to the authorizer as a whole. +func waitUntilPermissionsTakeEffect(namespace string, rules []rbacv1.PolicyRule, subjects []rbacv1.Subject) { + for _, subject := range subjects { + for _, rule := range rules { + resource, subresource, _ := strings.Cut(rule.Resources[0], "/") + sarSpec := authorizationv1.SubjectAccessReviewSpec{ + ResourceAttributes: &authorizationv1.ResourceAttributes{ + Namespace: namespace, + Verb: rule.Verbs[0], + Group: rule.APIGroups[0], + Resource: resource, + Subresource: subresource, + }, + } + switch subject.Kind { + case rbacv1.UserKind: + sarSpec.User = subject.Name + case rbacv1.GroupKind: + sarSpec.Groups = []string{subject.Name} + } + + Eventually(func() error { + sar := &authorizationv1.SubjectAccessReview{ + Spec: *sarSpec.DeepCopy(), + } + if err := hubUncachedClient.Create(ctx, sar); err != nil { + return fmt.Errorf("failed to create subject access review: %w", err) + } + if !sar.Status.Allowed { + return fmt.Errorf("%s %s is not yet allowed to %s %s (subresource %q, API group %q) in namespace %q", + subject.Kind, subject.Name, rule.Verbs[0], resource, subresource, rule.APIGroups[0], namespace) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Permissions have not taken effect") + } + } +} + +// wantDeniedByPolicy returns an error if the given error does not report that a request has been denied by the +// given validating admission policy with the given message. +// +// As an RBAC denial is also reported as a Forbidden error, the check looks for the policy name and the message in +// the error, so that a request denied for any other reason will not pass as a policy denial. +func wantDeniedByPolicy(err error, policyName, wantMsg string) error { + if err == nil { + return fmt.Errorf("request is allowed, want it to be denied by validating admission policy %s", policyName) + } + if !apierrors.IsForbidden(err) { + return fmt.Errorf("request has failed with a non-Forbidden error, want it to be denied by validating admission policy %s: %w", policyName, err) + } + errMsg := err.Error() + if !strings.Contains(errMsg, fmt.Sprintf("ValidatingAdmissionPolicy '%s'", policyName)) || !strings.Contains(errMsg, wantMsg) { + return fmt.Errorf("request has been denied for an unexpected reason, want it to be denied by validating admission policy %s with message %q: %w", policyName, wantMsg, err) + } + return nil +} + +// waitUntilPolicyIsEnforced waits until the given probe request is denied by the given validating admission policy +// with the given message. +// +// The API server picks up new (or updated) validating admission policies and their bindings periodically (about +// every second); requests sent right after a policy is created might not be checked against it. +func waitUntilPolicyIsEnforced(probe func() error, policyName, wantMsg string) { + Eventually(func() error { + return wantDeniedByPolicy(probe(), policyName, wantMsg) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Validating admission policy %s has not taken effect", policyName) +} + +// createNamespace creates a namespace with the given name. +// +// Note: due to restrictions of the test environment (no namespace controller is running), namespaces cannot be +// deleted; namespaces created by the test cases are not cleaned up. +func createNamespace(name string) { + ns := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + }, + } + Expect(hubUncachedClient.Create(ctx, ns)).To(Succeed(), "Failed to create namespace %s", name) +} diff --git a/pkg/controllers/membercluster/v1beta1/membercluster_controller.go b/pkg/controllers/membercluster/v1beta1/membercluster_controller.go index c34bed2a6..14924e5f2 100644 --- a/pkg/controllers/membercluster/v1beta1/membercluster_controller.go +++ b/pkg/controllers/membercluster/v1beta1/membercluster_controller.go @@ -47,6 +47,7 @@ import ( "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/condition" "go.goms.io/fleet/pkg/utils/controller" + "go.goms.io/fleet/pkg/utils/labels" ) const ( @@ -75,6 +76,8 @@ type Reconciler struct { MaxConcurrentReconciles int // the wait time in minutes before we force delete a member cluster. ForceDeleteWaitTime time.Duration + // Grant the member agent access to the Works of the experimental placement policy APIs or not. + PlacementPolicyAPIsEnabled bool // agents are used as hashset to query the expected agent type, so the value will be ignored. agents map[clusterv1beta1.AgentType]bool } @@ -282,6 +285,8 @@ func (r *Reconciler) ensureFinalizer(ctx context.Context, mc *clusterv1beta1.Mem // ensureMemberNameLabel makes sure that the member cluster has a label with its own name. // This enables selecting clusters by name in ResourceOverride and ClusterResourceOverride via labelSelector. +// It also seeds the cluster alias label and, if the fleet.azure.com/location label is present, the +// topology.kubernetes.io/region label, which the region shorthand of annotation-based placement selects on. func (r *Reconciler) ensureMemberNameLabel(ctx context.Context, mc *clusterv1beta1.MemberCluster) error { changed := false if mc.Labels == nil { @@ -302,10 +307,21 @@ func (r *Reconciler) ensureMemberNameLabel(ctx context.Context, mc *clusterv1bet changed = true } + // Azure-specific change: add the topology.kubernetes.io/region label using the same value + // as the fleet.azure.com/location label if it exists. Like the alias label above, the region + // label is seeded only when it is absent, so that a value set by an admin (or any other + // writer) is never overwritten. + if location, found := mc.Labels[labels.AzureLocationLabel]; found && location != "" { + if _, found := mc.Labels[corev1.LabelTopologyRegion]; !found { + mc.Labels[corev1.LabelTopologyRegion] = location + changed = true + } + } + if !changed { return nil } - klog.InfoS("Ensured the member cluster name and alias labels", "memberCluster", klog.KObj(mc)) + klog.InfoS("Ensured the member cluster name, alias, and region labels", "memberCluster", klog.KObj(mc)) return r.Update(ctx, mc, client.FieldOwner(utils.MCControllerFieldManagerName)) } @@ -414,6 +430,9 @@ func (r *Reconciler) syncRole(ctx context.Context, mc *clusterv1beta1.MemberClus }, Rules: []rbacv1.PolicyRule{utils.FleetClusterRule, utils.FleetPlacementRule, utils.FleetNetworkRule, utils.EventRule, utils.EventsK8sIoRule}, } + if r.PlacementPolicyAPIsEnabled { + expectedRole.Rules = append(expectedRole.Rules, utils.FleetPlacementPolicyWorkRule) + } // Creates role if not found. var currentRole rbacv1.Role diff --git a/pkg/controllers/membercluster/v1beta1/membercluster_controller_integration_test.go b/pkg/controllers/membercluster/v1beta1/membercluster_controller_integration_test.go index 17d194eba..d39bb7fb6 100644 --- a/pkg/controllers/membercluster/v1beta1/membercluster_controller_integration_test.go +++ b/pkg/controllers/membercluster/v1beta1/membercluster_controller_integration_test.go @@ -36,6 +36,7 @@ import ( placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/condition" + "go.goms.io/fleet/pkg/utils/labels" ) const ( @@ -711,6 +712,113 @@ var _ = Describe("Test MemberCluster Controller", func() { Expect(ns.DeletionTimestamp != nil).Should(BeTrue()) }) }) + + // Azure-specific: the controller seeds the topology.kubernetes.io/region label from the + // fleet.azure.com/location label, but never overwrites an existing region label. + Context("Test membercluster controller with the Azure location label", func() { + const ( + location = "eastus" + adminRegion = "us-east" + ) + + reconcileAndExpectSuccess := func() { + result, err := r.Reconcile(ctx, ctrl.Request{ + NamespacedName: memberClusterNamespacedName, + }) + Expect(result).Should(Equal(ctrl.Result{})) + Expect(err).Should(Succeed()) + } + + BeforeEach(func() { + ctx = context.Background() + memberClusterName = utils.RandStr() + namespaceName = fmt.Sprintf(utils.NamespaceNameFormat, memberClusterName) + memberClusterNamespacedName = types.NamespacedName{ + Name: memberClusterName, + } + + By("create the member cluster reconciler") + r = &Reconciler{ + Client: k8sClient, + ForceDeleteWaitTime: 15 * time.Minute, + } + Expect(r.SetupWithManager(mgr, memberClusterName+"-controller")).Should(Succeed()) + + By("create member cluster with the Azure location label for join") + mc := buildMemberCluster(memberClusterName) + mc.Labels = map[string]string{ + labels.AzureLocationLabel: location, + } + Expect(k8sClient.Create(ctx, &mc)).Should(Succeed()) + + By("trigger reconcile to initiate the join workflow") + reconcileAndExpectSuccess() + + checkIfMemberClusterResourcesExistsAndUpdateAgentStatusToTrue(ctx, memberClusterName, namespaceName) + + By("trigger reconcile again to update member cluster status to joined") + reconcileAndExpectSuccess() + }) + + AfterEach(func() { + var ns corev1.Namespace + Expect(k8sClient.Get(ctx, types.NamespacedName{Name: namespaceName}, &ns)).Should(Succeed()) + + By("Deleting the namespace") + Eventually(func() error { + return k8sClient.Delete(ctx, &ns) + }, timeout, interval).Should(SatisfyAny(Succeed(), &utils.NotFoundMatcher{})) + }) + + It("should seed the region label from the location label", func() { + Expect(k8sClient.Get(ctx, memberClusterNamespacedName, &mc)).Should(Succeed()) + wantLabels := map[string]string{ + placementv1beta1.MemberNameLabel: memberClusterName, + placementv1beta1.ClusterAliasLabel: memberClusterName, + labels.AzureLocationLabel: location, + corev1.LabelTopologyRegion: location, + } + Expect(cmp.Diff(wantLabels, mc.Labels)).Should(BeEmpty(), "member cluster labels mismatch (-want +got)") + }) + + It("should not overwrite a region label set by an admin", func() { + Expect(k8sClient.Get(ctx, memberClusterNamespacedName, &mc)).Should(Succeed()) + mc.Labels[corev1.LabelTopologyRegion] = adminRegion + Expect(k8sClient.Update(ctx, &mc)).Should(Succeed()) + + By("trigger reconcile") + reconcileAndExpectSuccess() + + Expect(k8sClient.Get(ctx, memberClusterNamespacedName, &mc)).Should(Succeed()) + Expect(mc.Labels[corev1.LabelTopologyRegion]).To(Equal(adminRegion)) + Expect(mc.Labels[labels.AzureLocationLabel]).To(Equal(location)) + }) + + It("should seed the region label again if it is removed", func() { + Expect(k8sClient.Get(ctx, memberClusterNamespacedName, &mc)).Should(Succeed()) + delete(mc.Labels, corev1.LabelTopologyRegion) + Expect(k8sClient.Update(ctx, &mc)).Should(Succeed()) + + By("trigger reconcile to seed the region label again") + reconcileAndExpectSuccess() + + Expect(k8sClient.Get(ctx, memberClusterNamespacedName, &mc)).Should(Succeed()) + Expect(mc.Labels[corev1.LabelTopologyRegion]).To(Equal(location)) + }) + + It("should not add the region label if the location label is absent", func() { + Expect(k8sClient.Get(ctx, memberClusterNamespacedName, &mc)).Should(Succeed()) + delete(mc.Labels, labels.AzureLocationLabel) + delete(mc.Labels, corev1.LabelTopologyRegion) + Expect(k8sClient.Update(ctx, &mc)).Should(Succeed()) + + By("trigger reconcile") + reconcileAndExpectSuccess() + + Expect(k8sClient.Get(ctx, memberClusterNamespacedName, &mc)).Should(Succeed()) + Expect(mc.Labels).NotTo(HaveKey(corev1.LabelTopologyRegion)) + }) + }) }) func buildMemberCluster(name string) clusterv1beta1.MemberCluster { diff --git a/pkg/controllers/membercluster/v1beta1/membercluster_controller_test.go b/pkg/controllers/membercluster/v1beta1/membercluster_controller_test.go index 032c63f41..58e97c370 100644 --- a/pkg/controllers/membercluster/v1beta1/membercluster_controller_test.go +++ b/pkg/controllers/membercluster/v1beta1/membercluster_controller_test.go @@ -41,6 +41,7 @@ import ( placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/controller" + "go.goms.io/fleet/pkg/utils/labels" ) const ( @@ -197,6 +198,144 @@ func TestEnsureMemberNameLabel(t *testing.T) { placementv1beta1.ClusterAliasLabel: "mc1", }, }, + "location label present, region label absent: region is seeded from location": { + r: &Reconciler{ + Client: &test.MockClient{ + MockUpdate: func(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error { + return nil + }, + }, + }, + memberCluster: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "mc1", + Labels: map[string]string{ + labels.AzureLocationLabel: "eastus", + }, + }, + }, + wantLabels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + labels.AzureLocationLabel: "eastus", + corev1.LabelTopologyRegion: "eastus", + }, + }, + // The day-2 scenario for the region label: a member cluster labeled by the controller + // before the region label was seeded. Only the region branch has anything to do, and it + // alone must drive the update; the update failure below proves that the update is issued. + "name and alias labels correct, region label absent: region alone drives the update": { + r: &Reconciler{ + Client: &test.MockClient{ + MockUpdate: test.NewMockUpdateFn(errors.New("update failed")), + }, + }, + memberCluster: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "mc1", + Labels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + labels.AzureLocationLabel: "eastus", + }, + }, + }, + wantErr: "update failed", + }, + // Like the alias, the region label is seeded rather than reasserted: a value set by an + // admin (or any other writer) is left alone, even if it differs from the location. + "region label already present with a different value is not overwritten": { + r: &Reconciler{ + Client: &test.MockClient{ + MockUpdate: test.NewMockUpdateFn(fmt.Errorf("update should not be called when the region label is already present")), + }, + }, + memberCluster: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "mc1", + Labels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + labels.AzureLocationLabel: "eastus", + corev1.LabelTopologyRegion: "us-east", + }, + }, + }, + wantLabels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + labels.AzureLocationLabel: "eastus", + corev1.LabelTopologyRegion: "us-east", + }, + }, + "region label already matches the location label": { + r: &Reconciler{ + Client: &test.MockClient{ + MockUpdate: test.NewMockUpdateFn(fmt.Errorf("update should not be called when the labels are already correct")), + }, + }, + memberCluster: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "mc1", + Labels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + labels.AzureLocationLabel: "eastus", + corev1.LabelTopologyRegion: "eastus", + }, + }, + }, + wantLabels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + labels.AzureLocationLabel: "eastus", + corev1.LabelTopologyRegion: "eastus", + }, + }, + "location label present but empty: region label is not seeded": { + r: &Reconciler{ + Client: &test.MockClient{ + MockUpdate: test.NewMockUpdateFn(fmt.Errorf("update should not be called when the location label is empty")), + }, + }, + memberCluster: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "mc1", + Labels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + labels.AzureLocationLabel: "", + }, + }, + }, + wantLabels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + labels.AzureLocationLabel: "", + }, + }, + "location label absent: an existing region label is left alone": { + r: &Reconciler{ + Client: &test.MockClient{ + MockUpdate: test.NewMockUpdateFn(fmt.Errorf("update should not be called when the location label is absent")), + }, + }, + memberCluster: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "mc1", + Labels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + corev1.LabelTopologyRegion: "us-east", + }, + }, + }, + wantLabels: map[string]string{ + placementv1beta1.MemberNameLabel: "mc1", + placementv1beta1.ClusterAliasLabel: "mc1", + corev1.LabelTopologyRegion: "us-east", + }, + }, "update error": { r: &Reconciler{ Client: &test.MockClient{ @@ -2031,3 +2170,43 @@ func TestHandleDelete(t *testing.T) { }) } } + +func TestSyncRolePlacementPolicyWorkRule(t *testing.T) { + baseRules := []rbacv1.PolicyRule{utils.FleetClusterRule, utils.FleetPlacementRule, utils.FleetNetworkRule, utils.EventRule, utils.EventsK8sIoRule} + tests := map[string]struct { + enabled bool + wantRules []rbacv1.PolicyRule + }{ + "placement policy APIs disabled": { + wantRules: baseRules, + }, + "placement policy APIs enabled": { + enabled: true, + wantRules: append(append([]rbacv1.PolicyRule{}, baseRules...), utils.FleetPlacementPolicyWorkRule), + }, + } + for testName, tt := range tests { + t.Run(testName, func(t *testing.T) { + var created *rbacv1.Role + r := &Reconciler{ + Client: &test.MockClient{ + MockGet: func(ctx context.Context, key client.ObjectKey, obj client.Object) error { + return apierrors.NewNotFound(schema.GroupResource{Resource: "roles"}, key.Name) + }, + MockCreate: func(ctx context.Context, obj client.Object, opts ...client.CreateOption) error { + created = obj.(*rbacv1.Role) + return nil + }, + }, + recorder: events.NewFakeRecorder(1), + PlacementPolicyAPIsEnabled: tt.enabled, + } + if _, err := r.syncRole(context.Background(), &clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: "mc1"}}, "fleet-member-mc1"); err != nil { + t.Fatalf("syncRole() = %v, want nil", err) + } + if diff := cmp.Diff(created.Rules, tt.wantRules); diff != "" { + t.Errorf("syncRole() created role rules diff (-got, +want):\n%s", diff) + } + }) + } +} diff --git a/pkg/controllers/placement/controller.go b/pkg/controllers/placement/controller.go index c051758c9..2403fbd1a 100644 --- a/pkg/controllers/placement/controller.go +++ b/pkg/controllers/placement/controller.go @@ -868,16 +868,19 @@ func emitPlacementStatusMetric(placementObj fleetv1beta1.PlacementObj) { // Check Placement Scheduled condition. status := "nil" reason := "nil" + metricTimestamp := placementObj.GetCreationTimestamp() scheduledConditionType := getPlacementScheduledConditionType(placementObj) cond := placementObj.GetCondition(scheduledConditionType) if !condition.IsConditionStatusTrue(cond, placementObj.GetGeneration()) { if cond != nil && cond.ObservedGeneration == placementObj.GetGeneration() { status = string(cond.Status) reason = cond.Reason + metricTimestamp = cond.LastTransitionTime } - hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), scheduledConditionType, status, reason).SetToCurrentTime() + hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), scheduledConditionType, status, reason).Set(float64(metricTimestamp.UnixNano()) / float64(time.Second)) return } + completionTimestamp := cond.LastTransitionTime // Check placement expected conditions. expectedCondTypes := determineExpectedPlacementAndResourcePlacementStatusCondType(placementObj) @@ -888,13 +891,16 @@ func emitPlacementStatusMetric(placementObj fleetv1beta1.PlacementObj) { if cond != nil && cond.ObservedGeneration == placementObj.GetGeneration() { status = string(cond.Status) reason = cond.Reason + metricTimestamp = cond.LastTransitionTime } - hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), conditionType, status, reason).SetToCurrentTime() + hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), conditionType, status, reason).Set(float64(metricTimestamp.UnixNano()) / float64(time.Second)) return } + completionTimestamp = cond.LastTransitionTime } // Emit the "Completed" condition metric to indicate that the placement has completed. - // This condition is used solely for metric reporting purposes. - hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), "Completed", string(metav1.ConditionTrue), "Completed").SetToCurrentTime() + // Completed is a synthetic metric label, not a persisted condition. Its timestamp is the + // transition time of the final real condition that established completion. + hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues(placementObj.GetNamespace(), placementObj.GetName(), strconv.FormatInt(placementObj.GetGeneration(), 10), "Completed", string(metav1.ConditionTrue), "Completed").Set(float64(completionTimestamp.UnixNano()) / float64(time.Second)) } diff --git a/pkg/controllers/placement/controller_integration_test.go b/pkg/controllers/placement/controller_integration_test.go index 20e3d3b33..01868ba52 100644 --- a/pkg/controllers/placement/controller_integration_test.go +++ b/pkg/controllers/placement/controller_integration_test.go @@ -17,7 +17,9 @@ package placement import ( "fmt" + "sort" "strconv" + "strings" "time" "github.com/google/go-cmp/cmp" @@ -40,7 +42,6 @@ import ( "go.goms.io/fleet/pkg/utils/condition" "go.goms.io/fleet/pkg/utils/resource" "go.goms.io/fleet/test/utils/crpstatussync" - metricsUtils "go.goms.io/fleet/test/utils/metrics" ) const ( @@ -2346,6 +2347,34 @@ func checkPlacementStatusMetric(wantMetrics []*prometheusclientmodel.Metric) { placementStatusMetrics = mf.GetMetric() } } - // Sort the emitted metrics for comparison - Expect(cmp.Diff(placementStatusMetrics, wantMetrics, metricsUtils.MetricsCmpOptions...)).Should(BeEmpty(), "Placement status metrics do not match diff (-got, +want):") + // Persisted condition and creation timestamps can legitimately be identical, so compare + // placement time series by their stable label identity rather than scrape order. + sortPlacementStatusMetricsByLabels(placementStatusMetrics) + sortPlacementStatusMetricsByLabels(wantMetrics) + cmpOptions := []cmp.Option{ + cmpopts.SortSlices(func(a, b *prometheusclientmodel.LabelPair) bool { + return a.GetName() < b.GetName() + }), + cmp.Comparer(func(a, b *prometheusclientmodel.Gauge) bool { + return (a.GetValue() > 0) == (b.GetValue() > 0) + }), + cmpopts.IgnoreUnexported(prometheusclientmodel.Metric{}, prometheusclientmodel.LabelPair{}, prometheusclientmodel.Gauge{}), + } + Expect(cmp.Diff(placementStatusMetrics, wantMetrics, cmpOptions...)).Should(BeEmpty(), "Placement status metrics do not match diff (-got, +want):") +} + +func sortPlacementStatusMetricsByLabels(metrics []*prometheusclientmodel.Metric) { + sort.Slice(metrics, func(i, j int) bool { + return placementStatusMetricLabelKey(metrics[i]) < placementStatusMetricLabelKey(metrics[j]) + }) +} + +func placementStatusMetricLabelKey(metric *prometheusclientmodel.Metric) string { + labels := metric.GetLabel() + pairs := make([]string, 0, len(labels)) + for _, label := range labels { + pairs = append(pairs, label.GetName()+"="+label.GetValue()) + } + sort.Strings(pairs) + return strings.Join(pairs, ";") } diff --git a/pkg/controllers/placement/controller_test.go b/pkg/controllers/placement/controller_test.go index a175ccd11..3860a7b20 100644 --- a/pkg/controllers/placement/controller_test.go +++ b/pkg/controllers/placement/controller_test.go @@ -27,6 +27,7 @@ import ( "github.com/google/go-cmp/cmp" "github.com/google/go-cmp/cmp/cmpopts" + "github.com/prometheus/client_golang/prometheus/testutil" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -39,6 +40,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client/fake" fleetv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + hubmetrics "go.goms.io/fleet/pkg/metrics/hub" "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/condition" "go.goms.io/fleet/pkg/utils/controller" @@ -119,6 +121,175 @@ func clusterResourcePlacementForTest() *fleetv1beta1.ClusterResourcePlacement { } } +func TestEmitPlacementStatusMetric(t *testing.T) { + generation := int64(3) + creationTime := metav1.NewTime(time.Date(2026, time.September, 1, 2, 3, 4, 123456789, time.UTC)) + scheduledTime := metav1.NewTime(time.Date(2026, time.September, 2, 3, 4, 5, 234567890, time.UTC)) + rolloutTime := metav1.NewTime(time.Date(2026, time.September, 3, 4, 5, 6, 345678901, time.UTC)) + overriddenTime := metav1.NewTime(time.Date(2026, time.September, 4, 5, 6, 7, 456789012, time.UTC)) + synchronizedTime := metav1.NewTime(time.Date(2026, time.September, 5, 6, 7, 8, 567890123, time.UTC)) + appliedTime := metav1.NewTime(time.Date(2026, time.September, 6, 7, 8, 9, 678901234, time.UTC)) + availableTime := metav1.NewTime(time.Date(2026, time.September, 7, 8, 9, 10, 789012345, time.UTC)) + + newPlacement := func(clusterScoped bool, conditions ...metav1.Condition) fleetv1beta1.PlacementObj { + objectMeta := metav1.ObjectMeta{ + Name: "test-placement", + Generation: generation, + CreationTimestamp: creationTime, + } + if clusterScoped { + return &fleetv1beta1.ClusterResourcePlacement{ + ObjectMeta: objectMeta, + Status: fleetv1beta1.PlacementStatus{ + Conditions: conditions, + }, + } + } + objectMeta.Namespace = "test-namespace" + return &fleetv1beta1.ResourcePlacement{ + ObjectMeta: objectMeta, + Status: fleetv1beta1.PlacementStatus{ + Conditions: conditions, + }, + } + } + newCondition := func(conditionType string, status metav1.ConditionStatus, reason string, transitionTime metav1.Time) metav1.Condition { + return metav1.Condition{ + Type: conditionType, + Status: status, + Reason: reason, + ObservedGeneration: generation, + LastTransitionTime: transitionTime, + } + } + completedConditions := func(clusterScoped bool) []metav1.Condition { + placementObj := newPlacement(clusterScoped) + return []metav1.Condition{ + newCondition(getPlacementScheduledConditionType(placementObj), metav1.ConditionTrue, "Scheduled", scheduledTime), + newCondition(getPlacementConditionType(placementObj, condition.RolloutStartedCondition), metav1.ConditionTrue, "RolloutStarted", rolloutTime), + newCondition(getPlacementConditionType(placementObj, condition.OverriddenCondition), metav1.ConditionTrue, "Overridden", overriddenTime), + newCondition(getPlacementConditionType(placementObj, condition.WorkSynchronizedCondition), metav1.ConditionTrue, "WorkSynchronized", synchronizedTime), + newCondition(getPlacementConditionType(placementObj, condition.AppliedCondition), metav1.ConditionTrue, "Applied", appliedTime), + newCondition(getPlacementConditionType(placementObj, condition.AvailableCondition), metav1.ConditionTrue, "Available", availableTime), + } + } + + tests := []struct { + name string + placementObj fleetv1beta1.PlacementObj + wantConditionType string + wantStatus string + wantReason string + wantTimestamp metav1.Time + }{ + { + name: "cluster-scoped missing scheduled condition uses creation timestamp", + placementObj: newPlacement(true), + wantConditionType: string(fleetv1beta1.ClusterResourcePlacementScheduledConditionType), + wantStatus: "nil", + wantReason: "nil", + wantTimestamp: creationTime, + }, + { + name: "namespace-scoped missing scheduled condition uses creation timestamp", + placementObj: newPlacement(false), + wantConditionType: string(fleetv1beta1.ResourcePlacementScheduledConditionType), + wantStatus: "nil", + wantReason: "nil", + wantTimestamp: creationTime, + }, + { + name: "cluster-scoped scheduling failure uses condition transition timestamp", + placementObj: newPlacement(true, + newCondition(string(fleetv1beta1.ClusterResourcePlacementScheduledConditionType), metav1.ConditionFalse, "SchedulingFailed", scheduledTime), + ), + wantConditionType: string(fleetv1beta1.ClusterResourcePlacementScheduledConditionType), + wantStatus: string(metav1.ConditionFalse), + wantReason: "SchedulingFailed", + wantTimestamp: scheduledTime, + }, + { + name: "namespace-scoped scheduling pending uses condition transition timestamp", + placementObj: newPlacement(false, + newCondition(string(fleetv1beta1.ResourcePlacementScheduledConditionType), metav1.ConditionUnknown, "SchedulingPending", scheduledTime), + ), + wantConditionType: string(fleetv1beta1.ResourcePlacementScheduledConditionType), + wantStatus: string(metav1.ConditionUnknown), + wantReason: "SchedulingPending", + wantTimestamp: scheduledTime, + }, + { + name: "cluster-scoped missing intermediate condition uses creation timestamp", + placementObj: newPlacement(true, + newCondition(string(fleetv1beta1.ClusterResourcePlacementScheduledConditionType), metav1.ConditionTrue, "Scheduled", scheduledTime), + ), + wantConditionType: string(fleetv1beta1.ClusterResourcePlacementRolloutStartedConditionType), + wantStatus: "nil", + wantReason: "nil", + wantTimestamp: creationTime, + }, + { + name: "namespace-scoped selects first non-true intermediate condition", + placementObj: newPlacement(false, + newCondition(string(fleetv1beta1.ResourcePlacementScheduledConditionType), metav1.ConditionTrue, "Scheduled", scheduledTime), + newCondition(string(fleetv1beta1.ResourcePlacementRolloutStartedConditionType), metav1.ConditionTrue, "RolloutStarted", rolloutTime), + newCondition(string(fleetv1beta1.ResourcePlacementOverriddenConditionType), metav1.ConditionFalse, "OverrideFailed", overriddenTime), + newCondition(string(fleetv1beta1.ResourcePlacementWorkSynchronizedConditionType), metav1.ConditionFalse, "SynchronizationFailed", synchronizedTime), + ), + wantConditionType: string(fleetv1beta1.ResourcePlacementOverriddenConditionType), + wantStatus: string(metav1.ConditionFalse), + wantReason: "OverrideFailed", + wantTimestamp: overriddenTime, + }, + { + name: "cluster-scoped completed uses final expected condition transition timestamp", + placementObj: newPlacement(true, completedConditions(true)...), + wantConditionType: "Completed", + wantStatus: string(metav1.ConditionTrue), + wantReason: "Completed", + wantTimestamp: availableTime, + }, + { + name: "namespace-scoped completed uses final expected condition transition timestamp", + placementObj: newPlacement(false, completedConditions(false)...), + wantConditionType: "Completed", + wantStatus: string(metav1.ConditionTrue), + wantReason: "Completed", + wantTimestamp: availableTime, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + hubmetrics.FleetPlacementStatusLastTimeStampSeconds.Reset() + emitPlacementStatusMetric(tc.placementObj) + + gauge := hubmetrics.FleetPlacementStatusLastTimeStampSeconds.WithLabelValues( + tc.placementObj.GetNamespace(), + tc.placementObj.GetName(), + strconv.FormatInt(tc.placementObj.GetGeneration(), 10), + tc.wantConditionType, + tc.wantStatus, + tc.wantReason, + ) + wantTimestamp := float64(tc.wantTimestamp.UnixNano()) / float64(time.Second) + firstTimestamp := testutil.ToFloat64(gauge) + if firstTimestamp != wantTimestamp { + t.Fatalf("emitPlacementStatusMetric() timestamp = %v, want %v", firstTimestamp, wantTimestamp) + } + if gotMetricCount := testutil.CollectAndCount(hubmetrics.FleetPlacementStatusLastTimeStampSeconds); gotMetricCount != 1 { + t.Fatalf("emitPlacementStatusMetric() metric count = %d, want 1", gotMetricCount) + } + + emitPlacementStatusMetric(tc.placementObj) + secondTimestamp := testutil.ToFloat64(gauge) + if secondTimestamp != firstTimestamp { + t.Errorf("emitPlacementStatusMetric() repeated timestamp = %v, want %v", secondTimestamp, firstTimestamp) + } + }) + } +} + func TestGetOrCreateClusterSchedulingPolicySnapshot(t *testing.T) { testPolicy := placementPolicyForTest() testPolicyHash := testPolicy.DeepCopy() diff --git a/pkg/controllers/placement/suite_test.go b/pkg/controllers/placement/suite_test.go index 201480558..c1b31377a 100644 --- a/pkg/controllers/placement/suite_test.go +++ b/pkg/controllers/placement/suite_test.go @@ -42,9 +42,9 @@ import ( "go.goms.io/fleet/pkg/controllers/bindingwatcher" "go.goms.io/fleet/pkg/controllers/placementwatcher" "go.goms.io/fleet/pkg/controllers/schedulingpolicysnapshot" - "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/controller" "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" ) var ( @@ -112,12 +112,9 @@ var _ = BeforeSuite(func() { Expect(err).Should(Succeed(), "failed to create manager") resourceSelectorResolver := controller.ResourceSelectorResolver{ - RestMapper: mgr.GetRESTMapper(), - InformerManager: informer.NewInformerManager(dynamicClient, 5*time.Minute, ctx.Done()), - ResourceConfig: utils.NewResourceConfig(false), - SkippedNamespaces: map[string]bool{ - "default": true, - }, + RestMapper: mgr.GetRESTMapper(), + InformerManager: informer.NewInformerManager(dynamicClient, 5*time.Minute, ctx.Done()), + ResourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(mgr.GetRESTMapper()), } resourceSnapshotResolver := controller.NewResourceSnapshotResolver(mgr.GetClient(), mgr.GetScheme()) reconciler := &Reconciler{ diff --git a/pkg/controllers/resourcechange/resourcechange_controller.go b/pkg/controllers/resourcechange/resourcechange_controller.go index 3fa9d9879..5a3c2a606 100644 --- a/pkg/controllers/resourcechange/resourcechange_controller.go +++ b/pkg/controllers/resourcechange/resourcechange_controller.go @@ -60,9 +60,14 @@ type Reconciler struct { // Event recorder to indicate the which placement picks up this object Recorder events.EventRecorder + + // EnqueuePlacementPolicies, when set, re-queues the FEP-0001 placement policies that may select + // the changed resource. It needs only the key, so it serves deletions as well as updates. It is + // best effort: a failure is logged and the change is processed for the placements regardless. + EnqueuePlacementPolicies func(context.Context, keys.ClusterWideKey) error } -func (r *Reconciler) Reconcile(_ context.Context, key controller.QueueKey) (ctrl.Result, error) { +func (r *Reconciler) Reconcile(ctx context.Context, key controller.QueueKey) (ctrl.Result, error) { startTime := time.Now() clusterWideKey, ok := key.(keys.ClusterWideKey) if !ok { @@ -77,6 +82,14 @@ func (r *Reconciler) Reconcile(_ context.Context, key controller.QueueKey) (ctrl klog.V(2).InfoS("ResourceChange reconciliation loop ends", "obj", clusterWideKey, "latency", time.Since(startTime).Milliseconds()) }() + if r.EnqueuePlacementPolicies != nil { + // Best effort: the policies also requeue periodically, so a failure here only delays + // their status, and must not hold up the placements below. + if err := r.EnqueuePlacementPolicies(ctx, clusterWideKey); err != nil { + klog.ErrorS(err, "Failed to re-queue the placement policies that may select the object", "obj", clusterWideKey) + } + } + // The clusterObj is set to be the object that the placement direct selects. clusterObj, isClusterScoped, err := r.getUnstructuredObject(clusterWideKey) switch { diff --git a/pkg/controllers/resourcechange/resourcechange_controller_test.go b/pkg/controllers/resourcechange/resourcechange_controller_test.go index 098babdcd..9c8b86e4e 100644 --- a/pkg/controllers/resourcechange/resourcechange_controller_test.go +++ b/pkg/controllers/resourcechange/resourcechange_controller_test.go @@ -20,12 +20,14 @@ import ( "context" "errors" "reflect" + "strings" "testing" "github.com/google/go-cmp/cmp" "github.com/google/go-cmp/cmp/cmpopts" appv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" @@ -2588,3 +2590,29 @@ func TestIsSelectNamespaceOnly(t *testing.T) { }) } } + +// TestReconcileEnqueuesPlacementPolicies pins that every resource change is handed to the +// placement policy hook first, with the key alone, and that a failing hook is only logged: the +// change still goes on to the placements, here as far as the REST mapping this test leaves empty. +func TestReconcileEnqueuesPlacementPolicies(t *testing.T) { + key := keys.ClusterWideKey{ResourceIdentifier: placementv1beta1.ResourceIdentifier{Version: "v1", Kind: "ConfigMap", Namespace: "team-a", Name: "app"}} + hookErr := errors.New("the policy list is unwell") + var got []keys.ClusterWideKey + r := &Reconciler{ + RestMapper: meta.NewDefaultRESTMapper(nil), + EnqueuePlacementPolicies: func(_ context.Context, k keys.ClusterWideKey) error { + got = append(got, k) + return hookErr + }, + } + _, err := r.Reconcile(context.Background(), key) + if errors.Is(err, hookErr) { + t.Fatalf("Reconcile() = %v, want the hook error logged, not returned", err) + } + if err == nil || !strings.Contains(err.Error(), "failed to get GVR") { + t.Errorf("Reconcile() = %v, want the REST mapping error from the placement path the change went on to", err) + } + if diff := cmp.Diff([]keys.ClusterWideKey{key}, got); diff != "" { + t.Errorf("Reconcile() handed the hook (-want +got):\n%s", diff) + } +} diff --git a/pkg/controllers/rollout/controller_test.go b/pkg/controllers/rollout/controller_test.go index 492ff81b1..4f0ccb3a0 100644 --- a/pkg/controllers/rollout/controller_test.go +++ b/pkg/controllers/rollout/controller_test.go @@ -1235,7 +1235,204 @@ func TestPickBindingsToRoll(t *testing.T) { wantWaitTime time.Duration wantErr error }{ - // TODO: add more tests + "test scheduled binding to bound, latest resources and nil overrides - rollout allowed": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster1), + } + }, + latestResourceSnapshotName: "snapshot-1", + crp: clusterResourcePlacementForTest("test", + createPlacementPolicyForTest(placementv1beta1.PickAllPlacementType, 0), + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, generateDefaultRollingUpdateConfig(), nil)), + wantTobeUpdatedBindings: []int{0}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-1", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, + "test scheduled binding to bound with PickFixed placement type - rollout allowed": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster1), + } + }, + latestResourceSnapshotName: "snapshot-2", + crp: clusterResourcePlacementForTest("test", + &placementv1beta1.PlacementPolicy{ + PlacementType: placementv1beta1.PickFixedPlacementType, + ClusterNames: []string{cluster1}, + }, + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, generateDefaultRollingUpdateConfig(), nil)), + wantTobeUpdatedBindings: []int{0}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-2", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, + "test multiple scheduled bindings with maxSurge set to zero and no ready bindings - rollout allowed": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster1), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster2), + } + }, + latestResourceSnapshotName: "snapshot-1", + crp: clusterResourcePlacementForTest("test", + createPlacementPolicyForTest(placementv1beta1.PickNPlacementType, 2), + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, &placementv1beta1.RollingUpdateConfig{ + MaxUnavailable: &intstr.IntOrString{ + Type: intstr.String, + StrVal: "20%", + }, + MaxSurge: &intstr.IntOrString{ + Type: intstr.Int, + IntVal: 0, + }, + UnavailablePeriodSeconds: ptr.To(1), + }, nil)), + // maxSurge bounds how far the can-be-ready set may exceed the target. Scheduled bindings + // are not counted as can-be-ready, so with nothing ready yet the budget is the full + // target number and both bindings are allowed to roll. + wantTobeUpdatedBindings: []int{0, 1}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster2, + ResourceSnapshotName: "snapshot-1", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, + "test scheduled binding with maxSurge set to zero and target met by ready bound bindings - rollout blocked": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster1), + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster2), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster3), + } + }, + latestResourceSnapshotName: "snapshot-1", + crp: clusterResourcePlacementForTest("test", + createPlacementPolicyForTest(placementv1beta1.PickNPlacementType, 2), + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, &placementv1beta1.RollingUpdateConfig{ + MaxUnavailable: &intstr.IntOrString{ + Type: intstr.String, + StrVal: "20%", + }, + MaxSurge: &intstr.IntOrString{ + Type: intstr.Int, + IntVal: 0, + }, + UnavailablePeriodSeconds: ptr.To(1), + }, nil)), + // The two ready bound bindings already fill the target, so with maxSurge zero there is + // no room to bring the scheduled binding up and it stays unselected this round. + wantTobeUpdatedBindings: []int{}, + wantStaleUnselectedBindings: []int{2}, + wantUpToDateBoundBindings: []int{0, 1}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster2, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster3, + ResourceSnapshotName: "snapshot-1", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, + "test multiple scheduled bindings with percentage maxSurge - surge budget rounds up": { + allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { + return []*placementv1beta1.ClusterResourceBinding{ + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster1), + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster2), + generateReadyClusterResourceBinding(placementv1beta1.BindingStateBound, "snapshot-1", cluster3), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster4), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster5), + generateClusterResourceBinding(placementv1beta1.BindingStateScheduled, "snapshot-1", cluster6), + } + }, + latestResourceSnapshotName: "snapshot-1", + crp: clusterResourcePlacementForTest("test", + createPlacementPolicyForTest(placementv1beta1.PickNPlacementType, 3), + createPlacementRolloutStrategyForTest(placementv1beta1.RollingUpdateRolloutStrategyType, &placementv1beta1.RollingUpdateConfig{ + MaxUnavailable: &intstr.IntOrString{ + Type: intstr.String, + StrVal: "20%", + }, + MaxSurge: &intstr.IntOrString{ + Type: intstr.String, + StrVal: "50%", + }, + UnavailablePeriodSeconds: ptr.To(1), + }, nil)), + // 50% of the target of 3 rounds up to 2, so the can-be-ready ceiling is 5. The three + // ready bound bindings already count towards it, leaving room for exactly two of the + // three scheduled bindings; the third stays unselected this round. + wantTobeUpdatedBindings: []int{3, 4}, + wantStaleUnselectedBindings: []int{5}, + wantUpToDateBoundBindings: []int{0, 1, 2}, + wantDesiredBindingsSpec: []placementv1beta1.ResourceBindingSpec{ + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster1, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster2, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster3, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster4, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster5, + ResourceSnapshotName: "snapshot-1", + }, + { + State: placementv1beta1.BindingStateBound, + TargetCluster: cluster6, + ResourceSnapshotName: "snapshot-1", + }, + }, + wantNeedRoll: true, + wantWaitTime: 0, + }, "test scheduled binding to bound, outdated resources and nil overrides - rollout allowed": { allBindingsFunc: func() []*placementv1beta1.ClusterResourceBinding { return []*placementv1beta1.ClusterResourceBinding{ diff --git a/pkg/controllers/updaterun/controller.go b/pkg/controllers/updaterun/controller.go index 56d50875e..33f3650a8 100644 --- a/pkg/controllers/updaterun/controller.go +++ b/pkg/controllers/updaterun/controller.go @@ -492,21 +492,26 @@ func handleApprovalRequestDelete(obj client.Object, q workqueue.TypedRateLimitin }) } +// removeWaitTimeFromUpdateRunStatus removes waitTime from the Approval tasks in the updateRun status. func removeWaitTimeFromUpdateRunStatus(updateRun placementv1beta1.UpdateRunObj) { - // Remove waitTime from the updateRun status for BeforeStageTask and AfterStageTask for type Approval. - updateRunStatus := updateRun.GetUpdateRunStatus() - if updateRunStatus.UpdateStrategySnapshot != nil { - for i := range updateRunStatus.UpdateStrategySnapshot.Stages { - for j := range updateRunStatus.UpdateStrategySnapshot.Stages[i].BeforeStageTasks { - if updateRunStatus.UpdateStrategySnapshot.Stages[i].BeforeStageTasks[j].Type == placementv1beta1.StageTaskTypeApproval { - updateRunStatus.UpdateStrategySnapshot.Stages[i].BeforeStageTasks[j].WaitTime = nil - } - } - for j := range updateRunStatus.UpdateStrategySnapshot.Stages[i].AfterStageTasks { - if updateRunStatus.UpdateStrategySnapshot.Stages[i].AfterStageTasks[j].Type == placementv1beta1.StageTaskTypeApproval { - updateRunStatus.UpdateStrategySnapshot.Stages[i].AfterStageTasks[j].WaitTime = nil - } - } + strategy := updateRun.GetUpdateRunStatus().UpdateStrategySnapshot + if strategy == nil { + return + } + for i := range strategy.Stages { + removeWaitTimeFromApprovalTasks(strategy.Stages[i].BeforeStageTasks) + removeWaitTimeFromApprovalTasks(strategy.Stages[i].AfterStageTasks) + } + if strategy.DeleteStage != nil { + removeWaitTimeFromApprovalTasks(strategy.DeleteStage.BeforeStageTasks) + } +} + +// removeWaitTimeFromApprovalTasks removes waitTime from the tasks of type Approval. +func removeWaitTimeFromApprovalTasks(tasks []placementv1beta1.StageTask) { + for i := range tasks { + if tasks[i].Type == placementv1beta1.StageTaskTypeApproval { + tasks[i].WaitTime = nil } } } diff --git a/pkg/controllers/updaterun/controller_integration_test.go b/pkg/controllers/updaterun/controller_integration_test.go index 0c2b80553..bcdbeb953 100644 --- a/pkg/controllers/updaterun/controller_integration_test.go +++ b/pkg/controllers/updaterun/controller_integration_test.go @@ -20,10 +20,13 @@ import ( "context" "encoding/json" "fmt" + "sort" "strconv" + "strings" "time" "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" . "github.com/onsi/ginkgo/v2" . "github.com/onsi/gomega" prometheusclientmodel "github.com/prometheus/client_model/go" @@ -46,7 +49,6 @@ import ( hubmetrics "go.goms.io/fleet/pkg/metrics/hub" "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/condition" - metricsutils "go.goms.io/fleet/test/utils/metrics" ) const ( @@ -73,6 +75,34 @@ var ( updateRunNamespacedName types.NamespacedName ) +var updateRunStatusMetricCmpOptions = []cmp.Option{ + // Metric slices are pre-sorted deterministically (see sortMetricsByGaugeThenLabel) before + // diffing, so we intentionally do not also sort *Metric here via cmpopts.SortSlices. + cmpopts.SortSlices(func(a, b *prometheusclientmodel.LabelPair) bool { + return a.GetName() < b.GetName() + }), + // No custom Comparer for *Gauge is registered here: leaving it to cmp's default + // (exported-field) comparison means gauge values, which now carry the exact + // persisted condition.LastTransitionTime, are compared for exact equality. + cmpopts.IgnoreUnexported(prometheusclientmodel.Metric{}, prometheusclientmodel.LabelPair{}, prometheusclientmodel.Gauge{}), +} + +// sortMetricsByGaugeThenLabel sorts metrics in place by gauge value, breaking ties by +// label identity. Gauge values now reflect persisted (second-precision) condition +// LastTransitionTime, so two distinct condition transitions can legitimately tie on +// gauge value; the label-identity tie-break guarantees a deterministic, matching order +// between the "got" (scraped) and "want" (test-authored) slices without relying on +// cmp's own slice-sorting/pairing for *Metric. +func sortMetricsByGaugeThenLabel(metrics []*prometheusclientmodel.Metric) { + sort.Slice(metrics, func(i, j int) bool { + gi, gj := metrics[i].GetGauge().GetValue(), metrics[j].GetGauge().GetValue() + if gi != gj { + return gi < gj + } + return metricLabelKey(metrics[i]) < metricLabelKey(metrics[j]) + }) +} + var _ = Describe("Test the clusterStagedUpdateRun controller", func() { BeforeEach(func() { @@ -238,7 +268,7 @@ func resetUpdateRunMetrics() { hubmetrics.FleetUpdateRunApprovalRequestLatencySeconds.Reset() } -// validateUpdateRunMetricsEmitted validates the update run status metrics are emitted and are emitted in the correct order. +// validateUpdateRunMetricsEmitted validates the update run status metrics are emitted with the expected labels and values. func validateUpdateRunMetricsEmitted(wantMetrics ...*prometheusclientmodel.Metric) { Eventually(func() error { metricFamilies, err := ctrlmetrics.Registry.Gather() @@ -251,8 +281,12 @@ func validateUpdateRunMetricsEmitted(wantMetrics ...*prometheusclientmodel.Metri gotMetrics = mf.GetMetric() } } - - if diff := cmp.Diff(gotMetrics, wantMetrics, metricsutils.MetricsCmpOptions...); diff != "" { + // Pre-sort both slices deterministically (by gauge value, then label identity) so + // that got/want are paired up correctly even when multiple metrics tie on gauge + // value; the cmp options below intentionally do not re-sort *Metric slices. + sortMetricsByGaugeThenLabel(gotMetrics) + sortMetricsByGaugeThenLabel(wantMetrics) + if diff := cmp.Diff(gotMetrics, wantMetrics, updateRunStatusMetricCmpOptions...); diff != "" { return fmt.Errorf("update run status metrics mismatch (-got, +want):\n%s", diff) } @@ -260,6 +294,16 @@ func validateUpdateRunMetricsEmitted(wantMetrics ...*prometheusclientmodel.Metri }, timeout, interval).Should(Succeed(), "failed to validate the update run status metrics") } +func metricLabelKey(metric *prometheusclientmodel.Metric) string { + labels := metric.GetLabel() + pairs := make([]string, 0, len(labels)) + for _, label := range labels { + pairs = append(pairs, label.GetName()+"="+label.GetValue()) + } + sort.Strings(pairs) + return strings.Join(pairs, ";") +} + // validateUpdateRunApprovalStageTaskMetric validates the update run approval stage task metric by checking labels and count. func validateUpdateRunApprovalStageTaskMetric(wantMetrics ...*prometheusclientmodel.Metric) { Eventually(func() error { @@ -390,12 +434,84 @@ func generateMetricsLabels( } } +// conditionTransitionTimeSeconds returns the gauge value that emitUpdateRunStatusMetric +// would have set for updateRun's condition of the given type and reason, derived from the +// condition's persisted LastTransitionTime (rather than the time the test happens to run at). +// +// Callers typically build the expected metric before the controller has necessarily +// finished reconciling (validateUpdateRunMetricsEmitted then polls until it converges), +// so this re-fetches updateRun from the API server and waits for the condition to appear +// rather than requiring it to already be present on the (possibly stale) object passed in. +// +// A single condition type (e.g. Progressing) cycles through multiple reasons over an +// updateRun's lifetime (Waiting -> Progressing -> Stuck, etc.), but the live object only +// ever exposes the *current* reason's LastTransitionTime; once the condition moves on to a +// new reason, the previous reason's transition time can no longer be recovered from the +// object (Kubernetes conditions retain only the most recent transition per type). The metric +// emitted while that earlier reason was current, however, remains persisted in the metrics +// registry under its own label set (each reason has a distinct time series that is never +// overwritten by a later reason). So when the requested reason is no longer the condition's +// current reason, this falls back to reading the already-persisted metric for that reason +// straight out of the registry instead of the (now unavailable) live condition. +func conditionTransitionTimeSeconds(updateRun *placementv1beta1.ClusterStagedUpdateRun, conditionType placementv1beta1.StagedUpdateRunConditionType, reason string) float64 { + var value float64 + Eventually(func() error { + if err := k8sClient.Get(ctx, types.NamespacedName{Name: updateRun.Name, Namespace: updateRun.Namespace}, updateRun); err != nil { + return err + } + cond := meta.FindStatusCondition(updateRun.Status.Conditions, string(conditionType)) + if cond == nil { + return fmt.Errorf("condition %s not found on updateRun %s", conditionType, updateRun.Name) + } + if cond.Reason == reason { + value = float64(cond.LastTransitionTime.UnixNano()) / float64(time.Second) + return nil + } + v, found, err := persistedConditionGaugeValue(updateRun, conditionType, reason) + if err != nil { + return err + } + if !found { + return fmt.Errorf("condition %s on updateRun %s currently has reason %q, want %q, and no metric for reason %q was found in the registry", conditionType, updateRun.Name, cond.Reason, reason, reason) + } + value = v + return nil + }, timeout, interval).Should(Succeed(), "failed to determine transition time for condition %s (reason %s) on updateRun %s", conditionType, reason, updateRun.Name) + return value +} + +// persistedConditionGaugeValue looks up the already-emitted update run status gauge for the +// given condition type and reason directly from the metrics registry. It returns found=false +// if no such metric has been emitted (yet). +func persistedConditionGaugeValue(updateRun *placementv1beta1.ClusterStagedUpdateRun, conditionType placementv1beta1.StagedUpdateRunConditionType, reason string) (float64, bool, error) { + metricFamilies, err := ctrlmetrics.Registry.Gather() + if err != nil { + return 0, false, fmt.Errorf("failed to gather metrics: %w", err) + } + for _, mf := range metricFamilies { + if mf.GetName() != "fleet_workload_update_run_status_last_timestamp_seconds" { + continue + } + for _, m := range mf.GetMetric() { + labels := make(map[string]string, len(m.GetLabel())) + for _, l := range m.GetLabel() { + labels[l.GetName()] = l.GetValue() + } + if labels["namespace"] == updateRun.Namespace && labels["name"] == updateRun.Name && + labels["condition"] == string(conditionType) && labels["reason"] == reason { + return m.GetGauge().GetValue(), true, nil + } + } + } + return 0, false, nil +} + func generateInitializationSucceededMetric(state placementv1beta1.State, updateRun *placementv1beta1.ClusterStagedUpdateRun) *prometheusclientmodel.Metric { return &prometheusclientmodel.Metric{ Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionInitialized), string(metav1.ConditionTrue), condition.UpdateRunInitializeSucceededReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionInitialized, condition.UpdateRunInitializeSucceededReason)), }, } } @@ -405,7 +521,7 @@ func generateInitializationFailedMetric(state placementv1beta1.State, updateRun Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionInitialized), string(metav1.ConditionFalse), condition.UpdateRunInitializeFailedReason, failureType), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionInitialized, condition.UpdateRunInitializeFailedReason)), }, } } @@ -415,7 +531,7 @@ func generateProgressingMetric(state placementv1beta1.State, updateRun *placemen Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionTrue), condition.UpdateRunProgressingReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunProgressingReason)), }, } } @@ -425,7 +541,7 @@ func generateWaitingMetric(state placementv1beta1.State, updateRun *placementv1b Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionFalse), condition.UpdateRunWaitingReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunWaitingReason)), }, } } @@ -435,7 +551,7 @@ func generateStuckMetric(state placementv1beta1.State, updateRun *placementv1bet Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionFalse), condition.UpdateRunStuckReason, string(hubmetrics.UpdateRunFailureTypeInternalError)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunStuckReason)), }, } } @@ -445,7 +561,7 @@ func generateFailedMetric(state placementv1beta1.State, updateRun *placementv1be Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionSucceeded), string(metav1.ConditionFalse), condition.UpdateRunFailedReason, failureType), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionSucceeded, condition.UpdateRunFailedReason)), }, } } @@ -455,7 +571,7 @@ func generateStoppingMetric(state placementv1beta1.State, updateRun *placementv1 Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionUnknown), condition.UpdateRunStoppingReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunStoppingReason)), }, } } @@ -465,7 +581,7 @@ func generateStoppedMetric(state placementv1beta1.State, updateRun *placementv1b Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionProgressing), string(metav1.ConditionFalse), condition.UpdateRunStoppedReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunStoppedReason)), }, } } @@ -475,7 +591,7 @@ func generateSucceededMetric(state placementv1beta1.State, updateRun *placementv Label: generateMetricsLabels(updateRun, string(state), string(placementv1beta1.StagedUpdateRunConditionSucceeded), string(metav1.ConditionTrue), condition.UpdateRunSucceededReason, string(hubmetrics.UpdateRunFailureTypeNone)), Gauge: &prometheusclientmodel.Gauge{ - Value: ptr.To(float64(time.Now().UnixNano()) / 1e9), + Value: new(conditionTransitionTimeSeconds(updateRun, placementv1beta1.StagedUpdateRunConditionSucceeded, condition.UpdateRunSucceededReason)), }, } } diff --git a/pkg/controllers/updaterun/controller_test.go b/pkg/controllers/updaterun/controller_test.go index f57f27c5f..390a13fbb 100644 --- a/pkg/controllers/updaterun/controller_test.go +++ b/pkg/controllers/updaterun/controller_test.go @@ -986,6 +986,35 @@ func TestRemoveWaitTimeFromUpdateRunStatus(t *testing.T) { }, }, }, + "should remove waitTime from the Approval task of the delete stage BeforeStageTasks": { + inputUpdateRun: &placementv1beta1.ClusterStagedUpdateRun{ + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{ + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + WaitTime: &waitTime, + }, + }, + }, + }, + }, + }, + wantUpdateRun: &placementv1beta1.ClusterStagedUpdateRun{ + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{ + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + }, + }, + }, + }, + }, + }, + }, "should handle multiple stages": { inputUpdateRun: &placementv1beta1.ClusterStagedUpdateRun{ Status: placementv1beta1.UpdateRunStatus{ diff --git a/pkg/controllers/updaterun/execution.go b/pkg/controllers/updaterun/execution.go index 164c6b6e0..6bebf9757 100644 --- a/pkg/controllers/updaterun/execution.go +++ b/pkg/controllers/updaterun/execution.go @@ -83,7 +83,8 @@ func (r *Reconciler) execute( // No need to wait to get to the next stage. return false, 0, nil } - approved, err := r.checkBeforeStageTasksStatus(ctx, updatingStageIndex, updateRun) + beforeStageTasks := updateRunStatus.UpdateStrategySnapshot.Stages[updatingStageIndex].BeforeStageTasks + approved, err := r.checkBeforeStageTasksStatus(ctx, updatingStageStatus, beforeStageTasks, updateRun) if err != nil { return false, 0, err } @@ -101,26 +102,28 @@ func (r *Reconciler) execute( return false, waitTime, err } // All the stages have finished, now start the delete stage. - finished, err = r.executeDeleteStage(ctx, updateRun, toBeDeletedBindings) - return finished, clusterUpdatingWaitTime, err + return r.executeDeleteStage(ctx, updateRun, toBeDeletedBindings) } -// checkBeforeStageTasksStatus checks if the before stage tasks have finished. +// checkBeforeStageTasksStatus checks if the before stage tasks of a stage, an update stage or the delete stage, +// have finished. // It returns if the before stage tasks have finished or error if the before stage tasks failed. -func (r *Reconciler) checkBeforeStageTasksStatus(ctx context.Context, updatingStageIndex int, updateRun placementv1beta1.UpdateRunObj) (bool, error) { +func (r *Reconciler) checkBeforeStageTasksStatus( + ctx context.Context, + stageStatus *placementv1beta1.StageUpdatingStatus, + tasks []placementv1beta1.StageTask, + updateRun placementv1beta1.UpdateRunObj, +) (bool, error) { updateRunRef := klog.KObj(updateRun) - updateRunStatus := updateRun.GetUpdateRunStatus() - updatingStage := &updateRunStatus.UpdateStrategySnapshot.Stages[updatingStageIndex] - if updatingStage.BeforeStageTasks == nil { - klog.V(2).InfoS("There is no before stage task for this stage", "stage", updatingStage.Name, "updateRun", updateRunRef) + if len(tasks) == 0 { + klog.V(2).InfoS("There is no before stage task for this stage", "stage", stageStatus.StageName, "updateRun", updateRunRef) return true, nil } - updatingStageStatus := &updateRunStatus.StagesStatus[updatingStageIndex] - for i, task := range updatingStage.BeforeStageTasks { + for i, task := range tasks { switch task.Type { case placementv1beta1.StageTaskTypeApproval: - approved, err := r.handleStageApprovalTask(ctx, &updatingStageStatus.BeforeStageTaskStatus[i], updatingStage, updateRun, placementv1beta1.BeforeStageTaskLabelValue) + approved, err := r.handleStageApprovalTask(ctx, &stageStatus.BeforeStageTaskStatus[i], stageStatus.StageName, updateRun, placementv1beta1.BeforeStageTaskLabelValue) if err != nil { return false, err } @@ -128,7 +131,7 @@ func (r *Reconciler) checkBeforeStageTasksStatus(ctx context.Context, updatingSt default: // Approval is the only supported before stage task. unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("found unsupported task type in before stage tasks: %s", task.Type)) - klog.ErrorS(unexpectedErr, "Task type is not supported in before stage tasks", "stage", updatingStage.Name, "updateRun", updateRunRef, "taskType", task.Type) + klog.ErrorS(unexpectedErr, "Task type is not supported in before stage tasks", "stage", stageStatus.StageName, "updateRun", updateRunRef, "taskType", task.Type) return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) } } @@ -272,7 +275,7 @@ func (r *Reconciler) executeUpdatingStage( // If cluster update has been running for more than "updateRunStuckThreshold", mark the update run as stuck. timeElapsed := time.Since(clusterStartedCond.LastTransitionTime.Time) if timeElapsed > updateRunStuckThreshold { - klog.V(2).InfoS("Time waiting for cluster update to finish passes threshold, mark the update run as stuck", "time elapsed", timeElapsed, "threshold", updateRunStuckThreshold, "cluster", clusterStatus.ClusterName, "stage", updatingStageStatus.StageName, "updateRun", updateRunRef) + klog.V(2).InfoS("Time waiting for cluster update to finish passes threshold, mark the update run as stuck", "timeElapsed", timeElapsed, "threshold", updateRunStuckThreshold, "cluster", clusterStatus.ClusterName, "stage", updatingStageStatus.StageName, "updateRun", updateRunRef) stuckClusterNames = append(stuckClusterNames, clusterStatus.ClusterName) } } @@ -335,80 +338,141 @@ func (r *Reconciler) handleStageCompletion( return waitTime, nil } -// executeDeleteStage executes the delete stage by deleting the bindings. +// executeDeleteStage executes the delete stage by deleting the bindings, once the before stage tasks have completed. +// It returns a boolean indicating if the delete stage is completed, +// the time to wait before rechecking the delete stage, and any error encountered. func (r *Reconciler) executeDeleteStage( ctx context.Context, updateRun placementv1beta1.UpdateRunObj, toBeDeletedBindings []placementv1beta1.BindingObj, -) (bool, error) { +) (bool, time.Duration, error) { updateRunRef := klog.KObj(updateRun) updateRunStatus := updateRun.GetUpdateRunStatus() existingDeleteStageStatus := updateRunStatus.DeletionStageStatus - existingDeleteStageClusterMap := make(map[string]*placementv1beta1.ClusterUpdatingStatus, len(existingDeleteStageStatus.Clusters)) - for i := range existingDeleteStageStatus.Clusters { - existingDeleteStageClusterMap[existingDeleteStageStatus.Clusters[i].ClusterName] = &existingDeleteStageStatus.Clusters[i] + approved, err := r.checkBeforeDeleteStageTasksStatus(ctx, updateRun, toBeDeletedBindings) + if err != nil { + return false, 0, err + } + if !approved { + return false, stageUpdatingWaitTime, nil } // Mark the delete stage as started in case it's not. - markStageUpdatingProgressStarted(updateRunStatus.DeletionStageStatus, updateRun.GetGeneration()) + markStageUpdatingProgressStarted(existingDeleteStageStatus, updateRun.GetGeneration()) + toBeStartedClusters, err := syncDeleteStageClusterStatuses(updateRun, toBeDeletedBindings) + if err != nil { + return false, 0, err + } + for _, binding := range toBeDeletedBindings { + curCluster, toBeStarted := toBeStartedClusters[binding.GetBindingSpec().TargetCluster] + if !toBeStarted { + continue + } + if err := r.Client.Delete(ctx, binding); err != nil { + klog.ErrorS(err, "Failed to delete a binding in the update run", "binding", klog.KObj(binding), "cluster", curCluster.ClusterName, "updateRun", updateRunRef) + return false, 0, controller.NewAPIServerError(false, err) + } + klog.V(2).InfoS("Deleted a binding pointing to a to be deleted cluster", "binding", klog.KObj(binding), "cluster", curCluster.ClusterName, "updateRun", updateRunRef) + markClusterUpdatingStarted(curCluster, updateRun.GetGeneration()) + } + klog.InfoS("The delete stage is progressing", "numberOfDeletingClusters", len(toBeDeletedBindings), "updateRun", updateRunRef) + if len(toBeDeletedBindings) == 0 { + markStageUpdatingSucceeded(existingDeleteStageStatus, updateRun.GetGeneration()) + } + return len(toBeDeletedBindings) == 0, clusterUpdatingWaitTime, nil +} + +// syncDeleteStageClusterStatuses checks the clusters in the delete stage against the bindings that are still to be +// deleted and marks the clusters with no binding left as succeeded in memory. +// It returns the clusters with a binding whose deletion has not started, by their names. +func syncDeleteStageClusterStatuses( + updateRun placementv1beta1.UpdateRunObj, + toBeDeletedBindings []placementv1beta1.BindingObj, +) (map[string]*placementv1beta1.ClusterUpdatingStatus, error) { + updateRunRef := klog.KObj(updateRun) + deleteStageStatus := updateRun.GetUpdateRunStatus().DeletionStageStatus + deletedClusters := make(map[string]*placementv1beta1.ClusterUpdatingStatus, len(deleteStageStatus.Clusters)) + for i := range deleteStageStatus.Clusters { + deletedClusters[deleteStageStatus.Clusters[i].ClusterName] = &deleteStageStatus.Clusters[i] + } + toBeStartedClusters := make(map[string]*placementv1beta1.ClusterUpdatingStatus, len(toBeDeletedBindings)) for _, binding := range toBeDeletedBindings { bindingSpec := binding.GetBindingSpec() - curCluster, exist := existingDeleteStageClusterMap[bindingSpec.TargetCluster] + curCluster, exist := deletedClusters[bindingSpec.TargetCluster] if !exist { // This is unexpected because we already checked in validation. - missingErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the to be deleted cluster `%s` is not in the deleting stage during execution", bindingSpec.TargetCluster)) + missingErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the to be deleted cluster `%s` is not in the deleting stage", bindingSpec.TargetCluster)) klog.ErrorS(missingErr, "The cluster in the deleting stage does not include all the to be deleted binding", "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, missingErr.Error()) + return nil, fmt.Errorf("%w: %s", errStagedUpdatedAborted, missingErr.Error()) } - // In validation, we already check the binding must exist in the status. - delete(existingDeleteStageClusterMap, bindingSpec.TargetCluster) + // The binding is still there, so the cluster is not deleted. + delete(deletedClusters, bindingSpec.TargetCluster) // Make sure the cluster is not marked as deleted as the binding is still there. if condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionSucceeded)), updateRun.GetGeneration()) { unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the deleted cluster `%s` in the deleting stage still has a binding", bindingSpec.TargetCluster)) klog.ErrorS(unexpectedErr, "The cluster in the deleting stage is not removed yet but marked as deleted", "cluster", curCluster.ClusterName, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) + return nil, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) } - if condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { - // The cluster status is marked as being deleted. - if binding.GetDeletionTimestamp().IsZero() { - // The cluster is marked as deleting but the binding is not deleting. - unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the cluster `%s` in the deleting stage is marked as deleting but its corresponding binding is not deleting", curCluster.ClusterName)) - klog.ErrorS(unexpectedErr, "The binding should be deleting before we mark a cluster deleting", "clusterStatus", curCluster, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) - } + if !condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { + // The cluster status is not deleting yet. + toBeStartedClusters[bindingSpec.TargetCluster] = curCluster continue } - // The cluster status is not deleting yet - if err := r.Client.Delete(ctx, binding); err != nil { - klog.ErrorS(err, "Failed to delete a binding in the update run", "binding", klog.KObj(binding), "cluster", curCluster.ClusterName, "updateRun", updateRunRef) - return false, controller.NewAPIServerError(false, err) + // The cluster status is marked as being deleted. + if binding.GetDeletionTimestamp().IsZero() { + // The cluster is marked as deleting but the binding is not deleting. + unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the cluster `%s` in the deleting stage is marked as deleting but its corresponding binding is not deleting", curCluster.ClusterName)) + klog.ErrorS(unexpectedErr, "The binding should be deleting before we mark a cluster deleting", "clusterStatus", curCluster, "updateRun", updateRunRef) + return nil, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) } - klog.V(2).InfoS("Deleted a binding pointing to a to be deleted cluster", "binding", klog.KObj(binding), "cluster", curCluster.ClusterName, "updateRun", updateRunRef) - markClusterUpdatingStarted(curCluster, updateRun.GetGeneration()) } // The rest of the clusters in the stage are not in the toBeDeletedBindings so it should be marked as delete succeeded. - for _, clusterStatus := range existingDeleteStageClusterMap { + for _, clusterStatus := range deletedClusters { // Make sure the cluster is marked as deleted. if !condition.IsConditionStatusTrue(meta.FindStatusCondition(clusterStatus.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { markClusterUpdatingStarted(clusterStatus, updateRun.GetGeneration()) } markClusterUpdatingSucceeded(clusterStatus, updateRun.GetGeneration()) } - klog.InfoS("The delete stage is progressing", "numberOfDeletingClusters", len(toBeDeletedBindings), "updateRun", updateRunRef) - if len(toBeDeletedBindings) == 0 { - markStageUpdatingSucceeded(updateRunStatus.DeletionStageStatus, updateRun.GetGeneration()) + return toBeStartedClusters, nil +} + +// checkBeforeDeleteStageTasksStatus checks if the before stage tasks of the delete stage have finished. +// It returns if the delete stage can start deleting the bindings or error if the before stage tasks failed. +func (r *Reconciler) checkBeforeDeleteStageTasksStatus( + ctx context.Context, + updateRun placementv1beta1.UpdateRunObj, + toBeDeletedBindings []placementv1beta1.BindingObj, +) (bool, error) { + updateRunStatus := updateRun.GetUpdateRunStatus() + deleteStage := updateRunStatus.UpdateStrategySnapshot.DeleteStage + deleteStageStatus := updateRunStatus.DeletionStageStatus + // The tasks only gate the start of the deletion. Same as the update stages with no clusters, the delete stage + // skips its tasks if there is nothing to delete. + if deleteStage == nil || len(deleteStage.BeforeStageTasks) == 0 || len(toBeDeletedBindings) == 0 || deleteStageStatus.StartTime != nil { + return true, nil + } + approved, err := r.checkBeforeStageTasksStatus(ctx, deleteStageStatus, deleteStage.BeforeStageTasks, updateRun) + if err != nil { + return false, err + } + if !approved { + markStageUpdatingWaiting(deleteStageStatus, updateRun.GetGeneration(), "Not all before-stage tasks are completed, waiting for approval") + markUpdateRunWaiting(updateRun, fmt.Sprintf(condition.UpdateRunWaitingMessageFmt, "before-stage", deleteStageStatus.StageName)) + return false, nil } - return len(toBeDeletedBindings) == 0, nil + markUpdateRunProgressing(updateRun) + return true, nil } // checkAfterStageTasksStatus checks if the after stage tasks have finished. // It returns if the after stage tasks have finished or error if the after stage tasks failed. -// It also returns the time to wait before rechecking the wait type of task. It turns -1 if the task is not a wait type. +// It also returns the time to wait before rechecking the wait type of task. It returns -1 if the task is not a wait type. func (r *Reconciler) checkAfterStageTasksStatus(ctx context.Context, updatingStageIndex int, updateRun placementv1beta1.UpdateRunObj) (bool, time.Duration, error) { updateRunRef := klog.KObj(updateRun) updateRunStatus := updateRun.GetUpdateRunStatus() updatingStageStatus := &updateRunStatus.StagesStatus[updatingStageIndex] updatingStage := &updateRunStatus.UpdateStrategySnapshot.Stages[updatingStageIndex] - if updatingStage.AfterStageTasks == nil { + if len(updatingStage.AfterStageTasks) == 0 { klog.V(2).InfoS("There is no after stage task for this stage", "stage", updatingStage.Name, "updateRun", updateRunRef) return true, 0, nil } @@ -417,6 +481,11 @@ func (r *Reconciler) checkAfterStageTasksStatus(ctx context.Context, updatingSta for i, task := range updatingStage.AfterStageTasks { switch task.Type { case placementv1beta1.StageTaskTypeTimedWait: + // The wait time is measured from the latest transition of the stage, which a stop and a resume of the + // update run move; a wait that has elapsed once stays elapsed. + if meta.IsStatusConditionTrue(updatingStageStatus.AfterStageTaskStatus[i].Conditions, string(placementv1beta1.StageTaskConditionWaitTimeElapsed)) { + continue + } waitStartTime := meta.FindStatusCondition(updatingStageStatus.Conditions, string(placementv1beta1.StageUpdatingConditionProgressing)).LastTransitionTime.Time // Check if the wait time has passed. waitTime := time.Until(waitStartTime.Add(task.WaitTime.Duration)) @@ -429,7 +498,7 @@ func (r *Reconciler) checkAfterStageTasksStatus(ctx context.Context, updatingSta klog.V(2).InfoS("The after stage wait task has completed", "stage", updatingStage.Name, "updateRun", updateRunRef) } case placementv1beta1.StageTaskTypeApproval: - approved, err := r.handleStageApprovalTask(ctx, &updatingStageStatus.AfterStageTaskStatus[i], updatingStage, updateRun, placementv1beta1.AfterStageTaskLabelValue) + approved, err := r.handleStageApprovalTask(ctx, &updatingStageStatus.AfterStageTaskStatus[i], updatingStage.Name, updateRun, placementv1beta1.AfterStageTaskLabelValue) if err != nil { return false, -1, err } @@ -449,7 +518,7 @@ func (r *Reconciler) checkAfterStageTasksStatus(ctx context.Context, updatingSta func (r *Reconciler) handleStageApprovalTask( ctx context.Context, stageTaskStatus *placementv1beta1.StageTaskStatus, - updatingStage *placementv1beta1.StageConfig, + stageName string, updateRun placementv1beta1.UpdateRunObj, stageTaskType string, ) (bool, error) { @@ -462,54 +531,54 @@ func (r *Reconciler) handleStageApprovalTask( } // Check if the approval request has been created. - approvalRequest := buildApprovalRequestObject(types.NamespacedName{Name: stageTaskStatus.ApprovalRequestName, Namespace: updateRun.GetNamespace()}, updatingStage.Name, updateRun.GetName(), stageTaskType) + approvalRequest := buildApprovalRequestObject(types.NamespacedName{Name: stageTaskStatus.ApprovalRequestName, Namespace: updateRun.GetNamespace()}, stageName, updateRun.GetName(), stageTaskType) requestRef := klog.KObj(approvalRequest) - if err := r.Client.Create(ctx, approvalRequest); err != nil { - if apierrors.IsAlreadyExists(err) { - // The approval task already exists. - markStageTaskRequestCreated(stageTaskStatus, updateRun.GetGeneration()) - if err = r.Client.Get(ctx, client.ObjectKeyFromObject(approvalRequest), approvalRequest); err != nil { - klog.ErrorS(err, "Failed to get the already existing approval request", "approvalRequest", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - return false, controller.NewAPIServerError(true, err) - } - approvalRequestSpec := approvalRequest.GetApprovalRequestSpec() - if approvalRequestSpec.TargetStage != updatingStage.Name || approvalRequestSpec.TargetUpdateRun != updateRun.GetName() { - unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the approval request task `%s/%s` is targeting update run `%s/%s` and stage `%s`, want target update run `%s/%s and stage `%s`", approvalRequest.GetNamespace(), approvalRequest.GetName(), approvalRequest.GetNamespace(), approvalRequestSpec.TargetUpdateRun, approvalRequestSpec.TargetStage, approvalRequest.GetNamespace(), updateRun.GetName(), updatingStage.Name)) - klog.ErrorS(unexpectedErr, "Found an approval request targeting wrong stage", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) - } - approvalRequestStatus := approvalRequest.GetApprovalRequestStatus() - approvalAccepted := condition.IsConditionStatusTrue(meta.FindStatusCondition(approvalRequestStatus.Conditions, string(placementv1beta1.ApprovalRequestConditionApprovalAccepted)), approvalRequest.GetGeneration()) - approved := condition.IsConditionStatusTrue(meta.FindStatusCondition(approvalRequestStatus.Conditions, string(placementv1beta1.ApprovalRequestConditionApproved)), approvalRequest.GetGeneration()) - if !approvalAccepted && !approved { - klog.V(2).InfoS("The approval request has not been approved yet", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - return false, nil - } - if approved { - klog.V(2).InfoS("The approval request has been approved", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - if !approvalAccepted { - if err = r.updateApprovalRequestAccepted(ctx, approvalRequest); err != nil { - klog.ErrorS(err, "Failed to accept the approved approval request", "approvalRequest", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - // retriable err - return false, err - } - } - } else { - // Approved state should not change once the approval is accepted. - klog.V(2).InfoS("The approval request has been approval-accepted, ignoring changing back to unapproved", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - } - markStageTaskRequestApproved(stageTaskStatus, updateRun, stageTaskType) - } else { - // retriable error - klog.ErrorS(err, "Failed to create the approval request", "approvalRequest", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) - return false, controller.NewAPIServerError(false, err) - } - } else { + err := r.Client.Create(ctx, approvalRequest) + if err == nil { // The approval request has been created for the first time. - klog.V(2).InfoS("The approval request has been created", "approvalRequestTask", requestRef, "stage", updatingStage.Name, "updateRun", updateRunRef) + klog.V(2).InfoS("The approval request has been created", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) markStageTaskRequestCreated(stageTaskStatus, updateRun.GetGeneration()) return false, nil } + if !apierrors.IsAlreadyExists(err) { + // retriable error + klog.ErrorS(err, "Failed to create the approval request", "approvalRequest", requestRef, "stage", stageName, "updateRun", updateRunRef) + return false, controller.NewAPIServerError(false, err) + } + + // The approval task already exists. + markStageTaskRequestCreated(stageTaskStatus, updateRun.GetGeneration()) + if err = r.Client.Get(ctx, client.ObjectKeyFromObject(approvalRequest), approvalRequest); err != nil { + klog.ErrorS(err, "Failed to get the already existing approval request", "approvalRequest", requestRef, "stage", stageName, "updateRun", updateRunRef) + return false, controller.NewAPIServerError(true, err) + } + approvalRequestSpec := approvalRequest.GetApprovalRequestSpec() + if approvalRequestSpec.TargetStage != stageName || approvalRequestSpec.TargetUpdateRun != updateRun.GetName() { + unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the approval request task `%s/%s` is targeting update run `%s/%s` and stage `%s`, want target update run `%s/%s` and stage `%s`", approvalRequest.GetNamespace(), approvalRequest.GetName(), approvalRequest.GetNamespace(), approvalRequestSpec.TargetUpdateRun, approvalRequestSpec.TargetStage, approvalRequest.GetNamespace(), updateRun.GetName(), stageName)) + klog.ErrorS(unexpectedErr, "Found an approval request targeting wrong stage", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) + } + approvalRequestStatus := approvalRequest.GetApprovalRequestStatus() + approvalAccepted := condition.IsConditionStatusTrue(meta.FindStatusCondition(approvalRequestStatus.Conditions, string(placementv1beta1.ApprovalRequestConditionApprovalAccepted)), approvalRequest.GetGeneration()) + approved := condition.IsConditionStatusTrue(meta.FindStatusCondition(approvalRequestStatus.Conditions, string(placementv1beta1.ApprovalRequestConditionApproved)), approvalRequest.GetGeneration()) + switch { + case !approvalAccepted && !approved: + klog.V(2).InfoS("The approval request has not been approved yet", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + return false, nil + case !approved: + // Approved state should not change once the approval is accepted. + klog.V(2).InfoS("The approval request has been approval-accepted, ignoring changing back to unapproved", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + case !approvalAccepted: + klog.V(2).InfoS("The approval request has been approved", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + if err = r.updateApprovalRequestAccepted(ctx, approvalRequest); err != nil { + klog.ErrorS(err, "Failed to accept the approved approval request", "approvalRequest", requestRef, "stage", stageName, "updateRun", updateRunRef) + // retriable err + return false, err + } + default: + klog.V(2).InfoS("The approval request has been approved", "approvalRequestTask", requestRef, "stage", stageName, "updateRun", updateRunRef) + } + markStageTaskRequestApproved(stageTaskStatus, updateRun, stageTaskType) return true, nil } @@ -633,42 +702,29 @@ func checkClusterUpdateResult( // buildApprovalRequestObject creates an approval request object for before-stage or after-stage tasks. // It returns a ClusterApprovalRequest if namespace is empty, otherwise returns an ApprovalRequest. func buildApprovalRequestObject(namespacedName types.NamespacedName, stageName, updateRunName, stageTaskType string) placementv1beta1.ApprovalRequestObj { - var approvalRequest placementv1beta1.ApprovalRequestObj + stageLabelValue := stageName + if stageName == placementv1beta1.UpdateRunDeleteStageName { + // The name of the delete stage is not a valid label value; the spec still targets the stage by its name. + stageLabelValue = placementv1beta1.UpdateRunDeleteStageTaskName + } + objectMeta := metav1.ObjectMeta{ + Name: namespacedName.Name, + Namespace: namespacedName.Namespace, + Labels: map[string]string{ + placementv1beta1.TargetUpdatingStageNameLabel: stageLabelValue, + placementv1beta1.TargetUpdateRunLabel: updateRunName, + placementv1beta1.TaskTypeLabel: stageTaskType, + placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", + }, + } + spec := placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: updateRunName, + TargetStage: stageName, + } if namespacedName.Namespace == "" { - approvalRequest = &placementv1beta1.ClusterApprovalRequest{ - ObjectMeta: metav1.ObjectMeta{ - Name: namespacedName.Name, - Labels: map[string]string{ - placementv1beta1.TargetUpdatingStageNameLabel: stageName, - placementv1beta1.TargetUpdateRunLabel: updateRunName, - placementv1beta1.TaskTypeLabel: stageTaskType, - placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", - }, - }, - Spec: placementv1beta1.ApprovalRequestSpec{ - TargetUpdateRun: updateRunName, - TargetStage: stageName, - }, - } - } else { - approvalRequest = &placementv1beta1.ApprovalRequest{ - ObjectMeta: metav1.ObjectMeta{ - Name: namespacedName.Name, - Namespace: namespacedName.Namespace, - Labels: map[string]string{ - placementv1beta1.TargetUpdatingStageNameLabel: stageName, - placementv1beta1.TargetUpdateRunLabel: updateRunName, - placementv1beta1.TaskTypeLabel: stageTaskType, - placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", - }, - }, - Spec: placementv1beta1.ApprovalRequestSpec{ - TargetUpdateRun: updateRunName, - TargetStage: stageName, - }, - } + return &placementv1beta1.ClusterApprovalRequest{ObjectMeta: objectMeta, Spec: spec} } - return approvalRequest + return &placementv1beta1.ApprovalRequest{ObjectMeta: objectMeta, Spec: spec} } // markUpdateRunProgressing marks the update run as progressing in memory. @@ -707,19 +763,13 @@ func markUpdateRunStuck(updateRun placementv1beta1.UpdateRunObj, stageName strin }) } +// generateStuckClustersString lists up to 3 of the stuck clusters. func generateStuckClustersString(stuckClusterNames []string) string { - if len(stuckClusterNames) == 0 { - return "" - } if len(stuckClusterNames) <= 3 { return strings.Join(stuckClusterNames, ", ") } // Print first 3 if more than 3 stuck clusters. - clustersToShow := stuckClusterNames - if len(stuckClusterNames) > 3 { - clustersToShow = stuckClusterNames[:3] - } - return strings.Join(clustersToShow, ", ") + "..." + return strings.Join(stuckClusterNames[:3], ", ") + "..." } // markUpdateRunWaiting marks the updateRun as waiting in memory. diff --git a/pkg/controllers/updaterun/execution_integration_test.go b/pkg/controllers/updaterun/execution_integration_test.go index 5bdbe9bf2..b3a314b55 100644 --- a/pkg/controllers/updaterun/execution_integration_test.go +++ b/pkg/controllers/updaterun/execution_integration_test.go @@ -880,6 +880,199 @@ var _ = Describe("UpdateRun execution tests - double stages", func() { }) }) + + Context("Cluster staged update run should wait for the before-stage tasks of the delete stage", Ordered, func() { + var wantApprovalRequest *placementv1beta1.ClusterApprovalRequest + + // Wait for a few rounds of reconciliation, which is every stageUpdatingWaitTime while waiting for the tasks. + validateToBeDeletedBindingsExistConsistently := func() { + Consistently(func() error { + for i := numTargetClusters; i < numTargetClusters+numUnscheduledClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding); err != nil { + return fmt.Errorf("failed to get binding %s: %w", resourceBindings[i].Name, err) + } + if !binding.DeletionTimestamp.IsZero() { + return fmt.Errorf("binding %s is being deleted", binding.Name) + } + } + return nil + }, 3*stageUpdatingWaitTime, interval).Should(Succeed(), "failed to validate the to-be-deleted bindings are kept") + } + + BeforeAll(func() { + By("Reassigning the strategy to have 1 stage selecting all clusters and the delete stage with a before-stage approval task") + sortingKey := "index" + updateStrategy.Spec.Stages = []placementv1beta1.StageConfig{ + { + Name: "stage1", + LabelSelector: &metav1.LabelSelector{}, // Empty label selector selects all clusters. + SortingLabelKey: &sortingKey, + }, + } + updateStrategy.Spec.DeleteStage = &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + }, + }, + } + Expect(k8sClient.Update(ctx, updateStrategy)).To(Succeed()) + + By("Creating a new clusterStagedUpdateRun") + Expect(k8sClient.Create(ctx, updateRun)).To(Succeed()) + + By("Validating the initialization succeeded and the execution has started") + initialized := generateInitializedStatus( + crp, updateRun, testResourceSnapshotIndex, policySnapshot, updateStrategy, 10, + generateTenClusterSingleStageStatus(clusterResourceOverride)[:1], generateTenClusterDeletionStageStatus(), + ) + wantStatus = generateExecutionStartedStatus(updateRun, initialized) + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + + // Build wantStatus based on the current updateRun status after initialization. + wantStatus = updateRun.Status.DeepCopy() + }) + + It("Should complete the 1st stage and wait for the before-stage tasks of the delete stage", func() { + By("Marking all bindings in the 1st stage as available and validating all clusters have completed") + for i := 0; i < numTargetClusters; i++ { + binding := resourceBindings[numTargetClusters-1-i] + validateBindingState(ctx, binding, resourceSnapshot.Name, updateRun, &updateRun.Status.StagesStatus[0].Clusters[i]) + meta.SetStatusCondition(&binding.Status.Conditions, generateTrueCondition(binding, placementv1beta1.ResourceBindingAvailable)) + Expect(k8sClient.Status().Update(ctx, binding)).Should(Succeed(), "failed to update the binding status") + + meta.SetStatusCondition(&wantStatus.StagesStatus[0].Clusters[i].Conditions, generateTrueCondition(updateRun, placementv1beta1.ClusterUpdatingConditionStarted)) + meta.SetStatusCondition(&wantStatus.StagesStatus[0].Clusters[i].Conditions, generateTrueCondition(updateRun, placementv1beta1.ClusterUpdatingConditionSucceeded)) + } + + By("Validating the 1st stage has completed and the delete stage is waiting for the approval") + wantStatus.StagesStatus[0].Conditions[0] = generateFalseConditionWithReason(updateRun, placementv1beta1.StageUpdatingConditionProgressing, condition.StageUpdatingSucceededReason) + wantStatus.StagesStatus[0].Conditions = append(wantStatus.StagesStatus[0].Conditions, generateTrueCondition(updateRun, placementv1beta1.StageUpdatingConditionSucceeded)) + wantStatus.DeletionStageStatus.Conditions = append(wantStatus.DeletionStageStatus.Conditions, generateFalseCondition(updateRun, placementv1beta1.StageUpdatingConditionProgressing)) + wantStatus.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions = append(wantStatus.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions, + generateTrueCondition(updateRun, placementv1beta1.StageTaskConditionApprovalRequestCreated)) + meta.SetStatusCondition(&wantStatus.Conditions, generateFalseCondition(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing)) + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + + By("Validating the delete stage has not started") + Expect(updateRun.Status.DeletionStageStatus.StartTime).Should(BeNil()) + + By("Validating the approvalRequest has been created") + wantApprovalRequest = &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.Name, placementv1beta1.UpdateRunDeleteStageTaskName), + Labels: map[string]string{ + placementv1beta1.TargetUpdatingStageNameLabel: placementv1beta1.UpdateRunDeleteStageTaskName, + placementv1beta1.TargetUpdateRunLabel: updateRun.Name, + placementv1beta1.TaskTypeLabel: placementv1beta1.BeforeStageTaskLabelValue, + placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", + }, + }, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: updateRun.Name, + TargetStage: updateRun.Status.DeletionStageStatus.StageName, + }, + } + Expect(updateRun.Status.DeletionStageStatus.BeforeStageTaskStatus[0].ApprovalRequestName).Should(Equal(wantApprovalRequest.Name)) + validateApprovalRequestCreated(wantApprovalRequest) + + By("Validating the to-be-deleted bindings are not deleted") + validateToBeDeletedBindingsExistConsistently() + + By("Checking update run status metrics are emitted") + validateUpdateRunMetricsEmitted(generateProgressingMetric(placementv1beta1.StateRun, updateRun), generateWaitingMetric(placementv1beta1.StateRun, updateRun)) + }) + + It("Should recreate the approvalRequest of the delete stage if it is deleted", func() { + By("Deleting the approvalRequest") + approvalRequest := &placementv1beta1.ClusterApprovalRequest{} + Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantApprovalRequest.Name}, approvalRequest)).Should(Succeed()) + oldUID := approvalRequest.UID + Expect(k8sClient.Delete(ctx, approvalRequest)).Should(Succeed()) + + By("Validating the approvalRequest has been recreated") + Eventually(func() error { + if err := k8sClient.Get(ctx, types.NamespacedName{Name: wantApprovalRequest.Name}, approvalRequest); err != nil { + return err + } + if approvalRequest.UID == oldUID { + return fmt.Errorf("approvalRequest %s is not deleted yet", approvalRequest.Name) + } + return nil + }, timeout, interval).Should(Succeed(), "failed to validate the approvalRequest is recreated") + validateApprovalRequestCreated(wantApprovalRequest) + + By("Validating the to-be-deleted bindings are not deleted") + validateToBeDeletedBindingsExistConsistently() + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + }) + + It("Should stop the update run without deleting the bindings when state is Stop", func() { + By("Updating updateRun state to Stop") + updateRun = updateClusterStagedUpdateRunState(updateRun.Name, placementv1beta1.StateStop) + // Update the test's want status to match the new generation. + updateAllStatusConditionsGeneration(wantStatus, updateRun.Generation) + + By("Validating the update run has stopped") + meta.SetStatusCondition(&wantStatus.DeletionStageStatus.Conditions, generateFalseConditionWithReason(updateRun, placementv1beta1.StageUpdatingConditionProgressing, condition.StageUpdatingStoppedReason)) + meta.SetStatusCondition(&wantStatus.Conditions, generateFalseConditionWithReason(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunStoppedReason)) + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + + By("Approving the approvalRequest") + approveClusterApprovalRequest(ctx, wantApprovalRequest.Name) + + By("Validating the to-be-deleted bindings are not deleted") + validateToBeDeletedBindingsExistConsistently() + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + }) + + It("Should delete all the clusterResourceBindings and complete the update run when state is Run", func() { + By("Updating updateRun state to Run") + updateRun = updateClusterStagedUpdateRunState(updateRun.Name, placementv1beta1.StateRun) + // Update the test's want status to match the new generation. + updateAllStatusConditionsGeneration(wantStatus, updateRun.Generation) + + By("Validating the approvalRequest has ApprovalAccepted status") + validateApprovalRequestAccepted(ctx, wantApprovalRequest.Name) + + By("Validating the to-be-deleted bindings are all deleted") + Eventually(func() error { + for i := numTargetClusters; i < numTargetClusters+numUnscheduledClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding) + if err == nil { + return fmt.Errorf("binding %s is not deleted", binding.Name) + } + if !apierrors.IsNotFound(err) { + return fmt.Errorf("get binding %s does not return a not-found error: %w", binding.Name, err) + } + } + return nil + }, timeout, interval).Should(Succeed(), "failed to validate the deletion of the to-be-deleted bindings") + + By("Validating the delete stage and the clusterStagedUpdateRun has completed") + wantStatus.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions = append(wantStatus.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions, + generateTrueCondition(updateRun, placementv1beta1.StageTaskConditionApprovalRequestApproved)) + for i := range wantStatus.DeletionStageStatus.Clusters { + wantStatus.DeletionStageStatus.Clusters[i].Conditions = append(wantStatus.DeletionStageStatus.Clusters[i].Conditions, + generateTrueCondition(updateRun, placementv1beta1.ClusterUpdatingConditionStarted), + generateTrueCondition(updateRun, placementv1beta1.ClusterUpdatingConditionSucceeded)) + } + meta.SetStatusCondition(&wantStatus.DeletionStageStatus.Conditions, generateFalseConditionWithReason(updateRun, placementv1beta1.StageUpdatingConditionProgressing, condition.StageUpdatingSucceededReason)) + wantStatus.DeletionStageStatus.Conditions = append(wantStatus.DeletionStageStatus.Conditions, generateTrueCondition(updateRun, placementv1beta1.StageUpdatingConditionSucceeded)) + meta.SetStatusCondition(&wantStatus.Conditions, generateFalseConditionWithReason(updateRun, placementv1beta1.StagedUpdateRunConditionProgressing, condition.UpdateRunSucceededReason)) + wantStatus.Conditions = append(wantStatus.Conditions, generateTrueCondition(updateRun, placementv1beta1.StagedUpdateRunConditionSucceeded)) + validateClusterStagedUpdateRunStatus(ctx, updateRun, wantStatus, "") + + By("Validating the delete stage has startTime and endTime set") + Expect(updateRun.Status.DeletionStageStatus.StartTime).ShouldNot(BeNil()) + Expect(updateRun.Status.DeletionStageStatus.EndTime).ShouldNot(BeNil()) + + By("Checking update run metrics are emitted") + validateUpdateRunApprovalStageTaskMetric(generateApprovalStageTaskMetric(updateRun, placementv1beta1.BeforeStageTaskLabelValue, 1)) + }) + }) }) var _ = Describe("UpdateRun execution tests - single stage", func() { @@ -1173,7 +1366,7 @@ var _ = Describe("UpdateRun execution tests - single stage", func() { validateUpdateRunMetricsEmitted(generateWaitingMetric(placementv1beta1.StateRun, updateRun), generateProgressingMetric(placementv1beta1.StateRun, updateRun)) }) - It("Should Should complete the 1st stage and complete the updateRun after the 3rd cluster succeeds", func() { + It("Should complete the 1st stage and complete the updateRun after the 3rd cluster succeeds", func() { By("Validating the 3rd clusterResourceBinding is updated to Bound") binding := resourceBindings[2] // cluster-2 validateBindingState(ctx, binding, resourceSnapshot.Name, updateRun, &updateRun.Status.StagesStatus[0].Clusters[2]) @@ -1959,7 +2152,7 @@ var _ = Describe("UpdateRun execution tests - single stage", func() { By("Validating the 1st stage has endTime set") Expect(updateRun.Status.StagesStatus[0].EndTime).ShouldNot(BeNil()) - By("Checking update run metrics are emitted") + By("Checking update run metrics are emitted") validateUpdateRunMetricsEmitted(generateInitializationSucceededMetric(placementv1beta1.StateInitialize, updateRun), generateProgressingMetric(placementv1beta1.StateRun, updateRun), generateSucceededMetric(placementv1beta1.StateRun, updateRun)) validateUpdateRunStageMetricsEmitted(generateStageClusterUpdatingMetric(updateRun)) }) diff --git a/pkg/controllers/updaterun/execution_test.go b/pkg/controllers/updaterun/execution_test.go index f84c540cc..c27c6eee9 100644 --- a/pkg/controllers/updaterun/execution_test.go +++ b/pkg/controllers/updaterun/execution_test.go @@ -26,6 +26,7 @@ import ( "github.com/google/go-cmp/cmp" "github.com/google/go-cmp/cmp/cmpopts" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -37,6 +38,7 @@ import ( placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils/condition" + "go.goms.io/fleet/pkg/utils/controller" ) func TestIsBindingSyncedWithClusterStatus(t *testing.T) { @@ -363,6 +365,12 @@ func TestCheckClusterUpdateResult(t *testing.T) { } func TestBuildApprovalRequestObject(t *testing.T) { + // Pin the values that show up on the approval requests of the delete stage, which users see and select with. + const ( + deleteStageName = "kubernetes-fleet.io/deleteStage" + deleteStageLabelValue = "delete-stage" + deleteStageApprovalRequestName = "test-update-run-before-delete-stage" + ) tests := []struct { name string namespacedName types.NamespacedName @@ -422,6 +430,56 @@ func TestBuildApprovalRequestObject(t *testing.T) { }, }, }, + { + name: "should create ClusterApprovalRequest with a valid stage label for the delete stage", + namespacedName: types.NamespacedName{ + Name: deleteStageApprovalRequestName, + }, + stageName: deleteStageName, + updateRunName: "test-update-run", + stageTaskType: placementv1beta1.BeforeStageTaskLabelValue, + want: &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{ + Name: deleteStageApprovalRequestName, + Labels: map[string]string{ + placementv1beta1.TargetUpdatingStageNameLabel: deleteStageLabelValue, + placementv1beta1.TargetUpdateRunLabel: "test-update-run", + placementv1beta1.TaskTypeLabel: placementv1beta1.BeforeStageTaskLabelValue, + placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", + }, + }, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: "test-update-run", + TargetStage: deleteStageName, + }, + }, + }, + { + name: "should create namespaced ApprovalRequest with a valid stage label for the delete stage", + namespacedName: types.NamespacedName{ + Name: deleteStageApprovalRequestName, + Namespace: testNamespaceName, + }, + stageName: deleteStageName, + updateRunName: "test-update-run", + stageTaskType: placementv1beta1.BeforeStageTaskLabelValue, + want: &placementv1beta1.ApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{ + Name: deleteStageApprovalRequestName, + Namespace: testNamespaceName, + Labels: map[string]string{ + placementv1beta1.TargetUpdatingStageNameLabel: deleteStageLabelValue, + placementv1beta1.TargetUpdateRunLabel: "test-update-run", + placementv1beta1.TaskTypeLabel: placementv1beta1.BeforeStageTaskLabelValue, + placementv1beta1.IsLatestUpdateRunApprovalLabel: "true", + }, + }, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: "test-update-run", + TargetStage: deleteStageName, + }, + }, + }, } for _, test := range tests { @@ -1251,7 +1309,9 @@ func TestCheckBeforeStageTasksStatus_NegativeCases(t *testing.T) { Client: fakeClient, } ctx := context.Background() - _, gotErr := r.checkBeforeStageTasksStatus(ctx, tt.stageIndex, tt.updateRun) + status := tt.updateRun.GetUpdateRunStatus() + tasks := status.UpdateStrategySnapshot.Stages[tt.stageIndex].BeforeStageTasks + _, gotErr := r.checkBeforeStageTasksStatus(ctx, &status.StagesStatus[tt.stageIndex], tasks, tt.updateRun) if gotErr == nil { t.Fatalf("checkBeforeStageTasksStatus() want error but got nil") } @@ -1265,6 +1325,318 @@ func TestCheckBeforeStageTasksStatus_NegativeCases(t *testing.T) { } } +func TestExecuteDeleteStage(t *testing.T) { + const ( + updateRunName = "test-update-run" + clusterName = "cluster-1" + ) + approvalRequestName := fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunName, placementv1beta1.UpdateRunDeleteStageTaskName) + now := metav1.Now() + + approvalTask := placementv1beta1.StageTask{Type: placementv1beta1.StageTaskTypeApproval} + approvalTaskStatus := placementv1beta1.StageTaskStatus{Type: placementv1beta1.StageTaskTypeApproval, ApprovalRequestName: approvalRequestName} + newCondition := func(condType any, status metav1.ConditionStatus, reason string) metav1.Condition { + return metav1.Condition{Type: fmt.Sprint(condType), Status: status, Reason: reason, ObservedGeneration: 1} + } + stageWaitingCond := newCondition(placementv1beta1.StageUpdatingConditionProgressing, metav1.ConditionFalse, condition.StageUpdatingWaitingReason) + stageStartedCond := newCondition(placementv1beta1.StageUpdatingConditionProgressing, metav1.ConditionTrue, condition.StageUpdatingStartedReason) + runWaitingCond := newCondition(placementv1beta1.StagedUpdateRunConditionProgressing, metav1.ConditionFalse, condition.UpdateRunWaitingReason) + runProgressingCond := newCondition(placementv1beta1.StagedUpdateRunConditionProgressing, metav1.ConditionTrue, condition.UpdateRunProgressingReason) + requestCreatedCond := newCondition(placementv1beta1.StageTaskConditionApprovalRequestCreated, metav1.ConditionTrue, condition.StageTaskApprovalRequestCreatedReason) + requestApprovedCond := newCondition(placementv1beta1.StageTaskConditionApprovalRequestApproved, metav1.ConditionTrue, condition.StageTaskApprovalRequestApprovedReason) + clusterStartedCond := newCondition(placementv1beta1.ClusterUpdatingConditionStarted, metav1.ConditionTrue, condition.ClusterUpdatingStartedReason) + approvedRequest := &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{Name: approvalRequestName, Generation: 1}, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: updateRunName, + TargetStage: placementv1beta1.UpdateRunDeleteStageName, + }, + Status: placementv1beta1.ApprovalRequestStatus{ + Conditions: []metav1.Condition{ + newCondition(placementv1beta1.ApprovalRequestConditionApproved, metav1.ConditionTrue, "Approved"), + }, + }, + } + noClusters := []placementv1beta1.ClusterUpdatingStatus{} + + tests := []struct { + name string + // tasks are the before stage tasks of the delete stage. + tasks []placementv1beta1.StageTask + // deleteStageStatus is the status of the delete stage, whose name and clusters are defaulted. + deleteStageStatus placementv1beta1.StageUpdatingStatus + approvalRequest *placementv1beta1.ClusterApprovalRequest + noBinding bool + wantFinished bool + wantWaitTime time.Duration + wantErr error + wantBindingKept bool + // wantDeleteStageStatus is the wanted status of the delete stage, whose name and clusters are defaulted. + wantDeleteStageStatus placementv1beta1.StageUpdatingStatus + wantRunConditions []metav1.Condition + wantApprovalRequest bool + }{ + { + name: "no delete stage configuration should delete the binding", + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName, Conditions: []metav1.Condition{clusterStartedCond}}}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + }, + { + name: "pending approval task should create the approval request and keep the binding", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + wantWaitTime: stageUpdatingWaitTime, + wantBindingKept: true, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{requestCreatedCond}, + }}, + Conditions: []metav1.Condition{stageWaitingCond}, + }, + wantRunConditions: []metav1.Condition{runWaitingCond}, + wantApprovalRequest: true, + }, + { + name: "approved approval task should delete the binding", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{stageWaitingCond}, + }, + approvalRequest: approvedRequest, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName, Conditions: []metav1.Condition{clusterStartedCond}}}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{requestCreatedCond, requestApprovedCond}, + }}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + wantRunConditions: []metav1.Condition{runProgressingCond}, + wantApprovalRequest: true, + }, + { + name: "accepted approval request that is unapproved afterwards should still delete the binding", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + approvalRequest: &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: approvedRequest.ObjectMeta, + Spec: approvedRequest.Spec, + Status: placementv1beta1.ApprovalRequestStatus{ + Conditions: []metav1.Condition{ + newCondition(placementv1beta1.ApprovalRequestConditionApproved, metav1.ConditionFalse, "Unapproved"), + newCondition(placementv1beta1.ApprovalRequestConditionApprovalAccepted, metav1.ConditionTrue, condition.ApprovalRequestApprovalAcceptedReason), + }, + }, + }, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName, Conditions: []metav1.Condition{clusterStartedCond}}}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{requestCreatedCond, requestApprovedCond}, + }}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + wantRunConditions: []metav1.Condition{runProgressingCond}, + wantApprovalRequest: true, + }, + { + // A delete stage that has started is not gated again, so no approval request is created. + name: "tasks should not be checked again after the delete stage has started", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + StartTime: &now, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName, Conditions: []metav1.Condition{clusterStartedCond}}}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{stageStartedCond}, + }, + }, + { + name: "tasks should be skipped when there is no cluster to delete", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: noClusters, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + noBinding: true, + wantFinished: true, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: noClusters, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{ + newCondition(placementv1beta1.StageUpdatingConditionProgressing, metav1.ConditionFalse, condition.StageUpdatingSucceededReason), + newCondition(placementv1beta1.StageUpdatingConditionSucceeded, metav1.ConditionTrue, condition.StageUpdatingSucceededReason), + }, + }, + }, + { + name: "tasks should be skipped when the bindings are already deleted", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + noBinding: true, + wantFinished: true, + wantWaitTime: clusterUpdatingWaitTime, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ + ClusterName: clusterName, + Conditions: []metav1.Condition{ + clusterStartedCond, + newCondition(placementv1beta1.ClusterUpdatingConditionSucceeded, metav1.ConditionTrue, condition.ClusterUpdatingSucceededReason), + }, + }}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + Conditions: []metav1.Condition{ + newCondition(placementv1beta1.StageUpdatingConditionProgressing, metav1.ConditionFalse, condition.StageUpdatingSucceededReason), + newCondition(placementv1beta1.StageUpdatingConditionSucceeded, metav1.ConditionTrue, condition.StageUpdatingSucceededReason), + }, + }, + }, + { + name: "approval request targeting another update run should abort the update run", + tasks: []placementv1beta1.StageTask{approvalTask}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{approvalTaskStatus}, + }, + approvalRequest: &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{Name: approvalRequestName, Generation: 1}, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: "another-update-run", + TargetStage: placementv1beta1.UpdateRunDeleteStageName, + }, + }, + wantErr: errStagedUpdatedAborted, + wantBindingKept: true, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{requestCreatedCond}, + }}, + }, + wantApprovalRequest: true, + }, + { + // The API rejects a TimedWait task on the delete stage, so this only happens with a corrupted snapshot. + name: "unsupported timed wait task should abort the update run", + tasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeTimedWait, WaitTime: &metav1.Duration{Duration: time.Hour}}}, + deleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{Type: placementv1beta1.StageTaskTypeTimedWait}}, + }, + wantErr: errStagedUpdatedAborted, + wantBindingKept: true, + wantDeleteStageStatus: placementv1beta1.StageUpdatingStatus{ + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{{Type: placementv1beta1.StageTaskTypeTimedWait}}, + }, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + binding := &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "test-binding"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: clusterName}, + } + clientBuilder := fake.NewClientBuilder().WithScheme(scheme).WithStatusSubresource(&placementv1beta1.ClusterApprovalRequest{}) + var toBeDeletedBindings []placementv1beta1.BindingObj + if !tt.noBinding { + clientBuilder = clientBuilder.WithObjects(binding) + toBeDeletedBindings = append(toBeDeletedBindings, binding) + } + if tt.approvalRequest != nil { + clientBuilder = clientBuilder.WithObjects(tt.approvalRequest.DeepCopy()) + } + fakeClient := clientBuilder.Build() + r := &Reconciler{Client: fakeClient} + + deleteStageStatus := tt.deleteStageStatus.DeepCopy() + deleteStageStatus.StageName = placementv1beta1.UpdateRunDeleteStageName + if deleteStageStatus.Clusters == nil { + deleteStageStatus.Clusters = []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName}} + } + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: updateRunName, Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{}, + DeletionStageStatus: deleteStageStatus, + }, + } + if tt.tasks != nil { + updateRun.Status.UpdateStrategySnapshot.DeleteStage = &placementv1beta1.DeleteStageConfig{BeforeStageTasks: tt.tasks} + } + // The metrics are global; an approved approval request records one. + t.Cleanup(func() { deleteUpdateRunMetrics(updateRun) }) + + gotFinished, gotWaitTime, gotErr := r.executeDeleteStage(ctx, updateRun, toBeDeletedBindings) + if !errors.Is(gotErr, tt.wantErr) { + t.Fatalf("executeDeleteStage() error = %v, want %v", gotErr, tt.wantErr) + } + if gotFinished != tt.wantFinished { + t.Errorf("executeDeleteStage() finished = %v, want %v", gotFinished, tt.wantFinished) + } + if gotWaitTime != tt.wantWaitTime { + t.Errorf("executeDeleteStage() waitTime = %v, want %v", gotWaitTime, tt.wantWaitTime) + } + + if !tt.noBinding { + err := fakeClient.Get(ctx, client.ObjectKeyFromObject(binding), &placementv1beta1.ClusterResourceBinding{}) + if gotBindingKept := err == nil; gotBindingKept != tt.wantBindingKept || (err != nil && !apierrors.IsNotFound(err)) { + t.Errorf("executeDeleteStage() binding kept = %v (get error: %v), want %v", gotBindingKept, err, tt.wantBindingKept) + } + } + err := fakeClient.Get(ctx, client.ObjectKey{Name: approvalRequestName}, &placementv1beta1.ClusterApprovalRequest{}) + if gotApprovalRequest := err == nil; gotApprovalRequest != tt.wantApprovalRequest || (err != nil && !apierrors.IsNotFound(err)) { + t.Errorf("executeDeleteStage() approval request exists = %v (get error: %v), want %v", gotApprovalRequest, err, tt.wantApprovalRequest) + } + + wantDeleteStageStatus := tt.wantDeleteStageStatus.DeepCopy() + wantDeleteStageStatus.StageName = placementv1beta1.UpdateRunDeleteStageName + if wantDeleteStageStatus.Clusters == nil { + wantDeleteStageStatus.Clusters = []placementv1beta1.ClusterUpdatingStatus{{ClusterName: clusterName}} + } + cmpOpts := []cmp.Option{ + cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message"), + cmpopts.IgnoreFields(placementv1beta1.StageUpdatingStatus{}, "StartTime", "EndTime"), + } + if diff := cmp.Diff(wantDeleteStageStatus, updateRun.Status.DeletionStageStatus, cmpOpts...); diff != "" { + t.Errorf("executeDeleteStage() delete stage status mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff(tt.wantRunConditions, updateRun.Status.Conditions, cmpOpts...); diff != "" { + t.Errorf("executeDeleteStage() update run conditions mismatch (-want +got):\n%s", diff) + } + // The delete stage is only considered started once the tasks are completed. + if gotStarted, wantStarted := updateRun.Status.DeletionStageStatus.StartTime != nil, !tt.wantBindingKept; gotStarted != wantStarted { + t.Errorf("executeDeleteStage() delete stage started = %v, want %v", gotStarted, wantStarted) + } + }) + } +} + func TestGenerateStuckClustersString(t *testing.T) { tests := []struct { name string @@ -1464,3 +1836,400 @@ func TestExecute_ZeroClustersSkipsEntireStage(t *testing.T) { }) } } + +// TestExecuteDeleteStage_MultipleClusters tests that executeDeleteStage only deletes the bindings whose deletion has +// not started yet, and marks the clusters with no binding left as deleted. +func TestExecuteDeleteStage_MultipleClusters(t *testing.T) { + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + newCondition := func(condType any, reason string) metav1.Condition { + return metav1.Condition{Type: fmt.Sprint(condType), Status: metav1.ConditionTrue, Reason: reason, ObservedGeneration: 1} + } + clusterStartedCond := newCondition(placementv1beta1.ClusterUpdatingConditionStarted, condition.ClusterUpdatingStartedReason) + clusterSucceededCond := newCondition(placementv1beta1.ClusterUpdatingConditionSucceeded, condition.ClusterUpdatingSucceededReason) + deletionTime := metav1.Now() + + // The deletion of the binding on cluster-1 has started already; the fake client keeps it as it has a finalizer. + deletingBinding := &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "binding-1", DeletionTimestamp: &deletionTime, Finalizers: []string{"test-finalizer"}}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-1"}, + } + binding := &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "binding-2"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-2"}, + } + var deletedBindings []string + fakeClient := fake.NewClientBuilder().WithScheme(scheme).WithObjects(deletingBinding, binding).WithInterceptorFuncs(interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + deletedBindings = append(deletedBindings, obj.GetName()) + return c.Delete(ctx, obj, opts...) + }, + }).Build() + r := &Reconciler{Client: fakeClient} + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: "test-update-run", Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{}, + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: placementv1beta1.UpdateRunDeleteStageName, + Clusters: []placementv1beta1.ClusterUpdatingStatus{ + {ClusterName: "cluster-1", Conditions: []metav1.Condition{clusterStartedCond}}, + {ClusterName: "cluster-2"}, + // The binding on cluster-3 is gone, so the cluster is deleted. + {ClusterName: "cluster-3", Conditions: []metav1.Condition{clusterStartedCond}}, + }, + }, + }, + } + + gotFinished, gotWaitTime, err := r.executeDeleteStage(ctx, updateRun, []placementv1beta1.BindingObj{deletingBinding, binding}) + if err != nil { + t.Fatalf("executeDeleteStage() error = %v, want no error", err) + } + if gotFinished || gotWaitTime != clusterUpdatingWaitTime { + t.Errorf("executeDeleteStage() = (%v, %v), want (false, %v)", gotFinished, gotWaitTime, clusterUpdatingWaitTime) + } + if diff := cmp.Diff([]string{"binding-2"}, deletedBindings); diff != "" { + t.Errorf("executeDeleteStage() deleted bindings mismatch (-want +got):\n%s", diff) + } + wantClusters := []placementv1beta1.ClusterUpdatingStatus{ + {ClusterName: "cluster-1", Conditions: []metav1.Condition{clusterStartedCond}}, + {ClusterName: "cluster-2", Conditions: []metav1.Condition{clusterStartedCond}}, + {ClusterName: "cluster-3", Conditions: []metav1.Condition{clusterStartedCond, clusterSucceededCond}}, + } + if diff := cmp.Diff(wantClusters, updateRun.Status.DeletionStageStatus.Clusters, cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message")); diff != "" { + t.Errorf("executeDeleteStage() delete stage clusters mismatch (-want +got):\n%s", diff) + } +} + +// TestExecuteDeleteStage_Errors tests that executeDeleteStage reports the API server errors as retriable and aborts +// the update run, without deleting any binding, if the delete stage status does not match the bindings. +func TestExecuteDeleteStage_Errors(t *testing.T) { + const updateRunName = "test-update-run" + approvalRequestName := fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunName, placementv1beta1.UpdateRunDeleteStageTaskName) + errAPIServer := apierrors.NewServiceUnavailable("api server error") + newCondition := func(condType any, reason string) metav1.Condition { + return metav1.Condition{Type: fmt.Sprint(condType), Status: metav1.ConditionTrue, Reason: reason, ObservedGeneration: 1} + } + clusterStartedCond := newCondition(placementv1beta1.ClusterUpdatingConditionStarted, condition.ClusterUpdatingStartedReason) + clusterSucceededCond := newCondition(placementv1beta1.ClusterUpdatingConditionSucceeded, condition.ClusterUpdatingSucceededReason) + approvalRequest := func(conds ...metav1.Condition) *placementv1beta1.ClusterApprovalRequest { + return &placementv1beta1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{Name: approvalRequestName, Generation: 1}, + Spec: placementv1beta1.ApprovalRequestSpec{ + TargetUpdateRun: updateRunName, + TargetStage: placementv1beta1.UpdateRunDeleteStageName, + }, + Status: placementv1beta1.ApprovalRequestStatus{Conditions: conds}, + } + } + isApprovalRequest := func(obj client.Object) bool { + _, ok := obj.(*placementv1beta1.ClusterApprovalRequest) + return ok + } + + tests := []struct { + name string + withApproval bool + approvalRequest *placementv1beta1.ClusterApprovalRequest + // clusters are the clusters in the delete stage; cluster-2 is always added without any condition. + clusters []placementv1beta1.ClusterUpdatingStatus + interceptors interceptor.Funcs + wantErr error + }{ + { + name: "failing to create the approval request should be retriable", + withApproval: true, + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + interceptors: interceptor.Funcs{ + Create: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + if isApprovalRequest(obj) { + return errAPIServer + } + return c.Create(ctx, obj, opts...) + }, + }, + wantErr: controller.ErrAPIServerError, + }, + { + name: "failing to get the existing approval request should be retriable", + withApproval: true, + approvalRequest: approvalRequest(), + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + interceptors: interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if isApprovalRequest(obj) { + return errAPIServer + } + return c.Get(ctx, key, obj, opts...) + }, + }, + wantErr: controller.ErrAPIServerError, + }, + { + name: "failing to accept the approval should be retriable", + withApproval: true, + approvalRequest: approvalRequest(newCondition(placementv1beta1.ApprovalRequestConditionApproved, "Approved")), + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + interceptors: interceptor.Funcs{ + SubResourceUpdate: func(ctx context.Context, c client.Client, subResourceName string, obj client.Object, opts ...client.SubResourceUpdateOption) error { + return errAPIServer + }, + }, + wantErr: controller.ErrAPIServerError, + }, + { + name: "failing to delete a binding should be retriable", + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + interceptors: interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + return errAPIServer + }, + }, + wantErr: controller.ErrAPIServerError, + }, + { + name: "binding on a cluster missing from the delete stage should abort the update run", + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "another-cluster"}}, + wantErr: errStagedUpdatedAborted, + }, + { + name: "deleted cluster that still has a binding should abort the update run", + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1", Conditions: []metav1.Condition{clusterStartedCond, clusterSucceededCond}}}, + wantErr: errStagedUpdatedAborted, + }, + { + name: "deleting cluster whose binding is not deleting should abort the update run", + clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1", Conditions: []metav1.Condition{clusterStartedCond}}}, + wantErr: errStagedUpdatedAborted, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + // The binding on cluster-2 comes after the binding on cluster-1, so that an abort caused by the + // cluster-1 shows that no binding is deleted before all the clusters are checked. + bindings := []placementv1beta1.BindingObj{ + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "binding-1"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-1"}, + }, + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "binding-2"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-2"}, + }, + } + clientBuilder := fake.NewClientBuilder().WithScheme(scheme). + WithStatusSubresource(&placementv1beta1.ClusterApprovalRequest{}). + WithObjects(bindings[0], bindings[1]) + if tt.approvalRequest != nil { + clientBuilder = clientBuilder.WithObjects(tt.approvalRequest) + } + var deletedBindings []string + interceptors := tt.interceptors + if interceptors.Delete == nil { + interceptors.Delete = func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + deletedBindings = append(deletedBindings, obj.GetName()) + return c.Delete(ctx, obj, opts...) + } + } + r := &Reconciler{Client: clientBuilder.WithInterceptorFuncs(interceptors).Build()} + + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: updateRunName, Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{}, + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: placementv1beta1.UpdateRunDeleteStageName, + Clusters: append(tt.clusters, placementv1beta1.ClusterUpdatingStatus{ClusterName: "cluster-2"}), + }, + }, + } + if tt.withApproval { + updateRun.Status.UpdateStrategySnapshot.DeleteStage = &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeApproval}}, + } + updateRun.Status.DeletionStageStatus.BeforeStageTaskStatus = []placementv1beta1.StageTaskStatus{ + {Type: placementv1beta1.StageTaskTypeApproval, ApprovalRequestName: approvalRequestName}, + } + } + t.Cleanup(func() { deleteUpdateRunMetrics(updateRun) }) + + gotFinished, _, gotErr := r.executeDeleteStage(ctx, updateRun, bindings) + if !errors.Is(gotErr, tt.wantErr) { + t.Fatalf("executeDeleteStage() error = %v, want %v", gotErr, tt.wantErr) + } + if tt.wantErr == controller.ErrAPIServerError && errors.Is(gotErr, errStagedUpdatedAborted) { + t.Errorf("executeDeleteStage() error = %v, want a retriable error that does not abort the update run", gotErr) + } + if gotFinished { + t.Errorf("executeDeleteStage() finished = true, want false") + } + if len(deletedBindings) != 0 { + t.Errorf("executeDeleteStage() deleted bindings %v, want none", deletedBindings) + } + }) + } +} + +// TestExecuteDeleteStage_Namespaced tests that the approval request of the delete stage of a namespaced update run +// is created in the namespace of the update run. +func TestExecuteDeleteStage_Namespaced(t *testing.T) { + const ( + namespace = "test-namespace" + updateRunName = "test-update-run" + ) + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + approvalRequestName := fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunName, placementv1beta1.UpdateRunDeleteStageTaskName) + binding := &placementv1beta1.ResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "test-binding", Namespace: namespace}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-1"}, + } + fakeClient := fake.NewClientBuilder().WithScheme(scheme).WithObjects(binding).Build() + r := &Reconciler{Client: fakeClient} + updateRun := &placementv1beta1.StagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: updateRunName, Namespace: namespace, Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{ + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeApproval}}, + }, + }, + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: placementv1beta1.UpdateRunDeleteStageName, + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "cluster-1"}}, + BeforeStageTaskStatus: []placementv1beta1.StageTaskStatus{ + {Type: placementv1beta1.StageTaskTypeApproval, ApprovalRequestName: approvalRequestName}, + }, + }, + }, + } + + gotFinished, gotWaitTime, err := r.executeDeleteStage(ctx, updateRun, []placementv1beta1.BindingObj{binding}) + if err != nil { + t.Fatalf("executeDeleteStage() error = %v, want no error", err) + } + if gotFinished || gotWaitTime != stageUpdatingWaitTime { + t.Errorf("executeDeleteStage() = (%v, %v), want (false, %v)", gotFinished, gotWaitTime, stageUpdatingWaitTime) + } + gotApprovalRequest := &placementv1beta1.ApprovalRequest{} + if err := fakeClient.Get(ctx, client.ObjectKey{Name: approvalRequestName, Namespace: namespace}, gotApprovalRequest); err != nil { + t.Fatalf("Get() approval request = %v, want no error", err) + } + wantApprovalRequest := buildApprovalRequestObject(types.NamespacedName{Name: approvalRequestName, Namespace: namespace}, + placementv1beta1.UpdateRunDeleteStageName, updateRunName, placementv1beta1.BeforeStageTaskLabelValue) + if diff := cmp.Diff(wantApprovalRequest, gotApprovalRequest, cmpopts.IgnoreFields(metav1.ObjectMeta{}, "ResourceVersion")); diff != "" { + t.Errorf("executeDeleteStage() approval request mismatch (-want +got):\n%s", diff) + } + if got := gotApprovalRequest.Labels[placementv1beta1.TargetUpdatingStageNameLabel]; got != placementv1beta1.UpdateRunDeleteStageTaskName { + t.Errorf("executeDeleteStage() approval request stage label = %q, want %q", got, placementv1beta1.UpdateRunDeleteStageTaskName) + } + if err := fakeClient.Get(ctx, client.ObjectKeyFromObject(binding), &placementv1beta1.ResourceBinding{}); err != nil { + t.Errorf("Get() binding = %v, want the binding to be kept", err) + } +} + +// TestExecute_DeleteStageAbort tests that an abort in the delete stage marks the delete stage as failed. +func TestExecute_DeleteStageAbort(t *testing.T) { + ctx := context.Background() + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + binding := &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "test-binding"}, + Spec: placementv1beta1.ResourceBindingSpec{TargetCluster: "cluster-1"}, + } + r := &Reconciler{Client: fake.NewClientBuilder().WithScheme(scheme).WithObjects(binding).Build()} + // The delete stage does not include the cluster of the binding. + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: "test-update-run", Generation: 1}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{}, + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: placementv1beta1.UpdateRunDeleteStageName, + Clusters: []placementv1beta1.ClusterUpdatingStatus{{ClusterName: "another-cluster"}}, + }, + }, + } + + _, _, err := r.execute(ctx, updateRun, 0, nil, []placementv1beta1.BindingObj{binding}) + if !errors.Is(err, errStagedUpdatedAborted) { + t.Fatalf("execute() error = %v, want %v", err, errStagedUpdatedAborted) + } + wantConds := []metav1.Condition{ + { + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: condition.StageUpdatingFailedReason, + }, + { + Type: string(placementv1beta1.StageUpdatingConditionSucceeded), + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: condition.StageUpdatingFailedReason, + }, + } + if diff := cmp.Diff(wantConds, updateRun.Status.DeletionStageStatus.Conditions, cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message")); diff != "" { + t.Errorf("execute() delete stage conditions mismatch (-want +got):\n%s", diff) + } +} + +// TestCheckAfterStageTasksStatus_ElapsedTimedWait tests that a timed wait task that has elapsed stays elapsed after +// the stage transitions again, e.g., when the update run is stopped and resumed. +func TestCheckAfterStageTasksStatus_ElapsedTimedWait(t *testing.T) { + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: "test-update-run", Generation: 2}, + Status: placementv1beta1.UpdateRunStatus{ + UpdateStrategySnapshot: &placementv1beta1.UpdateStrategySpec{ + Stages: []placementv1beta1.StageConfig{{ + Name: "stage-1", + AfterStageTasks: []placementv1beta1.StageTask{ + {Type: placementv1beta1.StageTaskTypeTimedWait, WaitTime: &metav1.Duration{Duration: time.Hour}}, + }, + }}, + }, + StagesStatus: []placementv1beta1.StageUpdatingStatus{{ + StageName: "stage-1", + Conditions: []metav1.Condition{{ + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionFalse, + ObservedGeneration: 2, + Reason: condition.StageUpdatingStoppedReason, + LastTransitionTime: metav1.Now(), + }}, + AfterStageTaskStatus: []placementv1beta1.StageTaskStatus{{ + Type: placementv1beta1.StageTaskTypeTimedWait, + Conditions: []metav1.Condition{{ + Type: string(placementv1beta1.StageTaskConditionWaitTimeElapsed), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.AfterStageTaskWaitTimeElapsedReason, + }}, + }}, + }}, + }, + } + r := &Reconciler{} + + gotPassed, gotWaitTime, err := r.checkAfterStageTasksStatus(context.Background(), 0, updateRun) + if err != nil { + t.Fatalf("checkAfterStageTasksStatus() error = %v, want no error", err) + } + if !gotPassed || gotWaitTime != 0 { + t.Errorf("checkAfterStageTasksStatus() = (%v, %v), want (true, 0)", gotPassed, gotWaitTime) + } +} diff --git a/pkg/controllers/updaterun/initialization.go b/pkg/controllers/updaterun/initialization.go index 96c89a95c..c71c67eab 100644 --- a/pkg/controllers/updaterun/initialization.go +++ b/pkg/controllers/updaterun/initialization.go @@ -285,6 +285,17 @@ func (r *Reconciler) generateStagesByStrategy( // Remove waitTime from the updateRun status for BeforeStageTask and AfterStageTask for type Approval. removeWaitTimeFromUpdateRunStatus(updateRun) + var deleteStageTasks []placementv1beta1.StageTask + if updateStrategySpec.DeleteStage != nil { + deleteStageTasks = updateStrategySpec.DeleteStage.BeforeStageTasks + } + if err := validateBeforeStageTask(deleteStageTasks); err != nil { + klog.ErrorS(err, "Failed to validate the before stage tasks of the delete stage", "updateStrategy", strategyKey, "updateRun", updateRunRef) + // no more retries here. + invalidBeforeStageErr := controller.NewUserError(fmt.Errorf("the before stage tasks are invalid, updateStrategy: `%s`, stage: %s, err: %s", strategyKey, placementv1beta1.UpdateRunDeleteStageName, err.Error())) + return fmt.Errorf("%w: %s", errValidationFailed, invalidBeforeStageErr.Error()) + } + // Compute the update stages. if err := r.computeRunStageStatus(ctx, scheduledBindings, updateRun); err != nil { return err @@ -305,6 +316,9 @@ func (r *Reconciler) generateStagesByStrategy( StageName: placementv1beta1.UpdateRunDeleteStageName, Clusters: toBeDeletedClusters, } + // Create the before stage tasks of the delete stage. + updateRunStatus.DeletionStageStatus.BeforeStageTaskStatus = buildStageTaskStatuses(deleteStageTasks, + fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), placementv1beta1.UpdateRunDeleteStageTaskName)) return nil } @@ -415,21 +429,11 @@ func (r *Reconciler) computeRunStageStatus( } // Create the before stage tasks. - curStageUpdatingStatus.BeforeStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.BeforeStageTasks)) - for i, task := range stage.BeforeStageTasks { - curStageUpdatingStatus.BeforeStageTaskStatus[i].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - curStageUpdatingStatus.BeforeStageTaskStatus[i].ApprovalRequestName = fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - } + curStageUpdatingStatus.BeforeStageTaskStatus = buildStageTaskStatuses(stage.BeforeStageTasks, + fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name)) // Create the after stage tasks. - curStageUpdatingStatus.AfterStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.AfterStageTasks)) - for i, task := range stage.AfterStageTasks { - curStageUpdatingStatus.AfterStageTaskStatus[i].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - curStageUpdatingStatus.AfterStageTaskStatus[i].ApprovalRequestName = fmt.Sprintf(placementv1beta1.AfterStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - } + curStageUpdatingStatus.AfterStageTaskStatus = buildStageTaskStatuses(stage.AfterStageTasks, + fmt.Sprintf(placementv1beta1.AfterStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name)) stagesStatus = append(stagesStatus, curStageUpdatingStatus) } updateRunStatus.StagesStatus = stagesStatus @@ -452,6 +456,19 @@ func (r *Reconciler) computeRunStageStatus( return nil } +// buildStageTaskStatuses builds the statuses of the given tasks of a stage, where approvalRequestName is the name of +// the approval request of the Approval task. +func buildStageTaskStatuses(tasks []placementv1beta1.StageTask, approvalRequestName string) []placementv1beta1.StageTaskStatus { + taskStatuses := make([]placementv1beta1.StageTaskStatus, len(tasks)) + for i, task := range tasks { + taskStatuses[i].Type = task.Type + if task.Type == placementv1beta1.StageTaskTypeApproval { + taskStatuses[i].ApprovalRequestName = approvalRequestName + } + } + return taskStatuses +} + // validateBeforeStageTask validates the beforeStageTasks in the stage defined in the UpdateStrategy. // The error returned from this function is not retriable. func validateBeforeStageTask(tasks []placementv1beta1.StageTask) error { diff --git a/pkg/controllers/updaterun/initialization_integration_test.go b/pkg/controllers/updaterun/initialization_integration_test.go index d7aea3463..0c4f4ef2d 100644 --- a/pkg/controllers/updaterun/initialization_integration_test.go +++ b/pkg/controllers/updaterun/initialization_integration_test.go @@ -697,7 +697,12 @@ var _ = Describe("Updaterun initialization tests", func() { }) It("Should generate the cluster update stage in the status as expected", func() { - By("Creating a clusterStagedUpdateStrategy") + By("Creating a clusterStagedUpdateStrategy with a before stage approval task for the delete stage") + updateStrategy.Spec.DeleteStage = &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + {Type: placementv1beta1.StageTaskTypeApproval}, + }, + } Expect(k8sClient.Create(ctx, updateStrategy)).To(Succeed()) By("Creating a new clusterStagedUpdateRun") @@ -1102,12 +1107,21 @@ func validateFailedInitCondition(ctx context.Context, updateRun *placementv1beta } // populateStageTaskStatuses populates the BeforeStageTaskStatus and AfterStageTaskStatus -// for all stages in the status based on the strategy configuration. +// for all stages, including the delete stage, in the status based on the strategy configuration. func populateStageTaskStatuses( status *placementv1beta1.UpdateRunStatus, updateRunName string, - stages []placementv1beta1.StageConfig, + strategy *placementv1beta1.UpdateStrategySpec, ) { + stages := strategy.Stages + if strategy.DeleteStage != nil && status.DeletionStageStatus != nil { + status.DeletionStageStatus.BeforeStageTaskStatus = buildTaskStatuses( + strategy.DeleteStage.BeforeStageTasks, + placementv1beta1.BeforeStageApprovalTaskNameFmt, + updateRunName, + placementv1beta1.UpdateRunDeleteStageTaskName, + ) + } for i := range status.StagesStatus { status.StagesStatus[i].BeforeStageTaskStatus = buildTaskStatuses( stages[i].BeforeStageTasks, @@ -1170,7 +1184,7 @@ func generateInitializedStatus( generateTrueCondition(updateRun, placementv1beta1.StagedUpdateRunConditionInitialized), }, } - populateStageTaskStatuses(status, updateRun.Name, updateStrategy.Spec.Stages) + populateStageTaskStatuses(status, updateRun.Name, &updateStrategy.Spec) return status } diff --git a/pkg/controllers/updaterun/initialization_test.go b/pkg/controllers/updaterun/initialization_test.go index 446f70de7..a72ea12d8 100644 --- a/pkg/controllers/updaterun/initialization_test.go +++ b/pkg/controllers/updaterun/initialization_test.go @@ -17,12 +17,17 @@ limitations under the License. package updaterun import ( + "context" + "errors" "fmt" + "strings" "testing" "time" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client/fake" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" ) @@ -169,3 +174,34 @@ func TestValidateAfterStageTask(t *testing.T) { }) } } + +// TestGenerateStagesByStrategy_InvalidDeleteStageTasks tests that an update run whose strategy has invalid before +// stage tasks in its delete stage fails the validation. +func TestGenerateStagesByStrategy_InvalidDeleteStageTasks(t *testing.T) { + scheme := runtime.NewScheme() + if err := placementv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + strategy := &placementv1beta1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{Name: "test-strategy"}, + Spec: placementv1beta1.UpdateStrategySpec{ + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeTimedWait}}, + }, + }, + } + r := &Reconciler{Client: fake.NewClientBuilder().WithScheme(scheme).WithObjects(strategy).Build()} + updateRun := &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: "test-update-run", Generation: 1}, + Spec: placementv1beta1.UpdateRunSpec{StagedUpdateStrategyName: strategy.Name}, + } + + err := r.generateStagesByStrategy(context.Background(), nil, nil, updateRun) + if !errors.Is(err, errValidationFailed) { + t.Fatalf("generateStagesByStrategy() error = %v, want %v", err, errValidationFailed) + } + wantErrMsg := fmt.Sprintf("the before stage tasks are invalid, updateStrategy: `/test-strategy`, stage: %s, err: task 0 of type TimedWait is not allowed in beforeStageTasks, allowed type: Approval", placementv1beta1.UpdateRunDeleteStageName) + if !strings.Contains(err.Error(), wantErrMsg) { + t.Errorf("generateStagesByStrategy() error = %v, want an error containing %q", err, wantErrMsg) + } +} diff --git a/pkg/controllers/updaterun/metrics.go b/pkg/controllers/updaterun/metrics.go index 384925529..06289b21a 100644 --- a/pkg/controllers/updaterun/metrics.go +++ b/pkg/controllers/updaterun/metrics.go @@ -89,25 +89,19 @@ func emitUpdateRunStatusMetric(updateRun placementv1beta1.UpdateRunObj) { updateRunStatus := updateRun.GetUpdateRunStatus() succeedCond := meta.FindStatusCondition(updateRunStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionSucceeded)) if succeedCond != nil && succeedCond.ObservedGeneration == generation { - failureType := determineFailureType(succeedCond) - hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues(updateRun.GetNamespace(), updateRun.GetName(), string(state), - string(placementv1beta1.StagedUpdateRunConditionSucceeded), string(succeedCond.Status), succeedCond.Reason, string(failureType)).SetToCurrentTime() + setUpdateRunStatusMetric(updateRun, state, succeedCond) return } progressingCond := meta.FindStatusCondition(updateRunStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionProgressing)) if progressingCond != nil && progressingCond.ObservedGeneration == generation { - failureType := determineFailureType(progressingCond) - hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues(updateRun.GetNamespace(), updateRun.GetName(), string(state), - string(placementv1beta1.StagedUpdateRunConditionProgressing), string(progressingCond.Status), progressingCond.Reason, string(failureType)).SetToCurrentTime() + setUpdateRunStatusMetric(updateRun, state, progressingCond) return } initializedCond := meta.FindStatusCondition(updateRunStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionInitialized)) if initializedCond != nil && initializedCond.ObservedGeneration == generation { - failureType := determineFailureType(initializedCond) - hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues(updateRun.GetNamespace(), updateRun.GetName(), string(state), - string(placementv1beta1.StagedUpdateRunConditionInitialized), string(initializedCond.Status), initializedCond.Reason, string(failureType)).SetToCurrentTime() + setUpdateRunStatusMetric(updateRun, state, initializedCond) return } @@ -115,6 +109,19 @@ func emitUpdateRunStatusMetric(updateRun placementv1beta1.UpdateRunObj) { klog.V(2).InfoS("There's no valid status condition on updateRun, status updating failed possibly", "updateRun", klog.KObj(updateRun)) } +func setUpdateRunStatusMetric(updateRun placementv1beta1.UpdateRunObj, state placementv1beta1.State, cond *metav1.Condition) { + failureType := determineFailureType(cond) + hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues( + updateRun.GetNamespace(), + updateRun.GetName(), + string(state), + cond.Type, + string(cond.Status), + cond.Reason, + string(failureType), + ).Set(float64(cond.LastTransitionTime.UnixNano()) / float64(time.Second)) +} + // recordApprovalRequestLatency records the time from approval request creation to user approval. func recordApprovalRequestLatency( stageTaskStatus *placementv1beta1.StageTaskStatus, diff --git a/pkg/controllers/updaterun/metrics_test.go b/pkg/controllers/updaterun/metrics_test.go index 0e1ac23ed..75ca0e81e 100644 --- a/pkg/controllers/updaterun/metrics_test.go +++ b/pkg/controllers/updaterun/metrics_test.go @@ -19,8 +19,10 @@ package updaterun import ( "fmt" "testing" + "time" "github.com/google/go-cmp/cmp" + "github.com/prometheus/client_golang/prometheus/testutil" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "go.goms.io/fleet/apis/placement/v1beta1" @@ -199,3 +201,125 @@ func TestIsFailureReason(t *testing.T) { }) } } + +func TestEmitUpdateRunStatusMetricUsesConditionTransitionTime(t *testing.T) { + tests := []struct { + name string + state v1beta1.State + conditions []metav1.Condition + wantType v1beta1.StagedUpdateRunConditionType + wantStatus metav1.ConditionStatus + wantReason string + wantFailure hubmetrics.UpdateRunFailureType + wantTime metav1.Time + }{ + { + name: "failed run keeps failure transition time", + state: v1beta1.StateRun, + conditions: []metav1.Condition{ + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionSucceeded, metav1.ConditionFalse, condition.UpdateRunFailedReason, "internal error", time.Unix(100, 123456789)), + }, + wantType: v1beta1.StagedUpdateRunConditionSucceeded, + wantStatus: metav1.ConditionFalse, + wantReason: condition.UpdateRunFailedReason, + wantFailure: hubmetrics.UpdateRunFailureTypeInternalError, + wantTime: metav1.NewTime(time.Unix(100, 123456789)), + }, + { + name: "initialized run uses initialization transition time", + state: v1beta1.StateInitialize, + conditions: []metav1.Condition{ + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionInitialized, metav1.ConditionTrue, condition.UpdateRunInitializeSucceededReason, "initialized", time.Unix(200, 0)), + }, + wantType: v1beta1.StagedUpdateRunConditionInitialized, + wantStatus: metav1.ConditionTrue, + wantReason: condition.UpdateRunInitializeSucceededReason, + wantFailure: hubmetrics.UpdateRunFailureTypeNone, + wantTime: metav1.NewTime(time.Unix(200, 0)), + }, + { + name: "executing run uses progressing transition time", + state: v1beta1.StateRun, + conditions: []metav1.Condition{ + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionInitialized, metav1.ConditionTrue, condition.UpdateRunInitializeSucceededReason, "initialized", time.Unix(200, 0)), + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionProgressing, metav1.ConditionTrue, condition.UpdateRunProgressingReason, "progressing", time.Unix(300, 0)), + }, + wantType: v1beta1.StagedUpdateRunConditionProgressing, + wantStatus: metav1.ConditionTrue, + wantReason: condition.UpdateRunProgressingReason, + wantFailure: hubmetrics.UpdateRunFailureTypeNone, + wantTime: metav1.NewTime(time.Unix(300, 0)), + }, + { + name: "succeeded run uses success transition time", + state: v1beta1.StateRun, + conditions: []metav1.Condition{ + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionInitialized, metav1.ConditionTrue, condition.UpdateRunInitializeSucceededReason, "initialized", time.Unix(200, 0)), + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionProgressing, metav1.ConditionFalse, condition.UpdateRunSucceededReason, "completed", time.Unix(400, 0)), + newUpdateRunMetricCondition(v1beta1.StagedUpdateRunConditionSucceeded, metav1.ConditionTrue, condition.UpdateRunSucceededReason, "succeeded", time.Unix(400, 0)), + }, + wantType: v1beta1.StagedUpdateRunConditionSucceeded, + wantStatus: metav1.ConditionTrue, + wantReason: condition.UpdateRunSucceededReason, + wantFailure: hubmetrics.UpdateRunFailureTypeNone, + wantTime: metav1.NewTime(time.Unix(400, 0)), + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.Reset() + t.Cleanup(hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.Reset) + + updateRun := &v1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-update-run", + Generation: 1, + }, + Spec: v1beta1.UpdateRunSpec{ + State: tc.state, + }, + Status: v1beta1.UpdateRunStatus{ + Conditions: tc.conditions, + }, + } + + metric := hubmetrics.FleetUpdateRunStatusLastTimestampSeconds.WithLabelValues( + "", + updateRun.Name, + string(tc.state), + string(tc.wantType), + string(tc.wantStatus), + tc.wantReason, + string(tc.wantFailure), + ) + wantTimestamp := float64(tc.wantTime.UnixNano()) / float64(time.Second) + + emitUpdateRunStatusMetric(updateRun) + if got := testutil.ToFloat64(metric); got != wantTimestamp { + t.Fatalf("emitUpdateRunStatusMetric() timestamp = %v, want %v", got, wantTimestamp) + } + + emitUpdateRunStatusMetric(updateRun) + if got := testutil.ToFloat64(metric); got != wantTimestamp { + t.Fatalf("emitUpdateRunStatusMetric() timestamp after unchanged reconciliation = %v, want %v", got, wantTimestamp) + } + }) + } +} + +func newUpdateRunMetricCondition( + conditionType v1beta1.StagedUpdateRunConditionType, + status metav1.ConditionStatus, + reason, message string, + transitionTime time.Time, +) metav1.Condition { + return metav1.Condition{ + Type: string(conditionType), + Status: status, + ObservedGeneration: 1, + LastTransitionTime: metav1.NewTime(transitionTime), + Reason: reason, + Message: message, + } +} diff --git a/pkg/controllers/updaterun/stop.go b/pkg/controllers/updaterun/stop.go index eab8ea088..224d689bb 100644 --- a/pkg/controllers/updaterun/stop.go +++ b/pkg/controllers/updaterun/stop.go @@ -109,7 +109,7 @@ func (r *Reconciler) stopUpdatingStage( // If cluster update has been running for more than "updateRunStuckThreshold", mark the update run as stuck. timeElapsed := time.Since(clusterStartedCond.LastTransitionTime.Time) if timeElapsed > updateRunStuckThreshold { - klog.V(2).InfoS("Time waiting for cluster update to finish passes threshold, mark the update run as stuck", "time elapsed", timeElapsed, "threshold", updateRunStuckThreshold, "cluster", clusterStatus.ClusterName, "stage", updatingStageStatus.StageName, "updateRun", updateRunRef) + klog.V(2).InfoS("Time waiting for cluster update to finish passes threshold, mark the update run as stuck", "timeElapsed", timeElapsed, "threshold", updateRunStuckThreshold, "cluster", clusterStatus.ClusterName, "stage", updatingStageStatus.StageName, "updateRun", updateRunRef) stuckClusterNames = append(stuckClusterNames, clusterStatus.ClusterName) } } @@ -142,51 +142,11 @@ func (r *Reconciler) stopDeleteStage( ) (bool, error) { updateRunRef := klog.KObj(updateRun) updateRunStatus := updateRun.GetUpdateRunStatus() - existingDeleteStageStatus := updateRunStatus.DeletionStageStatus - existingDeleteStageClusterMap := make(map[string]*placementv1beta1.ClusterUpdatingStatus, len(existingDeleteStageStatus.Clusters)) - for i := range existingDeleteStageStatus.Clusters { - existingDeleteStageClusterMap[existingDeleteStageStatus.Clusters[i].ClusterName] = &existingDeleteStageStatus.Clusters[i] - } // Mark the delete stage as stopping in case it's not. - markStageUpdatingStopping(existingDeleteStageStatus, updateRun.GetGeneration()) - - for _, binding := range toBeDeletedBindings { - bindingSpec := binding.GetBindingSpec() - curCluster, exist := existingDeleteStageClusterMap[bindingSpec.TargetCluster] - if !exist { - // This is unexpected because we already checked in validation. - missingErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the to be deleted cluster `%s` is not in the deleting stage during stopping", bindingSpec.TargetCluster)) - klog.ErrorS(missingErr, "The cluster in the deleting stage does not include all the to be deleted binding", "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, missingErr.Error()) - } - // In validation, we already check the binding must exist in the status. - delete(existingDeleteStageClusterMap, bindingSpec.TargetCluster) - // Make sure the cluster is not marked as deleted as the binding is still there. - if condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionSucceeded)), updateRun.GetGeneration()) { - // The cluster status is marked as deleted. - unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the deleted cluster `%s` in the deleting stage still has a binding", bindingSpec.TargetCluster)) - klog.ErrorS(unexpectedErr, "The cluster in the deleting stage is not removed yet but marked as deleted", "cluster", curCluster.ClusterName, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) - } - if condition.IsConditionStatusTrue(meta.FindStatusCondition(curCluster.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { - // The cluster status is marked as being deleted. - if binding.GetDeletionTimestamp().IsZero() { - // The cluster is marked as deleting but the binding is not deleting. - unexpectedErr := controller.NewUnexpectedBehaviorError(fmt.Errorf("the cluster `%s` in the deleting stage is marked as deleting but its corresponding binding is not deleting", curCluster.ClusterName)) - klog.ErrorS(unexpectedErr, "The binding should be deleting before we mark a cluster deleting", "clusterStatus", curCluster, "updateRun", updateRunRef) - return false, fmt.Errorf("%w: %s", errStagedUpdatedAborted, unexpectedErr.Error()) - } - continue - } - } - - // The rest of the clusters in the stage are not in the toBeDeletedBindings so it should be marked as delete succeeded. - for _, clusterStatus := range existingDeleteStageClusterMap { - // Make sure the cluster is marked as deleted. - if !condition.IsConditionStatusTrue(meta.FindStatusCondition(clusterStatus.Conditions, string(placementv1beta1.ClusterUpdatingConditionStarted)), updateRun.GetGeneration()) { - markClusterUpdatingStarted(clusterStatus, updateRun.GetGeneration()) - } - markClusterUpdatingSucceeded(clusterStatus, updateRun.GetGeneration()) + markStageUpdatingStopping(updateRunStatus.DeletionStageStatus, updateRun.GetGeneration()) + // The clusters whose deletion has not started are left as they are. + if _, err := syncDeleteStageClusterStatuses(updateRun, toBeDeletedBindings); err != nil { + return false, err } klog.V(2).InfoS("The delete stage is stopping", "numberOfDeletingClusters", len(toBeDeletedBindings), "updateRun", updateRunRef) @@ -204,7 +164,7 @@ func (r *Reconciler) stopDeleteStage( if allDeletingClustersDeleted { markStageUpdatingStopped(updateRunStatus.DeletionStageStatus, updateRun.GetGeneration()) } - return len(toBeDeletedBindings) == 0, nil + return allDeletingClustersDeleted, nil } // markUpdateRunStopping marks the update run as stopping in memory. @@ -216,11 +176,11 @@ func markUpdateRunStopping(updateRun placementv1beta1.UpdateRunObj) { Status: metav1.ConditionUnknown, ObservedGeneration: updateRun.GetGeneration(), Reason: condition.UpdateRunStoppingReason, - Message: "The update run is the process of stopping, waiting for all the updating/deleting clusters to finish updating before completing the stop process", + Message: "The update run is in the process of stopping, waiting for all the updating/deleting clusters to finish updating before completing the stop process", }) } -// markStageUpdatingStopping marks the stage updating status as pausing in memory. +// markStageUpdatingStopping marks the stage updating status as stopping in memory. func markStageUpdatingStopping(stageUpdatingStatus *placementv1beta1.StageUpdatingStatus, generation int64) { meta.SetStatusCondition(&stageUpdatingStatus.Conditions, metav1.Condition{ Type: string(placementv1beta1.StageUpdatingConditionProgressing), @@ -242,6 +202,8 @@ func markStageUpdatingStopped(stageUpdatingStatus *placementv1beta1.StageUpdatin }) } +// checkIfErrorStagedUpdateAborted marks the updating stage, or the delete stage if there is no updating stage, +// as failed in memory if the error aborts the update run. func checkIfErrorStagedUpdateAborted(err error, updateRun placementv1beta1.UpdateRunObj, updatingStageStatus *placementv1beta1.StageUpdatingStatus) { if errors.Is(err, errStagedUpdatedAborted) { if updatingStageStatus != nil { diff --git a/pkg/controllers/updaterun/stop_integration_test.go b/pkg/controllers/updaterun/stop_integration_test.go index acacd8ae4..daff8a52a 100644 --- a/pkg/controllers/updaterun/stop_integration_test.go +++ b/pkg/controllers/updaterun/stop_integration_test.go @@ -625,6 +625,157 @@ var _ = Describe("UpdateRun stop tests", func() { validateUpdateRunStageMetricsEmitted(generateStageClusterUpdatingMetric(updateRun)) }) }) + + Context("Cluster staged update run should stop and resume at the before-stage tasks of the delete stage", Ordered, func() { + var approvalRequestName string + + getUpdateRun := func() (*placementv1beta1.ClusterStagedUpdateRun, error) { + gotUpdateRun := &placementv1beta1.ClusterStagedUpdateRun{} + if err := k8sClient.Get(ctx, updateRunNamespacedName, gotUpdateRun); err != nil { + return nil, fmt.Errorf("failed to get the update run: %w", err) + } + return gotUpdateRun, nil + } + + toBeDeletedBindingsKeptActual := func() error { + for i := numTargetClusters; i < numTargetClusters+numUnscheduledClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding); err != nil { + return fmt.Errorf("failed to get binding %s: %w", resourceBindings[i].Name, err) + } + if !binding.DeletionTimestamp.IsZero() { + return fmt.Errorf("binding %s is being deleted", binding.Name) + } + } + return nil + } + + BeforeAll(func() { + By("Reassigning the strategy to have no stage tasks and the delete stage with a before-stage approval task") + updateStrategy.Spec.Stages[0].BeforeStageTasks = nil + updateStrategy.Spec.Stages[0].AfterStageTasks = nil + updateStrategy.Spec.DeleteStage = &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + }, + }, + } + Expect(k8sClient.Update(ctx, updateStrategy)).To(Succeed()) + + By("Creating a new clusterStagedUpdateRun") + updateRun.Spec.State = placementv1beta1.StateRun + Expect(k8sClient.Create(ctx, updateRun)).To(Succeed()) + approvalRequestName = fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.Name, placementv1beta1.UpdateRunDeleteStageTaskName) + }) + + It("Should complete the 1st stage once all bindings are available", func() { + Eventually(func() error { + for i := 0; i < numTargetClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + if err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding); err != nil { + return fmt.Errorf("failed to get binding %s: %w", resourceBindings[i].Name, err) + } + if binding.Spec.State != placementv1beta1.BindingStateBound { + return fmt.Errorf("binding %s is not bound yet", binding.Name) + } + if meta.IsStatusConditionTrue(binding.Status.Conditions, string(placementv1beta1.ResourceBindingAvailable)) { + continue + } + meta.SetStatusCondition(&binding.Status.Conditions, generateTrueCondition(binding, placementv1beta1.ResourceBindingAvailable)) + if err := k8sClient.Status().Update(ctx, binding); err != nil { + return fmt.Errorf("failed to mark binding %s as available: %w", binding.Name, err) + } + } + gotUpdateRun, err := getUpdateRun() + if err != nil { + return err + } + if !meta.IsStatusConditionTrue(gotUpdateRun.Status.StagesStatus[0].Conditions, string(placementv1beta1.StageUpdatingConditionSucceeded)) { + return fmt.Errorf("the 1st stage has not succeeded yet") + } + return nil + }, 3*timeout, interval).Should(Succeed(), "failed to complete the 1st stage") + }) + + It("Should wait at the delete stage once the approval request is created", func() { + Eventually(func() error { + gotUpdateRun, err := getUpdateRun() + if err != nil { + return err + } + deleteStageStatus := gotUpdateRun.Status.DeletionStageStatus + if !meta.IsStatusConditionTrue(deleteStageStatus.BeforeStageTaskStatus[0].Conditions, string(placementv1beta1.StageTaskConditionApprovalRequestCreated)) { + return fmt.Errorf("the approval request of the delete stage has not been created yet") + } + if deleteStageStatus.StartTime != nil { + return fmt.Errorf("the delete stage has started before its tasks completed") + } + return nil + }, timeout, interval).Should(Succeed(), "failed to wait at the delete stage") + + Expect(toBeDeletedBindingsKeptActual()).To(Succeed(), "the to-be-deleted bindings should be kept while waiting") + }) + + It("Should stop the update run at the delete stage and keep the bindings when state is Stop", func() { + updateRun = updateClusterStagedUpdateRunState(updateRun.Name, placementv1beta1.StateStop) + + By("Validating the delete stage and the update run have stopped") + Eventually(func() error { + gotUpdateRun, err := getUpdateRun() + if err != nil { + return err + } + stageCond := meta.FindStatusCondition(gotUpdateRun.Status.DeletionStageStatus.Conditions, string(placementv1beta1.StageUpdatingConditionProgressing)) + if stageCond == nil || stageCond.Reason != condition.StageUpdatingStoppedReason || stageCond.ObservedGeneration != gotUpdateRun.Generation { + return fmt.Errorf("the delete stage has not stopped yet: %v", stageCond) + } + runCond := meta.FindStatusCondition(gotUpdateRun.Status.Conditions, string(placementv1beta1.StagedUpdateRunConditionProgressing)) + if runCond == nil || runCond.Reason != condition.UpdateRunStoppedReason || runCond.ObservedGeneration != gotUpdateRun.Generation { + return fmt.Errorf("the update run has not stopped yet: %v", runCond) + } + return nil + }, timeout, interval).Should(Succeed(), "failed to stop the update run") + + By("Approving the approval request and validating the bindings are kept while stopped") + approveClusterApprovalRequest(ctx, approvalRequestName) + Consistently(toBeDeletedBindingsKeptActual, 2*stageUpdatingWaitTime, interval).Should(Succeed(), "the to-be-deleted bindings should be kept while stopped") + }) + + It("Should delete the bindings and complete the update run when state is Run", func() { + updateRun = updateClusterStagedUpdateRunState(updateRun.Name, placementv1beta1.StateRun) + + By("Validating the to-be-deleted bindings are deleted") + Eventually(func() error { + for i := numTargetClusters; i < numTargetClusters+numUnscheduledClusters; i++ { + binding := &placementv1beta1.ClusterResourceBinding{} + err := k8sClient.Get(ctx, types.NamespacedName{Name: resourceBindings[i].Name}, binding) + if err == nil { + return fmt.Errorf("binding %s is not deleted", binding.Name) + } + if !apierrors.IsNotFound(err) { + return fmt.Errorf("get binding %s does not return a not-found error: %w", binding.Name, err) + } + } + return nil + }, timeout, interval).Should(Succeed(), "failed to delete the to-be-deleted bindings") + + By("Validating the update run has succeeded") + Eventually(func() error { + gotUpdateRun, err := getUpdateRun() + if err != nil { + return err + } + if !meta.IsStatusConditionTrue(gotUpdateRun.Status.DeletionStageStatus.BeforeStageTaskStatus[0].Conditions, string(placementv1beta1.StageTaskConditionApprovalRequestApproved)) { + return fmt.Errorf("the approval task of the delete stage is not approved") + } + if !meta.IsStatusConditionTrue(gotUpdateRun.Status.Conditions, string(placementv1beta1.StagedUpdateRunConditionSucceeded)) { + return fmt.Errorf("the update run has not succeeded yet") + } + return nil + }, timeout, interval).Should(Succeed(), "failed to complete the update run") + }) + }) }) func updateClusterStagedUpdateRunState(updateRunName string, state placementv1beta1.State) *placementv1beta1.ClusterStagedUpdateRun { diff --git a/pkg/controllers/updaterun/stop_test.go b/pkg/controllers/updaterun/stop_test.go index f18d20de7..c7a24b99b 100644 --- a/pkg/controllers/updaterun/stop_test.go +++ b/pkg/controllers/updaterun/stop_test.go @@ -613,7 +613,61 @@ func TestStopDeleteStage(t *testing.T) { }, }, { - name: "cluster not marked as deleting and binding not deleting", + name: "deleted cluster and cluster not marked as deleting should stop without deleting the remaining binding", + updateRun: &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-updaterun", + Generation: 1, + }, + Status: placementv1beta1.UpdateRunStatus{ + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: "deletion", + Clusters: []placementv1beta1.ClusterUpdatingStatus{ + { + ClusterName: "cluster-1", + Conditions: []metav1.Condition{ + { + Type: string(placementv1beta1.ClusterUpdatingConditionStarted), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.ClusterUpdatingStartedReason, + }, + { + Type: string(placementv1beta1.ClusterUpdatingConditionSucceeded), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.ClusterUpdatingSucceededReason, + }, + }, + }, + { + ClusterName: "cluster-2", + }, + }, + }, + }, + }, + toBeDeletedBindings: []placementv1beta1.BindingObj{ + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{ + // No DeletionTimestamp set + }, + Spec: placementv1beta1.ResourceBindingSpec{ + TargetCluster: "cluster-2", + }, + }, + }, + wantFinished: true, + wantError: nil, + wantProgressCond: metav1.Condition{ + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: condition.StageUpdatingStoppedReason, + }, + }, + { + name: "cluster waiting on delete stage tasks should stop without deleting its binding", updateRun: &placementv1beta1.ClusterStagedUpdateRun{ ObjectMeta: metav1.ObjectMeta{ Name: "test-updaterun", @@ -640,7 +694,7 @@ func TestStopDeleteStage(t *testing.T) { }, }, }, - wantFinished: false, + wantFinished: true, wantError: nil, wantProgressCond: metav1.Condition{ Type: string(placementv1beta1.StageUpdatingConditionProgressing), @@ -649,6 +703,111 @@ func TestStopDeleteStage(t *testing.T) { Reason: condition.StageUpdatingStoppedReason, }, }, + { + name: "deleting cluster and cluster not marked as deleting should not finish", + updateRun: &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-updaterun", + Generation: 1, + }, + Status: placementv1beta1.UpdateRunStatus{ + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: "deletion", + Clusters: []placementv1beta1.ClusterUpdatingStatus{ + { + ClusterName: "cluster-1", + Conditions: []metav1.Condition{ + { + Type: string(placementv1beta1.ClusterUpdatingConditionStarted), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + LastTransitionTime: now, + Reason: condition.ClusterUpdatingStartedReason, + }, + }, + }, + { + ClusterName: "cluster-2", + }, + }, + }, + }, + }, + toBeDeletedBindings: []placementv1beta1.BindingObj{ + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{ + DeletionTimestamp: &deletionTime, + }, + Spec: placementv1beta1.ResourceBindingSpec{ + TargetCluster: "cluster-1", + }, + }, + &placementv1beta1.ClusterResourceBinding{ + Spec: placementv1beta1.ResourceBindingSpec{ + TargetCluster: "cluster-2", + }, + }, + }, + wantFinished: false, + wantError: nil, + wantProgressCond: metav1.Condition{ + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionUnknown, + ObservedGeneration: 1, + Reason: condition.StageUpdatingStoppingReason, + }, + }, + { + name: "deleted cluster that still has a binding - should abort", + updateRun: &placementv1beta1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-updaterun", + Generation: 1, + }, + Status: placementv1beta1.UpdateRunStatus{ + DeletionStageStatus: &placementv1beta1.StageUpdatingStatus{ + StageName: "deletion", + Clusters: []placementv1beta1.ClusterUpdatingStatus{ + { + ClusterName: "cluster-1", + Conditions: []metav1.Condition{ + { + Type: string(placementv1beta1.ClusterUpdatingConditionStarted), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.ClusterUpdatingStartedReason, + }, + { + Type: string(placementv1beta1.ClusterUpdatingConditionSucceeded), + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: condition.ClusterUpdatingSucceededReason, + }, + }, + }, + }, + }, + }, + }, + toBeDeletedBindings: []placementv1beta1.BindingObj{ + &placementv1beta1.ClusterResourceBinding{ + ObjectMeta: metav1.ObjectMeta{ + DeletionTimestamp: &deletionTime, + }, + Spec: placementv1beta1.ResourceBindingSpec{ + TargetCluster: "cluster-1", + }, + }, + }, + wantFinished: false, + wantError: errors.New("the deleted cluster `cluster-1` in the deleting stage still has a binding"), + wantProgressCond: metav1.Condition{ + Type: string(placementv1beta1.StageUpdatingConditionProgressing), + Status: metav1.ConditionUnknown, + ObservedGeneration: 1, + Reason: condition.StageUpdatingStoppingReason, + }, + }, { name: "multiple clusters with mixed states", updateRun: &placementv1beta1.ClusterStagedUpdateRun{ @@ -729,13 +888,13 @@ func TestStopDeleteStage(t *testing.T) { // Verify error expectation. if (tt.wantError != nil) != (gotErr != nil) { - t.Fatalf("stopUpdatingStage() want error: %v, got error: %v", tt.wantError, gotErr) + t.Fatalf("stopDeleteStage() want error: %v, got error: %v", tt.wantError, gotErr) } // Verify error message contains expected substring. if tt.wantError != nil && gotErr != nil { if !strings.Contains(gotErr.Error(), tt.wantError.Error()) { - t.Fatalf("stopUpdatingStage() want error: %v, got error: %v", tt.wantError, gotErr) + t.Fatalf("stopDeleteStage() want error: %v, got error: %v", tt.wantError, gotErr) } } diff --git a/pkg/controllers/updaterun/suite_test.go b/pkg/controllers/updaterun/suite_test.go index 0eea92111..51cf75db1 100644 --- a/pkg/controllers/updaterun/suite_test.go +++ b/pkg/controllers/updaterun/suite_test.go @@ -45,6 +45,7 @@ import ( "go.goms.io/fleet/pkg/utils" controller "go.goms.io/fleet/pkg/utils/controller" "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" ) var ( @@ -123,10 +124,9 @@ var _ = BeforeSuite(func() { // Setup our main reconciler. resourceSelectorResolver = controller.ResourceSelectorResolver{ - RestMapper: mgr.GetRESTMapper(), - InformerManager: dynamicInformerManager, - ResourceConfig: utils.NewResourceConfig(false), - SkippedNamespaces: map[string]bool{}, + RestMapper: mgr.GetRESTMapper(), + InformerManager: dynamicInformerManager, + ResourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(mgr.GetRESTMapper()), } err = (&Reconciler{ Client: k8sClient, diff --git a/pkg/controllers/updaterun/validation.go b/pkg/controllers/updaterun/validation.go index c54aa26d8..942bd0382 100644 --- a/pkg/controllers/updaterun/validation.go +++ b/pkg/controllers/updaterun/validation.go @@ -289,8 +289,8 @@ func validateDeleteStageStatus( } } - deleteStageFinishedCond := meta.FindStatusCondition(existingDeleteStageStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionSucceeded)) - deleteStageProgressingCond := meta.FindStatusCondition(existingDeleteStageStatus.Conditions, string(placementv1beta1.StagedUpdateRunConditionProgressing)) + deleteStageFinishedCond := meta.FindStatusCondition(existingDeleteStageStatus.Conditions, string(placementv1beta1.StageUpdatingConditionSucceeded)) + deleteStageProgressingCond := meta.FindStatusCondition(existingDeleteStageStatus.Conditions, string(placementv1beta1.StageUpdatingConditionProgressing)) // Check if there is any active updating stage if updatingStageIndex != -1 || lastFinishedStageIndex < totalStages-1 { // There are still stages updating before the delete stage, make sure the delete stage is not active/finished. diff --git a/pkg/controllers/workapplier/controller.go b/pkg/controllers/workapplier/controller.go index 02117d572..93afb8e0c 100644 --- a/pkg/controllers/workapplier/controller.go +++ b/pkg/controllers/workapplier/controller.go @@ -185,11 +185,13 @@ const ( ApplyOrReportDiffResTypeFoundGenerateName ManifestProcessingApplyOrReportDiffResultType = "FoundGenerateName" ApplyOrReportDiffResTypeDuplicated ManifestProcessingApplyOrReportDiffResultType = "Duplicated" ApplyOrReportDiffResTypeFailedToFindObjInMemberCluster ManifestProcessingApplyOrReportDiffResultType = "FailedToFindObjInMemberCluster" - ApplyOrReportDiffResTypeFailedToTakeOver ManifestProcessingApplyOrReportDiffResultType = "FailedToTakeOver" - ApplyOrReportDiffResTypeNotTakenOver ManifestProcessingApplyOrReportDiffResultType = "NotTakenOver" - ApplyOrReportDiffResTypeFailedToRunDriftDetection ManifestProcessingApplyOrReportDiffResultType = "FailedToRunDriftDetection" - ApplyOrReportDiffResTypeFoundDrifts ManifestProcessingApplyOrReportDiffResultType = "FoundDrifts" - ApplyOrReportDiffResTypeFoundDriftsInDegradedMode ManifestProcessingApplyOrReportDiffResultType = "FoundDriftsInDegradedMode" + // Note that the reason string below uses the same value as kept in the v1 work applier. + ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects ManifestProcessingApplyOrReportDiffResultType = "OwnedByOtherKubeFleetAPIObjects" + ApplyOrReportDiffResTypeFailedToTakeOver ManifestProcessingApplyOrReportDiffResultType = "FailedToTakeOver" + ApplyOrReportDiffResTypeNotTakenOver ManifestProcessingApplyOrReportDiffResultType = "NotTakenOver" + ApplyOrReportDiffResTypeFailedToRunDriftDetection ManifestProcessingApplyOrReportDiffResultType = "FailedToRunDriftDetection" + ApplyOrReportDiffResTypeFoundDrifts ManifestProcessingApplyOrReportDiffResultType = "FoundDrifts" + ApplyOrReportDiffResTypeFoundDriftsInDegradedMode ManifestProcessingApplyOrReportDiffResultType = "FoundDriftsInDegradedMode" // Note that the reason string below uses the same value as kept in the old work applier. ApplyOrReportDiffResTypeFailedToApply ManifestProcessingApplyOrReportDiffResultType = "ManifestApplyFailed" @@ -231,6 +233,7 @@ var ( ApplyOrReportDiffResTypeFoundGenerateName, ApplyOrReportDiffResTypeDuplicated, ApplyOrReportDiffResTypeFailedToFindObjInMemberCluster, + ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects, ApplyOrReportDiffResTypeFailedToTakeOver, ApplyOrReportDiffResTypeNotTakenOver, ApplyOrReportDiffResTypeFailedToRunDriftDetection, diff --git a/pkg/controllers/workapplier/controller_integration_test.go b/pkg/controllers/workapplier/controller_integration_test.go index d4d90d12f..d715ddd4e 100644 --- a/pkg/controllers/workapplier/controller_integration_test.go +++ b/pkg/controllers/workapplier/controller_integration_test.go @@ -38,6 +38,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" fleetv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/condition" @@ -5486,6 +5487,271 @@ var _ = Describe("drift detection and takeover", func() { }) }) + Context("objects owned by the newer KubeFleet APIs", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + // The owner reference points to an AppliedWork object from the newer KubeFleet APIs, which does + // not exist; this is OK as there is no garbage collector in the test environment. It is in the same + // form as the newer work applier sets it on the objects it places. + newerAPIAppliedWorkOwnerRef := metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.AppliedWorkKind, + Name: fmt.Sprintf(workNameTemplate, utils.RandStr()), + UID: "uid-newer-api-applied-work", + BlockOwnerDeletion: ptr.To(true), + } + + var preExistingConfigMap *corev1.ConfigMap + var regularConfigMapJSON []byte + + configMapManifestCondIdentifier := fleetv1beta1.WorkResourceIdentifier{ + Ordinal: 0, + Group: "", + Version: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + } + + // Any write (takeover, apply, etc.) would bump the resource version of the config map. + configMapUntouchedActual := func() error { + gotConfigMap := &corev1.ConfigMap{} + if err := memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, gotConfigMap); err != nil { + return fmt.Errorf("failed to retrieve the config map: %w", err) + } + + // Rebuild the config map to ignore fields that are irrelevant to the comparison. + rebuiltGotConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: gotConfigMap.Namespace, + Name: gotConfigMap.Name, + ResourceVersion: gotConfigMap.ResourceVersion, + Annotations: gotConfigMap.Annotations, + OwnerReferences: gotConfigMap.OwnerReferences, + }, + Data: gotConfigMap.Data, + } + wantConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: configMapName, + ResourceVersion: preExistingConfigMap.ResourceVersion, + OwnerReferences: []metav1.OwnerReference{newerAPIAppliedWorkOwnerRef}, + }, + Data: map[string]string{ + dummyLabelKey: dummyLabelValue2, + }, + } + if diff := cmp.Diff(rebuiltGotConfigMap, wantConfigMap); diff != "" { + return fmt.Errorf("config map diff (-got +want):\n%s", diff) + } + return nil + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the NS object") + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON = marshalK8sObjJSON(regularConfigMap) + + // Create the ConfigMap object on the member cluster side, with different data and + // an owner reference that points to an AppliedWork object from the newer KubeFleet APIs. + preExistingConfigMap = configMap.DeepCopy() + preExistingConfigMap.Namespace = nsName + preExistingConfigMap.Data = map[string]string{ + dummyLabelKey: dummyLabelValue2, + } + preExistingConfigMap.OwnerReferences = []metav1.OwnerReference{newerAPIAppliedWorkOwnerRef} + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing ConfigMap object") + + // Create a new Work object with the most permissive takeover setup. + applyStrategy := &fleetv1beta1.ApplyStrategy{ + WhenToTakeOver: fleetv1beta1.WhenToTakeOverTypeAlways, + AllowCoOwnership: true, + } + createWorkObject(workName, memberReservedNSName1, applyStrategy, nil, regularConfigMapJSON) + }) + + It("should add cleanup finalizer to the Work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the Work object") + }) + + It("should prepare an AppliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an AppliedWork object") + }) + + It("should update the Work object status (always take over, co-ownership allowed)", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: fleetv1beta1.WorkConditionTypeApplied, + Status: metav1.ConditionFalse, + Reason: condition.WorkNotAllManifestsAppliedReason, + }, + } + manifestConds := []fleetv1beta1.ManifestCondition{ + { + Identifier: configMapManifestCondIdentifier, + Conditions: []metav1.Condition{ + { + Type: fleetv1beta1.WorkConditionTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestConds, nil, nil) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should not take over or apply the ConfigMap object (always take over, co-ownership allowed)", func() { + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The ConfigMap object has been written to unexpectedly") + }) + + It("should update the AppliedWork object status", func() { + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, nil) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("can switch to the never take over option", func() { + applyStrategy := &fleetv1beta1.ApplyStrategy{ + WhenToTakeOver: fleetv1beta1.WhenToTakeOverTypeNever, + } + updateWorkObject(workName, applyStrategy, regularConfigMapJSON) + }) + + It("should update the Work object status (never take over)", func() { + // Prepare the status information. + // + // Note that the work-level condition must observe the latest generation of the Work object. + workConds := []metav1.Condition{ + { + Type: fleetv1beta1.WorkConditionTypeApplied, + Status: metav1.ConditionFalse, + Reason: condition.WorkNotAllManifestsAppliedReason, + }, + } + manifestConds := []fleetv1beta1.ManifestCondition{ + { + Identifier: configMapManifestCondIdentifier, + Conditions: []metav1.Condition{ + { + Type: fleetv1beta1.WorkConditionTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestConds, nil, nil) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should not touch the ConfigMap object (never take over)", func() { + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The ConfigMap object has been written to unexpectedly") + }) + + It("can switch to the ReportDiff mode", func() { + // Note that the takeover option is defaulted to Always. + applyStrategy := &fleetv1beta1.ApplyStrategy{ + Type: fleetv1beta1.ApplyStrategyTypeReportDiff, + AllowCoOwnership: true, + } + updateWorkObject(workName, applyStrategy, regularConfigMapJSON) + }) + + It("should update the Work object status (report diff)", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: fleetv1beta1.WorkConditionTypeDiffReported, + Status: metav1.ConditionFalse, + Reason: condition.WorkNotAllManifestsDiffReportedReason, + }, + } + manifestConds := []fleetv1beta1.ManifestCondition{ + { + Identifier: configMapManifestCondIdentifier, + Conditions: []metav1.Condition{ + { + Type: fleetv1beta1.WorkConditionTypeDiffReported, + Status: metav1.ConditionFalse, + Reason: string(ApplyOrReportDiffResTypeFailedToReportDiff), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestConds, nil, nil) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should not touch the ConfigMap object (report diff)", func() { + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The ConfigMap object has been written to unexpectedly") + }) + + It("can delete the Work object", func() { + deleteWorkObject(workName, memberReservedNSName1) + + // Ensure that the AppliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the AppliedWork object") + + workRemovedActual := testutilsactuals.WorkObjectRemovedActual(ctx, hubClient, workName, memberReservedNSName1) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the Work object") + }) + + It("should leave the ConfigMap object alone after the Work object is deleted", func() { + Expect(configMapUntouchedActual()).To(Succeed(), "The ConfigMap object has been written to unexpectedly") + }) + + AfterAll(func() { + // Delete the Work object and related resources, in case an earlier step has failed. + work := &fleetv1beta1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete the Work object") + + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the AppliedWork object") + + workRemovedActual := testutilsactuals.WorkObjectRemovedActual(ctx, hubClient, workName, memberReservedNSName1) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the Work object") + + // Remove the pre-existing ConfigMap object. + cm := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: configMapName, + }, + } + Expect(client.IgnoreNotFound(memberClient1.Delete(ctx, cm))).To(Succeed(), "Failed to delete the ConfigMap object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + Context("obscure sensitive data (drift detection)", Ordered, func() { workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) // The environment prepared by the envtest package does not support namespace diff --git a/pkg/controllers/workapplier/drift_detection_takeover.go b/pkg/controllers/workapplier/drift_detection_takeover.go index 99ae07e96..89f445aff 100644 --- a/pkg/controllers/workapplier/drift_detection_takeover.go +++ b/pkg/controllers/workapplier/drift_detection_takeover.go @@ -30,6 +30,7 @@ import ( "k8s.io/apimachinery/pkg/types" "k8s.io/klog/v2" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" fleetv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils/controller" ) @@ -40,6 +41,35 @@ const ( fleetReservedLabelAnnotationDomain = "kubernetes-fleet.io/" ) +// isOwnedByNewerKubeFleetAPIObjects checks if the object in the member cluster is owned by newer KubeFleet API +// objects, specifically the AppliedWork objects in the `placement.kubefleet.dev` API group. +// +// This is added as a safeguard to avoid scenarios where a manifest is applied via both the older ResourcePlacement +// APIs and the newer PlacementPolicy APIs. And the safeguard always runs regardless of the apply strategy in use. +// +// Owner references are matched by API group and kind only, as owner references are not migrated when an API +// version changes; an object might carry an owner reference of any version that the API group has ever served. +func isOwnedByNewerKubeFleetAPIObjects(inMemberClusterObj *unstructured.Unstructured) bool { + if inMemberClusterObj == nil { + return false + } + + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + owner := &curOwners[idx] + gv, err := schema.ParseGroupVersion(owner.APIVersion) + if err != nil { + // The API server rejects owner references with unparsable API versions; normally this should never + // happen. + continue + } + if gv.Group == placementv1alpha1.GroupVersion.Group && owner.Kind == placementv1alpha1.AppliedWorkKind { + return true + } + } + return false +} + // takeOverPreExistingObject takes over a pre-existing object in the member cluster. func (r *Reconciler) takeOverPreExistingObject( ctx context.Context, diff --git a/pkg/controllers/workapplier/drift_detection_takeover_test.go b/pkg/controllers/workapplier/drift_detection_takeover_test.go index 53bd2f99c..b3f135dc0 100644 --- a/pkg/controllers/workapplier/drift_detection_takeover_test.go +++ b/pkg/controllers/workapplier/drift_detection_takeover_test.go @@ -35,6 +35,7 @@ import ( "k8s.io/utils/ptr" ctrlfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" fleetv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" ) @@ -58,6 +59,96 @@ func toUnstructured(t *testing.T, obj runtime.Object) *unstructured.Unstructured return &unstructured.Unstructured{Object: unstructuredObj} } +// TestIsOwnedByNewerKubeFleetAPIObjects tests the isOwnedByNewerKubeFleetAPIObjects function. +func TestIsOwnedByNewerKubeFleetAPIObjects(t *testing.T) { + newerAPIAppliedWorkOwnerRef := metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.AppliedWorkKind, + Name: "applied-work-v1", + UID: "uid-applied-work-v1", + } + // An owner reference of a different (hypothetical) version in the newer KubeFleet API group. + newerAPIOtherVersionAppliedWorkOwnerRef := metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.Group + "/v1beta1", + Kind: placementv1alpha1.AppliedWorkKind, + Name: "applied-work-v1-other-version", + UID: "uid-applied-work-v1-other-version", + } + newerAPIWorkOwnerRef := metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "Work", + Name: "work-v1", + UID: "uid-work-v1", + } + // An owner reference of the AppliedWork kind and of the same version as the newer KubeFleet APIs, but + // in an API group that merely shares a prefix with the newer KubeFleet API group. + lookalikeGroupAppliedWorkOwnerRef := metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.Group + ".example.com/" + placementv1alpha1.GroupVersion.Version, + Kind: placementv1alpha1.AppliedWorkKind, + Name: "applied-work-lookalike", + UID: "uid-applied-work-lookalike", + } + + nsWithOwners := func(ownerRefs ...metav1.OwnerReference) *unstructured.Unstructured { + obj := nsUnstructured.DeepCopy() + obj.SetOwnerReferences(ownerRefs) + return obj + } + + testCases := []struct { + name string + inMemberClusterObj *unstructured.Unstructured + want bool + }{ + { + name: "no object", + }, + { + name: "no owners", + inMemberClusterObj: nsWithOwners(), + }, + { + name: "owned by a non-KubeFleet owner", + inMemberClusterObj: nsWithOwners(dummyOwnerRef), + }, + { + name: "owned by an AppliedWork object from the older KubeFleet APIs", + inMemberClusterObj: nsWithOwners(*appliedWorkOwnerRef), + }, + { + name: "owned by a non-AppliedWork object from the newer KubeFleet APIs", + inMemberClusterObj: nsWithOwners(newerAPIWorkOwnerRef), + }, + { + name: "owned by an AppliedWork object from an API group that shares a prefix with the newer KubeFleet API group", + inMemberClusterObj: nsWithOwners(lookalikeGroupAppliedWorkOwnerRef), + }, + { + name: "owned by an AppliedWork object from the newer KubeFleet APIs", + inMemberClusterObj: nsWithOwners(newerAPIAppliedWorkOwnerRef), + want: true, + }, + { + name: "owned by an AppliedWork object from a different version of the newer KubeFleet APIs", + inMemberClusterObj: nsWithOwners(newerAPIOtherVersionAppliedWorkOwnerRef), + want: true, + }, + { + name: "co-owned by AppliedWork objects from both the older and the newer KubeFleet APIs", + inMemberClusterObj: nsWithOwners(*appliedWorkOwnerRef, dummyOwnerRef, newerAPIAppliedWorkOwnerRef), + want: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := isOwnedByNewerKubeFleetAPIObjects(tc.inMemberClusterObj); got != tc.want { + t.Errorf("isOwnedByNewerKubeFleetAPIObjects() = %t, want %t", got, tc.want) + } + }) + } +} + // Note (chenyu1): The fake client Fleet uses for unit tests has trouble processing certain // requests at the moment; affected test cases will be covered in // integration tests (w/ real clients) instead. diff --git a/pkg/controllers/workapplier/process.go b/pkg/controllers/workapplier/process.go index fededa20c..a27c54de6 100644 --- a/pkg/controllers/workapplier/process.go +++ b/pkg/controllers/workapplier/process.go @@ -25,6 +25,7 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/klog/v2" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" fleetv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils/controller" "go.goms.io/fleet/pkg/utils/resource" @@ -248,6 +249,19 @@ func (r *Reconciler) takeOverInMemberClusterObjectIfApplicable( work *fleetv1beta1.Work, expectedAppliedWorkOwnerRef *metav1.OwnerReference, ) (shouldSkipProcessing bool) { + // Refuse to process objects that are owned by the newer KubeFleet APIs. This check runs before + // any write (takeover or apply) and regardless of the apply strategy in use, so that + // the object is never co-owned by both placement experiences. + if isOwnedByNewerKubeFleetAPIObjects(bundle.inMemberClusterObj) { + wrappedErr := fmt.Errorf("the object to process is already owned by another KubeFleet placement API object from the %s API group; the two placement experiences cannot be used to place the same manifest at the same time", placementv1alpha1.GroupVersion.Group) + bundle.applyOrReportDiffErr = wrappedErr + bundle.applyOrReportDiffResTyp = ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects + klog.ErrorS(wrappedErr, "Failed to check if takeover is needed", + "work", klog.KObj(work), "GVR", *bundle.gvr, "manifestObj", klog.KObj(bundle.manifestObj), + "inMemberClusterObj", klog.KObj(bundle.inMemberClusterObj), "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + return true + } + if !shouldInitiateTakeOverAttempt(bundle.inMemberClusterObj, work.Spec.ApplyStrategy, expectedAppliedWorkOwnerRef) { // Takeover is not necessary; proceed with the processing. klog.V(2).InfoS("Takeover is not needed; skip the step", "work", klog.KObj(work), "GVR", *bundle.gvr, "manifestObj", bundle.workResourceIdentifierStr) diff --git a/pkg/controllers/workapplier/status_test.go b/pkg/controllers/workapplier/status_test.go index 294b02cd3..1a423ba50 100644 --- a/pkg/controllers/workapplier/status_test.go +++ b/pkg/controllers/workapplier/status_test.go @@ -1326,6 +1326,23 @@ func TestSetManifestAppliedCondition(t *testing.T) { }, }, }, + { + name: "owned by other KubeFleet API objects", + manifestCond: &fleetv1beta1.ManifestCondition{}, + applyOrReportDiffResTyp: ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects, + applyOrReportDiffErr: fmt.Errorf("the object to process is already owned by another KubeFleet placement API object"), + observedInMemberClusterGeneration: 1, + wantManifestCond: &fleetv1beta1.ManifestCondition{ + Conditions: []metav1.Condition{ + { + Type: fleetv1beta1.WorkConditionTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects), + ObservedGeneration: 1, + }, + }, + }, + }, { name: "no apply performed", manifestCond: &fleetv1beta1.ManifestCondition{ diff --git a/pkg/controllers/workapplier/waves.go b/pkg/controllers/workapplier/waves.go index 6433584be..9a810b9dc 100644 --- a/pkg/controllers/workapplier/waves.go +++ b/pkg/controllers/workapplier/waves.go @@ -112,8 +112,8 @@ type bundleProcessingWave struct { bundles []*manifestProcessingBundle } -// organizeBundlesIntoProcessingWaves organizes the list of bundles into different -// waves of bundles for parallel processing based on their GVR information. +// organizeBundlesIntoProcessingWaves organizes the list of manifests (their processing states) into different +// waves for parallel processing based on their GVR information. func organizeBundlesIntoProcessingWaves(bundles []*manifestProcessingBundle, workRef klog.ObjectRef) []*bundleProcessingWave { // Pre-allocate the map; 7 is the total count of default wave numbers, though // not all wave numbers might be used. diff --git a/pkg/metrics/hub/metrics.go b/pkg/metrics/hub/metrics.go index 82ff70946..7aba2138d 100644 --- a/pkg/metrics/hub/metrics.go +++ b/pkg/metrics/hub/metrics.go @@ -40,10 +40,10 @@ const ( ) var ( - // FleetPlacementStatusLastTimeStampSeconds is a prometheus metric which keeps track of the last placement status. + // FleetPlacementStatusLastTimeStampSeconds is a prometheus metric which keeps track of the last placement status transition. FleetPlacementStatusLastTimeStampSeconds = prometheus.NewGaugeVec(prometheus.GaugeOpts{ Name: "fleet_workload_placement_status_last_timestamp_seconds", - Help: "Last update timestamp of placement status in seconds", + Help: "Last placement status timestamp in seconds; this is the condition transition timestamp when available, or the placement creation timestamp when the selected condition is missing", }, []string{"namespace", "name", "generation", "conditionType", "status", "reason"}) // FleetEvictionStatus is prometheus metrics which holds the @@ -53,13 +53,30 @@ var ( Help: "Last update timestamp of eviction complete status in seconds", }, []string{"name", "isCompleted", "isValid"}) + // FleetPlacementPolicyStatusLastTimestampSeconds is a prometheus metric which keeps track + // of the last scheduling status reported for a placement policy. The namespace label is + // empty for cluster-scoped placement policies. + FleetPlacementPolicyStatusLastTimestampSeconds = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "fleet_workload_placement_policy_status_last_timestamp_seconds", + Help: "Last update timestamp of placement policy scheduling status in seconds", + }, []string{"namespace", "name", "generation", "conditionType", "status", "reason"}) + + // FleetPlacementPolicyActiveClusterClaims is a prometheus metric which holds the number of + // cluster claims a placement policy currently has outstanding, including claims that are + // being withdrawn and not yet observed gone, held by a provisioner finalizer or not. The + // namespace label is empty for cluster-scoped placement policies. + FleetPlacementPolicyActiveClusterClaims = prometheus.NewGaugeVec(prometheus.GaugeOpts{ + Name: "fleet_workload_placement_policy_active_cluster_claims", + Help: "Number of cluster claims currently outstanding for a placement policy", + }, []string{"namespace", "name"}) + // FleetUpdateRunStatusLastTimestampSeconds is a prometheus metric which holds the - // last update timestamp of update run status in seconds. + // last condition transition timestamp of update run status in seconds. // The failure_type label indicates whether a failure is a user_error (customer configuration issue), // internal_error (requires investigation), or none (no failure). FleetUpdateRunStatusLastTimestampSeconds = prometheus.NewGaugeVec(prometheus.GaugeOpts{ Name: "fleet_workload_update_run_status_last_timestamp_seconds", - Help: "Last update timestamp of update run status in seconds", + Help: "Last transition timestamp of update run status in seconds", }, []string{"namespace", "name", "state", "condition", "status", "reason", "failureType"}) // FleetUpdateRunApprovalRequestLatencySeconds tracks how long users take to approve approval requests. @@ -107,6 +124,8 @@ var ( func init() { metrics.Registry.MustRegister( FleetPlacementStatusLastTimeStampSeconds, + FleetPlacementPolicyStatusLastTimestampSeconds, + FleetPlacementPolicyActiveClusterClaims, FleetEvictionStatus, FleetUpdateRunStatusLastTimestampSeconds, FleetUpdateRunApprovalRequestLatencySeconds, diff --git a/pkg/resourcewatcher/change_detector.go b/pkg/resourcewatcher/change_detector.go index ac7307c45..b63463c8a 100644 --- a/pkg/resourcewatcher/change_detector.go +++ b/pkg/resourcewatcher/change_detector.go @@ -29,10 +29,11 @@ import ( "k8s.io/klog/v2" "sigs.k8s.io/controller-runtime/pkg/manager" - "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/controller" "go.goms.io/fleet/pkg/utils/informer" "go.goms.io/fleet/pkg/utils/keys" + "go.goms.io/fleet/pkg/utils/resourceeligibility" + "go.goms.io/fleet/pkg/v1/controllers/annotationplacement" ) // make sure that our ChangeDetector implements controller runtime interfaces @@ -67,14 +68,17 @@ type ChangeDetector struct { // This controller will be used by both v1alpha1 & v1beta1 ClusterResourcePlacementController. ResourceChangeController controller.Controller + // AnnotatedSources, when set, receives every resource that carries or just lost the + // cluster-selectors annotation, for the annotation-based placement controller. + AnnotatedSources *annotationplacement.SourceQueue + // InformerManager manages all the dynamic informers created by the discovery client InformerManager informer.Manager - // ResourceConfig contains all the API resources that we won't select based on the allowed or skipped propagating APIs option. - ResourceConfig *utils.ResourceConfig - - // SkippedNamespaces contains all the namespaces that we won't select - SkippedNamespaces map[string]bool + // ResourceEligibilityCheckerForV0APIs determines if a given namespace or GVK (GVR) is eligible for placement for v0 APIs. + ResourceEligibilityCheckerForV0APIs resourceeligibility.Checker + // ResourceEligibilityCheckerForV1APIs determines if a given namespace, GVK (GVR), or object is eligible for placement for v1 APIs. + ResourceEligibilityCheckerForV1APIs resourceeligibility.Checker // ConcurrentPlacementWorker is the number of `placement` reconcilers that are // allowed to sync concurrently. @@ -138,7 +142,8 @@ func (d *ChangeDetector) discoverAPIResourcesLoop(ctx context.Context, period ti // discoverResources goes through all the api resources in the cluster and adds event handlers to informers func (d *ChangeDetector) discoverResources(dynamicResourceEventHandler cache.ResourceEventHandler) { - resourcesToWatch := discoverWatchableResources(d.DiscoveryClient, d.RESTMapper, d.ResourceConfig) + // Allow watching of a resource type if any of the eligibility checkers allows it. + resourcesToWatch := discoverWatchableResources(d.DiscoveryClient, d.ResourceEligibilityCheckerForV0APIs, d.ResourceEligibilityCheckerForV1APIs) // On the leader, add event handlers to informers that were already created by InformerPopulator // The informers exist on all pods, but only the leader adds handlers and processes events @@ -168,19 +173,46 @@ func (d *ChangeDetector) dynamicResourceFilter(obj any) bool { } cwKey, _ := key.(keys.ClusterWideKey) - if !utils.ShouldPropagateNamespace(cwKey.Namespace, d.SkippedNamespaces) { - klog.V(5).InfoS("Skip watching resource in namespace", "namespace", cwKey.Namespace, + // Block watching of a resource if the v0 APIs' resource eligibility checker disallows its owner namespace; the v1 APIs' checker is + // not consulted here as they are guaranteed to have the same namespace allow/deny list at the moment. + if d.ResourceEligibilityCheckerForV0APIs != nil && !d.ResourceEligibilityCheckerForV0APIs.AllowsNamespace(cwKey.Namespace) { + klog.V(5).InfoS("Skip watching resource in namespace due to placement eligibility", "namespace", cwKey.Namespace, "group", cwKey.Group, "version", cwKey.Version, "kind", cwKey.Kind, "object", cwKey.Name) return false } + // Block watching of a resource if neither the v0 APIs nor the v1 APIs allow it. + // + // Note that here the original ShouldPropagateObj method is used to determine eligibility for v0 APIs for compatibility reasons; + // the v0 APIs' eligibility checker does not check object-level eligibility (it's a no-op; all objects are considered eligible). if unstructuredObj, ok := obj.(*unstructured.Unstructured); ok { - shouldPropagate, err := controller.ShouldPropagateObj(d.InformerManager, unstructuredObj.DeepCopy(), d.EnableWorkload) - if err != nil || !shouldPropagate { - klog.V(5).InfoS("Skip watching resource in namespace", "namespace", cwKey.Namespace, - "group", cwKey.Group, "version", cwKey.Version, "kind", cwKey.Kind, "object", cwKey.Name) + shouldPropagateForV0APIs, err := controller.ShouldPropagateObj(d.InformerManager, unstructuredObj.DeepCopy(), d.EnableWorkload) + if err != nil { + klog.V(5).InfoS("Skip watching a resource as its eligibility check (for v0 APIs) failed", "err", err, + "group", cwKey.Group, "version", cwKey.Version, "kind", cwKey.Kind, "object", cwKey.Name, "namespace", cwKey.Namespace) return false } + if shouldPropagateForV0APIs { + return true + } + + // AllowsObject does not check the GVK; the GVK check is required so that kinds blocked only for the v1 APIs + // (e.g., Endpoints) do not pass. + if d.ResourceEligibilityCheckerForV1APIs != nil && d.ResourceEligibilityCheckerForV1APIs.AllowsGVK(cwKey.GroupVersionKind()) { + allowed, _, err := d.ResourceEligibilityCheckerForV1APIs.AllowsObject(unstructuredObj) + if err != nil { + klog.V(5).InfoS("Skip watching a resource as its eligibility check (for v1 APIs) failed", "err", err, + "group", cwKey.Group, "version", cwKey.Version, "kind", cwKey.Kind, "object", cwKey.Name, "namespace", cwKey.Namespace) + return false + } + if allowed { + return true + } + } + + klog.V(5).InfoS("Skip watching a resource as it failed all resource eligibility checks", "namespace", cwKey.Namespace, + "group", cwKey.Group, "version", cwKey.Version, "kind", cwKey.Kind, "object", cwKey.Name) + return false } return true diff --git a/pkg/resourcewatcher/change_detector_test.go b/pkg/resourcewatcher/change_detector_test.go index 78f3eeb73..802e13d3e 100644 --- a/pkg/resourcewatcher/change_detector_test.go +++ b/pkg/resourcewatcher/change_detector_test.go @@ -19,9 +19,11 @@ package resourcewatcher import ( "testing" + appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" fakediscovery "k8s.io/client-go/discovery/fake" "k8s.io/client-go/kubernetes/fake" @@ -29,6 +31,7 @@ import ( "k8s.io/client-go/tools/cache" "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/resourceeligibility" testinformer "go.goms.io/fleet/test/utils/informer" testresource "go.goms.io/fleet/test/utils/resource" ) @@ -37,7 +40,8 @@ func TestChangeDetector_discoverResources(t *testing.T) { tests := []struct { name string discoveryResources []*metav1.APIResourceList - resourceConfig *utils.ResourceConfig + v0Checker func(t *testing.T) resourceeligibility.Checker + v1Checker func(t *testing.T) resourceeligibility.Checker }{ { name: "discovers and adds handlers for watchable resources", @@ -50,7 +54,8 @@ func TestChangeDetector_discoverResources(t *testing.T) { }, }, }, - resourceConfig: nil, // Allow all resources + v0Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, + v1Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, }, { name: "skips resources without list/watch verbs", @@ -62,10 +67,11 @@ func TestChangeDetector_discoverResources(t *testing.T) { }, }, }, - resourceConfig: nil, + v0Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, + v1Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, }, { - name: "respects resource config filtering", + name: "respects eligibility checker filtering", discoveryResources: []*metav1.APIResourceList{ { GroupVersion: "v1", @@ -75,11 +81,12 @@ func TestChangeDetector_discoverResources(t *testing.T) { }, }, }, - resourceConfig: func() *utils.ResourceConfig { - rc := utils.NewResourceConfig(false) // Skip mode - _ = rc.Parse("v1/Secret") // Skip secrets - return rc - }(), + v0Checker: func(t *testing.T) resourceeligibility.Checker { + return secretDenyingChecker(t, resourceeligibility.DefaultForV0APIs(nil)) + }, + v1Checker: func(t *testing.T) resourceeligibility.Checker { + return secretDenyingChecker(t, resourceeligibility.DefaultForV1APIs(nil)) + }, }, { name: "discovers apps group resources", @@ -92,7 +99,8 @@ func TestChangeDetector_discoverResources(t *testing.T) { }, }, }, - resourceConfig: nil, + v0Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, + v1Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, }, } @@ -144,10 +152,11 @@ func TestChangeDetector_discoverResources(t *testing.T) { // Create ChangeDetector with the interface type detector := &ChangeDetector{ - DiscoveryClient: fakeDiscovery, - RESTMapper: restMapper, - InformerManager: fakeInformerManager, - ResourceConfig: tt.resourceConfig, + DiscoveryClient: fakeDiscovery, + RESTMapper: restMapper, + InformerManager: fakeInformerManager, + ResourceEligibilityCheckerForV0APIs: tt.v0Checker(t), + ResourceEligibilityCheckerForV1APIs: tt.v1Checker(t), } // Test discoverResources which discovers resources and adds handlers @@ -181,6 +190,14 @@ func TestChangeDetector_dynamicResourceFilter(t *testing.T) { u.SetName(name) return u } + unstructuredReplicaSetWithNonControllerOwner := func(namespace, name string) *unstructured.Unstructured { + u := &unstructured.Unstructured{} + u.SetGroupVersionKind(appsv1.SchemeGroupVersion.WithKind(utils.ReplicaSetKind)) + u.SetNamespace(namespace) + u.SetName(name) + u.SetOwnerReferences([]metav1.OwnerReference{{APIVersion: "v1", Kind: utils.ConfigMapKind, Name: "owner"}}) + return u + } typedSecret := func(namespace, name string) *corev1.Secret { return &corev1.Secret{ TypeMeta: metav1.TypeMeta{ @@ -194,11 +211,20 @@ func TestChangeDetector_dynamicResourceFilter(t *testing.T) { } } + // Mirrors the namespace deny list the hub agent builds in production. + newChecker := func(skippedNSNames ...string) resourceeligibility.Checker { + c := resourceeligibility.New(resourceeligibility.CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(resourceeligibility.NamespaceEligibilityModeDenyList, skippedNSNames, []string{utils.KubeNSNamePrefix, utils.FleetNSNamePrefix}) + return c + } + tests := []struct { - name string - obj any - skippedNamespaces map[string]bool - want bool + name string + obj any + // v0Checker defaults to newChecker() when unset. + v0Checker resourceeligibility.Checker + v1Checker resourceeligibility.Checker + want bool }{ { name: "non-runtime object is filtered out", @@ -240,24 +266,51 @@ func TestChangeDetector_dynamicResourceFilter(t *testing.T) { want: false, }, { - name: "object in user-skipped namespace is filtered out", - obj: unstructuredConfigMap("skip-me", "cm"), - skippedNamespaces: map[string]bool{"skip-me": true}, - want: false, + name: "object in user-skipped namespace is filtered out", + obj: unstructuredConfigMap("skip-me", "cm"), + v0Checker: newChecker("skip-me"), + want: false, + }, + { + // Only the v0 APIs' checker is consulted for namespace eligibility. + name: "object in user-skipped namespace is filtered out even if it is eligible for the v1 APIs", + obj: unstructuredConfigMap("skip-me", "cm"), + v0Checker: newChecker("skip-me"), + v1Checker: resourceeligibility.AllowAll(nil), + want: false, }, { - name: "unstructured ConfigMap kube-root-ca.crt is filtered out by ShouldPropagateObj", + name: "unstructured ConfigMap kube-root-ca.crt is filtered out by ShouldPropagateObj when no v1 APIs checker is set", obj: unstructuredConfigMap("default", "kube-root-ca.crt"), want: false, }, + { + name: "unstructured ConfigMap kube-root-ca.crt is filtered out when the v1 APIs reject the object too", + obj: unstructuredConfigMap("default", "kube-root-ca.crt"), + v1Checker: resourceeligibility.AllowAll(nil), + want: false, + }, + { + // ShouldPropagateObj rejects a ReplicaSet with any owner; the v1 APIs reject only controller owners. + name: "object rejected by ShouldPropagateObj is allowed when the v1 APIs accept it", + obj: unstructuredReplicaSetWithNonControllerOwner("default", "rs"), + v1Checker: resourceeligibility.AllowAll(nil), + want: true, + }, + { + name: "object rejected by ShouldPropagateObj is filtered out when its GVK is ineligible for the v1 APIs", + obj: unstructuredReplicaSetWithNonControllerOwner("default", "rs"), + v1Checker: gvkDenyingChecker(t, appsv1.SchemeGroupVersion.WithKind(utils.ReplicaSetKind)), + want: false, + }, { name: "unstructured ConfigMap with regular name is allowed", obj: unstructuredConfigMap("default", "user-config"), want: true, }, { - // Cluster-scoped objects have an empty namespace; ShouldPropagateNamespace returns true - // for "" (no reserved prefix match, not in skip-list). + // Cluster-scoped objects have an empty namespace, which matches no reserved prefix and + // no deny list entry. name: "cluster-scoped unstructured object is allowed", obj: unstructuredClusterRole("admin"), want: true, @@ -277,9 +330,14 @@ func TestChangeDetector_dynamicResourceFilter(t *testing.T) { fakeInformerManager := &testinformer.FakeManager{ APIResources: make(map[schema.GroupVersionKind]bool), } + v0Checker := tt.v0Checker + if v0Checker == nil { + v0Checker = newChecker() + } detector := &ChangeDetector{ - InformerManager: fakeInformerManager, - SkippedNamespaces: tt.skippedNamespaces, + InformerManager: fakeInformerManager, + ResourceEligibilityCheckerForV0APIs: v0Checker, + ResourceEligibilityCheckerForV1APIs: tt.v1Checker, } got := detector.dynamicResourceFilter(tt.obj) @@ -290,6 +348,131 @@ func TestChangeDetector_dynamicResourceFilter(t *testing.T) { } } +// TestChangeDetector_dynamicResourceFilter_PlacementPolicyAPIs verifies the filter with the built-in checkers, wired +// as in production. With the placement policy (v1) APIs disabled, the v1 checker is unset and ShouldPropagateObj alone +// decides, as it did before the v1 APIs were added; with them enabled, an object that ShouldPropagateObj rejects +// still passes if the v1 APIs accept both its GVK and the object itself. +func TestChangeDetector_dynamicResourceFilter_PlacementPolicyAPIs(t *testing.T) { + newObj := func(apiVersion, kind, namespace, name string, ownerRefs ...metav1.OwnerReference) *unstructured.Unstructured { + u := &unstructured.Unstructured{} + u.SetAPIVersion(apiVersion) + u.SetKind(kind) + u.SetNamespace(namespace) + u.SetName(name) + u.SetOwnerReferences(ownerRefs) + return u + } + isController := true + appsGroupVersion := appsv1.SchemeGroupVersion.String() + deploymentOwner := metav1.OwnerReference{APIVersion: appsGroupVersion, Kind: "Deployment", Name: "web", UID: "uid", Controller: &isController} + nonControllerOwner := metav1.OwnerReference{APIVersion: "v1", Kind: utils.ConfigMapKind, Name: "owner", UID: "uid"} + deleting := func(u *unstructured.Unstructured) *unstructured.Unstructured { + now := metav1.Now() + u.SetDeletionTimestamp(&now) + return u + } + + tests := []struct { + name string + obj *unstructured.Unstructured + wantV1Disabled bool + wantV1Enabled bool + }{ + { + name: "regular config map", + obj: newObj("v1", "ConfigMap", "app", "user-config"), + wantV1Disabled: true, + wantV1Enabled: true, + }, + { + name: "kube-root-ca.crt config map", + obj: newObj("v1", "ConfigMap", "app", "kube-root-ca.crt"), + wantV1Disabled: false, + wantV1Enabled: false, + }, + { + name: "default service account", + obj: newObj("v1", "ServiceAccount", "app", "default"), + wantV1Disabled: false, + wantV1Enabled: false, + }, + { + name: "replica set owned by a deployment", + obj: newObj(appsGroupVersion, "ReplicaSet", "app", "web-abc", deploymentOwner), + wantV1Disabled: false, + wantV1Enabled: false, + }, + { + // ShouldPropagateObj rejects a ReplicaSet with any owner; the v1 APIs reject only controller owners. + name: "replica set with a non-controller owner", + obj: newObj(appsGroupVersion, "ReplicaSet", "app", "rs-owned", nonControllerOwner), + wantV1Disabled: false, + wantV1Enabled: true, + }, + { + name: "replica set with a non-controller owner being deleted", + obj: deleting(newObj(appsGroupVersion, "ReplicaSet", "app", "rs-owned", nonControllerOwner)), + wantV1Disabled: false, + wantV1Enabled: false, + }, + { + name: "standalone replica set", + obj: newObj(appsGroupVersion, "ReplicaSet", "app", "rs"), + wantV1Disabled: true, + wantV1Enabled: true, + }, + { + // The v1 APIs deny endpoints altogether. + name: "endpoints backed by a service", + obj: newObj("v1", "Endpoints", "app", "svc"), + wantV1Disabled: false, + wantV1Enabled: false, + }, + { + name: "standalone endpoints", + obj: newObj("v1", "Endpoints", "app", "standalone"), + wantV1Disabled: true, + wantV1Enabled: true, + }, + { + name: "config map in a reserved namespace", + obj: newObj("v1", "ConfigMap", "kube-system", "cm"), + wantV1Disabled: false, + wantV1Enabled: false, + }, + } + + for _, tt := range tests { + for _, v1Enabled := range []bool{false, true} { + subtestName := tt.name + "/placement policy APIs disabled" + want := tt.wantV1Disabled + if v1Enabled { + subtestName = tt.name + "/placement policy APIs enabled" + want = tt.wantV1Enabled + } + t.Run(subtestName, func(t *testing.T) { + fakeInformerManager := &testinformer.FakeManager{ + APIResources: make(map[schema.GroupVersionKind]bool), + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.ServiceGVR: {Objects: []runtime.Object{newObj("v1", "Service", "app", "svc")}}, + }, + } + detector := &ChangeDetector{ + InformerManager: fakeInformerManager, + ResourceEligibilityCheckerForV0APIs: resourceeligibility.DefaultForV0APIs(nil), + } + if v1Enabled { + detector.ResourceEligibilityCheckerForV1APIs = resourceeligibility.DefaultForV1APIs(nil) + } + + if got := detector.dynamicResourceFilter(tt.obj); got != want { + t.Errorf("dynamicResourceFilter(%s %s/%s) = %v, want %v", tt.obj.GetKind(), tt.obj.GetNamespace(), tt.obj.GetName(), got, want) + } + }) + } + } +} + // TestNewFilteringHandlerOnAllEvents verifies that the helper composes a // FilteringResourceEventHandler that gates each event type on the supplied filter: // callbacks fire only when the filter accepts the object. diff --git a/pkg/resourcewatcher/event_handlers.go b/pkg/resourcewatcher/event_handlers.go index db8ea1db6..97547bc96 100644 --- a/pkg/resourcewatcher/event_handlers.go +++ b/pkg/resourcewatcher/event_handlers.go @@ -24,6 +24,8 @@ import ( "k8s.io/client-go/tools/cache" "k8s.io/klog/v2" "sigs.k8s.io/controller-runtime/pkg/client" + + "go.goms.io/fleet/pkg/v1/controllers/annotationplacement" ) // handleTombStoneObj handles the case that the delete object is a tombStone instead of the real object @@ -60,6 +62,7 @@ func (d *ChangeDetector) onResourceAdded(obj interface{}) { klog.V(3).InfoS("A resource is added", "obj", klog.KObj(metaInfo), "gvk", runtimeObject.GetObjectKind().GroupVersionKind().String()) d.ResourceChangeController.Enqueue(obj) + d.reportAnnotated(obj, annotationplacement.HasClusterSelectorsAnnotation(metaInfo)) } // onResourceUpdated handles object update event and push the updated object to the resource queue. @@ -83,6 +86,8 @@ func (d *ChangeDetector) onResourceUpdated(oldObj, newObj interface{}) { klog.V(3).InfoS("A resource is updated", "obj", oldObjMeta.GetName(), "namespace", oldObjMeta.GetNamespace(), "gvk", runtimeObject.GetObjectKind().GroupVersionKind().String()) d.ResourceChangeController.Enqueue(newObj) + // The old object counts too: removing the annotation is what deletes the generated policy. + d.reportAnnotated(newObj, annotationplacement.HasClusterSelectorsAnnotation(newObjMeta) || annotationplacement.HasClusterSelectorsAnnotation(oldObjMeta)) return } klog.V(4).InfoS("Received a resource updated event with no change", "obj", oldObjMeta.GetName(), @@ -98,4 +103,18 @@ func (d *ChangeDetector) onResourceDeleted(obj interface{}) { } klog.V(3).InfoS("A resource is deleted", "obj", klog.KObj(clientObj), "gvk", clientObj.GetObjectKind().GroupVersionKind().String()) d.ResourceChangeController.Enqueue(clientObj) + // A resource that stops passing the watcher's filter arrives here too, and its generated policy + // has to go; the reconciler tells that apart from a real deletion. + d.reportAnnotated(clientObj, annotationplacement.HasClusterSelectorsAnnotation(clientObj)) +} + +// reportAnnotated hands a resource to annotation-based placement, if it is running and the +// resource is one it cares about. +func (d *ChangeDetector) reportAnnotated(obj interface{}, annotated bool) { + if d.AnnotatedSources == nil || !annotated { + return + } + if clientObj, ok := obj.(client.Object); ok { + d.AnnotatedSources.Add(clientObj) + } } diff --git a/pkg/resourcewatcher/event_handlers_test.go b/pkg/resourcewatcher/event_handlers_test.go index c6f852d3b..7eb953e87 100644 --- a/pkg/resourcewatcher/event_handlers_test.go +++ b/pkg/resourcewatcher/event_handlers_test.go @@ -19,17 +19,22 @@ package resourcewatcher import ( "context" "reflect" + "slices" "testing" + "github.com/google/go-cmp/cmp" + corev1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/tools/cache" + "k8s.io/client-go/util/workqueue" "sigs.k8s.io/controller-runtime/pkg/client" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" fleetv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" - "go.goms.io/fleet/pkg/utils/controller" + "go.goms.io/fleet/pkg/v1/controllers/annotationplacement" ) func TestHandleTombStoneObj(t *testing.T) { @@ -137,3 +142,51 @@ func (t *fakeController) Enqueue(_ interface{}) { func (t *fakeController) Run(_ context.Context, _ int) error { return nil } + +type noopController struct{} + +func (noopController) Enqueue(interface{}) {} +func (noopController) Run(context.Context, int) error { return nil } + +// TestReportAnnotated pins which events reach annotation-based placement: an annotated add, an +// update that keeps or removes the annotation, and an annotated delete; never an unannotated one. +func TestReportAnnotated(t *testing.T) { + configMap := func(name string, annotated bool, rv string) *corev1.ConfigMap { + cm := &corev1.ConfigMap{ + TypeMeta: metav1.TypeMeta{Kind: "ConfigMap", APIVersion: "v1"}, + ObjectMeta: metav1.ObjectMeta{Namespace: "app", Name: name, ResourceVersion: rv}, + } + if annotated { + cm.Annotations = map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: "region=east"} + } + return cm + } + sources := &annotationplacement.SourceQueue{} + d := &ChangeDetector{ResourceChangeController: noopController{}, AnnotatedSources: sources} + + d.onResourceAdded(configMap("added", true, "1")) + d.onResourceAdded(configMap("plain", false, "1")) + d.onResourceUpdated(configMap("removed", true, "1"), configMap("removed", false, "2")) + d.onResourceUpdated(configMap("plain-updated", false, "1"), configMap("plain-updated", false, "2")) + d.onResourceDeleted(cache.DeletedFinalStateUnknown{Obj: configMap("deleted", true, "1")}) + d.onResourceDeleted(configMap("plain-deleted", false, "1")) + + wq := workqueue.NewTypedRateLimitingQueue(workqueue.DefaultTypedControllerRateLimiter[annotationplacement.Request]()) + defer wq.ShutDown() + if err := sources.Start(context.Background(), wq); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + var got []string + for wq.Len() > 0 { + req, _ := wq.Get() + got = append(got, req.Name) + wq.Done(req) + } + slices.Sort(got) + if diff := cmp.Diff([]string{"added", "deleted", "removed"}, got); diff != "" { + t.Errorf("reported resources mismatch (-want +got):\n%s", diff) + } + + // Off when the feature is: no queue, no panic. + (&ChangeDetector{ResourceChangeController: noopController{}}).onResourceAdded(configMap("added", true, "1")) +} diff --git a/pkg/resourcewatcher/informer_populator.go b/pkg/resourcewatcher/informer_populator.go index 1faef543b..999571e9e 100644 --- a/pkg/resourcewatcher/informer_populator.go +++ b/pkg/resourcewatcher/informer_populator.go @@ -26,8 +26,8 @@ import ( "k8s.io/klog/v2" "sigs.k8s.io/controller-runtime/pkg/manager" - "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" ) const ( @@ -54,8 +54,10 @@ type InformerPopulator struct { // InformerManager manages all the dynamic informers created by the discovery client InformerManager informer.Manager - // ResourceConfig contains all the API resources that we won't select based on the allowed or skipped propagating APIs option. - ResourceConfig *utils.ResourceConfig + // ResourceEligibilityCheckerForV0APIs determines if a given namespace, GVK (GVR), or object is eligible for placement for v0 APIs. + ResourceEligibilityCheckerForV0APIs resourceeligibility.Checker + // ResourceEligibilityCheckerForV1APIs determines if a given namespace, GVK (GVR), or object is eligible for placement for v1 APIs. + ResourceEligibilityCheckerForV1APIs resourceeligibility.Checker } // Start runs the informer populator, discovering resources and creating informers. @@ -81,7 +83,7 @@ func (p *InformerPopulator) Start(ctx context.Context) error { // discoverAndCreateInformers discovers API resources and creates informers WITHOUT adding event handlers func (p *InformerPopulator) discoverAndCreateInformers() { - resourcesToWatch := discoverWatchableResources(p.DiscoveryClient, p.RESTMapper, p.ResourceConfig) + resourcesToWatch := discoverWatchableResources(p.DiscoveryClient, p.ResourceEligibilityCheckerForV0APIs, p.ResourceEligibilityCheckerForV1APIs) // Create informers directly without adding event handlers. // This avoids adding any event handlers on follower pods diff --git a/pkg/resourcewatcher/informer_populator_test.go b/pkg/resourcewatcher/informer_populator_test.go index 47c592c55..63f4a9d81 100644 --- a/pkg/resourcewatcher/informer_populator_test.go +++ b/pkg/resourcewatcher/informer_populator_test.go @@ -21,13 +21,16 @@ import ( "testing" "time" + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" fakediscovery "k8s.io/client-go/discovery/fake" "k8s.io/client-go/kubernetes/fake" "k8s.io/client-go/restmapper" - "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/resourceeligibility" testinformer "go.goms.io/fleet/test/utils/informer" testresource "go.goms.io/fleet/test/utils/resource" ) @@ -39,6 +42,37 @@ const ( testSleep = 150 * time.Millisecond ) +// fakeDiscoveryWithPreferredResources serves Resources from ServerPreferredResources, which the +// embedded FakeDiscovery always answers with nil. +type fakeDiscoveryWithPreferredResources struct { + *fakediscovery.FakeDiscovery +} + +func (f *fakeDiscoveryWithPreferredResources) ServerPreferredResources() ([]*metav1.APIResourceList, error) { + return f.Resources, nil +} + +// gvkDenyingChecker returns a checker that allows everything except the given GVKs. +func gvkDenyingChecker(t *testing.T, gvks ...schema.GroupVersionKind) resourceeligibility.Checker { + t.Helper() + c := resourceeligibility.AllowAll(nil) + if err := c.SetGVKCheckList(resourceeligibility.GVKEligibilityModeDenyList, gvks); err != nil { + t.Fatalf("SetGVKCheckList(%v) = %v, want no error", gvks, err) + } + return c +} + +// secretDenyingChecker returns a checker that wraps the given built-in checker and additionally denies secrets. +func secretDenyingChecker(t *testing.T, wrapped resourceeligibility.Checker) resourceeligibility.Checker { + t.Helper() + // Only the GVK check runs during discovery, which does not use the RESTMapper. + c := resourceeligibility.New(resourceeligibility.CheckerOptions{RestMapper: nil, Wrapped: wrapped, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(resourceeligibility.GVKEligibilityModeDenyList, []schema.GroupVersionKind{testresource.GVKSecret()}); err != nil { + t.Fatalf("SetGVKCheckList() = %v, want no error", err) + } + return c +} + func TestInformerPopulator_NeedLeaderElection(t *testing.T) { populator := &InformerPopulator{} @@ -52,9 +86,10 @@ func TestInformerPopulator_discoverAndCreateInformers(t *testing.T) { tests := []struct { name string discoveryResources []*metav1.APIResourceList - resourceConfig *utils.ResourceConfig + v0Checker func(t *testing.T) resourceeligibility.Checker + v1Checker func(t *testing.T) resourceeligibility.Checker expectedInformerCreated bool - expectedResourceCount int + wantResourceGVKs []schema.GroupVersionKind }{ { name: "creates informers for watchable resources", @@ -66,9 +101,10 @@ func TestInformerPopulator_discoverAndCreateInformers(t *testing.T) { }, }, }, - resourceConfig: nil, // Allow all resources + v0Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, + v1Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, expectedInformerCreated: true, - expectedResourceCount: 1, + wantResourceGVKs: []schema.GroupVersionKind{testresource.GVKConfigMap()}, }, { name: "skips resources without list/watch verbs", @@ -80,12 +116,12 @@ func TestInformerPopulator_discoverAndCreateInformers(t *testing.T) { }, }, }, - resourceConfig: nil, + v0Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, + v1Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.AllowAll(nil) }, expectedInformerCreated: false, - expectedResourceCount: 0, }, { - name: "respects resource config filtering", + name: "skips resources denied by both eligibility checkers", discoveryResources: []*metav1.APIResourceList{ { GroupVersion: "v1", @@ -94,13 +130,100 @@ func TestInformerPopulator_discoverAndCreateInformers(t *testing.T) { }, }, }, - resourceConfig: func() *utils.ResourceConfig { - rc := utils.NewResourceConfig(false) // Skip mode - _ = rc.Parse("v1/Secret") // Skip secrets - return rc - }(), + v0Checker: func(t *testing.T) resourceeligibility.Checker { + return secretDenyingChecker(t, resourceeligibility.DefaultForV0APIs(nil)) + }, + v1Checker: func(t *testing.T) resourceeligibility.Checker { + return secretDenyingChecker(t, resourceeligibility.DefaultForV1APIs(nil)) + }, expectedInformerCreated: false, - expectedResourceCount: 0, + }, + { + name: "creates informers for resources allowed by either eligibility checker", + discoveryResources: []*metav1.APIResourceList{ + { + GroupVersion: "v1", + APIResources: []metav1.APIResource{ + testresource.APIResourceConfigMap(), + testresource.APIResourceSecret(), + }, + }, + }, + v0Checker: func(t *testing.T) resourceeligibility.Checker { + return gvkDenyingChecker(t, testresource.GVKSecret()) + }, + v1Checker: func(t *testing.T) resourceeligibility.Checker { + return gvkDenyingChecker(t, testresource.GVKConfigMap()) + }, + expectedInformerCreated: true, + wantResourceGVKs: []schema.GroupVersionKind{testresource.GVKConfigMap(), testresource.GVKSecret()}, + }, + { + // Endpoints are denied only for the v1 APIs, and pods are denied for both. + name: "uses the built-in eligibility checkers for the v0 and v1 APIs", + discoveryResources: []*metav1.APIResourceList{ + { + GroupVersion: "v1", + APIResources: []metav1.APIResource{ + testresource.APIResourceConfigMap(), + testresource.APIResourcePod(), + testresource.APIResourceWithVerbs("endpoints", "Endpoints", true, []string{"get", "list", "watch"}), + }, + }, + }, + v0Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.DefaultForV0APIs(nil) }, + v1Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.DefaultForV1APIs(nil) }, + expectedInformerCreated: true, + wantResourceGVKs: []schema.GroupVersionKind{testresource.GVKConfigMap(), corev1.SchemeGroupVersion.WithKind("Endpoints")}, + }, + { + // The v1 APIs deny a superset of what the v0 APIs deny, so disabling them watches the same resources. + name: "uses the built-in eligibility checker for the v0 APIs only when the v1 APIs are disabled", + discoveryResources: []*metav1.APIResourceList{ + { + GroupVersion: "v1", + APIResources: []metav1.APIResource{ + testresource.APIResourceConfigMap(), + testresource.APIResourcePod(), + testresource.APIResourceWithVerbs("endpoints", "Endpoints", true, []string{"get", "list", "watch"}), + }, + }, + }, + v0Checker: func(_ *testing.T) resourceeligibility.Checker { return resourceeligibility.DefaultForV0APIs(nil) }, + expectedInformerCreated: true, + wantResourceGVKs: []schema.GroupVersionKind{testresource.GVKConfigMap(), corev1.SchemeGroupVersion.WithKind("Endpoints")}, + }, + { + name: "creates informers for all watchable resources when no eligibility checker is set", + discoveryResources: []*metav1.APIResourceList{ + { + GroupVersion: "v1", + APIResources: []metav1.APIResource{ + testresource.APIResourceConfigMap(), + testresource.APIResourceSecret(), + }, + }, + }, + expectedInformerCreated: true, + wantResourceGVKs: []schema.GroupVersionKind{testresource.GVKConfigMap(), testresource.GVKSecret()}, + }, + { + // An unset checker must not be treated as one that allows everything. + name: "ignores an unset eligibility checker", + discoveryResources: []*metav1.APIResourceList{ + { + GroupVersion: "v1", + APIResources: []metav1.APIResource{ + testresource.APIResourceConfigMap(), + testresource.APIResourceSecret(), + }, + }, + }, + v0Checker: func(t *testing.T) resourceeligibility.Checker { + return gvkDenyingChecker(t, testresource.GVKSecret()) + }, + expectedInformerCreated: true, + wantResourceGVKs: []schema.GroupVersionKind{testresource.GVKConfigMap()}, }, } @@ -150,16 +273,31 @@ func TestInformerPopulator_discoverAndCreateInformers(t *testing.T) { // Track calls to CreateInformerForResource populator := &InformerPopulator{ - DiscoveryClient: fakeDiscovery, + DiscoveryClient: &fakeDiscoveryWithPreferredResources{FakeDiscovery: fakeDiscovery}, RESTMapper: restMapper, InformerManager: fakeInformerManager, - ResourceConfig: tt.resourceConfig, + } + if tt.v0Checker != nil { + populator.ResourceEligibilityCheckerForV0APIs = tt.v0Checker(t) + } + if tt.v1Checker != nil { + populator.ResourceEligibilityCheckerForV1APIs = tt.v1Checker(t) } // Run discovery populator.discoverAndCreateInformers() - // Note: FakeManager doesn't track calls, so we verify no panics occurred + gotResourceGVKs := make([]schema.GroupVersionKind, 0, len(fakeInformerManager.CreatedInformers)) + for _, res := range fakeInformerManager.CreatedInformers { + gotResourceGVKs = append(gotResourceGVKs, res.GroupVersionKind) + } + sortGVKs := cmpopts.SortSlices(func(a, b schema.GroupVersionKind) bool { return a.String() < b.String() }) + if diff := cmp.Diff(gotResourceGVKs, tt.wantResourceGVKs, sortGVKs, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("discoverAndCreateInformers() created informers mismatch (-got, +want):\n%s", diff) + } + if got := len(fakeInformerManager.CreatedInformers) > 0; got != tt.expectedInformerCreated { + t.Errorf("discoverAndCreateInformers() informer created = %v, want %v", got, tt.expectedInformerCreated) + } }) } } @@ -197,10 +335,11 @@ func TestInformerPopulator_Start(t *testing.T) { } populator := &InformerPopulator{ - DiscoveryClient: fakeDiscovery, - RESTMapper: restMapper, - InformerManager: fakeInformerManager, - ResourceConfig: nil, + DiscoveryClient: fakeDiscovery, + RESTMapper: restMapper, + InformerManager: fakeInformerManager, + ResourceEligibilityCheckerForV0APIs: resourceeligibility.AllowAll(restMapper), + ResourceEligibilityCheckerForV1APIs: resourceeligibility.AllowAll(restMapper), } // Create a context that will cancel after a short time @@ -269,11 +408,10 @@ func TestInformerPopulator_Integration(t *testing.T) { } restMapper := restmapper.NewDiscoveryRESTMapper(groupResources) - // Create resource config that skips secrets - resourceConfig := utils.NewResourceConfig(false) - err := resourceConfig.Parse("v1/Secret") - if err != nil { - t.Fatalf("Failed to parse resource config: %v", err) + // Create an eligibility checker that skips secrets + eligibilityChecker := resourceeligibility.New(resourceeligibility.CheckerOptions{RestMapper: restMapper, Wrapped: resourceeligibility.DefaultForV0APIs(restMapper), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := eligibilityChecker.SetGVKCheckList(resourceeligibility.GVKEligibilityModeDenyList, []schema.GroupVersionKind{{Version: "v1", Kind: "Secret"}}); err != nil { + t.Fatalf("Failed to set the GVK deny list: %v", err) } fakeInformerManager := &testinformer.FakeManager{ @@ -282,10 +420,11 @@ func TestInformerPopulator_Integration(t *testing.T) { } populator := &InformerPopulator{ - DiscoveryClient: fakeDiscovery, - RESTMapper: restMapper, - InformerManager: fakeInformerManager, - ResourceConfig: resourceConfig, + DiscoveryClient: fakeDiscovery, + RESTMapper: restMapper, + InformerManager: fakeInformerManager, + ResourceEligibilityCheckerForV0APIs: eligibilityChecker, + ResourceEligibilityCheckerForV1APIs: eligibilityChecker, } // Run discovery @@ -328,10 +467,11 @@ func TestInformerPopulator_PeriodicDiscovery(t *testing.T) { } populator := &InformerPopulator{ - DiscoveryClient: fakeDiscovery, - RESTMapper: restMapper, - InformerManager: fakeInformerManager, - ResourceConfig: nil, + DiscoveryClient: fakeDiscovery, + RESTMapper: restMapper, + InformerManager: fakeInformerManager, + ResourceEligibilityCheckerForV0APIs: resourceeligibility.AllowAll(restMapper), + ResourceEligibilityCheckerForV1APIs: resourceeligibility.AllowAll(restMapper), } // Override the discovery period for testing diff --git a/pkg/resourcewatcher/resource_collector.go b/pkg/resourcewatcher/resource_collector.go index cebb7f5be..b71203fb0 100644 --- a/pkg/resourcewatcher/resource_collector.go +++ b/pkg/resourcewatcher/resource_collector.go @@ -17,15 +17,14 @@ limitations under the License. package resourcewatcher import ( - "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/errors" "k8s.io/client-go/discovery" "k8s.io/klog/v2" metricsV1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1" - "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" ) // getWatchableResources returns all api resources from discoveryClient that we can watch. @@ -87,9 +86,9 @@ func getWatchableResources(discoveryClient discovery.ServerResourcesInterface) ( } // discoverWatchableResources discovers all API resources in the cluster and filters them -// based on the resource configuration. This is a shared helper used by both InformerPopulator +// based on their placement eligibility. This is a shared helper used by both InformerPopulator // and ChangeDetector to ensure consistent resource discovery logic. -func discoverWatchableResources(discoveryClient discovery.DiscoveryInterface, restMapper meta.RESTMapper, resourceConfig *utils.ResourceConfig) []informer.APIResourceMeta { +func discoverWatchableResources(discoveryClient discovery.DiscoveryInterface, resourceEligibilityCheckers ...resourceeligibility.Checker) []informer.APIResourceMeta { newResources, err := getWatchableResources(discoveryClient) if err != nil { klog.ErrorS(err, "Failed to get all the api resources from the cluster") @@ -97,7 +96,21 @@ func discoverWatchableResources(discoveryClient discovery.DiscoveryInterface, re var resourcesToWatch []informer.APIResourceMeta for _, res := range newResources { - if utils.ShouldProcessResource(res.GroupVersionResource, restMapper, resourceConfig) { + // A resource is watchable if any of the set eligibility checkers allows it; unset (nil) checkers are + // ignored, and if no checker is set at all, every resource is watchable. + allowed := false + anyCheckerPresent := false + for _, checker := range resourceEligibilityCheckers { + if checker == nil { + continue + } + anyCheckerPresent = true + if checker.AllowsGVK(res.GroupVersionKind) { + allowed = true + break + } + } + if allowed || !anyCheckerPresent { resourcesToWatch = append(resourcesToWatch, res) } } diff --git a/pkg/utils/apiresources.go b/pkg/utils/apiresources.go index f44e003bd..a1204bb5d 100644 --- a/pkg/utils/apiresources.go +++ b/pkg/utils/apiresources.go @@ -29,6 +29,7 @@ import ( metricsV1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1" clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" ) @@ -207,6 +208,12 @@ func NewResourceConfig(isAllowList bool) *ResourceConfig { // disable cluster group by default r.AddGroup(clusterv1beta1.GroupVersion.Group) + // Disable the placement.kubefleet.dev group wholesale: everything in it is KubeFleet's own + // bookkeeping (placement policies, bindings, snapshots, works, claims), including the policies + // that annotation-based placement generates. Treating those as placeable resources would let a + // placement that selects a whole namespace propagate KubeFleet's own generated objects. + r.AddGroup(kfplacementv1alpha1.GroupVersion.Group) + // disable some fleet networking resources r.AddGroupKind(serviceImportGK) r.AddGroupKind(trafficManagerProfileGK) diff --git a/pkg/utils/authorization/authorization.go b/pkg/utils/authorization/authorization.go new file mode 100644 index 000000000..38f2a018d --- /dev/null +++ b/pkg/utils/authorization/authorization.go @@ -0,0 +1,248 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package authorization provides authorization checks for resources selected by Fleet APIs. +package authorization + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "strings" + + "golang.org/x/sync/errgroup" + authenticationv1 "k8s.io/api/authentication/v1" + authorizationv1 "k8s.io/api/authorization/v1" + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/runtime/schema" + ctrlclient "sigs.k8s.io/controller-runtime/pkg/client" +) + +const ( + // DefaultMaxConcurrentReviews limits the number of SubjectAccessReviews an admission request issues concurrently. + DefaultMaxConcurrentReviews = 10 +) + +// SubjectAccessReviewClient creates SubjectAccessReviews. +type SubjectAccessReviewClient interface { + Create(context.Context, ctrlclient.Object, ...ctrlclient.CreateOption) error +} + +// Decision identifies the outcome of an authorization batch. +type Decision string + +const ( + // DecisionAllowed means all required permissions were granted. + DecisionAllowed Decision = "Allowed" + // DecisionDenied means at least one required permission was not granted. + DecisionDenied Decision = "Denied" + // DecisionIndeterminate means authorization could not be completed. + DecisionIndeterminate Decision = "Indeterminate" +) + +// Result reports the aggregate outcome of an authorization batch. +// A denied decision may include Err for additional review failures without changing the decision. +type Result struct { + Decision Decision + DenialReason string + Err error +} + +// Reviewer resolves resource kinds and reviews selected-resource access. +type Reviewer struct { + client SubjectAccessReviewClient + restMapper meta.RESTMapper + maxConcurrentReviews int +} + +// NewReviewer creates a selected-resource authorization reviewer. +func NewReviewer(client SubjectAccessReviewClient, restMapper meta.RESTMapper, maxConcurrentReviews int) (*Reviewer, error) { + if client == nil { + return nil, errors.New("subject access review client must not be nil") + } + if restMapper == nil { + return nil, errors.New("REST mapper must not be nil") + } + if maxConcurrentReviews < 1 { + return nil, errors.New("maximum concurrent reviews must be greater than zero") + } + return &Reviewer{ + client: client, + restMapper: restMapper, + maxConcurrentReviews: maxConcurrentReviews, + }, nil +} + +// ResourceAttributesFor resolves a resource kind to the plural resource used by Kubernetes authorization. +func (r *Reviewer) ResourceAttributesFor(gvk schema.GroupVersionKind, verb, namespace, name string) (authorizationv1.ResourceAttributes, error) { + mapping, err := r.restMapper.RESTMapping(gvk.GroupKind(), gvk.Version) + if err != nil { + return authorizationv1.ResourceAttributes{}, fmt.Errorf("failed to resolve resource mapping for %s: %w", gvk.String(), err) + } + return authorizationv1.ResourceAttributes{ + Namespace: namespace, + Verb: verb, + Group: mapping.Resource.Group, + Version: mapping.Resource.Version, + Resource: mapping.Resource.Resource, + Name: name, + }, nil +} + +// Authorize verifies that the request user is allowed to perform all the supplied resource actions. +func (r *Reviewer) Authorize(ctx context.Context, userInfo authenticationv1.UserInfo, attributes ...authorizationv1.ResourceAttributes) Result { + userInfo = *userInfo.DeepCopy() + uniqueAttributes := make([]authorizationv1.ResourceAttributes, 0, len(attributes)) + seen := make(map[string]struct{}, len(attributes)) + for i := range attributes { + key, err := reviewKey(attributes[i]) + if err != nil { + return Result{Decision: DecisionIndeterminate, Err: fmt.Errorf("failed to build authorization review key: %w", err)} + } + if _, ok := seen[key]; ok { + continue + } + seen[key] = struct{}{} + uniqueAttributes = append(uniqueAttributes, attributes[i]) + } + + var reviews errgroup.Group + reviews.SetLimit(r.maxConcurrentReviews) + results := make([]Result, len(uniqueAttributes)) + for i := range uniqueAttributes { + attributes := uniqueAttributes[i] + if err := ctx.Err(); err != nil { + for j := i; j < len(uniqueAttributes); j++ { + results[j] = Result{ + Decision: DecisionIndeterminate, + Err: fmt.Errorf("authorization review for %s was not submitted: %w", describeAttributes(uniqueAttributes[j]), err), + } + } + break + } + reviews.Go(func() error { + results[i] = r.review(ctx, userInfo, attributes) + return nil + }) + } + // Review outcomes are stored in results; group functions do not return errors. + _ = reviews.Wait() + + reasons := make([]string, 0, len(results)) + errs := make([]error, 0, len(results)) + for i := range results { + switch results[i].Decision { + case DecisionAllowed: + continue + case DecisionDenied: + if results[i].DenialReason != "" { + reasons = append(reasons, results[i].DenialReason) + } else { + errs = append(errs, fmt.Errorf("authorization review for %s returned a denial without a reason", describeAttributes(uniqueAttributes[i]))) + } + case DecisionIndeterminate: + if results[i].Err == nil { + errs = append(errs, fmt.Errorf("authorization review for %s returned an indeterminate decision without an error", describeAttributes(uniqueAttributes[i]))) + } + default: + errs = append(errs, fmt.Errorf("authorization review for %s returned an unknown decision %q", describeAttributes(uniqueAttributes[i]), results[i].Decision)) + } + if results[i].Err != nil { + errs = append(errs, results[i].Err) + } + } + if len(reasons) != 0 { + return Result{ + Decision: DecisionDenied, + DenialReason: strings.Join(reasons, "; "), + Err: errors.Join(errs...), + } + } + if len(errs) != 0 { + return Result{Decision: DecisionIndeterminate, Err: errors.Join(errs...)} + } + return Result{Decision: DecisionAllowed} +} + +func (r *Reviewer) review(ctx context.Context, userInfo authenticationv1.UserInfo, attributes authorizationv1.ResourceAttributes) Result { + review := &authorizationv1.SubjectAccessReview{ + Spec: authorizationv1.SubjectAccessReviewSpec{ + User: userInfo.Username, + UID: userInfo.UID, + Groups: append([]string(nil), userInfo.Groups...), + Extra: authorizationExtra(userInfo.Extra), + ResourceAttributes: attributes.DeepCopy(), + }, + } + if err := r.client.Create(ctx, review); err != nil { + return Result{Decision: DecisionIndeterminate, Err: fmt.Errorf("failed to create SubjectAccessReview for %s: %w", describeAttributes(attributes), err)} + } + if review.Status.Allowed && review.Status.Denied { + return Result{Decision: DecisionIndeterminate, Err: fmt.Errorf("SubjectAccessReview for %s returned conflicting allowed and denied decisions", describeAttributes(attributes))} + } + if review.Status.Allowed { + return Result{Decision: DecisionAllowed} + } + if review.Status.EvaluationError != "" { + return Result{Decision: DecisionIndeterminate, Err: fmt.Errorf("SubjectAccessReview for %s reported an evaluation error: %s", describeAttributes(attributes), review.Status.EvaluationError)} + } + decision := "no opinion" + if review.Status.Denied { + decision = "denied" + } + if review.Status.Reason == "" { + return Result{Decision: DecisionDenied, DenialReason: fmt.Sprintf("SubjectAccessReview %s access to %s", decision, describeAttributes(attributes))} + } + return Result{Decision: DecisionDenied, DenialReason: fmt.Sprintf("SubjectAccessReview %s access to %s: %s", decision, describeAttributes(attributes), review.Status.Reason)} +} + +func reviewKey(attributes authorizationv1.ResourceAttributes) (string, error) { + key, err := json.Marshal(attributes) + if err != nil { + return "", err + } + return string(key), nil +} + +func authorizationExtra(extra map[string]authenticationv1.ExtraValue) map[string]authorizationv1.ExtraValue { + if extra == nil { + return nil + } + converted := make(map[string]authorizationv1.ExtraValue, len(extra)) + for key, values := range extra { + converted[key] = append(authorizationv1.ExtraValue(nil), values...) + } + return converted +} + +func describeAttributes(attributes authorizationv1.ResourceAttributes) string { + resource := attributes.Resource + if attributes.Group != "" { + resource = resource + "." + attributes.Group + } + if attributes.Subresource != "" { + resource = resource + "/" + attributes.Subresource + } + target := resource + if attributes.Name != "" { + target = fmt.Sprintf("%s %q", resource, attributes.Name) + } + if attributes.Namespace != "" { + target = fmt.Sprintf("%s in namespace %q", target, attributes.Namespace) + } + return fmt.Sprintf("%s %s", attributes.Verb, target) +} diff --git a/pkg/utils/authorization/authorization_test.go b/pkg/utils/authorization/authorization_test.go new file mode 100644 index 000000000..a005765ff --- /dev/null +++ b/pkg/utils/authorization/authorization_test.go @@ -0,0 +1,520 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package authorization + +import ( + "context" + "errors" + "fmt" + "strings" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + authenticationv1 "k8s.io/api/authentication/v1" + authorizationv1 "k8s.io/api/authorization/v1" + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/runtime/schema" + ctrlclient "sigs.k8s.io/controller-runtime/pkg/client" +) + +type fakeSubjectAccessReviewClient struct { + create func(context.Context, *authorizationv1.SubjectAccessReview) error +} + +func (f *fakeSubjectAccessReviewClient) Create(ctx context.Context, obj ctrlclient.Object, _ ...ctrlclient.CreateOption) error { + review, ok := obj.(*authorizationv1.SubjectAccessReview) + if !ok { + return fmt.Errorf("object type = %T, want *authorizationv1.SubjectAccessReview", obj) + } + return f.create(ctx, review) +} + +func TestResourceAttributesFor(t *testing.T) { + gvk := schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "Deployment"} + mapper := meta.NewDefaultRESTMapper([]schema.GroupVersion{gvk.GroupVersion()}) + mapper.AddSpecific( + gvk, + schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"}, + schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployment"}, + meta.RESTScopeNamespace, + ) + reviewer, err := NewReviewer(&fakeSubjectAccessReviewClient{}, mapper, DefaultMaxConcurrentReviews) + if err != nil { + t.Fatalf("NewReviewer() returned error %v, want nil", err) + } + + got, err := reviewer.ResourceAttributesFor(gvk, "get", "workloads", "frontend") + if err != nil { + t.Fatalf("ResourceAttributesFor() returned error %v, want nil", err) + } + want := authorizationv1.ResourceAttributes{ + Namespace: "workloads", + Verb: "get", + Group: "apps", + Version: "v1", + Resource: "deployments", + Name: "frontend", + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("ResourceAttributesFor() mismatch (-want +got):\n%s", diff) + } +} + +func TestResourceAttributesForUnknownKind(t *testing.T) { + mapper := meta.NewDefaultRESTMapper(nil) + reviewer, err := NewReviewer(&fakeSubjectAccessReviewClient{}, mapper, DefaultMaxConcurrentReviews) + if err != nil { + t.Fatalf("NewReviewer() returned error %v, want nil", err) + } + + _, err = reviewer.ResourceAttributesFor( + schema.GroupVersionKind{Group: "unknown.example.io", Version: "v1", Kind: "Unknown"}, + "get", + "", + "example", + ) + if err == nil { + t.Fatal("ResourceAttributesFor() returned nil error, want mapping error") + } +} + +func TestAuthorizePreservesUserInfo(t *testing.T) { + attributes := authorizationv1.ResourceAttributes{ + Namespace: "workloads", + Verb: "get", + Group: "", + Version: "v1", + Resource: "secrets", + Name: "database", + } + userInfo := authenticationv1.UserInfo{ + Username: "user@example.com", + UID: "requester-uid", + Groups: []string{"system:authenticated", "developers"}, + Extra: map[string]authenticationv1.ExtraValue{ + "oid": {"object-id"}, + "tenant-id": {"tenant"}, + "multiValue": {"first", "second"}, + }, + } + var gotSpec authorizationv1.SubjectAccessReviewSpec + client := &fakeSubjectAccessReviewClient{ + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + gotSpec = *review.Spec.DeepCopy() + review.Status = authorizationv1.SubjectAccessReviewStatus{Allowed: true} + return nil + }, + } + reviewer := newTestReviewer(t, client, DefaultMaxConcurrentReviews) + + result := reviewer.Authorize(context.Background(), userInfo, attributes) + if result.Decision != DecisionAllowed || result.Err != nil { + t.Fatalf("Authorize() = %+v, want allowed result", result) + } + + wantSpec := authorizationv1.SubjectAccessReviewSpec{ + User: userInfo.Username, + UID: userInfo.UID, + Groups: userInfo.Groups, + Extra: map[string]authorizationv1.ExtraValue{ + "oid": {"object-id"}, + "tenant-id": {"tenant"}, + "multiValue": {"first", "second"}, + }, + ResourceAttributes: attributes.DeepCopy(), + } + if diff := cmp.Diff(wantSpec, gotSpec); diff != "" { + t.Errorf("Authorize() SubjectAccessReview mismatch (-want +got):\n%s", diff) + } +} + +func TestAuthorizeReturnsDenialsAndErrors(t *testing.T) { + apiErr := errors.New("API server unavailable") + testCases := map[string]struct { + create func(context.Context, *authorizationv1.SubjectAccessReview) error + wantErr string + reason string + }{ + "denied": { + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + review.Status = authorizationv1.SubjectAccessReviewStatus{Denied: true, Reason: "RBAC denied the request"} + return nil + }, + reason: "denied access", + }, + "no opinion": { + create: func(context.Context, *authorizationv1.SubjectAccessReview) error { + return nil + }, + reason: "no opinion", + }, + "evaluation error": { + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + review.Status = authorizationv1.SubjectAccessReviewStatus{EvaluationError: "authorizer failed"} + return nil + }, + wantErr: "evaluation error", + }, + "conflicting decision": { + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + review.Status = authorizationv1.SubjectAccessReviewStatus{Allowed: true, Denied: true} + return nil + }, + wantErr: "conflicting", + }, + "API error": { + create: func(context.Context, *authorizationv1.SubjectAccessReview) error { + return apiErr + }, + wantErr: apiErr.Error(), + }, + } + + for name, tc := range testCases { + t.Run(name, func(t *testing.T) { + reviewer := newTestReviewer(t, &fakeSubjectAccessReviewClient{create: tc.create}, DefaultMaxConcurrentReviews) + result := reviewer.Authorize(context.Background(), authenticationv1.UserInfo{Username: "requester"}, authorizationv1.ResourceAttributes{ + Verb: "get", + Resource: "namespaces", + Name: "workloads", + }) + wantDecision := DecisionDenied + if tc.wantErr != "" { + wantDecision = DecisionIndeterminate + } + if result.Decision != wantDecision { + t.Errorf("Authorize() decision = %q, want %q", result.Decision, wantDecision) + } + if tc.wantErr != "" && (result.Err == nil || !strings.Contains(result.Err.Error(), tc.wantErr)) { + t.Errorf("Authorize() error = %v, want error containing %q", result.Err, tc.wantErr) + } + if tc.reason != "" && (result.Err != nil || !strings.Contains(result.DenialReason, tc.reason)) { + t.Errorf("Authorize() = %+v, want denial reason containing %q", result, tc.reason) + } + }) + } +} + +func TestAuthorizeAllowsConclusiveDecisionWithEvaluationError(t *testing.T) { + client := &fakeSubjectAccessReviewClient{ + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + review.Status = authorizationv1.SubjectAccessReviewStatus{ + Allowed: true, + EvaluationError: "another authorizer failed", + } + return nil + }, + } + reviewer := newTestReviewer(t, client, DefaultMaxConcurrentReviews) + + result := reviewer.Authorize(context.Background(), authenticationv1.UserInfo{Username: "requester"}, authorizationv1.ResourceAttributes{ + Verb: "get", + Resource: "namespaces", + Name: "workloads", + }) + if result.Decision != DecisionAllowed || result.Err != nil { + t.Errorf("Authorize() = %+v, want allowed result", result) + } +} + +func TestAuthorizeHonorsContextTimeout(t *testing.T) { + client := &fakeSubjectAccessReviewClient{ + create: func(ctx context.Context, _ *authorizationv1.SubjectAccessReview) error { + <-ctx.Done() + return ctx.Err() + }, + } + reviewer := newTestReviewer(t, client, DefaultMaxConcurrentReviews) + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + + result := reviewer.Authorize(ctx, authenticationv1.UserInfo{Username: "requester"}, authorizationv1.ResourceAttributes{Verb: "get", Resource: "namespaces", Name: "workloads"}) + if !errors.Is(result.Err, context.DeadlineExceeded) { + t.Errorf("Authorize() error = %v, want %v", result.Err, context.DeadlineExceeded) + } +} + +func TestAuthorizeRecordsUnsubmittedReviews(t *testing.T) { + var calls atomic.Int32 + client := &fakeSubjectAccessReviewClient{ + create: func(context.Context, *authorizationv1.SubjectAccessReview) error { + calls.Add(1) + return nil + }, + } + reviewer := newTestReviewer(t, client, DefaultMaxConcurrentReviews) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + result := reviewer.Authorize( + ctx, + authenticationv1.UserInfo{Username: "requester"}, + authorizationv1.ResourceAttributes{Verb: "get", Resource: "configmaps", Name: "first"}, + authorizationv1.ResourceAttributes{Verb: "get", Resource: "configmaps", Name: "second"}, + ) + + if result.Decision != DecisionIndeterminate { + t.Errorf("Authorize() = %+v, want error result", result) + } + if !errors.Is(result.Err, context.Canceled) { + t.Errorf("Authorize() error = %v, want joined error containing %v", result.Err, context.Canceled) + } + for _, name := range []string{"first", "second"} { + want := fmt.Sprintf(`authorization review for get configmaps %q was not submitted`, name) + if !strings.Contains(result.Err.Error(), want) { + t.Errorf("Authorize() error = %q, want error containing %q", result.Err, want) + } + } + if got := calls.Load(); got != 0 { + t.Errorf("SubjectAccessReview calls = %d, want 0", got) + } +} + +func TestAuthorizeReturnsDenialWhenConcurrencyIsLimited(t *testing.T) { + client := &fakeSubjectAccessReviewClient{ + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + review.Status = authorizationv1.SubjectAccessReviewStatus{Denied: true, Reason: "RBAC denied the request"} + return nil + }, + } + reviewer := newTestReviewer(t, client, 1) + + result := reviewer.Authorize( + context.Background(), + authenticationv1.UserInfo{Username: "requester"}, + authorizationv1.ResourceAttributes{Verb: "get", Resource: "configmaps", Name: "first"}, + authorizationv1.ResourceAttributes{Verb: "get", Resource: "configmaps", Name: "second"}, + ) + if result.Decision != DecisionDenied || result.Err != nil || !strings.Contains(result.DenialReason, "RBAC denied the request") { + t.Errorf("Authorize() = %+v, want original authorization denial", result) + } +} + +func TestAuthorizeCollectsReviewResults(t *testing.T) { + apiErr := errors.New("API server unavailable") + var calls atomic.Int32 + client := &fakeSubjectAccessReviewClient{ + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + calls.Add(1) + switch review.Spec.ResourceAttributes.Name { + case "first": + review.Status = authorizationv1.SubjectAccessReviewStatus{Denied: true, Reason: "first denied"} + return nil + case "second": + return apiErr + case "third": + review.Status = authorizationv1.SubjectAccessReviewStatus{Denied: true, Reason: "third denied"} + return nil + default: + return fmt.Errorf("unexpected resource %q", review.Spec.ResourceAttributes.Name) + } + }, + } + reviewer := newTestReviewer(t, client, DefaultMaxConcurrentReviews) + + result := reviewer.Authorize( + context.Background(), + authenticationv1.UserInfo{Username: "requester"}, + authorizationv1.ResourceAttributes{Verb: "get", Resource: "configmaps", Name: "first"}, + authorizationv1.ResourceAttributes{Verb: "get", Resource: "configmaps", Name: "second"}, + authorizationv1.ResourceAttributes{Verb: "get", Resource: "configmaps", Name: "third"}, + ) + + if result.Decision != DecisionDenied { + t.Errorf("Authorize() = %+v, want denied result", result) + } + wantReason := `SubjectAccessReview denied access to get configmaps "first": first denied; SubjectAccessReview denied access to get configmaps "third": third denied` + if result.DenialReason != wantReason { + t.Errorf("Authorize() denial reason = %q, want %q", result.DenialReason, wantReason) + } + if !errors.Is(result.Err, apiErr) { + t.Errorf("Authorize() error = %v, want joined error containing %v", result.Err, apiErr) + } + if got := calls.Load(); got != 3 { + t.Errorf("SubjectAccessReview calls = %d, want 3", got) + } +} + +func TestAuthorizeDeduplicatesReviews(t *testing.T) { + var calls atomic.Int32 + client := &fakeSubjectAccessReviewClient{ + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + calls.Add(1) + review.Status = authorizationv1.SubjectAccessReviewStatus{Allowed: true} + return nil + }, + } + reviewer := newTestReviewer(t, client, DefaultMaxConcurrentReviews) + attributes := authorizationv1.ResourceAttributes{Verb: "get", Resource: "namespaces", Name: "workloads"} + + result := reviewer.Authorize(context.Background(), authenticationv1.UserInfo{Username: "requester"}, attributes, attributes) + if result.Decision != DecisionAllowed || result.Err != nil { + t.Fatalf("Authorize() = %+v, want allowed result", result) + } + if got := calls.Load(); got != 1 { + t.Errorf("SubjectAccessReview calls = %d, want 1", got) + } +} + +func TestAuthorizeDoesNotCacheAcrossCalls(t *testing.T) { + var calls atomic.Int32 + client := &fakeSubjectAccessReviewClient{ + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + calls.Add(1) + review.Status = authorizationv1.SubjectAccessReviewStatus{Allowed: true} + return nil + }, + } + reviewer := newTestReviewer(t, client, DefaultMaxConcurrentReviews) + userInfo := authenticationv1.UserInfo{Username: "requester"} + attributes := authorizationv1.ResourceAttributes{Verb: "get", Resource: "namespaces", Name: "workloads"} + + result := reviewer.Authorize(context.Background(), userInfo, attributes) + if result.Decision != DecisionAllowed || result.Err != nil { + t.Fatalf("Authorize() = %+v, want allowed result", result) + } + result = reviewer.Authorize(context.Background(), userInfo, attributes) + if result.Decision != DecisionAllowed || result.Err != nil { + t.Fatalf("Authorize() = %+v, want allowed result", result) + } + if got := calls.Load(); got != 2 { + t.Errorf("SubjectAccessReview calls = %d, want 2", got) + } +} + +func TestAuthorizeLimitsConcurrency(t *testing.T) { + const ( + reviewCount = 5 + maxConcurrentReview = 2 + ) + started := make(chan struct{}, reviewCount) + release := make(chan struct{}) + var closeRelease sync.Once + defer closeRelease.Do(func() { close(release) }) + + client := &fakeSubjectAccessReviewClient{ + create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + started <- struct{}{} + <-release + review.Status = authorizationv1.SubjectAccessReviewStatus{Allowed: true} + return nil + }, + } + reviewer := newTestReviewer(t, client, maxConcurrentReview) + attributes := make([]authorizationv1.ResourceAttributes, reviewCount) + for i := range attributes { + attributes[i] = authorizationv1.ResourceAttributes{ + Verb: "get", + Resource: "configmaps", + Name: string(rune('a' + i)), + } + } + + result := make(chan Result, 1) + go func() { + result <- reviewer.Authorize(context.Background(), authenticationv1.UserInfo{Username: "requester"}, attributes...) + }() + + for i := 0; i < maxConcurrentReview; i++ { + select { + case <-started: + case <-time.After(time.Second): + t.Fatalf("Authorize() started %d reviews, want %d", i, maxConcurrentReview) + } + } + select { + case <-started: + t.Fatalf("Authorize() exceeded concurrency limit %d", maxConcurrentReview) + case <-time.After(50 * time.Millisecond): + } + + closeRelease.Do(func() { close(release) }) + select { + case got := <-result: + if got.Decision != DecisionAllowed || got.Err != nil { + t.Errorf("Authorize() = %+v, want allowed result", got) + } + case <-time.After(time.Second): + t.Fatal("Authorize() did not finish after reviews were released") + } +} + +func TestDescribeAttributes(t *testing.T) { + testCases := map[string]struct { + attributes authorizationv1.ResourceAttributes + want string + }{ + "core cluster-scoped named resource": { + attributes: authorizationv1.ResourceAttributes{ + Verb: "get", + Resource: "namespaces", + Name: "workloads", + }, + want: `get namespaces "workloads"`, + }, + "grouped namespaced named resource": { + attributes: authorizationv1.ResourceAttributes{ + Namespace: "workloads", + Verb: "get", + Group: "apps", + Resource: "deployments", + Name: "frontend", + }, + want: `get deployments.apps "frontend" in namespace "workloads"`, + }, + "subresource": { + attributes: authorizationv1.ResourceAttributes{ + Namespace: "workloads", + Verb: "update", + Group: "apps", + Resource: "deployments", + Subresource: "status", + Name: "frontend", + }, + want: `update deployments.apps/status "frontend" in namespace "workloads"`, + }, + "namespaced list": { + attributes: authorizationv1.ResourceAttributes{ + Namespace: "workloads", + Verb: "list", + Resource: "secrets", + }, + want: `list secrets in namespace "workloads"`, + }, + } + + for name, tc := range testCases { + t.Run(name, func(t *testing.T) { + got := describeAttributes(tc.attributes) + if got != tc.want { + t.Errorf("describeAttributes(%v) = %q, want %q", tc.attributes, got, tc.want) + } + }) + } +} + +func newTestReviewer(t *testing.T, client SubjectAccessReviewClient, maxConcurrentReviews int) *Reviewer { + t.Helper() + reviewer, err := NewReviewer(client, meta.NewDefaultRESTMapper(nil), maxConcurrentReviews) + if err != nil { + t.Fatalf("NewReviewer() returned error %v, want nil", err) + } + return reviewer +} diff --git a/pkg/utils/authorization/placement.go b/pkg/utils/authorization/placement.go new file mode 100644 index 000000000..d1a86c0da --- /dev/null +++ b/pkg/utils/authorization/placement.go @@ -0,0 +1,134 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package authorization + +import ( + "context" + "errors" + "fmt" + + authenticationv1 "k8s.io/api/authentication/v1" + authorizationv1 "k8s.io/api/authorization/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils/resourceeligibility" +) + +// ResourcePlacementAuthorizer reviews source-resource access for ResourcePlacement selectors. +type ResourcePlacementAuthorizer struct { + restMapper meta.RESTMapper + reviewer *Reviewer + resourceEligibilityChecker resourceeligibility.Checker +} + +// NewResourcePlacementAuthorizer creates an authorizer for namespaced ResourcePlacements. +func NewResourcePlacementAuthorizer(restMapper meta.RESTMapper, reviewer *Reviewer, checker resourceeligibility.Checker) (*ResourcePlacementAuthorizer, error) { + if restMapper == nil { + return nil, errors.New("REST mapper must not be nil") + } + if reviewer == nil { + return nil, errors.New("authorization reviewer must not be nil") + } + if checker == nil { + return nil, errors.New("resource eligibility checker must not be nil") + } + return &ResourcePlacementAuthorizer{ + restMapper: restMapper, + reviewer: reviewer, + resourceEligibilityChecker: checker, + }, nil +} + +// Authorize requires get access for named selections and list access for enumerated scopes. +func (a *ResourcePlacementAuthorizer) Authorize(ctx context.Context, user authenticationv1.UserInfo, placement placementv1beta1.PlacementObj) Result { + if a == nil || a.restMapper == nil || a.reviewer == nil || a.resourceEligibilityChecker == nil { + return Result{Decision: DecisionIndeterminate, Err: fmt.Errorf("placement authorization dependencies are incomplete")} + } + attributes, err := a.resolve(placement) + if err != nil { + return Result{Decision: DecisionIndeterminate, Err: err} + } + return a.reviewer.Authorize(ctx, user, attributes...) +} + +func (a *ResourcePlacementAuthorizer) resolve(placement placementv1beta1.PlacementObj) ([]authorizationv1.ResourceAttributes, error) { + switch placement := placement.(type) { + case *placementv1beta1.ResourcePlacement: + return a.resolveResourcePlacement(placement) + case *placementv1beta1.ClusterResourcePlacement: + return nil, errors.New("cluster resource placement authorization is not implemented") + default: + return nil, fmt.Errorf("unsupported placement type %T", placement) + } +} + +func (a *ResourcePlacementAuthorizer) resolveResourcePlacement(rp *placementv1beta1.ResourcePlacement) ([]authorizationv1.ResourceAttributes, error) { + if rp == nil { + return nil, fmt.Errorf("resource placement must not be nil") + } + if rp.Namespace == "" { + return nil, fmt.Errorf("resource placement must specify a namespace") + } + var attributes []authorizationv1.ResourceAttributes + for _, selector := range rp.Spec.ResourceSelectors { + gvk := schema.GroupVersionKind{Group: selector.Group, Version: selector.Version, Kind: selector.Kind} + mapping, err := a.restMapper.RESTMapping(gvk.GroupKind(), gvk.Version) + if err != nil { + return nil, fmt.Errorf("failed to resolve selector %s: %w", gvk, err) + } + if mapping.Scope.Name() != meta.RESTScopeNameNamespace { + return nil, fmt.Errorf("resource placement cannot select cluster-scoped resource %s", gvk) + } + if !a.resourceEligibilityChecker.AllowsGVK(gvk) { + continue + } + check, err := a.selectorAttributes(mapping.Resource, rp.Namespace, selector) + if err != nil { + return nil, err + } + attributes = append(attributes, check) + } + return attributes, nil +} + +func (a *ResourcePlacementAuthorizer) selectorAttributes(gvr schema.GroupVersionResource, namespace string, selector placementv1beta1.ResourceSelectorTerm) (authorizationv1.ResourceAttributes, error) { + if namespace != "" && !a.resourceEligibilityChecker.AllowsNamespace(namespace) { + return authorizationv1.ResourceAttributes{}, fmt.Errorf("namespace %q is not eligible for placement", namespace) + } + if selector.Name != "" { + if selector.LabelSelector != nil { + return authorizationv1.ResourceAttributes{}, fmt.Errorf("name and labelSelector are mutually exclusive") + } + return resourceAttributes(gvr, "get", namespace, selector.Name), nil + } + if selector.LabelSelector != nil { + if _, err := metav1.LabelSelectorAsSelector(selector.LabelSelector); err != nil { + return authorizationv1.ResourceAttributes{}, fmt.Errorf("invalid resource label selector: %w", err) + } + } + return resourceAttributes(gvr, "list", namespace, ""), nil +} + +func resourceAttributes(gvr schema.GroupVersionResource, verb, namespace, name string) authorizationv1.ResourceAttributes { + return authorizationv1.ResourceAttributes{ + Group: gvr.Group, Version: gvr.Version, Resource: gvr.Resource, + Verb: verb, Namespace: namespace, Name: name, + } +} diff --git a/pkg/utils/authorization/placement_test.go b/pkg/utils/authorization/placement_test.go new file mode 100644 index 000000000..1da433358 --- /dev/null +++ b/pkg/utils/authorization/placement_test.go @@ -0,0 +1,262 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package authorization + +import ( + "context" + "errors" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + authenticationv1 "k8s.io/api/authentication/v1" + authorizationv1 "k8s.io/api/authorization/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/resourceeligibility" +) + +const ( + rpNamespace = "team" + rpSource = "config" + rpRequester = "requester" +) + +func resourcePlacementTestMapper() *meta.DefaultRESTMapper { + mapper := meta.NewDefaultRESTMapper([]schema.GroupVersion{corev1.SchemeGroupVersion}) + mapper.Add(corev1.SchemeGroupVersion.WithKind(utils.ConfigMapKind), meta.RESTScopeNamespace) + mapper.Add(utils.NamespaceGVK, meta.RESTScopeRoot) + return mapper +} + +func TestNewResourcePlacementAuthorizer(t *testing.T) { + mapper := resourcePlacementTestMapper() + reviewer := newTestReviewer(t, &fakeSubjectAccessReviewClient{}, 1) + checker := resourceeligibility.AllowAll(mapper) + tests := []struct { + name string + mapper meta.RESTMapper + reviewer *Reviewer + checker resourceeligibility.Checker + wantErr string + }{ + {name: "valid", mapper: mapper, reviewer: reviewer, checker: checker}, + {name: "missing mapper", reviewer: reviewer, checker: checker, wantErr: "REST mapper must not be nil"}, + {name: "missing reviewer", mapper: mapper, checker: checker, wantErr: "authorization reviewer must not be nil"}, + {name: "missing checker", mapper: mapper, reviewer: reviewer, wantErr: "resource eligibility checker must not be nil"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := NewResourcePlacementAuthorizer(tt.mapper, tt.reviewer, tt.checker) + if tt.wantErr != "" { + if got != nil || err == nil || err.Error() != tt.wantErr { + t.Fatalf("NewResourcePlacementAuthorizer() = (%v, %v), want (nil, %q)", got, err, tt.wantErr) + } + } else if got == nil || err != nil { + t.Fatalf("NewResourcePlacementAuthorizer() = (%v, %v), want initialized authorizer and nil error", got, err) + } + }) + } +} + +func TestResourcePlacementAuthorizeUninitialized(t *testing.T) { + for name, authorizer := range map[string]*ResourcePlacementAuthorizer{"nil": nil, "zero value": {}} { + t.Run(name, func(t *testing.T) { + result := authorizer.Authorize(context.Background(), authenticationv1.UserInfo{}, &placementv1beta1.ResourcePlacement{}) + if result.Decision != DecisionIndeterminate || result.Err == nil { + t.Errorf("Authorize() = %+v, want initialization error", result) + } + }) + } +} + +func TestResourcePlacementAuthorizeSelections(t *testing.T) { + named := placementv1beta1.ResourceSelectorTerm{Version: "v1", Kind: utils.ConfigMapKind, Name: rpSource} + all := placementv1beta1.ResourceSelectorTerm{Version: "v1", Kind: utils.ConfigMapKind} + labeled := placementv1beta1.ResourceSelectorTerm{ + Version: "v1", Kind: utils.ConfigMapKind, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{"app": "selected"}}, + } + emptyLabels := placementv1beta1.ResourceSelectorTerm{ + Version: "v1", Kind: utils.ConfigMapKind, + LabelSelector: &metav1.LabelSelector{}, + } + get := resourceAttributes(corev1.SchemeGroupVersion.WithResource("configmaps"), "get", rpNamespace, rpSource) + list := resourceAttributes(corev1.SchemeGroupVersion.WithResource("configmaps"), "list", rpNamespace, "") + tests := []struct { + name string + selectors []placementv1beta1.ResourceSelectorTerm + want []authorizationv1.ResourceAttributes + }{ + {name: "named object", selectors: []placementv1beta1.ResourceSelectorTerm{named}, want: []authorizationv1.ResourceAttributes{get}}, + {name: "label selection needs only list", selectors: []placementv1beta1.ResourceSelectorTerm{labeled}, want: []authorizationv1.ResourceAttributes{list}}, + {name: "all of kind needs only list", selectors: []placementv1beta1.ResourceSelectorTerm{all}, want: []authorizationv1.ResourceAttributes{list}}, + {name: "empty label selector", selectors: []placementv1beta1.ResourceSelectorTerm{emptyLabels}, want: []authorizationv1.ResourceAttributes{list}}, + {name: "overlapping selectors deduplicate reviews", selectors: []placementv1beta1.ResourceSelectorTerm{all, labeled, named, all, named}, want: []authorizationv1.ResourceAttributes{list, get}}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + mapper := resourcePlacementTestMapper() + var got []authorizationv1.ResourceAttributes + reviewer := newTestReviewer(t, &fakeSubjectAccessReviewClient{create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + got = append(got, *review.Spec.ResourceAttributes) + review.Status.Allowed = true + return nil + }}, 1) + a, err := NewResourcePlacementAuthorizer(mapper, reviewer, resourceeligibility.AllowAll(mapper)) + if err != nil { + t.Fatal(err) + } + p := &placementv1beta1.ResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{Namespace: rpNamespace}, + Spec: placementv1beta1.PlacementSpec{ResourceSelectors: tt.selectors}, + } + result := a.Authorize(context.Background(), authenticationv1.UserInfo{Username: rpRequester}, p) + if diff := cmp.Diff(Result{Decision: DecisionAllowed}, result); diff != "" { + t.Fatalf("Authorize() mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff(tt.want, got, cmpopts.SortSlices(func(a, b authorizationv1.ResourceAttributes) bool { + return a.Verb < b.Verb + })); diff != "" { + t.Errorf("Authorize() checks mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestResourcePlacementAuthorizeRejectsUnsupportedTypes(t *testing.T) { + const deferredCRPErr = "cluster resource placement authorization is not implemented" + for name, tt := range map[string]struct { + placement placementv1beta1.PlacementObj + wantErr string + }{ + "nil": {wantErr: "unsupported placement type "}, + "typed nil RP": {placement: (*placementv1beta1.ResourcePlacement)(nil), wantErr: "resource placement must not be nil"}, + "CRP": {placement: &placementv1beta1.ClusterResourcePlacement{}, wantErr: deferredCRPErr}, + "typed nil CRP": {placement: (*placementv1beta1.ClusterResourcePlacement)(nil), wantErr: deferredCRPErr}, + } { + t.Run(name, func(t *testing.T) { + mapper := resourcePlacementTestMapper() + a, err := NewResourcePlacementAuthorizer(mapper, newTestReviewer(t, &fakeSubjectAccessReviewClient{}, 1), resourceeligibility.AllowAll(mapper)) + if err != nil { + t.Fatal(err) + } + result := a.Authorize(context.Background(), authenticationv1.UserInfo{}, tt.placement) + if result.Decision != DecisionIndeterminate || result.Err == nil || result.Err.Error() != tt.wantErr { + t.Errorf("Authorize() = %+v, want indeterminate decision with error %q", result, tt.wantErr) + } + }) + } +} + +func TestResourcePlacementAuthorizeFailures(t *testing.T) { + tests := []struct { + name string + modify func(*ResourcePlacementAuthorizer, *placementv1beta1.ResourcePlacement) + denied bool + }{ + {name: "unknown kind", modify: func(_ *ResourcePlacementAuthorizer, p *placementv1beta1.ResourcePlacement) { + p.Spec.ResourceSelectors[0].Kind = "Unknown" + }}, + {name: "missing namespace", modify: func(_ *ResourcePlacementAuthorizer, p *placementv1beta1.ResourcePlacement) { p.Namespace = "" }}, + {name: "cluster-scoped selector", modify: func(_ *ResourcePlacementAuthorizer, p *placementv1beta1.ResourcePlacement) { + p.Spec.ResourceSelectors[0].Kind = utils.NamespaceGVK.Kind + }}, + {name: "name and label selector", modify: func(_ *ResourcePlacementAuthorizer, p *placementv1beta1.ResourcePlacement) { + p.Spec.ResourceSelectors[0].LabelSelector = &metav1.LabelSelector{} + }}, + {name: "invalid label selector", modify: func(_ *ResourcePlacementAuthorizer, p *placementv1beta1.ResourcePlacement) { + p.Spec.ResourceSelectors[0].Name = "" + p.Spec.ResourceSelectors[0].LabelSelector = &metav1.LabelSelector{MatchExpressions: []metav1.LabelSelectorRequirement{{Key: "app", Operator: "invalid"}}} + }}, + {name: "namespace excluded", modify: func(a *ResourcePlacementAuthorizer, _ *placementv1beta1.ResourcePlacement) { + a.resourceEligibilityChecker.SetNamespaceCheckList(resourceeligibility.NamespaceEligibilityModeDenyList, []string{rpNamespace}, nil) + }}, + {name: "SAR error", modify: func(a *ResourcePlacementAuthorizer, _ *placementv1beta1.ResourcePlacement) { + a.reviewer.client = &fakeSubjectAccessReviewClient{create: func(context.Context, *authorizationv1.SubjectAccessReview) error { + return errors.New("SAR unavailable") + }} + }}, + {name: "get denied", denied: true, modify: func(a *ResourcePlacementAuthorizer, _ *placementv1beta1.ResourcePlacement) { + a.reviewer.client = &fakeSubjectAccessReviewClient{create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + review.Status.Denied = true + return nil + }} + }}, + {name: "list denied", denied: true, modify: func(a *ResourcePlacementAuthorizer, p *placementv1beta1.ResourcePlacement) { + p.Spec.ResourceSelectors[0].Name = "" + a.reviewer.client = &fakeSubjectAccessReviewClient{create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + review.Status.Denied = true + return nil + }} + }}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + mapper := resourcePlacementTestMapper() + reviewer := newTestReviewer(t, &fakeSubjectAccessReviewClient{create: func(_ context.Context, review *authorizationv1.SubjectAccessReview) error { + review.Status.Allowed = true + return nil + }}, 1) + a, err := NewResourcePlacementAuthorizer(mapper, reviewer, resourceeligibility.AllowAll(mapper)) + if err != nil { + t.Fatal(err) + } + p := &placementv1beta1.ResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{Namespace: rpNamespace}, + Spec: placementv1beta1.PlacementSpec{ResourceSelectors: []placementv1beta1.ResourceSelectorTerm{{Version: "v1", Kind: utils.ConfigMapKind, Name: rpSource}}}, + } + tt.modify(a, p) + result := a.Authorize(context.Background(), authenticationv1.UserInfo{}, p) + wantDecision := DecisionIndeterminate + if tt.denied { + wantDecision = DecisionDenied + } + if result.Decision != wantDecision || (tt.denied && result.DenialReason == "") || (!tt.denied && result.Err == nil) { + t.Errorf("Authorize() = %+v, want %s with details", result, wantDecision) + } + }) + } +} + +func TestResourcePlacementAuthorizeSkipsIneligibleKinds(t *testing.T) { + mapper := resourcePlacementTestMapper() + checker := resourceeligibility.AllowAll(mapper) + if err := checker.SetGVKCheckList(resourceeligibility.GVKEligibilityModeDenyList, []schema.GroupVersionKind{corev1.SchemeGroupVersion.WithKind(utils.ConfigMapKind)}); err != nil { + t.Fatal(err) + } + reviewer := newTestReviewer(t, &fakeSubjectAccessReviewClient{create: func(context.Context, *authorizationv1.SubjectAccessReview) error { + t.Error("Authorize() submitted a SubjectAccessReview for an ineligible kind") + return errors.New("unexpected SubjectAccessReview") + }}, 1) + a, err := NewResourcePlacementAuthorizer(mapper, reviewer, checker) + if err != nil { + t.Fatal(err) + } + p := &placementv1beta1.ResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{Namespace: rpNamespace}, + Spec: placementv1beta1.PlacementSpec{ResourceSelectors: []placementv1beta1.ResourceSelectorTerm{{Version: "v1", Kind: utils.ConfigMapKind}}}, + } + if result := a.Authorize(context.Background(), authenticationv1.UserInfo{}, p); result.Decision != DecisionAllowed || result.Err != nil { + t.Errorf("Authorize() = %+v, want allowed with no reviews", result) + } +} diff --git a/pkg/utils/common.go b/pkg/utils/common.go index 9d57b42d6..156b8add1 100644 --- a/pkg/utils/common.go +++ b/pkg/utils/common.go @@ -43,6 +43,7 @@ import ( fleetnetworkingv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" placementv1 "go.goms.io/fleet/apis/placement/v1" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils/condition" @@ -118,6 +119,13 @@ var ( APIGroups: []string{placementv1beta1.GroupVersion.Group}, Resources: []string{"*"}, } + // FleetPlacementPolicyWorkRule lets the member agent apply the Works of the experimental + // placement policy APIs; it is granted only when the hub agent runs with those APIs enabled. + FleetPlacementPolicyWorkRule = rbacv1.PolicyRule{ + Verbs: []string{"get", "list", "watch", "update", "patch"}, + APIGroups: []string{kfplacementv1alpha1.GroupVersion.Group}, + Resources: []string{"works", "works/status"}, + } // EventRule grants access to core/v1 Events. The Fleet controllers have // moved to the events.k8s.io recorder, but this rule is still required: // the fleet-networking agents that share this role emit core/v1 Events. @@ -353,6 +361,13 @@ var ( Kind: "Work", } + // PlacementPolicyWorkMetaGVK is the GVK of the Works of the experimental placement policy APIs. + PlacementPolicyWorkMetaGVK = metav1.GroupVersionKind{ + Group: kfplacementv1alpha1.GroupVersion.Group, + Version: kfplacementv1alpha1.GroupVersion.Version, + Kind: kfplacementv1alpha1.WorkKind, + } + ClusterResourceOverrideSnapshotKind = schema.GroupVersionKind{ Group: placementv1beta1.GroupVersion.Group, Version: placementv1beta1.GroupVersion.Version, @@ -520,18 +535,6 @@ func IsFleetMemberNamespace(namespace string) bool { return strings.HasPrefix(namespace, FleetMemberNamespacePrefix) } -// ShouldPropagateNamespace decides if we should propagate the resources in the namespace. -func ShouldPropagateNamespace(namespace string, skippedNamespaces map[string]bool) bool { - if IsReservedNamespace(namespace) { - return false - } - - if skippedNamespaces[namespace] { - return false - } - return true -} - // GenerateGroupString generates a string which prints groups in which a user belongs, // it compresses the string to just display three groups if length of groups is more than 10. func GenerateGroupString(groups []string) string { diff --git a/pkg/utils/controller/resource_selector_resolver.go b/pkg/utils/controller/resource_selector_resolver.go index 6fa1b083f..9c5083ac2 100644 --- a/pkg/utils/controller/resource_selector_resolver.go +++ b/pkg/utils/controller/resource_selector_resolver.go @@ -40,6 +40,11 @@ import ( placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" +) + +const ( + defaultNSName = "default" ) var ( @@ -92,11 +97,8 @@ var ( ) type ResourceSelectorResolver struct { - // SkippedNamespaces contains the namespaces that should be skipped when selecting resources. - SkippedNamespaces map[string]bool - - // ResourceConfig contains the resource configuration. - ResourceConfig *utils.ResourceConfig + // ResourceEligibilityChecker determines if a given namespace, GVK (GVR) is eligible for placement. + ResourceEligibilityChecker resourceeligibility.Checker // InformerManager is the informer manager. InformerManager informer.Manager @@ -272,8 +274,8 @@ func (rs *ResourceSelectorResolver) gatherSelectedResource(placementKey types.Na Kind: selector.Kind, } - if rs.ResourceConfig.IsResourceDisabled(gvk) { - klog.V(2).InfoS("Skip select resource", "group version kind", gvk.String()) + if !rs.ResourceEligibilityChecker.AllowsGVK(gvk) { + klog.V(2).InfoS("Skip select resource as it is not eligible for placement", "GVK", gvk.String()) continue } @@ -435,8 +437,12 @@ func (rs *ResourceSelectorResolver) fetchNamespaceResources(selector placementv1 func (rs *ResourceSelectorResolver) fetchAllResourcesInOneNamespace(namespaceName string, placeName string) ([]runtime.Object, error) { var resources []runtime.Object - if !utils.ShouldPropagateNamespace(namespaceName, rs.SkippedNamespaces) { - err := fmt.Errorf("invalid clusterRresourcePlacement %s: namespace %s is not allowed to propagate", placeName, namespaceName) + if !rs.ResourceEligibilityChecker.AllowsNamespace(namespaceName) { + err := fmt.Errorf("invalid clusterResourcePlacement %s: namespace %s and any resource within it are not allowed for placement", placeName, namespaceName) + return nil, NewUserError(err) + } + if namespaceName == defaultNSName { + err := fmt.Errorf("invalid clusterResourcePlacement %s: the default namespace cannot be selected for placement; to place resources within the default namespace, consider using the ResourcePlacement API", placeName) return nil, NewUserError(err) } @@ -458,7 +464,18 @@ func (rs *ResourceSelectorResolver) fetchAllResourcesInOneNamespace(namespaceNam trackedResource := rs.InformerManager.GetNameSpaceScopedResources() for _, gvr := range trackedResource { - if !utils.ShouldProcessResource(gvr, rs.RestMapper, rs.ResourceConfig) { + eligible, err := rs.ResourceEligibilityChecker.AllowsGVR(gvr) + switch { + case meta.IsNoMatchError(err): + // The informer manager never drops a GVR once it has been tracked, so aborting here would + // block the placement indefinitely after a CRD is uninstalled. + klog.V(2).InfoS("Skip selecting the resource as no kind is registered for its GVR", "GVR", gvr.String()) + continue + case err != nil: + klog.ErrorS(err, "failed to check if the GVR is eligible for placement", "GVR", gvr.String()) + return nil, NewUnexpectedBehaviorError(err) + case !eligible: + klog.V(2).InfoS("Skip selecting the resource as its GVR is not eligible for placement", "GVR", gvr.String()) continue } if !rs.InformerManager.IsInformerSynced(gvr) { @@ -592,7 +609,8 @@ func (rs *ResourceSelectorResolver) fetchResources(selector placementv1beta1.Res // fetchSelectedNamespace retrieves the namespace name based on the namespace selector. // This function assumes the selector is a name-based selector (not label-based) as guaranteed by CEL validation. -// Returns the namespace name if found and not skipped, or empty string if not found/skipped. +// Returns the namespace name if found and not skipped, or empty string if not found/skipped. Selecting a reserved/denied namespace will +// trigger a user error. func (rs *ResourceSelectorResolver) fetchSelectedNamespace(selector placementv1beta1.ResourceSelectorTerm, placementName string) (string, bool, error) { klog.V(2).InfoS("start to fetch selected namespace", "selector", selector, "placement", placementName) @@ -614,12 +632,6 @@ func (rs *ResourceSelectorResolver) fetchSelectedNamespace(selector placementv1b return "", false, NewUnexpectedBehaviorError(fmt.Errorf("failed to access the namespace object: %w", err)) } - // Check if this namespace should be propagated - if !utils.ShouldPropagateNamespace(ns.GetName(), rs.SkippedNamespaces) { - klog.V(2).InfoS("skip namespace that is not allowed to propagate", "namespace", ns.GetName(), "placement", placementName) - return "", false, nil - } - return ns.GetName(), true, nil } @@ -632,6 +644,18 @@ func (rs *ResourceSelectorResolver) ShouldPropagateObj(namespace, placementName return false, nil } + // Deny reserved namespaces (themselves, not including resources within them) for placement. + if uObj.GroupVersionKind() == utils.NamespaceGVK { + if !rs.ResourceEligibilityChecker.AllowsNamespace(uObj.GetName()) { + klog.V(2).InfoS("Block reserved/denied namespaces for placement", "namespace", uObj.GetName(), "placement", placementName) + return false, NewUserError(fmt.Errorf("namespace %s and any resource within it are not allowed for placement", uObj.GetName())) + } + if uObj.GetName() == defaultNSName { + klog.V(2).InfoS("Block the default namespace for placement", "namespace", uObj.GetName(), "placement", placementName) + return false, NewUserError(fmt.Errorf("the default namespace cannot be selected for placement; to place resources within the default namespace, consider using the ResourcePlacement API")) + } + } + shouldInclude, err := ShouldPropagateObj(rs.InformerManager, uObj, rs.EnableWorkload) if err != nil { klog.ErrorS(err, "Cannot determine if we should propagate an object", "namespace", namespace, "placement", placementName, "object", uObjKObj) diff --git a/pkg/utils/controller/resource_selector_resolver_test.go b/pkg/utils/controller/resource_selector_resolver_test.go index 9ffd31208..12b403cb0 100644 --- a/pkg/utils/controller/resource_selector_resolver_test.go +++ b/pkg/utils/controller/resource_selector_resolver_test.go @@ -39,6 +39,7 @@ import ( fleetv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/resourceeligibility" testinformer "go.goms.io/fleet/test/utils/informer" ) @@ -473,14 +474,25 @@ func TestGatherSelectedResource(t *testing.T) { } kubeSystemNamespace.SetGroupVersionKind(utils.NamespaceGVK) + defaultNamespace := &unstructured.Unstructured{ // the default namespace object, which itself cannot be placed + Object: map[string]interface{}{ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": map[string]interface{}{ + "name": "default", + }, + }, + } + defaultNamespace.SetGroupVersionKind(utils.NamespaceGVK) + tests := []struct { - name string - placementName types.NamespacedName - selectors []fleetv1beta1.ResourceSelectorTerm - resourceConfig *utils.ResourceConfig - informerManager *testinformer.FakeManager - want []*unstructured.Unstructured - wantError error + name string + placementName types.NamespacedName + selectors []fleetv1beta1.ResourceSelectorTerm + resourceEligibilityChecker resourceeligibility.Checker + informerManager *testinformer.FakeManager + want []*unstructured.Unstructured + wantError error }{ { name: "should handle empty selectors", @@ -499,8 +511,12 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(true), // make this allow list - nothing is allowed - want: nil, + resourceEligibilityChecker: func() resourceeligibility.Checker { + c := resourceeligibility.New(resourceeligibility.CheckerOptions{RestMapper: newFakeRESTMapper(), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + _ = c.SetGVKCheckList(resourceeligibility.GVKEligibilityModeAllowList, nil) + return c + }(), // make this allow list - nothing is allowed + want: nil, }, { name: "should skip disabled resources for resource placement", @@ -513,8 +529,12 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(true), // make this allow list - nothing is allowed - want: nil, + resourceEligibilityChecker: func() resourceeligibility.Checker { + c := resourceeligibility.New(resourceeligibility.CheckerOptions{RestMapper: newFakeRESTMapper(), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + _ = c.SetGVKCheckList(resourceeligibility.GVKEligibilityModeAllowList, nil) + return c + }(), // make this allow list - nothing is allowed + want: nil, }, { name: "should return error for cluster-scoped resource", @@ -527,7 +547,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-clusterrole", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: &testinformer.FakeManager{ IsClusterScopedResource: false, Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{}, @@ -546,7 +566,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -569,7 +589,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -594,7 +614,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -618,7 +638,7 @@ func TestGatherSelectedResource(t *testing.T) { Kind: "Deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -649,7 +669,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deleting-deployment", // deleting deployment }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -676,7 +696,7 @@ func TestGatherSelectedResource(t *testing.T) { }, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -707,7 +727,7 @@ func TestGatherSelectedResource(t *testing.T) { }, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -736,7 +756,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", // same deployment selected twice }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -764,7 +784,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-configmap", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -789,7 +809,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: &testinformer.FakeManager{ IsClusterScopedResource: true, Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{}, @@ -814,7 +834,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-ns", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -845,7 +865,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-ns", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -874,7 +894,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -905,10 +925,10 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: func() *utils.ResourceConfig { - cfg := utils.NewResourceConfig(false) - cfg.AddGroupVersionKind(utils.DeploymentGVK) - return cfg + resourceEligibilityChecker: func() resourceeligibility.Checker { + c := resourceeligibility.New(resourceeligibility.CheckerOptions{RestMapper: newFakeRESTMapper(), Wrapped: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + _ = c.SetGVKCheckList(resourceeligibility.GVKEligibilityModeDenyList, []schema.GroupVersionKind{utils.DeploymentGVK}) + return c }(), informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ @@ -935,7 +955,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -962,7 +982,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceOnly, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -988,7 +1008,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceOnly, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -1015,7 +1035,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceOnly, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -1045,7 +1065,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -1059,6 +1079,107 @@ func TestGatherSelectedResource(t *testing.T) { }(), wantError: ErrUserError, }, + { + name: "should return error when selecting the default namespace by name with namespace with resources scope", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "default", + SelectionScope: fleetv1beta1.NamespaceWithResources, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list + informerManager: func() *testinformer.FakeManager { + return &testinformer.FakeManager{ + IsClusterScopedResource: false, + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.NamespaceGVR: {Objects: []runtime.Object{testNamespace, defaultNamespace}}, + utils.DeploymentGVR: {Objects: []runtime.Object{testDeployment}}, + utils.ConfigMapGVR: {Objects: []runtime.Object{testConfigMap}}, + }, + NamespaceScopedResources: []schema.GroupVersionResource{utils.DeploymentGVR, utils.ConfigMapGVR}, + } + }(), + wantError: ErrUserError, + }, + { + name: "should return error when selecting a reserved namespace by name with namespace only scope", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "kube-system", + SelectionScope: fleetv1beta1.NamespaceOnly, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list + informerManager: func() *testinformer.FakeManager { + return &testinformer.FakeManager{ + IsClusterScopedResource: false, + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.NamespaceGVR: {Objects: []runtime.Object{testNamespace, prodNamespace, kubeSystemNamespace}}, + }, + } + }(), + wantError: ErrUserError, + }, + { + name: "should return error when selecting the default namespace by name with namespace only scope", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "default", + SelectionScope: fleetv1beta1.NamespaceOnly, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list + informerManager: func() *testinformer.FakeManager { + return &testinformer.FakeManager{ + IsClusterScopedResource: false, + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.NamespaceGVR: {Objects: []runtime.Object{testNamespace, prodNamespace, defaultNamespace}}, + }, + } + }(), + wantError: ErrUserError, + }, + { + // A label selector that matches a reserved namespace fails the whole placement, rather than skipping the + // reserved namespace. + name: "should return error when the label selector matches a reserved namespace with namespace only scope", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "environment": "test", + }, + }, + SelectionScope: fleetv1beta1.NamespaceOnly, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list + informerManager: func() *testinformer.FakeManager { + return &testinformer.FakeManager{ + IsClusterScopedResource: false, + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.NamespaceGVR: {Objects: []runtime.Object{testNamespace, prodNamespace, testDeletingNamespace, kubeSystemNamespace}}, + }, + } + }(), + wantError: ErrUserError, + }, { name: "should return empty result when informer manager returns not found error for cluster scoped placement", placementName: types.NamespacedName{Name: "test-placement"}, @@ -1071,7 +1192,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1097,7 +1218,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1122,7 +1243,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1148,7 +1269,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1175,7 +1296,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "kube-root-ca.crt", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1196,7 +1317,7 @@ func TestGatherSelectedResource(t *testing.T) { Kind: "ConfigMap", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1218,7 +1339,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1241,7 +1362,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-cluster-role", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: false, @@ -1265,7 +1386,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResources, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1293,7 +1414,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-endpoints", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // endpoints are only denied for the v1 APIs informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1320,7 +1441,7 @@ func TestGatherSelectedResource(t *testing.T) { Kind: "Endpoints", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // endpoints are only denied for the v1 APIs informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1361,7 +1482,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-configmap", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ @@ -1397,7 +1518,7 @@ func TestGatherSelectedResource(t *testing.T) { }, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { // Create a deployment with matching labels for this test deploymentWithLabels := testDeployment.DeepCopy() @@ -1447,7 +1568,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ IsClusterScopedResource: true, @@ -1479,7 +1600,7 @@ func TestGatherSelectedResource(t *testing.T) { SelectionScope: fleetv1beta1.NamespaceWithResourceSelectors, }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ @@ -1511,7 +1632,7 @@ func TestGatherSelectedResource(t *testing.T) { Name: "test-configmap", }, }, - resourceConfig: utils.NewResourceConfig(false), // default deny list + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list informerManager: func() *testinformer.FakeManager { return &testinformer.FakeManager{ Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ @@ -1524,14 +1645,70 @@ func TestGatherSelectedResource(t *testing.T) { // Should error because no namespaces match the selector wantError: ErrUserError, }, + { + name: "should skip a tracked resource whose kind is no longer registered", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "test-ns", + SelectionScope: fleetv1beta1.NamespaceWithResources, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), // default deny list + informerManager: func() *testinformer.FakeManager { + return &testinformer.FakeManager{ + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.NamespaceGVR: {Objects: []runtime.Object{testNamespace, prodNamespace}}, + utils.DeploymentGVR: {Objects: []runtime.Object{testDeployment}}, + }, + // The informer manager keeps tracking a GVR after its CRD is uninstalled. + NamespaceScopedResources: []schema.GroupVersionResource{ + utils.DeploymentGVR, + {Group: "example.com", Version: "v1", Resource: "widgets"}, + }, + } + }(), + want: []*unstructured.Unstructured{testNamespace, testDeployment}, + }, + { + name: "should return error when the RESTMapper fails to resolve a tracked resource", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "test-ns", + SelectionScope: fleetv1beta1.NamespaceWithResources, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(&fakeRESTMapper{kindsForErr: errors.New("mapper failure")}), + informerManager: func() *testinformer.FakeManager { + return &testinformer.FakeManager{ + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.NamespaceGVR: {Objects: []runtime.Object{testNamespace, prodNamespace}}, + utils.DeploymentGVR: {Objects: []runtime.Object{testDeployment}}, + }, + NamespaceScopedResources: []schema.GroupVersionResource{utils.DeploymentGVR}, + } + }(), + wantError: ErrUnexpectedBehavior, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { + checker := tt.resourceEligibilityChecker + if checker == nil { + checker = resourceeligibility.New(resourceeligibility.CheckerOptions{RestMapper: newFakeRESTMapper(), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + } rsr := &ResourceSelectorResolver{ - ResourceConfig: tt.resourceConfig, - InformerManager: tt.informerManager, - RestMapper: newFakeRESTMapper(), + ResourceEligibilityChecker: checker, + InformerManager: tt.informerManager, + RestMapper: newFakeRESTMapper(), } got, err := rsr.gatherSelectedResource(tt.placementName, tt.selectors) @@ -1551,6 +1728,8 @@ func TestGatherSelectedResource(t *testing.T) { // fakeRESTMapper is a minimal RESTMapper implementation for testing type fakeRESTMapper struct { mappings map[schema.GroupKind]*meta.RESTMapping + // kindsForErr, when set, is returned by KindsFor for every resource. + kindsForErr error } // newFakeRESTMapper creates a new fakeRESTMapper with default mappings @@ -1615,10 +1794,17 @@ func (f *fakeRESTMapper) KindFor(resource schema.GroupVersionResource) (schema.G case resource.Group == "" && resource.Resource == "endpoints": return schema.GroupVersionKind{Group: "", Version: "v1", Kind: "Endpoints"}, nil } - return schema.GroupVersionKind{}, errors.New("kind not found") + // Mirrors the real RESTMapper, which reports unknown resources as a no-match error. + return schema.GroupVersionKind{}, &meta.NoKindMatchError{ + GroupKind: schema.GroupKind{Group: resource.Group, Kind: resource.Resource}, + SearchedVersions: []string{resource.Version}, + } } func (f *fakeRESTMapper) KindsFor(resource schema.GroupVersionResource) ([]schema.GroupVersionKind, error) { + if f.kindsForErr != nil { + return nil, f.kindsForErr + } kind, err := f.KindFor(resource) if err != nil { return nil, err @@ -2191,14 +2377,24 @@ func TestFetchSelectedNamespace(t *testing.T) { } skippedNs.SetGroupVersionKind(utils.NamespaceGVK) + defaultNs := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": map[string]interface{}{ + "name": "default", + }, + }, + } + defaultNs.SetGroupVersionKind(utils.NamespaceGVK) + tests := []struct { - name string - selector fleetv1beta1.ResourceSelectorTerm - skippedNamespaces map[string]bool - informerManager *testinformer.FakeManager - want string - wantFound bool - wantErr bool + name string + selector fleetv1beta1.ResourceSelectorTerm + informerManager *testinformer.FakeManager + want string + wantFound bool + wantError error }{ { name: "select namespace by name", @@ -2208,7 +2404,6 @@ func TestFetchSelectedNamespace(t *testing.T) { Kind: "Namespace", Name: "test-ns-1", }, - skippedNamespaces: nil, informerManager: &testinformer.FakeManager{ Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ utils.NamespaceGVR: {Objects: []runtime.Object{testNs1, testNs2}}, @@ -2218,21 +2413,38 @@ func TestFetchSelectedNamespace(t *testing.T) { wantFound: true, }, { - name: "filter out skipped namespaces", + name: "reserved namespace selected by name", selector: fleetv1beta1.ResourceSelectorTerm{ Group: "", Version: "v1", Kind: "Namespace", Name: "kube-system", // Use name-based selection instead of label selector }, - skippedNamespaces: map[string]bool{"kube-system": true}, informerManager: &testinformer.FakeManager{ Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ utils.NamespaceGVR: {Objects: []runtime.Object{testNs1, testNs2, skippedNs}}, }, }, - want: "", // Empty because kube-system is skipped + want: "", + wantFound: false, + wantError: ErrUserError, + }, + { + name: "default namespace selected by name", + selector: fleetv1beta1.ResourceSelectorTerm{ + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "default", + }, + informerManager: &testinformer.FakeManager{ + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.NamespaceGVR: {Objects: []runtime.Object{testNs1, testNs2, defaultNs}}, + }, + }, + want: "", wantFound: false, + wantError: ErrUserError, }, { name: "no namespaces match selector", @@ -2242,7 +2454,6 @@ func TestFetchSelectedNamespace(t *testing.T) { Kind: "Namespace", Name: "non-existent", }, - skippedNamespaces: nil, informerManager: &testinformer.FakeManager{ Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ utils.NamespaceGVR: {Objects: []runtime.Object{testNs1, testNs2}}, @@ -2256,16 +2467,14 @@ func TestFetchSelectedNamespace(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { rsr := &ResourceSelectorResolver{ - SkippedNamespaces: tt.skippedNamespaces, - ResourceConfig: utils.NewResourceConfig(false), - InformerManager: tt.informerManager, - RestMapper: newFakeRESTMapper(), + ResourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), + InformerManager: tt.informerManager, + RestMapper: newFakeRESTMapper(), } gotNamespace, gotFound, err := rsr.fetchSelectedNamespace(tt.selector, "test-placement") - if (err != nil) != tt.wantErr { - t.Errorf("fetchSelectedNamespace() error = %v, wantErr %v", err, tt.wantErr) - return + if gotErr, wantErr := err != nil, tt.wantError != nil; gotErr != wantErr || !errors.Is(err, tt.wantError) { + t.Fatalf("fetchSelectedNamespace() error = %v, want error %v", err, tt.wantError) } if gotNamespace != tt.want { t.Errorf("fetchSelectedNamespace() namespace = %v, want %v", gotNamespace, tt.want) @@ -2301,15 +2510,89 @@ func TestGatherSelectedResource_ErrorCases(t *testing.T) { } testDeployment.SetGroupVersionKind(utils.DeploymentGVK) + defaultNamespace := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": map[string]interface{}{ + "name": "default", + }, + }, + } + defaultNamespace.SetGroupVersionKind(utils.NamespaceGVK) + + // All selection scopes share the same error message when the default namespace is selected. + defaultNamespaceErrMsg := "the default namespace cannot be selected for placement; to place resources within the default namespace, consider using the ResourcePlacement API" + defaultNamespaceInformerManager := &testinformer.FakeManager{ + IsClusterScopedResource: true, + APIResources: map[schema.GroupVersionKind]bool{ + utils.NamespaceGVK: true, // cluster-scoped + }, + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + utils.NamespaceGVR: {Objects: []runtime.Object{testNamespace, defaultNamespace}}, + }, + } + tests := []struct { - name string - placementName types.NamespacedName - selectors []fleetv1beta1.ResourceSelectorTerm - resourceConfig *utils.ResourceConfig - informerManager *testinformer.FakeManager - wantError error - wantErrMsg string + name string + placementName types.NamespacedName + selectors []fleetv1beta1.ResourceSelectorTerm + resourceEligibilityChecker resourceeligibility.Checker + informerManager *testinformer.FakeManager + wantError error + wantErrMsg string }{ + { + name: "CRP selecting the default namespace with NamespaceOnly mode", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "default", + SelectionScope: fleetv1beta1.NamespaceOnly, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), + informerManager: defaultNamespaceInformerManager, + wantError: ErrUserError, + wantErrMsg: defaultNamespaceErrMsg, + }, + { + name: "CRP selecting the default namespace with NamespaceWithResources mode", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "default", + SelectionScope: fleetv1beta1.NamespaceWithResources, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), + informerManager: defaultNamespaceInformerManager, + wantError: ErrUserError, + wantErrMsg: defaultNamespaceErrMsg, + }, + { + name: "CRP selecting the default namespace with NamespaceWithResourceSelectors mode", + placementName: types.NamespacedName{Name: "test-placement"}, + selectors: []fleetv1beta1.ResourceSelectorTerm{ + { + Group: "", + Version: "v1", + Kind: "Namespace", + Name: "default", + SelectionScope: fleetv1beta1.NamespaceWithResourceSelectors, + }, + }, + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), + informerManager: defaultNamespaceInformerManager, + wantError: ErrUserError, + wantErrMsg: defaultNamespaceErrMsg, + }, { name: "ResourcePlacement trying to select namespace should fail", placementName: types.NamespacedName{Name: "test-placement", Namespace: "some-ns"}, @@ -2321,7 +2604,7 @@ func TestGatherSelectedResource_ErrorCases(t *testing.T) { Name: "test-ns", }, }, - resourceConfig: utils.NewResourceConfig(false), + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), informerManager: &testinformer.FakeManager{ Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ utils.NamespaceGVR: {Objects: []runtime.Object{testNamespace}}, @@ -2341,7 +2624,7 @@ func TestGatherSelectedResource_ErrorCases(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), informerManager: &testinformer.FakeManager{ IsClusterScopedResource: true, APIResources: map[schema.GroupVersionKind]bool{ @@ -2380,7 +2663,7 @@ func TestGatherSelectedResource_ErrorCases(t *testing.T) { Name: "test-deployment", }, }, - resourceConfig: utils.NewResourceConfig(false), + resourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), informerManager: &testinformer.FakeManager{ IsClusterScopedResource: true, APIResources: map[schema.GroupVersionKind]bool{ @@ -2400,10 +2683,14 @@ func TestGatherSelectedResource_ErrorCases(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { + checker := tt.resourceEligibilityChecker + if checker == nil { + checker = resourceeligibility.New(resourceeligibility.CheckerOptions{RestMapper: newFakeRESTMapper(), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + } rsr := &ResourceSelectorResolver{ - ResourceConfig: tt.resourceConfig, - InformerManager: tt.informerManager, - RestMapper: newFakeRESTMapper(), + ResourceEligibilityChecker: checker, + InformerManager: tt.informerManager, + RestMapper: newFakeRESTMapper(), } _, err := rsr.gatherSelectedResource(tt.placementName, tt.selectors) @@ -2508,7 +2795,7 @@ func TestShouldPropagateObj(t *testing.T) { }, }, enableWorkload: false, - want: true, // ShouldPropagateObj doesn't filter Pods - they're filtered by NewResourceConfig + want: true, // ShouldPropagateObj doesn't filter Pods - they're filtered by the resource eligibility checker }, { name: "controllerrevision owned by daemonset should NOT propagate", @@ -2674,3 +2961,82 @@ func TestShouldPropagateObj(t *testing.T) { }) } } + +func TestResourceSelectorResolver_ShouldPropagateObj(t *testing.T) { + namespaceObj := func(name string, deleting bool) *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(utils.NamespaceGVK) + obj.SetName(name) + if deleting { + obj.SetDeletionTimestamp(&metav1.Time{Time: time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC)}) + } + return obj + } + + tests := []struct { + name string + obj *unstructured.Unstructured + want bool + wantError error + wantErrMsg string + }{ + { + name: "eligible namespace", + obj: namespaceObj("test-ns", false), + want: true, + }, + { + name: "reserved namespace", + obj: namespaceObj("kube-system", false), + want: false, + wantError: ErrUserError, + wantErrMsg: "namespace kube-system and any resource within it are not allowed for placement", + }, + { + name: "default namespace", + obj: namespaceObj("default", false), + want: false, + wantError: ErrUserError, + wantErrMsg: "consider using the ResourcePlacement API", + }, + { + // The deletion check comes first; a namespace being deleted is skipped silently. + name: "reserved namespace being deleted", + obj: namespaceObj("kube-system", true), + want: false, + }, + { + // Only the namespace object itself is blocked; resources within the default namespace are eligible. + name: "config map in the default namespace", + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(utils.ConfigMapGVK) + obj.SetNamespace("default") + obj.SetName("test-configmap") + return obj + }(), + want: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + rsr := &ResourceSelectorResolver{ + ResourceEligibilityChecker: resourceeligibility.DefaultForV0APIs(newFakeRESTMapper()), + InformerManager: &testinformer.FakeManager{}, + RestMapper: newFakeRESTMapper(), + } + + got, err := rsr.ShouldPropagateObj(tt.obj.GetNamespace(), "test-placement", tt.obj) + if gotErr, wantErr := err != nil, tt.wantError != nil; gotErr != wantErr || !errors.Is(err, tt.wantError) { + t.Fatalf("ShouldPropagateObj() error = %v, want error %v", err, tt.wantError) + } + if tt.wantError != nil && !strings.Contains(err.Error(), tt.wantErrMsg) { + t.Errorf("ShouldPropagateObj() error = %v, wantErrMsg %v", err, tt.wantErrMsg) + } + if got != tt.want { + t.Errorf("ShouldPropagateObj() = %v, want %v", got, tt.want) + } + }) + } +} diff --git a/pkg/utils/errors/errors.go b/pkg/utils/errors/errors.go index 16236df86..eb7ea4e5b 100644 --- a/pkg/utils/errors/errors.go +++ b/pkg/utils/errors/errors.go @@ -206,6 +206,21 @@ func Args(err error, kvs ...interface{}) []interface{} { return compositeKVs } +// Category returns the error category of the given error. It will return ErrCategoryUncategorized +// if the error is nil, is not an *Error with a category, or does not wrap an *Error with a category. +func Category(err error) ErrCategory { + var kferr *Error + // Note: errors.As returns false for nil errors, and might yield a typed nil *Error. + if !errors.As(err, &kferr) || kferr == nil { + return ErrCategoryUncategorized + } + if kferr.category != "" { + return kferr.category + } + // Each recursive call descends further down the error chain, so it always terminates. + return Category(kferr.wrapped) +} + // CallerFrameOverview is a simple struct that summarizes a single call frame. type CallerFrameOverview struct { Function string diff --git a/pkg/utils/errors/errors_test.go b/pkg/utils/errors/errors_test.go index 277cbc2e3..98fde4717 100644 --- a/pkg/utils/errors/errors_test.go +++ b/pkg/utils/errors/errors_test.go @@ -286,6 +286,108 @@ func TestArgs(t *testing.T) { } } +func TestCategory(t *testing.T) { + var typedNilErr *Error + testCases := []struct { + name string + err error + wantCategory ErrCategory + }{ + { + name: "nil error", + err: nil, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "plain error (not an *Error)", + err: fmt.Errorf("plain error"), + wantCategory: ErrCategoryUncategorized, + }, + { + name: "typed nil *Error", + err: typedNilErr, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "*Error with no category set and no wrapped error", + err: &Error{}, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "*Error with explicit category", + err: &Error{category: ErrCategoryUser}, + wantCategory: ErrCategoryUser, + }, + { + name: "*Error with explicit category, wrapping an *Error of a different category", + err: &Error{category: ErrCategoryAPIServer, wrapped: &Error{category: ErrCategoryUser}}, + wantCategory: ErrCategoryAPIServer, + }, + { + name: "*Error with no category set, wrapping a plain error", + err: &Error{wrapped: fmt.Errorf("plain error")}, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "*Error with no category set, wrapping a typed nil *Error", + err: &Error{wrapped: typedNilErr}, + wantCategory: ErrCategoryUncategorized, + }, + { + name: "*Error with no category set, wrapping a categorized *Error", + err: &Error{wrapped: &Error{category: ErrCategoryTransient}}, + wantCategory: ErrCategoryTransient, + }, + { + name: "*Error with no category set, wrapping a categorized *Error through a plain error", + err: &Error{wrapped: fmt.Errorf("plain error: %w", &Error{category: ErrCategoryUnexpected})}, + wantCategory: ErrCategoryUnexpected, + }, + { + name: "*Error with no category set, wrapping multiple *Errors with no category set", + err: &Error{wrapped: &Error{wrapped: &Error{wrapped: &Error{category: ErrCategoryAPIServer}}}}, + wantCategory: ErrCategoryAPIServer, + }, + { + name: "plain error wrapping a typed nil *Error", + err: fmt.Errorf("plain error: %w", typedNilErr), + wantCategory: ErrCategoryUncategorized, + }, + { + name: "plain error wrapping a categorized *Error", + err: fmt.Errorf("plain error: %w", &Error{category: ErrCategoryTransient}), + wantCategory: ErrCategoryTransient, + }, + { + name: "plain error wrapping an *Error with no category set, which wraps a categorized *Error", + err: fmt.Errorf("plain error: %w", &Error{wrapped: &Error{category: ErrCategoryUser}}), + wantCategory: ErrCategoryUser, + }, + { + name: "error created by NewTransientError", + err: NewTransientError(nil, "transient error"), + wantCategory: ErrCategoryTransient, + }, + { + name: "error created by Wraps over a categorized *Error", + err: Wraps(NewAPIServerError(nil, "API server error", false), "high-level description"), + wantCategory: ErrCategoryAPIServer, + }, + { + name: "error created by Wraps over a plain error", + err: Wraps(fmt.Errorf("plain error"), "high-level description"), + wantCategory: ErrCategoryUncategorized, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if gotCategory := Category(tc.err); gotCategory != tc.wantCategory { + t.Errorf("Category(%v) = %v, want %v", tc.err, gotCategory, tc.wantCategory) + } + }) + } +} + func TestUnwrap(t *testing.T) { innerErr := fmt.Errorf("inner error") testCases := []struct { diff --git a/pkg/utils/labels/labels.go b/pkg/utils/labels/labels.go index d2a8a2b76..c8ecb6803 100644 --- a/pkg/utils/labels/labels.go +++ b/pkg/utils/labels/labels.go @@ -41,9 +41,12 @@ func ExtractResourceSnapshotIndexFromWork(work client.Object) (int, error) { func ExtractIndex(object client.Object, labelKey string) (int, error) { indexStr := object.GetLabels()[labelKey] v, err := strconv.Atoi(indexStr) - if err != nil || v < 0 { + if err != nil { return -1, fmt.Errorf("invalid resource index %q, error: %w", indexStr, err) } + if v < 0 { + return -1, fmt.Errorf("invalid resource index %q: must be a non-negative integer", indexStr) + } return v, nil } @@ -57,9 +60,12 @@ func ParsePolicyIndexFromLabel(policySnapshot client.Object) (int, error) { indexLabel := labels[fleetv1beta1.PolicyIndexLabel] v, err := strconv.Atoi(indexLabel) - if err != nil || v < 0 { + if err != nil { return -1, fmt.Errorf("invalid policy index %q, error: %w", indexLabel, err) } + if v < 0 { + return -1, fmt.Errorf("invalid policy index %q: must be a non-negative integer", indexLabel) + } return v, nil } diff --git a/pkg/utils/labels/labels_test.go b/pkg/utils/labels/labels_test.go index 9bbe49ce3..1cff0549f 100644 --- a/pkg/utils/labels/labels_test.go +++ b/pkg/utils/labels/labels_test.go @@ -20,6 +20,7 @@ import ( "testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" fleetv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" @@ -27,6 +28,8 @@ import ( const ( snapshotName = "test-snapshot" + + nonIntegerIndex = "abc" ) func TestExtractResourceIndexFromClusterResourceSnapshot(t *testing.T) { @@ -266,3 +269,133 @@ func TestExtractLabelFromMemberCluster(t *testing.T) { }) } } + +func TestExtractIndex_ErrorMessage(t *testing.T) { + testCases := []struct { + name string + labelValue string + wantErrMsg string + }{ + { + name: "not an integer", + labelValue: nonIntegerIndex, + wantErrMsg: `invalid resource index "abc", error: strconv.Atoi: parsing "abc": invalid syntax`, + }, + { + name: "negative integer", + labelValue: "-1", + wantErrMsg: `invalid resource index "-1": must be a non-negative integer`, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + snapshot := &fleetv1beta1.ClusterResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{fleetv1beta1.ResourceIndexLabel: tc.labelValue}, + }, + } + _, err := ExtractIndex(snapshot, fleetv1beta1.ResourceIndexLabel) + if err == nil { + t.Fatalf("ExtractIndex() error = nil, want %q", tc.wantErrMsg) + } + if err.Error() != tc.wantErrMsg { + t.Errorf("ExtractIndex() error = %q, want %q", err.Error(), tc.wantErrMsg) + } + }) + } +} + +func TestParsePolicyIndexFromLabel(t *testing.T) { + testCases := []struct { + name string + policySnapshot client.Object + wantIndex int + wantErrMsg string + }{ + { + name: "valid index on a cluster-scoped snapshot", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{fleetv1beta1.PolicyIndexLabel: "3"}, + }, + }, + wantIndex: 3, + }, + { + name: "valid index on a namespaced snapshot", + policySnapshot: &fleetv1beta1.SchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Namespace: "test-namespace", + Labels: map[string]string{fleetv1beta1.PolicyIndexLabel: "0"}, + }, + }, + wantIndex: 0, + }, + { + name: "no labels", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{Name: snapshotName}, + }, + wantIndex: -1, + wantErrMsg: "no labels found on policy snapshot", + }, + { + name: "index label missing", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{"other": "label"}, + }, + }, + wantIndex: -1, + wantErrMsg: `invalid policy index "", error: strconv.Atoi: parsing "": invalid syntax`, + }, + { + name: "not an integer", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{fleetv1beta1.PolicyIndexLabel: nonIntegerIndex}, + }, + }, + wantIndex: -1, + wantErrMsg: `invalid policy index "abc", error: strconv.Atoi: parsing "abc": invalid syntax`, + }, + { + name: "negative integer", + policySnapshot: &fleetv1beta1.ClusterSchedulingPolicySnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Labels: map[string]string{fleetv1beta1.PolicyIndexLabel: "-1"}, + }, + }, + wantIndex: -1, + wantErrMsg: `invalid policy index "-1": must be a non-negative integer`, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotIndex, err := ParsePolicyIndexFromLabel(tc.policySnapshot) + if gotIndex != tc.wantIndex { + t.Errorf("ParsePolicyIndexFromLabel() index = %d, want %d", gotIndex, tc.wantIndex) + } + if tc.wantErrMsg == "" { + if err != nil { + t.Fatalf("ParsePolicyIndexFromLabel() error = %v, want nil", err) + } + return + } + if err == nil { + t.Fatalf("ParsePolicyIndexFromLabel() error = nil, want %q", tc.wantErrMsg) + } + if err.Error() != tc.wantErrMsg { + t.Errorf("ParsePolicyIndexFromLabel() error = %q, want %q", err.Error(), tc.wantErrMsg) + } + }) + } +} diff --git a/pkg/utils/naming/naming.go b/pkg/utils/naming/naming.go new file mode 100644 index 000000000..5d0eea89c --- /dev/null +++ b/pkg/utils/naming/naming.go @@ -0,0 +1,149 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package naming derives the names and label values of objects that KubeFleet generates on a +// user's behalf. +// +// A controller that names an object after another one runs into the same two problems every time: +// a name legal for a Kubernetes object is not always legal where the controller puts it (object +// names run to 253 bytes, label values stop at 63), and shortening a name to fit can both produce +// a syntactically invalid result and collapse two distinct identities onto one string. Getting +// either wrong fails at the API server rather than at the call site, which surfaces as a +// reconciler that retries forever while the user sees nothing happen at all. +// +// The helpers here exist so that the answer is written once. They are deliberately primitive: +// callers compose the identity that is hashed, since only the caller knows what makes one of its +// objects distinct from another. +package naming + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "strings" + + "k8s.io/apimachinery/pkg/util/validation" +) + +const ( + // HashLength is the number of hexadecimal characters of Hash's output, i.e. 64 bits of a + // SHA-256 sum. A narrower hash is tempting and wrong: these hashes are what keep two distinct + // objects from generating one name, and whichever of them loses the race is not merely + // misnamed, it silently never gets its object at all. + HashLength = 16 + + // labelValuePrefixMaxLength is how much of an over-long value survives in front of the hash + // that LabelValue appends, allowing for the separator between the two. + labelValuePrefixMaxLength = validation.LabelValueMaxLength - HashLength - 1 + + // separators are the characters a Kubernetes name may contain but may not begin or end with. + separators = "-_." +) + +// Hash returns a short, stable, collision-resistant digest of s. +// +// Callers should hash the whole identity of an object, never the shortened form of it that appears +// in a generated name: shortening is precisely what collapses two identities into one, so a hash +// taken afterwards would agree where the identities differ. +func Hash(s string) string { + sum := sha256.Sum256([]byte(s)) + return hex.EncodeToString(sum[:])[:HashLength] +} + +// TrimSeparators removes the separators that a name may not end with, which truncating an +// otherwise valid name can leave exposed. +// +// The result can be empty only if the fragment consisted entirely of separators; since Kubernetes +// object names must begin with an alphanumeric character, a fragment taken from the front of one +// never can. +func TrimSeparators(fragment string) string { + return strings.TrimRight(fragment, separators) +} + +// Truncate shortens s to at most maxLength bytes, leaving a result that is still legal at the end +// of a Kubernetes name. +// +// It does not make the result unique; a caller that truncates must append Hash of the full, +// untruncated identity for that. +// +// A budget of zero or less leaves nothing, which is returned as the empty string rather than as a +// panic: this package exists so that a caller's arithmetic mistake surfaces as a name it can +// inspect, not as a crash in a reconcile loop. +func Truncate(s string, maxLength int) string { + if maxLength <= 0 { + return "" + } + if len(s) <= maxLength { + return s + } + return TrimSeparators(s[:maxLength]) +} + +// Sanitize renders s legal as a single label of a generated DNS-1123 name: it lowercases the value +// and replaces every character outside [a-z0-9-] with a dash. A name legal for its own API but not +// as a Kubernetes object name -- an RBAC name like system:aggregate-to-admin, whose colon a DNS-1123 +// name forbids -- becomes usable this way. +// +// The dot is mapped to a dash along with everything else, rather than kept as a label separator: a +// dot is only legal between two alphanumerics, so preserving it would let an invalid character next +// to it (or another dot) produce an empty or dash-bounded label that the API server still rejects -- +// the very failure Sanitize exists to prevent. Its one caller joins the sanitized parts with dashes +// anyway, so no dot is needed to keep them apart. +// +// It is lossy: distinct inputs can sanitize to the same string, so a caller must still append Hash +// of the full, unsanitized identity for uniqueness. +func Sanitize(s string) string { + return TrimSeparators(strings.TrimLeft(mapInvalid(strings.ToLower(s), isDNS1123Char), separators)) +} + +// sanitizeLabelValue is Sanitize's counterpart for a label value, whose legal character set is +// wider (it keeps case and underscores). It exists so LabelValue never returns a value the API +// server would reject; like Sanitize it is lossy and relies on a hash for identity. +func sanitizeLabelValue(value string) string { + return TrimSeparators(strings.TrimLeft(mapInvalid(value, isLabelValueChar), separators)) +} + +func mapInvalid(s string, valid func(rune) bool) string { + return strings.Map(func(r rune) rune { + if valid(r) { + return r + } + return '-' + }, s) +} + +func isDNS1123Char(r rune) bool { + return (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') || r == '-' +} + +func isLabelValueChar(r rune) bool { + return (r >= 'a' && r <= 'z') || (r >= 'A' && r <= 'Z') || (r >= '0' && r <= '9') || r == '-' || r == '_' || r == '.' +} + +// LabelValue renders value so that it fits in a label value, sanitizing characters the API server +// would reject and shortening it to a prefix and a hash of the whole when it is too long. +// +// The result is lossy by design -- both sanitization and shortening can collapse distinct values -- +// and callers must treat it that way: selecting on the label with the original value matches nothing +// once it has been rewritten, so anything that needs to resolve an exact identity has to read it +// from a field that has neither a character nor a length limit. +func LabelValue(value string) string { + sanitized := sanitizeLabelValue(value) + if len(sanitized) <= validation.LabelValueMaxLength { + return sanitized + } + return fmt.Sprintf("%s-%s", TrimSeparators(sanitized[:labelValuePrefixMaxLength]), Hash(value)) +} diff --git a/pkg/utils/naming/naming_test.go b/pkg/utils/naming/naming_test.go new file mode 100644 index 000000000..3b633e6ac --- /dev/null +++ b/pkg/utils/naming/naming_test.go @@ -0,0 +1,256 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package naming + +import ( + "strings" + "testing" + + "k8s.io/apimachinery/pkg/util/validation" +) + +func TestHash(t *testing.T) { + testCases := []struct { + name string + a, b string + want bool + }{ + {name: "same input hashes the same", a: "apps/Deployment/prod/app", b: "apps/Deployment/prod/app", want: true}, + {name: "namespace is part of the identity", a: "apps/Deployment/prod/app", b: "apps/Deployment/staging/app", want: false}, + {name: "api group is part of the identity", a: "apps/Widget/prod/app", b: "example.com/Widget/prod/app", want: false}, + {name: "separator placement matters", a: "apps/Deployment/prod/app", b: "apps/Deployment/prod-app/", want: false}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotA, gotB := Hash(tc.a), Hash(tc.b) + if (gotA == gotB) != tc.want { + t.Errorf("Hash(%q) = %v, Hash(%q) = %v, want equal to be %v", tc.a, gotA, tc.b, gotB, tc.want) + } + }) + } +} + +// TestHashShape pins the properties every generated name depends on: a fixed width, and a value +// made only of characters legal anywhere in a Kubernetes name. +func TestHashShape(t *testing.T) { + for _, input := range []string{"", "app", strings.Repeat("x", 4096), "ünïcödé/名前"} { + got := Hash(input) + if len(got) != HashLength { + t.Errorf("len(Hash(%q)) = %d, want %d", input, len(got), HashLength) + } + if strings.Trim(got, "0123456789abcdef") != "" { + t.Errorf("Hash(%q) = %v, want only lowercase hexadecimal characters", input, got) + } + } +} + +func TestTrimSeparators(t *testing.T) { + testCases := []struct { + name string + fragment string + want string + }{ + {name: "nothing to trim", fragment: "app", want: "app"}, + {name: "trailing dash", fragment: "app-", want: "app"}, + {name: "trailing dot", fragment: "app.", want: "app"}, + {name: "trailing underscore", fragment: "app_", want: "app"}, + {name: "run of separators", fragment: "app-._-", want: "app"}, + {name: "leading separators are left alone", fragment: "-app", want: "-app"}, + {name: "separators in the middle are left alone", fragment: "my-app.v2", want: "my-app.v2"}, + {name: "entirely separators", fragment: "---", want: ""}, + {name: "empty", fragment: "", want: ""}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := TrimSeparators(tc.fragment); got != tc.want { + t.Errorf("TrimSeparators(%q) = %v, want %v", tc.fragment, got, tc.want) + } + }) + } +} + +func TestTruncate(t *testing.T) { + testCases := []struct { + name string + s string + maxLength int + want string + }{ + {name: "shorter than the limit", s: "app", maxLength: 10, want: "app"}, + {name: "exactly at the limit", s: "app", maxLength: 3, want: "app"}, + {name: "over the limit", s: "application", maxLength: 4, want: "appl"}, + // The cut landing on a separator is the case that produced an invalid generated name in + // practice: the character after the cut would begin a new label. + {name: "cut lands on a dash", s: "my-application", maxLength: 3, want: "my"}, + {name: "cut lands on a dot", s: "my.application", maxLength: 3, want: "my"}, + {name: "cut lands inside a run of separators", s: "my-._application", maxLength: 5, want: "my"}, + {name: "cut lands just past a run of separators", s: "my-._application", maxLength: 6, want: "my-._a"}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := Truncate(tc.s, tc.maxLength) + if got != tc.want { + t.Errorf("Truncate(%q, %d) = %v, want %v", tc.s, tc.maxLength, got, tc.want) + } + if len(got) > tc.maxLength { + t.Errorf("len(Truncate(%q, %d)) = %d, want at most %d", tc.s, tc.maxLength, len(got), tc.maxLength) + } + }) + } +} + +func TestLabelValue(t *testing.T) { + testCases := []struct { + name string + value string + want string + }{ + {name: "short value is kept as is", value: "app", want: "app"}, + { + name: "value at the limit is kept as is", + value: strings.Repeat("a", validation.LabelValueMaxLength), + want: strings.Repeat("a", validation.LabelValueMaxLength), + }, + { + // The separator at the truncation point is trimmed, not kept in front of the hash's own. + name: "separator at the truncation point is trimmed before the hash", + value: strings.Repeat("a", labelValuePrefixMaxLength-1) + "-" + strings.Repeat("b", 40), + want: strings.Repeat("a", labelValuePrefixMaxLength-1) + "-" + Hash(strings.Repeat("a", labelValuePrefixMaxLength-1)+"-"+strings.Repeat("b", 40)), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := LabelValue(tc.value); got != tc.want { + t.Errorf("LabelValue(%q) = %v, want %v", tc.value, got, tc.want) + } + }) + } +} + +// TestLabelValueShortening covers the values that do not fit, where the result is not a value the +// test can spell out but a set of properties it must hold: it has to be a legal label value, it +// has to stay within the limit, and it has to distinguish inputs that share a prefix. +func TestLabelValueShortening(t *testing.T) { + long := strings.Repeat("a", validation.LabelValueMaxLength+1) + testCases := []string{ + long, + long + "b", + // A value whose truncation point falls on a separator, which must not survive into the + // shortened form. + strings.Repeat("a", labelValuePrefixMaxLength-1) + "-" + strings.Repeat("b", 40), + strings.Repeat("a", labelValuePrefixMaxLength-1) + "." + strings.Repeat("c", 40), + strings.Repeat("a", labelValuePrefixMaxLength-1) + "_" + strings.Repeat("d", 40), + } + + seen := make(map[string]string, len(testCases)) + for _, value := range testCases { + got := LabelValue(value) + if len(got) > validation.LabelValueMaxLength { + t.Errorf("len(LabelValue(%q)) = %d, want at most %d", value, len(got), validation.LabelValueMaxLength) + } + if errs := validation.IsValidLabelValue(got); len(errs) > 0 { + t.Errorf("LabelValue(%q) = %v, want a valid label value: %s", value, got, strings.Join(errs, "; ")) + } + if previous, collided := seen[got]; collided { + t.Errorf("LabelValue(%q) = %v, want a value distinct from that of %q", value, got, previous) + } + seen[got] = value + } +} + +// TestSanitize pins the transform that lets a name legal for its own API but not as a Kubernetes +// object name pass through into a generated name: every result must be legal as a DNS-1123 name +// segment, and the characters a name already allows must survive unchanged. +func TestSanitize(t *testing.T) { + testCases := []struct { + name string + value string + want string + }{ + {name: "already legal", value: "my-app", want: "my-app"}, + {name: "uppercase is lowercased", value: "MyApp", want: "myapp"}, + // The RBAC name that motivated the sanitizer: a colon is legal in the name but not in a + // Kubernetes object name. + {name: "rbac name with a colon", value: "system:aggregate-to-admin", want: "system-aggregate-to-admin"}, + {name: "underscores become dashes", value: "my_app", want: "my-app"}, + // A dot becomes a dash like any other separator: kept as a dot, an invalid character beside it + // would leave a label bounded by a dash, which the API server rejects. + {name: "dot becomes a dash", value: "my.app", want: "my-app"}, + {name: "dot next to an illegal character does not leave a bad label", value: "my.:app", want: "my--app"}, + {name: "adjacent dots do not leave an empty label", value: "a..b", want: "a--b"}, + {name: "leading separators are trimmed", value: ":::app", want: "app"}, + {name: "trailing separators are trimmed", value: "app:::", want: "app"}, + {name: "entirely illegal collapses to empty", value: ":::", want: ""}, + {name: "empty", value: "", want: ""}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := Sanitize(tc.value) + if got != tc.want { + t.Errorf("Sanitize(%q) = %v, want %v", tc.value, got, tc.want) + } + if got != "" { + if errs := validation.IsDNS1123Subdomain(got); len(errs) > 0 { + t.Errorf("Sanitize(%q) = %v, want a valid DNS-1123 subdomain: %s", tc.value, got, strings.Join(errs, "; ")) + } + } + }) + } +} + +// TestLabelValueSanitizes covers values that carry characters a label value forbids. The wider +// label-value character set keeps case and underscores that Sanitize would drop, but a colon still +// has to go, and the result must be a value the API server accepts. +func TestLabelValueSanitizes(t *testing.T) { + testCases := []struct { + name string + value string + want string + }{ + {name: "case and underscores are kept", value: "My_App", want: "My_App"}, + {name: "colon becomes a dash", value: "system:aggregate-to-admin", want: "system-aggregate-to-admin"}, + {name: "leading separators are trimmed", value: ":app", want: "app"}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := LabelValue(tc.value) + if got != tc.want { + t.Errorf("LabelValue(%q) = %v, want %v", tc.value, got, tc.want) + } + if errs := validation.IsValidLabelValue(got); len(errs) > 0 { + t.Errorf("LabelValue(%q) = %v, want a valid label value: %s", tc.value, got, strings.Join(errs, "; ")) + } + }) + } +} + +// TestTruncateNonPositiveBudget covers a caller whose own arithmetic left nothing for the value. +// The package exists to keep budget mistakes from reaching the API server; it must not turn one +// into a panic inside a reconcile loop either. +func TestTruncateNonPositiveBudget(t *testing.T) { + for _, maxLength := range []int{0, -1, -64} { + if got := Truncate("application", maxLength); got != "" { + t.Errorf("Truncate(%q, %d) = %v, want the empty string", "application", maxLength, got) + } + } +} diff --git a/pkg/utils/resourceeligibility/defaults.go b/pkg/utils/resourceeligibility/defaults.go new file mode 100644 index 000000000..b552ab258 --- /dev/null +++ b/pkg/utils/resourceeligibility/defaults.go @@ -0,0 +1,200 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package resourceeligibility + +import ( + coordv1 "k8s.io/api/coordination/v1" + corev1 "k8s.io/api/core/v1" + eventsv1 "k8s.io/api/events/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + metricsv1beta1 "k8s.io/metrics/pkg/apis/metrics/v1beta1" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils" +) + +var ( + // GVKsDeniedByDefault lists the GVKs that KubeFleet blocks from placement regardless of the API + // version in use; it covers the KubeFleet APIs themselves plus a few Kubernetes system APIs whose + // objects are node-local or control-plane managed. + // + // An empty version or kind acts as a wildcard; see gvkMatcherTrie.Register. + GVKsDeniedByDefault = []schema.GroupVersionKind{ + // Deny all resources in the cluster.kubernetes-fleet.io API group. + { + Group: clusterv1beta1.GroupVersion.Group, + }, + // Deny node resources. + { + Group: corev1.GroupName, + Kind: "Node", + }, + // Deny pod resources. + { + Group: corev1.GroupName, + Kind: utils.PodKind, + }, + // Deny all resources in the events.k8s.io API group. + { + Group: eventsv1.GroupName, + }, + // Deny all resources in the coordination.k8s.io API group. + { + Group: coordv1.GroupName, + }, + // Deny all resources in the metrics.k8s.io API group. + { + Group: metricsv1beta1.GroupName, + }, + // Deny the fleet networking resources that are managed by the fleet networking controllers. + { + Group: utils.NetworkingGroupName, + Kind: "ServiceImport", + }, + { + Group: utils.NetworkingGroupName, + Kind: "TrafficManagerProfile", + }, + { + Group: utils.NetworkingGroupName, + Kind: "TrafficManagerBackend", + }, + // Deny all non-envelope resources in the placement.kubernetes-fleet.io API group. + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourcePlacementKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ResourcePlacementKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourceBindingKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ResourceBindingKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourceSnapshotKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ResourceSnapshotKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterSchedulingPolicySnapshotKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.SchedulingPolicySnapshotKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.WorkKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterStagedUpdateRunKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterStagedUpdateStrategyKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterApprovalRequestKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.StagedUpdateRunKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.StagedUpdateStrategyKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ApprovalRequestKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourcePlacementEvictionKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourcePlacementDisruptionBudgetKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourceOverrideKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourceOverrideSnapshotKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ResourceOverrideKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ResourceOverrideSnapshotKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourcePlacementStatusKind, + }, + // Deny all resources in the placement.kubefleet.dev API group. + // + // TO-DO (chenyu1): deny resources by kinds when envelopes are added to the API group. + { + Group: kfplacementv1alpha1.GroupVersion.Group, + }, + // Deny all resources in the rollout.kubefleet.dev API group; member clusters do not have its CRDs. + { + Group: kfrolloutv1alpha1.GroupVersion.Group, + }, + } + + // GVKsDeniedForV1APIs lists the GVKs that are blocked from placement only when the new placement APIs (placement + // policies) are in use. + GVKsDeniedForV1APIs = []schema.GroupVersionKind{ + // Deny endpoints resources; the API is deprecated and the objects are typically managed by the control plane. + // + // Note (chenyu1): v0 placement APIs handle this API type differently. + { + Group: corev1.GroupName, + Kind: "Endpoints", + }, + // Deny envelope APIs in the placement.kubernetes-fleet.io API group. + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ClusterResourceEnvelopeKind, + }, + { + Group: placementv1beta1.GroupVersion.Group, + Kind: placementv1beta1.ResourceEnvelopeKind, + }, + } +) diff --git a/pkg/utils/resourceeligibility/guard.go b/pkg/utils/resourceeligibility/guard.go new file mode 100644 index 000000000..a3d834a00 --- /dev/null +++ b/pkg/utils/resourceeligibility/guard.go @@ -0,0 +1,440 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package resourceeligibility features utilities for enforcing resource-level placement constraints in KubeFleet. +package resourceeligibility + +import ( + "fmt" + "strings" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/sets" + + "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/errors" +) + +// NamespaceEligibilityMode controls how a namespace is evaluated for placement eligibility. +type NamespaceEligibilityMode string + +const ( + // NamespaceEligibilityModeAllowList means that namespaces that match with the checker are eligible for placement. + NamespaceEligibilityModeAllowList NamespaceEligibilityMode = "AllowList" + // NamespaceEligibilityModeDenyList means that namespaces that match with the checker are ineligible for placement. + NamespaceEligibilityModeDenyList NamespaceEligibilityMode = "DenyList" +) + +// GVKEligibilityMode controls how a GroupVersionKind (GVK) is evaluated for placement eligibility. +type GVKEligibilityMode string + +const ( + // GVKEligibilityModeAllowList means that GVKs that match with the checker are eligible for placement. + GVKEligibilityModeAllowList GVKEligibilityMode = "AllowList" + // GVKEligibilityModeDenyList means that GVKs that match with the checker are ineligible for placement. + GVKEligibilityModeDenyList GVKEligibilityMode = "DenyList" +) + +// Checker reports whether a namespace, a resource type, or a resource object is eligible for placement. +// +// A Checker may wrap another Checker; the wrapped Checker is consulted after the current one, which +// allows a set of custom, user-supplied constraints to be layered on top of the KubeFleet built-in ones. +// Note that wrapping can only narrow eligibility, never widen it. +// +// Once set up, a Checker should not be modified anymore. +type Checker interface { + SetNamespaceCheckList(mode NamespaceEligibilityMode, nsNames, nsNamePrefixes []string) + SetGVKCheckList(mode GVKEligibilityMode, gvks []schema.GroupVersionKind) error + + Wraps(c Checker) + DeepCopy() Checker + + AllowsNamespace(namespace string) bool + AllowsGVK(gvk schema.GroupVersionKind) bool + AllowsGVR(gvr schema.GroupVersionResource) (bool, error) + AllowsObject(obj *unstructured.Unstructured) (bool, string, error) +} + +// Verify that checker implements the Checker interface. +var _ Checker = &checker{} + +// checker is the default Checker implementation; it evaluates a namespace check list and a GVK check +// list, each in either allow list or deny list mode, before deferring to any wrapped Checker. +type checker struct { + nsEligibilityMode NamespaceEligibilityMode + customNSList sets.Set[string] + customNSPrefixList sets.Set[string] + + gvkEligibilityMode GVKEligibilityMode + customGVKList gvkMatcherTrie + + skipNSCheck bool + skipGVKRCheck bool + skipObjectCheck bool + + wrapped Checker + + restMapper meta.RESTMapper +} + +// Baseline returns a Checker that features default namespace and GVK deny lists. The checker also has object checks +// enabled. +func Baseline(restMapper meta.RESTMapper) Checker { + return baseline(restMapper, false) +} + +func baseline(restMapper meta.RESTMapper, skipObjectCheck bool) Checker { + c := New(CheckerOptions{ + RestMapper: restMapper, + Wrapped: nil, + SkipNSCheck: false, + SkipGVKRCheck: false, + SkipObjectCheck: skipObjectCheck, + }) + + // About the `default` namespace: it is not explicitly denied here; the namespace itself cannot be selected for placement, + // but KubeFleet considers it OK to have resources within the namespace to be placed. + // + // For the v0 APIs, this is enforced individually when processing the placements. + // For the v1 APIs (the new placement experience), this is enforced through the AllowsObject check. See the definition + // below for more information. + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, nil, []string{utils.KubeNSNamePrefix, utils.FleetNSNamePrefix}) + if err := c.SetGVKCheckList(GVKEligibilityModeDenyList, GVKsDeniedByDefault); err != nil { + // The deny list is defined in this package, so a rejection here is a programming error. + panic(fmt.Sprintf("failed to register the default GVK deny list: %v", err)) + } + return c +} + +// AllowAll returns a Checker that allows all namespaces and GVKs for placement. +// +// This function is added for testing purposes only. +func AllowAll(restMapper meta.RESTMapper) Checker { + return &checker{ + nsEligibilityMode: NamespaceEligibilityModeDenyList, + gvkEligibilityMode: GVKEligibilityModeDenyList, + restMapper: restMapper, + } +} + +// DefaultForV0APIs returns a Checker for placement with v0 APIs. +func DefaultForV0APIs(restMapper meta.RESTMapper) Checker { + checkerOpts := CheckerOptions{ + RestMapper: restMapper, + Wrapped: baseline(restMapper, true), + SkipNSCheck: true, + SkipGVKRCheck: false, + SkipObjectCheck: true, + } + c := New(checkerOpts) + + // TO-DO (chenyu1): deny envelope APIs in the kubefleet.dev API group when they are added. + return c +} + +// DefaultForV1APIs returns a Checker for placement with v1 APIs. +func DefaultForV1APIs(restMapper meta.RESTMapper) Checker { + checkerOpts := CheckerOptions{ + RestMapper: restMapper, + Wrapped: Baseline(restMapper), + SkipNSCheck: true, + SkipGVKRCheck: false, + SkipObjectCheck: true, + } + c := New(checkerOpts) + + if err := c.SetGVKCheckList(GVKEligibilityModeDenyList, GVKsDeniedForV1APIs); err != nil { + // The deny list is defined in this package, so a rejection here is a programming error. + panic(fmt.Sprintf("failed to register the default GVK deny list for the v1 APIs: %v", err)) + } + return c +} + +// CheckerOptions is the options for creating a new Checker. +type CheckerOptions struct { + RestMapper meta.RESTMapper + + SkipNSCheck bool + SkipGVKRCheck bool + SkipObjectCheck bool + + Wrapped Checker +} + +// New creates a new Checker. +func New(opts CheckerOptions) Checker { + return &checker{ + nsEligibilityMode: NamespaceEligibilityModeDenyList, + gvkEligibilityMode: GVKEligibilityModeDenyList, + wrapped: opts.Wrapped, + restMapper: opts.RestMapper, + skipNSCheck: opts.SkipNSCheck, + skipGVKRCheck: opts.SkipGVKRCheck, + skipObjectCheck: opts.SkipObjectCheck, + } +} + +// SetNamespaceCheckList sets the namespace check list for the checker, along with the mode in which +// the list is evaluated. +// +// Empty entries are ignored. +// +// For simplicity reasons, the setter will overwrite any existing check list and no concurrency control is added; +// the implementation assumes that a Checker is set up once before usage and is not modified at all afterwards. +func (c *checker) SetNamespaceCheckList(mode NamespaceEligibilityMode, nsNames, nsNamePrefixes []string) { + c.nsEligibilityMode = mode + c.customNSList = newNonEmptyStringSet(nsNames) + c.customNSPrefixList = newNonEmptyStringSet(nsNamePrefixes) +} + +// newNonEmptyStringSet builds a set from the given items, dropping empty ones; an empty prefix would +// otherwise match every namespace. +func newNonEmptyStringSet(items []string) sets.Set[string] { + s := sets.New[string]() + for _, item := range items { + if item != "" { + s.Insert(item) + } + } + return s +} + +// SetGVKCheckList sets the GVK check list for the checker, along with the mode in which the list is evaluated. +// +// For simplicity reasons, the setter will overwrite any existing check list and no concurrency control is added; +// the implementation assumes that a Checker is set up once before usage and is not modified at all afterwards. +func (c *checker) SetGVKCheckList(mode GVKEligibilityMode, gvks []schema.GroupVersionKind) error { + // Build the trie separately so that a rejected GVK leaves the checker untouched. + list := gvkMatcherTrie{} + for _, gvk := range gvks { + if err := list.Register(gvk); err != nil { + return err + } + } + + c.gvkEligibilityMode = mode + c.customGVKList = list + return nil +} + +// Wraps sets another Checker to be consulted for eligibility checks after this checker. +// +// For simplicity reasons, the setter will overwrite any existing wrapped checker and no concurrency control is added; +// the implementation assumes that a Checker is set up once before usage and is not modified at all afterwards. +func (c *checker) Wraps(wrapped Checker) { + c.wrapped = wrapped +} + +// DeepCopy creates a deep copy of the checker. +// +// The deep copy will have its own copies of the check lists and any wrapped checker, +// but will share the REST mapper reference. +func (c *checker) DeepCopy() Checker { + cp := &checker{ + nsEligibilityMode: c.nsEligibilityMode, + customNSList: c.customNSList.Clone(), + customNSPrefixList: c.customNSPrefixList.Clone(), + gvkEligibilityMode: c.gvkEligibilityMode, + customGVKList: c.customGVKList.DeepCopy(), + skipNSCheck: c.skipNSCheck, + skipGVKRCheck: c.skipGVKRCheck, + skipObjectCheck: c.skipObjectCheck, + // The REST mapper is a shared dependency rather than owned state, so the reference is kept. + restMapper: c.restMapper, + } + if c.wrapped != nil { + cp.wrapped = c.wrapped.DeepCopy() + } + return cp +} + +// AllowsNamespace checks if a given namespace (the namespace itself and any resource within it) +// is eligible for placement. +func (c *checker) AllowsNamespace(namespace string) bool { + if namespace == "" { + // Allow cluster-scoped resources (which have an empty namespace) for placement. + return true + } + if c.skipNSCheck { + // This checker has been set to skip the namespace check; delegate the decision to the wrapped checker if available. + if c.wrapped != nil { + return c.wrapped.AllowsNamespace(namespace) + } + // If no wrapper checker is available, return a decision based on the namespace eligibility mode. + // + // If the checker uses a namespace deny list, allow all namespaces; otherwise, deny all namespaces. + return c.nsEligibilityMode != NamespaceEligibilityModeAllowList + } + + hasAnyPrefix := func(prefixes sets.Set[string]) bool { + for prefix := range prefixes { + if strings.HasPrefix(namespace, prefix) { + return true + } + } + return false + } + + // Check namespace eligibility based on the configured allow list or deny list. + eligible := true + switch c.nsEligibilityMode { + case NamespaceEligibilityModeAllowList: + eligible = c.customNSList.Has(namespace) || hasAnyPrefix(c.customNSPrefixList) + case NamespaceEligibilityModeDenyList: + eligible = !c.customNSList.Has(namespace) && !hasAnyPrefix(c.customNSPrefixList) + } + if !eligible { + return false + } + + // Consult the wrapped checker to check if the eligibility is further restricted. + if c.wrapped != nil { + return c.wrapped.AllowsNamespace(namespace) + } + return true +} + +// AllowsGVK checks if a given GVK is eligible for placement. +func (c *checker) AllowsGVK(gvk schema.GroupVersionKind) bool { + if c.skipGVKRCheck { + // This checker has been set to skip the GVK check; delegate the decision to the wrapped checker if available. + if c.wrapped != nil { + return c.wrapped.AllowsGVK(gvk) + } + // If no wrapper checker is available, return a decision based on the GVK eligibility mode. + // + // If the checker uses a GVK deny list, allow all GVKs; otherwise, deny all GVKs. + return c.gvkEligibilityMode != GVKEligibilityModeAllowList + } + + // Check GVK eligibility based on the configured allow list or deny list. + eligible := true + switch c.gvkEligibilityMode { + case GVKEligibilityModeAllowList: + eligible = c.customGVKList.Match(gvk) + case GVKEligibilityModeDenyList: + eligible = !c.customGVKList.Match(gvk) + } + if !eligible { + return false + } + + // Consult the wrapped checker to check if the eligibility is further restricted. + if c.wrapped != nil { + return c.wrapped.AllowsGVK(gvk) + } + return true +} + +// AllowsGVR checks if a given GVR is eligible for placement. +func (c *checker) AllowsGVR(gvr schema.GroupVersionResource) (bool, error) { + if c.skipGVKRCheck { + // This checker has been set to skip the GVKR check; delegate the decision to the wrapped checker if available. + if c.wrapped != nil { + return c.wrapped.AllowsGVR(gvr) + } + // If no wrapper checker is available, return a decision based on the GVKR eligibility mode. + // + // If the checker uses a GVK deny list, allow all GVRs; otherwise, deny all GVRs. + return c.gvkEligibilityMode != GVKEligibilityModeAllowList, nil + } + + gvks, err := c.restMapper.KindsFor(gvr) + if err != nil { + if meta.IsNoMatchError(err) { + return false, errors.NewUserError(err, "no match is found given the GVR", "gvr", gvr) + } + return false, errors.NewUnexpectedError(err, "failed to map the resource to its kinds", "gvr", gvr) + } + if len(gvks) == 0 { + noMatchErr := &meta.NoResourceMatchError{ + PartialResource: gvr, + } + return false, errors.NewUserError(noMatchErr, "no match is found given the GVR", "gvr", gvr) + } + + // A resource is eligible only if every kind it maps to is eligible. + for _, gvk := range gvks { + if !c.AllowsGVK(gvk) { + return false, nil + } + } + + // Consult the wrapped checker to check if the eligibility is further restricted. + if c.wrapped != nil { + return c.wrapped.AllowsGVR(gvr) + } + return true, nil +} + +// AllowsObject checks if the given unstructured resource object is eligible for placement. +// +// This method will NOT check if the resource is eligible for placement given its namespaces, GVK (GVR); call +// the other methods for verification. +// +// Note (chenyu1): the logic defined here applies only to the v1 APIs (placement policies), as the v1 APIs +// require explicit resource selectors. +func (c *checker) AllowsObject(obj *unstructured.Unstructured) (bool, string, error) { + if c.skipObjectCheck { + // This checker has been set to skip the object check; delegate the decision to the wrapped checker if available. + if c.wrapped != nil { + return c.wrapped.AllowsObject(obj) + } + // If no wrapper is set up, allow all objects. + return true, "", nil + } + + switch { + case obj.GetDeletionTimestamp() != nil: + return false, "the resource is being deleted and cannot be placed", nil + case hasControllerOwner(obj): + return false, "the resource is under the management of a controller; place the owner object instead if applicable", nil + case obj.GroupVersionKind() == corev1.SchemeGroupVersion.WithKind(utils.ConfigMapKind) && obj.GetName() == "kube-root-ca.crt": + return false, "the built-in CA certificate config map is created automatically in every namespace and cannot be placed", nil + case obj.GroupVersionKind() == corev1.SchemeGroupVersion.WithKind("ServiceAccount") && obj.GetName() == "default": + return false, "the default service account is created automatically in every namespace and cannot be placed", nil + case obj.GroupVersionKind() == corev1.SchemeGroupVersion.WithKind(utils.ServiceKind) && obj.GetNamespace() == "default" && obj.GetName() == "kubernetes": + return false, "the Kubernetes API service is created automatically in the default namespace and cannot be placed", nil + case obj.GroupVersionKind() == corev1.SchemeGroupVersion.WithKind("Secret"): + secretType, found, err := unstructured.NestedString(obj.Object, "type") + if err != nil { + return false, "", errors.NewUnexpectedError(err, "failed to retrieve the type of the secret") + } + if found && secretType == string(corev1.SecretTypeServiceAccountToken) { + return false, "service account token secrets are created automatically along with their service accounts and cannot be placed", nil + } + case obj.GroupVersionKind() == utils.NamespaceGVK && obj.GetName() == "default": + return false, "the default namespace itself cannot be placed", nil + } + + // Consult the wrapped checker to check if the eligibility is further restricted. + if c.wrapped != nil { + return c.wrapped.AllowsObject(obj) + } + return true, "", nil +} + +func hasControllerOwner(obj *unstructured.Unstructured) bool { + for _, ownerRef := range obj.GetOwnerReferences() { + if ownerRef.Controller != nil && *ownerRef.Controller { + return true + } + } + return false +} diff --git a/pkg/utils/resourceeligibility/guard_test.go b/pkg/utils/resourceeligibility/guard_test.go new file mode 100644 index 000000000..ff31267f1 --- /dev/null +++ b/pkg/utils/resourceeligibility/guard_test.go @@ -0,0 +1,1295 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package resourceeligibility + +import ( + "errors" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/utils/ptr" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils" +) + +var ( + checkerCmpOptions = cmp.Options{ + cmp.AllowUnexported(checker{}, gvkMatcherTrie{}, gvkMatcherTrieNode{}), + // The REST mapper is a shared dependency rather than owned state. + cmpopts.IgnoreInterfaces(struct{ meta.RESTMapper }{}), + cmpopts.EquateEmpty(), + } + + deploymentGVK = schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "Deployment"} + configMapGVK = corev1.SchemeGroupVersion.WithKind(utils.ConfigMapKind) + nodeGVK = corev1.SchemeGroupVersion.WithKind("Node") + + deploymentGVR = schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} + nodeGVR = corev1.SchemeGroupVersion.WithResource("nodes") + unknownGVR = schema.GroupVersionResource{Group: "example.com", Version: "v1", Resource: "widgets"} +) + +// newTestRESTMapper returns a REST mapper with a few commonly used resources registered. +func newTestRESTMapper() meta.RESTMapper { + m := meta.NewDefaultRESTMapper([]schema.GroupVersion{corev1.SchemeGroupVersion, {Group: "apps", Version: "v1"}}) + m.Add(deploymentGVK, meta.RESTScopeNamespace) + m.Add(configMapGVK, meta.RESTScopeNamespace) + m.Add(nodeGVK, meta.RESTScopeRoot) + return m +} + +// fakeRESTMapper is a REST mapper stub that allows the KindsFor results to be set directly. +type fakeRESTMapper struct { + meta.RESTMapper + + gvks []schema.GroupVersionKind + err error +} + +func (m *fakeRESTMapper) KindsFor(_ schema.GroupVersionResource) ([]schema.GroupVersionKind, error) { + return m.gvks, m.err +} + +// TestSetGVKCheckList tests the SetGVKCheckList method of the checker type. +func TestSetGVKCheckList(t *testing.T) { + testCases := []struct { + name string + mode GVKEligibilityMode + gvks []schema.GroupVersionKind + wantErr bool + wantChecker *checker + }{ + { + name: "valid GVKs, allow list", + mode: GVKEligibilityModeAllowList, + gvks: []schema.GroupVersionKind{deploymentGVK, {Group: "batch"}}, + wantChecker: &checker{ + nsEligibilityMode: NamespaceEligibilityModeDenyList, + gvkEligibilityMode: GVKEligibilityModeAllowList, + customGVKList: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + }), + }), + "batch": node(map[string]*gvkMatcherTrieNode{ + wildcard: node(map[string]*gvkMatcherTrieNode{ + wildcard: leaf(), + }), + }), + }, + }, + }, + }, + { + name: "valid GVKs, deny list", + mode: GVKEligibilityModeDenyList, + gvks: []schema.GroupVersionKind{nodeGVK}, + wantChecker: &checker{ + nsEligibilityMode: NamespaceEligibilityModeDenyList, + gvkEligibilityMode: GVKEligibilityModeDenyList, + customGVKList: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "Node": leaf(), + }), + }), + }, + }, + }, + }, + { + name: "invalid GVK leaves the checker untouched, allow list", + mode: GVKEligibilityModeAllowList, + gvks: []schema.GroupVersionKind{deploymentGVK, {Group: wildcard}}, + wantErr: true, + wantChecker: &checker{ + nsEligibilityMode: NamespaceEligibilityModeDenyList, + gvkEligibilityMode: GVKEligibilityModeDenyList, + }, + }, + { + name: "invalid GVK leaves the checker untouched, deny list", + mode: GVKEligibilityModeDenyList, + gvks: []schema.GroupVersionKind{{Group: wildcard, Kind: "Node"}}, + wantErr: true, + wantChecker: &checker{ + nsEligibilityMode: NamespaceEligibilityModeDenyList, + gvkEligibilityMode: GVKEligibilityModeDenyList, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + gotErr := c.SetGVKCheckList(tc.mode, tc.gvks) + if (gotErr != nil) != tc.wantErr { + t.Fatalf("SetGVKCheckList(%s) error = %v, wantErr %t", tc.mode, gotErr, tc.wantErr) + } + if diff := cmp.Diff(c, tc.wantChecker, checkerCmpOptions); diff != "" { + t.Errorf("checker after SetGVKCheckList(%s) mismatch (-got, +want):\n%s", tc.mode, diff) + } + }) + } +} + +// TestSetNamespaceCheckList tests the SetNamespaceCheckList method of the checker type. +func TestSetNamespaceCheckList(t *testing.T) { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work"}, []string{"app-"}) + wantAllowListChecker := &checker{ + nsEligibilityMode: NamespaceEligibilityModeAllowList, + customNSList: sets.New("work"), + customNSPrefixList: sets.New("app-"), + gvkEligibilityMode: GVKEligibilityModeDenyList, + } + if diff := cmp.Diff(c, wantAllowListChecker, checkerCmpOptions); diff != "" { + t.Errorf("checker after SetNamespaceCheckList(%s) mismatch (-got, +want):\n%s", NamespaceEligibilityModeAllowList, diff) + } + + // A second call overwrites the mode and the list from the first one. + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"default"}, []string{utils.KubeNSNamePrefix}) + wantDenyListChecker := &checker{ + nsEligibilityMode: NamespaceEligibilityModeDenyList, + customNSList: sets.New("default"), + customNSPrefixList: sets.New(utils.KubeNSNamePrefix), + gvkEligibilityMode: GVKEligibilityModeDenyList, + } + if diff := cmp.Diff(c, wantDenyListChecker, checkerCmpOptions); diff != "" { + t.Errorf("checker after SetNamespaceCheckList(%s) mismatch (-got, +want):\n%s", NamespaceEligibilityModeDenyList, diff) + } +} + +// TestSetNamespaceCheckListSkipsEmptyEntries tests that the namespace check list setter of the checker +// type drops empty names and prefixes; an empty prefix would otherwise match every namespace. +func TestSetNamespaceCheckListSkipsEmptyEntries(t *testing.T) { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"", "work"}, []string{"", "app-"}) + wantAllowListChecker := &checker{ + nsEligibilityMode: NamespaceEligibilityModeAllowList, + customNSList: sets.New("work"), + customNSPrefixList: sets.New("app-"), + gvkEligibilityMode: GVKEligibilityModeDenyList, + } + if diff := cmp.Diff(c, wantAllowListChecker, checkerCmpOptions); diff != "" { + t.Errorf("checker after SetNamespaceCheckList(%s) mismatch (-got, +want):\n%s", NamespaceEligibilityModeAllowList, diff) + } + + c = New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{""}, []string{""}) + wantDenyListChecker := &checker{ + nsEligibilityMode: NamespaceEligibilityModeDenyList, + customNSList: sets.New[string](), + customNSPrefixList: sets.New[string](), + gvkEligibilityMode: GVKEligibilityModeDenyList, + } + if diff := cmp.Diff(c, wantDenyListChecker, checkerCmpOptions); diff != "" { + t.Errorf("checker after SetNamespaceCheckList(%s) mismatch (-got, +want):\n%s", NamespaceEligibilityModeDenyList, diff) + } + if got := c.AllowsNamespace("work"); !got { + t.Errorf("AllowsNamespace(work) = %t, want %t", got, true) + } +} + +// TestCheckerWraps tests the Wraps method of the checker type. +func TestCheckerWraps(t *testing.T) { + inner := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := inner.SetGVKCheckList(GVKEligibilityModeDenyList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(DenyList) = %v, want no error", err) + } + + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if got := c.AllowsGVK(deploymentGVK); !got { + t.Fatalf("AllowsGVK(%v) = %t, want %t", deploymentGVK, got, true) + } + + c.Wraps(inner) + if got := c.AllowsGVK(deploymentGVK); got { + t.Errorf("AllowsGVK(%v) = %t, want %t", deploymentGVK, got, false) + } +} + +// TestCheckerDeepCopy tests the DeepCopy method of the checker type. +func TestCheckerDeepCopy(t *testing.T) { + restMapper := newTestRESTMapper() + c := New(CheckerOptions{RestMapper: restMapper, Wrapped: Baseline(restMapper), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"default"}, []string{utils.KubeNSNamePrefix}) + if err := c.SetGVKCheckList(GVKEligibilityModeDenyList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(DenyList) = %v, want no error", err) + } + + cp := c.DeepCopy() + if diff := cmp.Diff(cp, c, checkerCmpOptions); diff != "" { + t.Fatalf("DeepCopy() mismatch (-got, +want):\n%s", diff) + } + + // Mutate the copy; the original checker should stay intact. + cp.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work"}, nil) + if err := cp.SetGVKCheckList(GVKEligibilityModeAllowList, []schema.GroupVersionKind{configMapGVK}); err != nil { + t.Fatalf("SetGVKCheckList(AllowList) = %v, want no error", err) + } + if got := c.AllowsNamespace("work-1"); !got { + t.Errorf("AllowsNamespace(work-1) on the original checker = %t, want %t", got, true) + } + if got := c.AllowsGVK(configMapGVK); !got { + t.Errorf("AllowsGVK(%v) on the original checker = %t, want %t", configMapGVK, got, true) + } + if got := c.AllowsGVK(deploymentGVK); got { + t.Errorf("AllowsGVK(%v) on the original checker = %t, want %t", deploymentGVK, got, false) + } + + // Mutate the wrapped checker of the copy; the original checker should stay intact. + cpChecker, ok := cp.(*checker) + if !ok { + t.Fatalf("DeepCopy() returned a checker of type %T, want *checker", cp) + } + cpChecker.wrapped.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"work"}, nil) + if got := c.AllowsNamespace("work"); !got { + t.Errorf("AllowsNamespace(work) on the original checker = %t, want %t", got, true) + } + + c = New(CheckerOptions{RestMapper: restMapper, SkipNSCheck: true, SkipGVKRCheck: true, SkipObjectCheck: true}) + cp = c.DeepCopy() + if diff := cmp.Diff(cp, c, checkerCmpOptions); diff != "" { + t.Fatalf("DeepCopy() with skip flags mismatch (-got, +want):\n%s", diff) + } +} + +// TestAllowsNamespace tests the AllowsNamespace method of the checker type. +func TestAllowsNamespace(t *testing.T) { + testCases := []struct { + name string + checker Checker + namespace string + want bool + }{ + { + name: "no list configured", + checker: New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}), + namespace: "work", + want: true, + }, + { + name: "skip check without wrapped checker allows everything in deny list mode", + checker: New(CheckerOptions{RestMapper: nil, SkipNSCheck: true, SkipGVKRCheck: false, SkipObjectCheck: false}), + namespace: "work", + want: true, + }, + { + name: "skip check delegates to wrapped checker", + checker: func() Checker { + inner := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c := New(CheckerOptions{RestMapper: nil, Wrapped: inner, SkipNSCheck: true, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"work"}, nil) + return c + }(), + namespace: "work", + want: true, + }, + { + name: "skip check without wrapped checker denies everything in allow list mode", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: true, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work"}, nil) + return c + }(), + namespace: "work", + want: false, + }, + { + name: "skip check without wrapped checker allows the empty namespace", + checker: New(CheckerOptions{RestMapper: nil, SkipNSCheck: true, SkipGVKRCheck: false, SkipObjectCheck: false}), + namespace: "", + want: true, + }, + { + name: "skip check delegates to wrapped checker that denies the namespace", + checker: func() Checker { + inner := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + inner.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"work"}, nil) + return New(CheckerOptions{RestMapper: nil, Wrapped: inner, SkipNSCheck: true, SkipGVKRCheck: false, SkipObjectCheck: false}) + }(), + namespace: "work", + want: false, + }, + { + name: "deny list, name match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"default"}, []string{utils.KubeNSNamePrefix}) + return c + }(), + namespace: "default", + want: false, + }, + { + name: "deny list, prefix match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"default"}, []string{utils.KubeNSNamePrefix}) + return c + }(), + namespace: "kube-system", + want: false, + }, + { + name: "deny list, no match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"default"}, []string{utils.KubeNSNamePrefix}) + return c + }(), + namespace: "work", + want: true, + }, + { + name: "allow list, name match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work"}, []string{"app-"}) + return c + }(), + namespace: "work", + want: true, + }, + { + name: "allow list, prefix match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work"}, []string{"app-"}) + return c + }(), + namespace: "app-1", + want: true, + }, + { + name: "allow list, no match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work"}, []string{"app-"}) + return c + }(), + namespace: "other", + want: false, + }, + { + name: "wrapped checker further restricts the eligibility", + checker: func() Checker { + inner := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + inner.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"work"}, nil) + c := New(CheckerOptions{RestMapper: nil, Wrapped: inner, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work", "app"}, nil) + return c + }(), + namespace: "work", + want: false, + }, + { + name: "wrapped checker allows the namespace", + checker: func() Checker { + inner := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + inner.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"work"}, nil) + c := New(CheckerOptions{RestMapper: nil, Wrapped: inner, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work", "app"}, nil) + return c + }(), + namespace: "app", + want: true, + }, + { + name: "baseline checker denies the fleet reserved namespaces", + checker: Baseline(nil), + namespace: utils.FleetSystemNamespace, + want: false, + }, + { + // The default namespace itself is blocked by the object check; resources within it are eligible. + name: "baseline checker allows the default namespace", + checker: Baseline(nil), + namespace: "default", + want: true, + }, + { + name: "allow all checker", + checker: AllowAll(nil), + namespace: "kube-system", + want: true, + }, + { + // Cluster-scoped resources carry an empty namespace; they stay eligible even when the + // configured allow list does not name them. + name: "empty namespace is always eligible", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, Wrapped: Baseline(nil), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c.SetNamespaceCheckList(NamespaceEligibilityModeAllowList, []string{"work"}, nil) + return c + }(), + namespace: "", + want: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := tc.checker.AllowsNamespace(tc.namespace); got != tc.want { + t.Errorf("AllowsNamespace(%s) = %t, want %t", tc.namespace, got, tc.want) + } + }) + } +} + +// TestAllowsGVK tests the AllowsGVK method of the checker type. +func TestAllowsGVK(t *testing.T) { + testCases := []struct { + name string + checker Checker + gvk schema.GroupVersionKind + want bool + }{ + { + name: "no list configured", + checker: New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}), + gvk: deploymentGVK, + want: true, + }, + { + name: "skip check without wrapped checker allows everything in deny list mode", + checker: New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: true, SkipObjectCheck: false}), + gvk: deploymentGVK, + want: true, + }, + { + name: "skip check delegates to wrapped checker", + checker: func() Checker { + inner := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + c := New(CheckerOptions{RestMapper: nil, Wrapped: inner, SkipNSCheck: false, SkipGVKRCheck: true, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeDenyList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(DenyList) = %v, want no error", err) + } + return c + }(), + gvk: deploymentGVK, + want: true, + }, + { + name: "skip check without wrapped checker denies everything in allow list mode", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: true, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeAllowList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(AllowList) = %v, want no error", err) + } + return c + }(), + gvk: deploymentGVK, + want: false, + }, + { + name: "skip check delegates to wrapped checker that denies the GVK", + checker: func() Checker { + inner := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := inner.SetGVKCheckList(GVKEligibilityModeDenyList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(DenyList) = %v, want no error", err) + } + return New(CheckerOptions{RestMapper: nil, Wrapped: inner, SkipNSCheck: false, SkipGVKRCheck: true, SkipObjectCheck: false}) + }(), + gvk: deploymentGVK, + want: false, + }, + { + name: "deny list, match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeDenyList, []schema.GroupVersionKind{nodeGVK}); err != nil { + t.Fatalf("SetGVKCheckList(DenyList) = %v, want no error", err) + } + return c + }(), + gvk: nodeGVK, + want: false, + }, + { + name: "deny list, no match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeDenyList, []schema.GroupVersionKind{nodeGVK}); err != nil { + t.Fatalf("SetGVKCheckList(DenyList) = %v, want no error", err) + } + return c + }(), + gvk: configMapGVK, + want: true, + }, + { + name: "allow list, match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeAllowList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(AllowList) = %v, want no error", err) + } + return c + }(), + gvk: deploymentGVK, + want: true, + }, + { + name: "allow list, no match", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeAllowList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(AllowList) = %v, want no error", err) + } + return c + }(), + gvk: configMapGVK, + want: false, + }, + { + name: "wrapped checker further restricts the eligibility", + checker: func() Checker { + inner := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := inner.SetGVKCheckList(GVKEligibilityModeDenyList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(DenyList) = %v, want no error", err) + } + c := New(CheckerOptions{RestMapper: nil, Wrapped: inner, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeAllowList, []schema.GroupVersionKind{deploymentGVK, configMapGVK}); err != nil { + t.Fatalf("SetGVKCheckList(AllowList) = %v, want no error", err) + } + return c + }(), + gvk: deploymentGVK, + want: false, + }, + { + name: "custom allow list for the core API group does not override the wrapped baseline checker", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, Wrapped: Baseline(nil), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeAllowList, []schema.GroupVersionKind{{Group: corev1.GroupName}}); err != nil { + t.Fatalf("SetGVKCheckList(AllowList) = %v, want no error", err) + } + return c + }(), + gvk: nodeGVK, + want: false, + }, + { + name: "custom allow list for the core API group allows other core resources via the wrapped baseline checker", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, Wrapped: Baseline(nil), SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeAllowList, []schema.GroupVersionKind{{Group: corev1.GroupName}}); err != nil { + t.Fatalf("SetGVKCheckList(AllowList) = %v, want no error", err) + } + return c + }(), + gvk: configMapGVK, + want: true, + }, + { + name: "baseline checker denies node resources", + checker: Baseline(nil), + gvk: nodeGVK, + want: false, + }, + { + name: "baseline checker denies the fleet cluster API group", + checker: Baseline(nil), + gvk: clusterv1beta1.GroupVersion.WithKind("MemberCluster"), + want: false, + }, + { + name: "baseline checker denies the fleet networking resources", + checker: Baseline(nil), + gvk: schema.GroupVersionKind{Group: utils.NetworkingGroupName, Version: "v1alpha1", Kind: "ServiceImport"}, + want: false, + }, + { + name: "baseline checker allows other resources in the fleet networking API group", + checker: Baseline(nil), + gvk: schema.GroupVersionKind{Group: utils.NetworkingGroupName, Version: "v1alpha1", Kind: "InternalServiceExport"}, + want: true, + }, + { + name: "baseline checker denies the placement policy API group", + checker: Baseline(nil), + gvk: kfplacementv1alpha1.GroupVersion.WithKind(kfplacementv1alpha1.PlacementPolicyKind), + want: false, + }, + { + name: "baseline checker denies the staged update run API group", + checker: Baseline(nil), + gvk: kfrolloutv1alpha1.GroupVersion.WithKind(kfrolloutv1alpha1.StagedUpdateRunKind), + want: false, + }, + { + name: "baseline checker allows regular resources", + checker: Baseline(nil), + gvk: deploymentGVK, + want: true, + }, + { + name: "v0 default checker allows envelope resources", + checker: DefaultForV0APIs(nil), + gvk: placementv1beta1.GroupVersion.WithKind(placementv1beta1.ResourceEnvelopeKind), + want: true, + }, + { + name: "v1 default checker denies envelope resources", + checker: DefaultForV1APIs(nil), + gvk: placementv1beta1.GroupVersion.WithKind(placementv1beta1.ResourceEnvelopeKind), + want: false, + }, + { + name: "v1 default checker denies endpoints resources", + checker: DefaultForV1APIs(nil), + gvk: corev1.SchemeGroupVersion.WithKind("Endpoints"), + want: false, + }, + { + name: "v1 default checker inherits the baseline restrictions", + checker: DefaultForV1APIs(nil), + gvk: nodeGVK, + want: false, + }, + { + name: "allow all checker", + checker: AllowAll(nil), + gvk: nodeGVK, + want: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := tc.checker.AllowsGVK(tc.gvk); got != tc.want { + t.Errorf("AllowsGVK(%v) = %t, want %t", tc.gvk, got, tc.want) + } + }) + } +} + +// TestAllowsGVR tests the AllowsGVR method of the checker type. +func TestAllowsGVR(t *testing.T) { + testCases := []struct { + name string + checker Checker + gvr schema.GroupVersionResource + want bool + wantErr bool + wantNoMatch bool + }{ + { + name: "eligible resource", + checker: Baseline(newTestRESTMapper()), + gvr: deploymentGVR, + want: true, + }, + { + name: "ineligible resource", + checker: Baseline(newTestRESTMapper()), + gvr: nodeGVR, + want: false, + }, + { + name: "allow all checker", + checker: AllowAll(newTestRESTMapper()), + gvr: nodeGVR, + want: true, + }, + { + name: "skip check without wrapped checker allows everything in deny list mode", + checker: New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: true, SkipObjectCheck: false}), + gvr: deploymentGVR, + want: true, + }, + { + name: "skip check delegates to wrapped checker", + checker: New(CheckerOptions{ + RestMapper: &fakeRESTMapper{err: errors.New("outer mapper must not be called")}, + Wrapped: Baseline(newTestRESTMapper()), + SkipNSCheck: false, + SkipGVKRCheck: true, + SkipObjectCheck: false, + }), + gvr: deploymentGVR, + want: true, + }, + { + name: "skip check without wrapped checker denies everything in allow list mode", + checker: func() Checker { + c := New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: true, SkipObjectCheck: false}) + if err := c.SetGVKCheckList(GVKEligibilityModeAllowList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(AllowList) = %v, want no error", err) + } + return c + }(), + gvr: deploymentGVR, + want: false, + }, + { + name: "skip check propagates the error of the wrapped checker", + checker: New(CheckerOptions{ + RestMapper: newTestRESTMapper(), + Wrapped: Baseline(&fakeRESTMapper{err: errors.New("mapper failure")}), + SkipNSCheck: false, + SkipGVKRCheck: true, + SkipObjectCheck: false, + }), + gvr: deploymentGVR, + want: false, + wantErr: true, + }, + { + name: "wrapped checker further restricts the eligibility", + checker: New(CheckerOptions{ + RestMapper: newTestRESTMapper(), + Wrapped: Baseline(newTestRESTMapper()), + SkipNSCheck: false, + SkipGVKRCheck: false, + SkipObjectCheck: false, + }), + gvr: nodeGVR, + want: false, + }, + { + name: "wrapped checker error is propagated", + checker: New(CheckerOptions{ + RestMapper: newTestRESTMapper(), + Wrapped: Baseline(&fakeRESTMapper{err: errors.New("mapper failure")}), + SkipNSCheck: false, + SkipGVKRCheck: false, + SkipObjectCheck: false, + }), + gvr: deploymentGVR, + want: false, + wantErr: true, + }, + { + name: "unregistered resource", + checker: Baseline(newTestRESTMapper()), + gvr: unknownGVR, + want: false, + wantErr: true, + wantNoMatch: true, + }, + { + name: "no kind is mapped to the resource", + checker: Baseline(&fakeRESTMapper{}), + gvr: deploymentGVR, + want: false, + wantErr: true, + wantNoMatch: true, + }, + { + name: "REST mapper failure", + checker: Baseline(&fakeRESTMapper{err: errors.New("mapper failure")}), + gvr: deploymentGVR, + want: false, + wantErr: true, + }, + { + name: "multiple kinds are mapped to the resource, all eligible", + checker: Baseline(&fakeRESTMapper{ + gvks: []schema.GroupVersionKind{deploymentGVK, configMapGVK}, + }), + gvr: deploymentGVR, + want: true, + }, + { + name: "multiple kinds are mapped to the resource, one ineligible", + checker: Baseline(&fakeRESTMapper{ + gvks: []schema.GroupVersionKind{deploymentGVK, nodeGVK}, + }), + gvr: deploymentGVR, + want: false, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, gotErr := tc.checker.AllowsGVR(tc.gvr) + if (gotErr != nil) != tc.wantErr { + t.Fatalf("AllowsGVR(%v) error = %v, wantErr %t", tc.gvr, gotErr, tc.wantErr) + } + if gotNoMatch := meta.IsNoMatchError(gotErr); gotNoMatch != tc.wantNoMatch { + t.Errorf("IsNoMatchError(AllowsGVR(%v)) = %t, want %t", tc.gvr, gotNoMatch, tc.wantNoMatch) + } + if got != tc.want { + t.Errorf("AllowsGVR(%v) = %t, want %t", tc.gvr, got, tc.want) + } + }) + } +} + +// TestAllowsObject tests the AllowsObject method of the checker type. +func TestAllowsObject(t *testing.T) { + now := metav1.Now() + + configMapObj := func(name string) *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(configMapGVK) + obj.SetNamespace("work") + obj.SetName(name) + return obj + } + + testCases := []struct { + name string + checker Checker + obj *unstructured.Unstructured + want bool + reason string + wantErr bool + }{ + { + name: "regular resource", + checker: DefaultForV1APIs(nil), + obj: configMapObj("app-config"), + want: true, + }, + { + name: "resource being deleted", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := configMapObj("app-config") + obj.SetDeletionTimestamp(&now) + return obj + }(), + want: false, + reason: "the resource is being deleted and cannot be placed", + }, + { + name: "skip check without wrapped checker allows otherwise ineligible object", + checker: New(CheckerOptions{RestMapper: nil, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: true}), + obj: func() *unstructured.Unstructured { + obj := configMapObj("app-config") + obj.SetDeletionTimestamp(&now) + return obj + }(), + want: true, + }, + { + name: "skip check delegates to wrapped checker", + checker: New(CheckerOptions{RestMapper: nil, Wrapped: &alwaysIneligibleChecker{}, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: true}), + obj: configMapObj("app-config"), + want: false, + reason: "ineligible", + }, + { + name: "baseline checker rejects a resource being deleted", + checker: Baseline(nil), + obj: func() *unstructured.Unstructured { + obj := configMapObj("app-config") + obj.SetDeletionTimestamp(&now) + return obj + }(), + want: false, + reason: "the resource is being deleted and cannot be placed", + }, + { + name: "baseline checker that skips the object check allows a resource being deleted", + checker: baseline(nil, true), + obj: func() *unstructured.Unstructured { + obj := configMapObj("app-config") + obj.SetDeletionTimestamp(&now) + return obj + }(), + want: true, + }, + { + name: "v0 default checker allows a resource being deleted", + checker: DefaultForV0APIs(nil), + obj: func() *unstructured.Unstructured { + obj := configMapObj("app-config") + obj.SetDeletionTimestamp(&now) + return obj + }(), + want: true, + }, + { + name: "v0 default checker allows a resource owned by another object", + checker: DefaultForV0APIs(nil), + obj: func() *unstructured.Unstructured { + obj := configMapObj("app-config") + obj.SetOwnerReferences([]metav1.OwnerReference{ + { + APIVersion: "apps/v1", + Kind: "Deployment", + Name: "app", + UID: "1", + }, + }) + return obj + }(), + want: true, + }, + { + name: "skip check propagates the error of the wrapped checker", + checker: New(CheckerOptions{RestMapper: nil, Wrapped: &failingChecker{}, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: true}), + obj: configMapObj("app-config"), + want: false, + wantErr: true, + }, + { + name: "wrapped checker error is propagated", + checker: New(CheckerOptions{RestMapper: nil, Wrapped: &failingChecker{}, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}), + obj: configMapObj("app-config"), + want: false, + wantErr: true, + }, + { + name: "resource owned by a controller", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := configMapObj("app-config") + obj.SetOwnerReferences([]metav1.OwnerReference{ + { + APIVersion: "apps/v1", + Kind: "Deployment", + Name: "app", + UID: "1", + Controller: ptr.To(true), + }, + }) + return obj + }(), + want: false, + reason: "the resource is under the management of a controller; place the owner object instead if applicable", + }, + { + name: "resource owned by another object that is not its controller", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := configMapObj("app-config") + obj.SetOwnerReferences([]metav1.OwnerReference{ + { + APIVersion: "apps/v1", + Kind: "Deployment", + Name: "app", + UID: "1", + }, + { + APIVersion: "apps/v1", + Kind: "Deployment", + Name: "another-app", + UID: "2", + Controller: ptr.To(false), + }, + }) + return obj + }(), + want: true, + }, + { + name: "default namespace", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(utils.NamespaceGVK) + obj.SetName("default") + return obj + }(), + want: false, + reason: "the default namespace itself cannot be placed", + }, + { + name: "non-default namespace", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(utils.NamespaceGVK) + obj.SetName("work") + return obj + }(), + want: true, + }, + { + name: "built-in CA certificate config map", + checker: DefaultForV1APIs(nil), + obj: configMapObj("kube-root-ca.crt"), + want: false, + reason: "the built-in CA certificate config map is created automatically in every namespace and cannot be placed", + }, + { + name: "default service account", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind("ServiceAccount")) + obj.SetNamespace("work") + obj.SetName("default") + return obj + }(), + want: false, + reason: "the default service account is created automatically in every namespace and cannot be placed", + }, + { + name: "non-default service account", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind("ServiceAccount")) + obj.SetNamespace("work") + obj.SetName("app") + return obj + }(), + want: true, + }, + { + name: "Kubernetes API service", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind(utils.ServiceKind)) + obj.SetNamespace("default") + obj.SetName("kubernetes") + return obj + }(), + want: false, + reason: "the Kubernetes API service is created automatically in the default namespace and cannot be placed", + }, + { + name: "service of the same name in another namespace", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind(utils.ServiceKind)) + obj.SetNamespace("work") + obj.SetName("kubernetes") + return obj + }(), + want: true, + }, + { + name: "service account token secret", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "type": string(corev1.SecretTypeServiceAccountToken), + }, + } + obj.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind("Secret")) + obj.SetNamespace("work") + obj.SetName("app-token") + return obj + }(), + want: false, + reason: "service account token secrets are created automatically along with their service accounts and cannot be placed", + }, + { + name: "regular secret", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "type": string(corev1.SecretTypeOpaque), + }, + } + obj.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind("Secret")) + obj.SetNamespace("work") + obj.SetName("app-secret") + return obj + }(), + want: true, + }, + { + name: "secret with a malformed type field", + checker: DefaultForV1APIs(nil), + obj: func() *unstructured.Unstructured { + obj := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "type": int64(1), + }, + } + obj.SetGroupVersionKind(corev1.SchemeGroupVersion.WithKind("Secret")) + obj.SetNamespace("work") + obj.SetName("app-secret") + return obj + }(), + want: false, + wantErr: true, + }, + { + name: "wrapped checker further restricts the eligibility", + checker: New(CheckerOptions{RestMapper: nil, Wrapped: &alwaysIneligibleChecker{}, SkipNSCheck: false, SkipGVKRCheck: false, SkipObjectCheck: false}), + obj: configMapObj("app-config"), + want: false, + reason: "ineligible", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, gotReason, gotErr := tc.checker.AllowsObject(tc.obj) + if (gotErr != nil) != tc.wantErr { + t.Fatalf("AllowsObject() error = %v, wantErr %t", gotErr, tc.wantErr) + } + if got != tc.want { + t.Errorf("AllowsObject() = %t, want %t", got, tc.want) + } + if gotReason != tc.reason { + t.Errorf("AllowsObject() reason = %q, want %q", gotReason, tc.reason) + } + }) + } +} + +// alwaysIneligibleChecker is a Checker stub that rejects every resource object. +type alwaysIneligibleChecker struct { + Checker +} + +func (c *alwaysIneligibleChecker) AllowsObject(_ *unstructured.Unstructured) (bool, string, error) { + return false, "ineligible", nil +} + +// failingChecker is a Checker stub that fails every resource object check. +type failingChecker struct { + Checker +} + +func (c *failingChecker) AllowsObject(_ *unstructured.Unstructured) (bool, string, error) { + return false, "", errors.New("check failure") +} + +// TestSkipFlagsAreIndependent tests that each skip flag of the checker type disables only its own check. +func TestSkipFlagsAreIndependent(t *testing.T) { + now := metav1.Now() + deletingConfigMap := &unstructured.Unstructured{} + deletingConfigMap.SetGroupVersionKind(configMapGVK) + deletingConfigMap.SetNamespace("work") + deletingConfigMap.SetName("app-config") + deletingConfigMap.SetDeletionTimestamp(&now) + + testCases := []struct { + name string + skipNSCheck bool + skipGVKRCheck bool + skipObjectCheck bool + wantNS bool + wantGVK bool + wantGVR bool + wantObject bool + }{ + { + name: "no check is skipped", + }, + { + name: "namespace check is skipped", + skipNSCheck: true, + wantNS: true, + }, + { + name: "GVK and GVR checks are skipped", + skipGVKRCheck: true, + wantGVK: true, + wantGVR: true, + }, + { + name: "object check is skipped", + skipObjectCheck: true, + wantObject: true, + }, + { + name: "all checks are skipped", + skipNSCheck: true, + skipGVKRCheck: true, + skipObjectCheck: true, + wantNS: true, + wantGVK: true, + wantGVR: true, + wantObject: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + // Every check is set to reject its input unless it is skipped. + c := New(CheckerOptions{RestMapper: newTestRESTMapper(), SkipNSCheck: tc.skipNSCheck, SkipGVKRCheck: tc.skipGVKRCheck, SkipObjectCheck: tc.skipObjectCheck}) + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"work"}, nil) + if err := c.SetGVKCheckList(GVKEligibilityModeDenyList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList(DenyList) = %v, want no error", err) + } + + if got := c.AllowsNamespace("work"); got != tc.wantNS { + t.Errorf("AllowsNamespace(work) = %t, want %t", got, tc.wantNS) + } + if got := c.AllowsGVK(deploymentGVK); got != tc.wantGVK { + t.Errorf("AllowsGVK(%v) = %t, want %t", deploymentGVK, got, tc.wantGVK) + } + gotGVR, err := c.AllowsGVR(deploymentGVR) + if err != nil { + t.Fatalf("AllowsGVR(%v) = %v, want no error", deploymentGVR, err) + } + if gotGVR != tc.wantGVR { + t.Errorf("AllowsGVR(%v) = %t, want %t", deploymentGVR, gotGVR, tc.wantGVR) + } + gotObject, _, err := c.AllowsObject(deletingConfigMap) + if err != nil { + t.Fatalf("AllowsObject() = %v, want no error", err) + } + if gotObject != tc.wantObject { + t.Errorf("AllowsObject() = %t, want %t", gotObject, tc.wantObject) + } + }) + } +} + +// TestCheckerDeepCopyWithWrappedSkippedCheckers tests that the DeepCopy method of the checker type preserves +// the skip flags at every level of a chain of wrapped checkers. +func TestCheckerDeepCopyWithWrappedSkippedCheckers(t *testing.T) { + restMapper := newTestRESTMapper() + inner := New(CheckerOptions{RestMapper: restMapper, SkipNSCheck: false, SkipGVKRCheck: true, SkipObjectCheck: true}) + c := New(CheckerOptions{RestMapper: restMapper, Wrapped: inner, SkipNSCheck: true, SkipGVKRCheck: false, SkipObjectCheck: true}) + c.SetNamespaceCheckList(NamespaceEligibilityModeDenyList, []string{"work"}, nil) + + cp := c.DeepCopy() + if diff := cmp.Diff(cp, c, checkerCmpOptions); diff != "" { + t.Fatalf("DeepCopy() mismatch (-got, +want):\n%s", diff) + } + + // The copy should behave in the same way as the original checker. + if got, want := cp.AllowsNamespace("work"), c.AllowsNamespace("work"); got != want { + t.Errorf("AllowsNamespace(work) on the copy = %t, want %t", got, want) + } + + // Mutate the wrapped checker of the copy; the wrapped checker of the original should stay intact. + cpWrapped, ok := cp.(*checker).wrapped.(*checker) + if !ok { + t.Fatalf("DeepCopy() returned a wrapped checker of type %T, want *checker", cp.(*checker).wrapped) + } + cpWrapped.skipGVKRCheck = false + if !inner.(*checker).skipGVKRCheck { + t.Errorf("skipGVKRCheck on the original wrapped checker = %t, want %t", false, true) + } +} diff --git a/pkg/utils/resourceeligibility/parse.go b/pkg/utils/resourceeligibility/parse.go new file mode 100644 index 000000000..d7bd1ced0 --- /dev/null +++ b/pkg/utils/resourceeligibility/parse.go @@ -0,0 +1,128 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package resourceeligibility + +import ( + "strings" + + "k8s.io/apimachinery/pkg/runtime/schema" + + "go.goms.io/fleet/pkg/utils/errors" +) + +const ( + entrySeparator = ";" + gvkSegmentSeparator = "/" + kindSeparator = "," + + // coreAPIVersion is the API version (v1) of the core API group. + coreAPIVersion = "v1" +) + +// ParseGVKs parses a semicolon-separated list of GVK strings for matching purposes, e.g., +// "apps/v1/Deployment;v1/ConfigMap". +// +// An empty input yields no GVKs. Surrounding whitespace on each entry is ignored. +// +// See the comments on the ParseOneGVK function for the accepted forms of each GVK entry. +func ParseGVKs(input string) ([]schema.GroupVersionKind, error) { + input = strings.TrimSpace(input) + if input == "" { + return nil, nil + } + + entries := strings.Split(input, entrySeparator) + gvks := make([]schema.GroupVersionKind, 0, len(entries)) + for _, entry := range entries { + entryGVKs, err := ParseOneGVK(entry) + if err != nil { + return nil, err + } + gvks = append(gvks, entryGVKs...) + } + return gvks, nil +} + +// ParseOneGVK parses a single GVK string entry into one or more GVKs for matching purposes. The input +// should be in one of the following forms: +// +// * "[API-GROUP]", e.g., "apps"; +// * "[API-GROUP]/[API-VERSION]", e.g., "apps/v1"; +// * "[API-GROUP]/[API-VERSION]/[KIND]", e.g., "apps/v1/Deployment"; +// * "[API-GROUP]/[API-VERSION]/[KINDS]", e.g., "apps/v1/Deployment,StatefulSet"; +// +// For the core API group, which always has an empty group name, the input should be in one of the following forms: +// * "v1", e.g., "v1"; +// * "v1/[KIND]", e.g., "v1/ConfigMap"; +// * "v1/[KINDS]", e.g., "v1/ConfigMap,Secret"; +// +// The API version or kind segments can be left empty or set to the wildcard value "*" to match with any value. +// The API group segment, however, must name a specific group. In addition, when multiple kinds are specified, +// none of them can be empty, e.g., "apps/v1/Deployment," is invalid. +// +// Surrounding whitespace on each segment and kind is ignored. +func ParseOneGVK(input string) ([]schema.GroupVersionKind, error) { + segments := strings.Split(input, gvkSegmentSeparator) + for i := range segments { + segments[i] = strings.TrimSpace(segments[i]) + } + + // The core API group has an empty group name, so its entries start with the version. + if segments[0] == coreAPIVersion { + switch len(segments) { + case 1: + return []schema.GroupVersionKind{{Version: coreAPIVersion}}, nil + case 2: + kinds := strings.Split(segments[1], kindSeparator) + gvks := make([]schema.GroupVersionKind, 0, len(kinds)) + for _, kind := range kinds { + kind = strings.TrimSpace(kind) + if kind == "" && len(kinds) > 1 { + return nil, errors.NewUserError(nil, "kind cannot be empty", "input", input, "kinds", kinds) + } + gvks = append(gvks, schema.GroupVersionKind{Version: coreAPIVersion, Kind: kind}) + } + return gvks, nil + default: + return nil, errors.NewUserError(nil, "core API group entry must be in the form of v1 or v1/[,...]", "input", input) + } + } + + if segments[0] == "" || segments[0] == wildcard { + return nil, errors.NewUserError(nil, "the API group segment cannot be empty or a wildcard", "input", input) + } + + switch len(segments) { + case 1: + return []schema.GroupVersionKind{{Group: segments[0]}}, nil + case 2: + return []schema.GroupVersionKind{{Group: segments[0], Version: segments[1]}}, nil + case 3: + kinds := strings.Split(segments[2], kindSeparator) + gvks := make([]schema.GroupVersionKind, 0, len(kinds)) + for _, kind := range kinds { + kind = strings.TrimSpace(kind) + if kind == "" && len(kinds) > 1 { + return nil, errors.NewUserError(nil, "kind cannot be empty", "input", input, "kinds", kinds) + } + gvks = append(gvks, schema.GroupVersionKind{Group: segments[0], Version: segments[1], Kind: kind}) + } + return gvks, nil + default: + return nil, errors.NewUserError(nil, "entry must be in the form of , /, or //[,...]", "input", input) + } +} diff --git a/pkg/utils/resourceeligibility/parse_test.go b/pkg/utils/resourceeligibility/parse_test.go new file mode 100644 index 000000000..b61ab5d94 --- /dev/null +++ b/pkg/utils/resourceeligibility/parse_test.go @@ -0,0 +1,323 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package resourceeligibility + +import ( + "testing" + + "github.com/google/go-cmp/cmp" + "k8s.io/apimachinery/pkg/runtime/schema" +) + +// TestParseOneGVK tests the ParseOneGVK function. +func TestParseOneGVK(t *testing.T) { + testCases := []struct { + name string + input string + wantGVKs []schema.GroupVersionKind + wantErr bool + }{ + { + name: "group only", + input: "apps", + wantGVKs: []schema.GroupVersionKind{{Group: "apps"}}, + }, + { + name: "group and version", + input: "apps/v1", + wantGVKs: []schema.GroupVersionKind{{Group: "apps", Version: "v1"}}, + }, + { + name: "group, version, and kind", + input: "apps/v1/Deployment", + wantGVKs: []schema.GroupVersionKind{{Group: "apps", Version: "v1", Kind: "Deployment"}}, + }, + { + name: "group, version, and multiple kinds", + input: "apps/v1/Deployment,StatefulSet", + wantGVKs: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + }, + }, + { + name: "core API group, version only", + input: "v1", + wantGVKs: []schema.GroupVersionKind{{Version: "v1"}}, + }, + { + name: "core API group, version and kind", + input: "v1/ConfigMap", + wantGVKs: []schema.GroupVersionKind{{Version: "v1", Kind: "ConfigMap"}}, + }, + { + name: "core API group, version and multiple kinds", + input: "v1/ConfigMap,Secret", + wantGVKs: []schema.GroupVersionKind{ + {Version: "v1", Kind: "ConfigMap"}, + {Version: "v1", Kind: "Secret"}, + }, + }, + { + name: "core API group, empty kind", + input: "v1/", + wantGVKs: []schema.GroupVersionKind{{Version: "v1"}}, + }, + { + name: "core API group, too many segments", + input: "v1/ConfigMap/extra", + wantErr: true, + }, + { + name: "core API group, multiple kinds with a leading empty kind", + input: "v1/,ConfigMap", + wantErr: true, + }, + { + name: "core API group, multiple kinds with a trailing empty kind", + input: "v1/ConfigMap,", + wantErr: true, + }, + { + name: "core API group, multiple kinds with an empty kind in between", + input: "v1/ConfigMap,,Secret", + wantErr: true, + }, + { + // A whitespace-only kind must not be trimmed into an empty kind, which would act as a wildcard. + name: "core API group, multiple kinds with a whitespace-only kind in between", + input: "v1/ConfigMap, ,Secret", + wantErr: true, + }, + { + name: "core API group, multiple kinds that are all empty", + input: "v1/,", + wantErr: true, + }, + { + name: "wildcard version and kind", + input: "apps/*/*", + wantGVKs: []schema.GroupVersionKind{{Group: "apps", Version: wildcard, Kind: wildcard}}, + }, + { + name: "empty version segment", + input: "apps//Deployment", + wantGVKs: []schema.GroupVersionKind{{Group: "apps", Kind: "Deployment"}}, + }, + { + name: "empty kind segment", + input: "apps/v1/", + wantGVKs: []schema.GroupVersionKind{{Group: "apps", Version: "v1"}}, + }, + { + name: "whitespace-only kind segment", + input: "apps/v1/ ", + wantGVKs: []schema.GroupVersionKind{{Group: "apps", Version: "v1"}}, + }, + { + name: "multiple kinds with a leading empty kind", + input: "apps/v1/,Deployment", + wantErr: true, + }, + { + name: "multiple kinds with a trailing empty kind", + input: "apps/v1/Deployment,", + wantErr: true, + }, + { + name: "multiple kinds with an empty kind in between", + input: "apps/v1/Deployment,,StatefulSet", + wantErr: true, + }, + { + // A whitespace-only kind must not be trimmed into an empty kind, which would act as a wildcard. + name: "multiple kinds with a whitespace-only kind in between", + input: "apps/v1/Deployment, ,StatefulSet", + wantErr: true, + }, + { + name: "multiple kinds that are all empty", + input: "apps/v1/,", + wantErr: true, + }, + { + name: "empty version segment, multiple kinds with an empty kind", + input: "apps//Deployment,", + wantErr: true, + }, + { + name: "wildcard API group", + input: "*", + wantErr: true, + }, + { + name: "wildcard API group with a version", + input: "*/v1", + wantErr: true, + }, + { + name: "wildcard API group with a version and a kind", + input: "*/v1/Deployment", + wantErr: true, + }, + { + name: "group with dots", + input: "networking.k8s.io/v1/Ingress", + wantGVKs: []schema.GroupVersionKind{{Group: "networking.k8s.io", Version: "v1", Kind: "Ingress"}}, + }, + { + name: "empty input", + input: "", + wantErr: true, + }, + { + name: "empty group segment", + input: "/v1/Deployment", + wantErr: true, + }, + { + name: "too many segments", + input: "apps/v1/Deployment/extra", + wantErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotGVKs, gotErr := ParseOneGVK(tc.input) + if (gotErr != nil) != tc.wantErr { + t.Fatalf("ParseOneGVK(%s) error = %v, wantErr %t", tc.input, gotErr, tc.wantErr) + } + if diff := cmp.Diff(gotGVKs, tc.wantGVKs); diff != "" { + t.Errorf("ParseOneGVK(%s) mismatch (-got, +want):\n%s", tc.input, diff) + } + }) + } +} + +// TestParseGVKs tests the ParseGVKs function. +func TestParseGVKs(t *testing.T) { + testCases := []struct { + name string + input string + wantGVKs []schema.GroupVersionKind + wantErr bool + }{ + { + name: "single entry", + input: "apps/v1/Deployment", + wantGVKs: []schema.GroupVersionKind{{Group: "apps", Version: "v1", Kind: "Deployment"}}, + }, + { + name: "multiple entries of mixed forms", + input: "apps/v1/Deployment;v1/ConfigMap;batch/v1;networking.k8s.io", + wantGVKs: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Version: "v1", Kind: "ConfigMap"}, + {Group: "batch", Version: "v1"}, + {Group: "networking.k8s.io"}, + }, + }, + { + name: "multiple kinds in entries", + input: "apps/v1/Deployment,StatefulSet;v1/ConfigMap,Secret", + wantGVKs: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + {Version: "v1", Kind: "ConfigMap"}, + {Version: "v1", Kind: "Secret"}, + }, + }, + { + name: "duplicate entries", + input: "apps/v1/Deployment;apps/v1/Deployment", + wantGVKs: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "Deployment"}, + }, + }, + { + name: "empty input", + input: "", + }, + { + name: "whitespace-only input", + input: " ", + }, + { + // The flag help text documents entries separated by "; ". + name: "surrounding whitespace on entries", + input: " networking.k8s.io/v1beta1/Ingress,IngressClass; v1/ConfigMap ", + wantGVKs: []schema.GroupVersionKind{ + {Group: "networking.k8s.io", Version: "v1beta1", Kind: "Ingress"}, + {Group: "networking.k8s.io", Version: "v1beta1", Kind: "IngressClass"}, + {Version: "v1", Kind: "ConfigMap"}, + }, + }, + { + name: "whitespace-only entry in between", + input: "apps/v1; ;batch/v1", + wantErr: true, + }, + { + name: "surrounding whitespace on segments and kinds", + input: "apps / v1 / Deployment , StatefulSet ; v1 / ConfigMap", + wantGVKs: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + {Version: "v1", Kind: "ConfigMap"}, + }, + }, + { + name: "empty entry in between", + input: "apps/v1;;batch/v1", + wantErr: true, + }, + { + name: "trailing separator", + input: "apps/v1;", + wantErr: true, + }, + { + name: "one invalid entry", + input: "apps/v1;apps/v1/Deployment/extra", + wantErr: true, + }, + { + name: "one entry with an empty kind among multiple kinds", + input: "apps/v1/Deployment,;v1/ConfigMap", + wantErr: true, + }, + { + name: "one core API group entry with a whitespace-only kind among multiple kinds", + input: "apps/v1/Deployment; v1/ConfigMap, ,Secret", + wantErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotGVKs, gotErr := ParseGVKs(tc.input) + if (gotErr != nil) != tc.wantErr { + t.Fatalf("ParseGVKs(%s) error = %v, wantErr %t", tc.input, gotErr, tc.wantErr) + } + if diff := cmp.Diff(gotGVKs, tc.wantGVKs); diff != "" { + t.Errorf("ParseGVKs(%s) mismatch (-got, +want):\n%s", tc.input, diff) + } + }) + } +} diff --git a/pkg/utils/resourceeligibility/trie.go b/pkg/utils/resourceeligibility/trie.go new file mode 100644 index 000000000..6c93d3f37 --- /dev/null +++ b/pkg/utils/resourceeligibility/trie.go @@ -0,0 +1,104 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package resourceeligibility + +import ( + "k8s.io/apimachinery/pkg/runtime/schema" + + "go.goms.io/fleet/pkg/utils/errors" +) + +const ( + wildcard = "*" +) + +type gvkMatcherTrie gvkMatcherTrieNode + +type gvkMatcherTrieNode struct { + children map[string]*gvkMatcherTrieNode +} + +// Register adds the given GVK to the matcher trie. +func (t *gvkMatcherTrie) Register(gvk schema.GroupVersionKind) error { + group := gvk.Group + if group == wildcard { + return errors.NewUserError(nil, "cannot use wildcards for API groups", "gvk", gvk) + } + + version := gvk.Version + if version == "" { + version = wildcard + } + kind := gvk.Kind + if kind == "" { + kind = wildcard + } + + node := (*gvkMatcherTrieNode)(t) + for _, edge := range []string{group, version, kind} { + if node.children == nil { + node.children = make(map[string]*gvkMatcherTrieNode) + } + child, found := node.children[edge] + if !found { + child = &gvkMatcherTrieNode{} + node.children[edge] = child + } + node = child + } + return nil +} + +// DeepCopy returns a deep copy of the matcher trie. +func (t *gvkMatcherTrie) DeepCopy() gvkMatcherTrie { + return gvkMatcherTrie(*(*gvkMatcherTrieNode)(t).deepCopy()) +} + +func (n *gvkMatcherTrieNode) deepCopy() *gvkMatcherTrieNode { + cp := &gvkMatcherTrieNode{} + if n.children == nil { + return cp + } + + cp.children = make(map[string]*gvkMatcherTrieNode, len(n.children)) + for edge, child := range n.children { + cp.children[edge] = child.deepCopy() + } + return cp +} + +// Match reports whether the given GVK is covered by an entry registered in the trie. +func (t *gvkMatcherTrie) Match(gvk schema.GroupVersionKind) bool { + edges := []string{gvk.Group, gvk.Version, gvk.Kind} + + // Do a DFS walk. + var walk func(node *gvkMatcherTrieNode, depth int) bool + walk = func(node *gvkMatcherTrieNode, depth int) bool { + if depth == len(edges) { + // All the edges have been consumed; a registered path has been found. + return true + } + + for _, edge := range []string{wildcard, edges[depth]} { + if child, found := node.children[edge]; found && walk(child, depth+1) { + return true + } + } + return false + } + return walk((*gvkMatcherTrieNode)(t), 0) +} diff --git a/pkg/utils/resourceeligibility/trie_test.go b/pkg/utils/resourceeligibility/trie_test.go new file mode 100644 index 000000000..d8c6cc6ea --- /dev/null +++ b/pkg/utils/resourceeligibility/trie_test.go @@ -0,0 +1,464 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package resourceeligibility + +import ( + "testing" + + "github.com/google/go-cmp/cmp" + "k8s.io/apimachinery/pkg/runtime/schema" +) + +var ( + trieCmpOptions = cmp.Options{ + cmp.AllowUnexported(gvkMatcherTrie{}, gvkMatcherTrieNode{}), + } +) + +// node is a helper that builds a trie node with the given children. +func node(children map[string]*gvkMatcherTrieNode) *gvkMatcherTrieNode { + return &gvkMatcherTrieNode{children: children} +} + +// leaf is a helper that builds a childless trie node. +func leaf() *gvkMatcherTrieNode { + return &gvkMatcherTrieNode{} +} + +// TestGVKMatcherTrieRegister tests the Register method of the gvkMatcherTrie type. +func TestGVKMatcherTrieRegister(t *testing.T) { + testCases := []struct { + name string + gvks []schema.GroupVersionKind + wantErr bool + wantTrie gvkMatcherTrie + }{ + { + name: "full GVK", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + }), + }), + }, + }, + }, + { + name: "group only (empty version and kind are treated as wildcards)", + gvks: []schema.GroupVersionKind{ + {Group: "apps"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + wildcard: node(map[string]*gvkMatcherTrieNode{ + wildcard: leaf(), + }), + }), + }, + }, + }, + { + name: "group and version only", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + wildcard: leaf(), + }), + }), + }, + }, + }, + { + name: "core API group (empty group name)", + gvks: []schema.GroupVersionKind{ + {Version: "v1", Kind: "ConfigMap"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "ConfigMap": leaf(), + }), + }), + }, + }, + }, + { + name: "explicit wildcards for version and kind", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: wildcard, Kind: wildcard}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + wildcard: node(map[string]*gvkMatcherTrieNode{ + wildcard: leaf(), + }), + }), + }, + }, + }, + { + name: "multiple GVKs sharing a group", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + {Group: "apps", Version: "v1beta1", Kind: "Deployment"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + "StatefulSet": leaf(), + }), + "v1beta1": node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + }), + }), + }, + }, + }, + { + name: "multiple GVKs across groups", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "batch", Version: "v1", Kind: "Job"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + }), + }), + "batch": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "Job": leaf(), + }), + }), + }, + }, + }, + { + name: "duplicate registration", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "Deployment"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + }), + }), + }, + }, + }, + { + name: "a wildcard registered before a more specific GVK", + gvks: []schema.GroupVersionKind{ + {Group: "apps"}, + {Group: "apps", Version: "v1", Kind: "Deployment"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + wildcard: node(map[string]*gvkMatcherTrieNode{ + wildcard: leaf(), + }), + "v1": node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + }), + }), + }, + }, + }, + { + // A wildcard version with a specific kind does not cover a specific version with a + // different kind, so the two must not share a branch. + name: "a partial wildcard does not absorb a more specific GVK", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + wildcard: node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + }), + "v1": node(map[string]*gvkMatcherTrieNode{ + "StatefulSet": leaf(), + }), + }), + }, + }, + }, + { + name: "a wildcard registered after a more specific GVK", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps"}, + }, + wantTrie: gvkMatcherTrie{ + children: map[string]*gvkMatcherTrieNode{ + "apps": node(map[string]*gvkMatcherTrieNode{ + "v1": node(map[string]*gvkMatcherTrieNode{ + "Deployment": leaf(), + }), + wildcard: node(map[string]*gvkMatcherTrieNode{ + wildcard: leaf(), + }), + }), + }, + }, + }, + { + name: "wildcard API group", + gvks: []schema.GroupVersionKind{ + {Group: wildcard, Version: "v1", Kind: "Deployment"}, + }, + wantErr: true, + wantTrie: gvkMatcherTrie{}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + trie := gvkMatcherTrie{} + var gotErr error + for _, gvk := range tc.gvks { + if err := trie.Register(gvk); err != nil { + gotErr = err + break + } + } + + if (gotErr != nil) != tc.wantErr { + t.Fatalf("Register() error = %v, wantErr %v", gotErr, tc.wantErr) + } + if diff := cmp.Diff(trie, tc.wantTrie, trieCmpOptions); diff != "" { + t.Errorf("trie after Register() mismatch (-got, +want):\n%s", diff) + } + }) + } +} + +// TestGVKMatcherTrieDeepCopy tests the DeepCopy method of the gvkMatcherTrie type. +func TestGVKMatcherTrieDeepCopy(t *testing.T) { + testCases := []struct { + name string + gvks []schema.GroupVersionKind + }{ + { + name: "empty trie", + }, + { + name: "trie with multiple entries", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + {Group: "batch"}, + {Version: "v1", Kind: "ConfigMap"}, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + trie := gvkMatcherTrie{} + for _, gvk := range tc.gvks { + if err := trie.Register(gvk); err != nil { + t.Fatalf("Register(%v) = %v, want no error", gvk, err) + } + } + + cp := trie.DeepCopy() + if diff := cmp.Diff(cp, trie, trieCmpOptions); diff != "" { + t.Fatalf("DeepCopy() mismatch (-got, +want):\n%s", diff) + } + + // Register a new GVK in the copy; the original trie should stay intact. + newGVK := schema.GroupVersionKind{Group: "networking.k8s.io", Version: "v1", Kind: "Ingress"} + if err := cp.Register(newGVK); err != nil { + t.Fatalf("Register(%v) = %v, want no error", newGVK, err) + } + if trie.Match(newGVK) { + t.Errorf("Match(%v) on the original trie = true, want false", newGVK) + } + + // Mutate the original trie; the copy should stay intact. + anotherGVK := schema.GroupVersionKind{Group: "rbac.authorization.k8s.io", Version: "v1", Kind: "Role"} + if err := trie.Register(anotherGVK); err != nil { + t.Fatalf("Register(%v) = %v, want no error", anotherGVK, err) + } + if cp.Match(anotherGVK) { + t.Errorf("Match(%v) on the copied trie = true, want false", anotherGVK) + } + }) + } +} + +// TestGVKMatcherTrieMatch tests the Match method of the gvkMatcherTrie type. +func TestGVKMatcherTrieMatch(t *testing.T) { + registered := []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "batch", Version: "v1"}, + {Group: "networking.k8s.io"}, + {Version: "v1", Kind: "ConfigMap"}, + } + + testCases := []struct { + name string + gvks []schema.GroupVersionKind + gvk schema.GroupVersionKind + wantMatch bool + }{ + { + name: "exact match", + gvks: registered, + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "Deployment"}, + wantMatch: true, + }, + { + name: "kind not registered", + gvks: registered, + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "StatefulSet"}, + wantMatch: false, + }, + { + name: "version not registered", + gvks: registered, + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1beta1", Kind: "Deployment"}, + wantMatch: false, + }, + { + name: "group not registered", + gvks: registered, + gvk: schema.GroupVersionKind{Group: "policy", Version: "v1", Kind: "PodDisruptionBudget"}, + wantMatch: false, + }, + { + name: "kind wildcard match", + gvks: registered, + gvk: schema.GroupVersionKind{Group: "batch", Version: "v1", Kind: "Job"}, + wantMatch: true, + }, + { + name: "kind wildcard match, version mismatch", + gvks: registered, + gvk: schema.GroupVersionKind{Group: "batch", Version: "v1beta1", Kind: "Job"}, + wantMatch: false, + }, + { + name: "version and kind wildcard match", + gvks: registered, + gvk: schema.GroupVersionKind{Group: "networking.k8s.io", Version: "v1beta1", Kind: "Ingress"}, + wantMatch: true, + }, + { + name: "core API group match", + gvks: registered, + gvk: schema.GroupVersionKind{Version: "v1", Kind: "ConfigMap"}, + wantMatch: true, + }, + { + name: "core API group, kind not registered", + gvks: registered, + gvk: schema.GroupVersionKind{Version: "v1", Kind: "Secret"}, + wantMatch: false, + }, + { + name: "empty trie", + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "Deployment"}, + wantMatch: false, + }, + { + name: "a more specific entry is registered alongside a wildcard one", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + {Group: "apps"}, + }, + gvk: schema.GroupVersionKind{Group: "apps", Version: "v2", Kind: "Deployment"}, + wantMatch: true, + }, + { + name: "a wildcard version with a specific kind, matching version and kind", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + }, + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "StatefulSet"}, + wantMatch: true, + }, + { + // The StatefulSet entry is pinned to v1, so it must not be widened by the + // wildcard version registered for Deployment. + name: "a wildcard version with a specific kind does not widen another kind", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + }, + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1beta1", Kind: "StatefulSet"}, + wantMatch: false, + }, + { + name: "a wildcard version still matches its own kind in any version", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Kind: "Deployment"}, + {Group: "apps", Version: "v1", Kind: "StatefulSet"}, + }, + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1beta1", Kind: "Deployment"}, + wantMatch: true, + }, + { + name: "partial path registered only", + gvks: []schema.GroupVersionKind{ + {Group: "apps", Version: "v1", Kind: "Deployment"}, + }, + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1"}, + wantMatch: false, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + trie := gvkMatcherTrie{} + for _, gvk := range tc.gvks { + if err := trie.Register(gvk); err != nil { + t.Fatalf("Register(%v) = %v, want no error", gvk, err) + } + } + + if gotMatch := trie.Match(tc.gvk); gotMatch != tc.wantMatch { + t.Errorf("Match(%v) = %t, want %t", tc.gvk, gotMatch, tc.wantMatch) + } + }) + } +} diff --git a/pkg/utils/validator/placement.go b/pkg/utils/validator/placement.go index f2efa9f3b..66b66fc5a 100644 --- a/pkg/utils/validator/placement.go +++ b/pkg/utils/validator/placement.go @@ -26,7 +26,9 @@ import ( "strings" admissionv1 "k8s.io/api/admission/v1" + authenticationv1 "k8s.io/api/authentication/v1" corev1 "k8s.io/api/core/v1" + apiequality "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -42,6 +44,7 @@ import ( placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/propertyprovider" "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/authorization" "go.goms.io/fleet/pkg/utils/controller" "go.goms.io/fleet/pkg/utils/informer" ) @@ -758,8 +761,15 @@ func supportedResourceCapacityTypes() []string { return capacityTypes } +// PlacementAuthorizer reviews access to the current resources selected by a placement. +type PlacementAuthorizer interface { + Authorize(context.Context, authenticationv1.UserInfo, placementv1beta1.PlacementObj) authorization.Result +} + // HandlePlacementValidation provides consolidated webhook validation logic for placement objects. // This function accepts higher-order functions for type-specific operations. +// A nil authorizer leaves selected-resource authorization disabled. +// Updates with unchanged specs skip selected-resource authorization, but still undergo validation. func HandlePlacementValidation( ctx context.Context, req admission.Request, @@ -768,6 +778,7 @@ func HandlePlacementValidation( decodeFunc func(admission.Request, webhook.AdmissionDecoder) (placementv1beta1.PlacementObj, error), decodeOldFunc func(admission.Request, webhook.AdmissionDecoder) (placementv1beta1.PlacementObj, error), validateFunc func(placementv1beta1.PlacementObj) error, + authorizer PlacementAuthorizer, ) admission.Response { if req.Operation == admissionv1.Create || req.Operation == admissionv1.Update { klog.V(2).InfoS("handling placement", "resourceType", resourceType, "operation", req.Operation, "namespacedName", types.NamespacedName{Name: req.Name, Namespace: req.Namespace}) @@ -778,12 +789,12 @@ func HandlePlacementValidation( return admission.Errored(http.StatusBadRequest, err) } + specUnchanged := false if req.Operation == admissionv1.Update { oldPlacement, err := decodeOldFunc(req, decoder) if err != nil { return admission.Errored(http.StatusBadRequest, err) } - // Special case: allow updates to old placement objects with invalid fields so that we can // update the placement to remove finalizer then delete it. if err := validateFunc(oldPlacement); err != nil { @@ -793,6 +804,8 @@ func HandlePlacementValidation( return admission.Denied(fmt.Sprintf(DenyUpdateOldInvalidFmt, resourceType, err)) } + specUnchanged = apiequality.Semantic.DeepEqual(oldPlacement.GetPlacementSpec(), placement.GetPlacementSpec()) + // Handle update case where placement type should be immutable. if IsPlacementPolicyTypeUpdated(oldPlacement.GetPlacementSpec().Policy, placement.GetPlacementSpec().Policy) { return admission.Denied("placement type is immutable") @@ -808,6 +821,35 @@ func HandlePlacementValidation( klog.V(2).InfoS("v1beta1 placement has invalid fields, request is denied", "resourceType", resourceType, "operation", req.Operation, "namespacedName", types.NamespacedName{Name: placement.GetName(), Namespace: req.Namespace}) return admission.Denied(fmt.Sprintf(DenyCreateUpdateInvalidFmt, resourceType, err)) } + if authorizer != nil && !specUnchanged { + result := authorizer.Authorize(ctx, req.UserInfo, placement) + if result.Err != nil { + klog.ErrorS(result.Err, "failed to authorize selected placement resources", "resourceType", resourceType, "userName", req.UserInfo.Username) + } + var invalidResultErr error + switch result.Decision { + case authorization.DecisionAllowed: + if result.Err != nil || result.DenialReason != "" { + invalidResultErr = fmt.Errorf("selected-resource authorization returned an allowed decision with denial or error details") + } + case authorization.DecisionDenied: + if result.DenialReason != "" { + return admission.Denied(result.DenialReason) + } + invalidResultErr = fmt.Errorf("selected-resource authorization returned a denial without a reason") + case authorization.DecisionIndeterminate: + if result.Err != nil { + return admission.Errored(http.StatusInternalServerError, result.Err) + } + invalidResultErr = fmt.Errorf("selected-resource authorization returned an indeterminate decision without an error") + default: + invalidResultErr = fmt.Errorf("selected-resource authorization returned an unknown decision %q", result.Decision) + } + if invalidResultErr != nil { + klog.ErrorS(invalidResultErr, "failed to authorize selected placement resources", "resourceType", resourceType) + return admission.Errored(http.StatusInternalServerError, invalidResultErr) + } + } } return admission.Allowed(fmt.Sprintf(AllowModifyFmt, resourceType)) diff --git a/pkg/utils/validator/placement_authorization_test.go b/pkg/utils/validator/placement_authorization_test.go new file mode 100644 index 000000000..7f3c63874 --- /dev/null +++ b/pkg/utils/validator/placement_authorization_test.go @@ -0,0 +1,127 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package validator + +import ( + "context" + "errors" + "fmt" + "net/http" + "testing" + + "github.com/google/go-cmp/cmp" + admissionv1 "k8s.io/api/admission/v1" + authenticationv1 "k8s.io/api/authentication/v1" + "sigs.k8s.io/controller-runtime/pkg/webhook" + "sigs.k8s.io/controller-runtime/pkg/webhook/admission" + + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils/authorization" +) + +type stubPlacementAuthorizer struct { + result authorization.Result +} + +func (a *stubPlacementAuthorizer) Authorize(context.Context, authenticationv1.UserInfo, placementv1beta1.PlacementObj) authorization.Result { + return a.result +} + +func TestHandlePlacementValidationAuthorizationDecision(t *testing.T) { + const denialReason = "requester cannot get selected resources" + reviewErr := errors.New("authorization API unavailable") + type outcome struct { + Allowed bool + Code int32 + Message string + } + tests := []struct { + name string + result authorization.Result + want outcome + }{ + { + name: "allowed", + result: authorization.Result{Decision: authorization.DecisionAllowed}, + want: outcome{Allowed: true, Code: http.StatusOK, Message: fmt.Sprintf(AllowModifyFmt, "RP")}, + }, + { + name: "denied", + result: authorization.Result{Decision: authorization.DecisionDenied, DenialReason: denialReason}, + want: outcome{Code: http.StatusForbidden, Message: denialReason}, + }, + { + name: "denial takes precedence over additional review errors", + result: authorization.Result{Decision: authorization.DecisionDenied, DenialReason: denialReason, Err: reviewErr}, + want: outcome{Code: http.StatusForbidden, Message: denialReason}, + }, + { + name: "indeterminate", + result: authorization.Result{Decision: authorization.DecisionIndeterminate, Err: reviewErr}, + want: outcome{Code: http.StatusInternalServerError, Message: reviewErr.Error()}, + }, + { + name: "zero decision", + want: outcome{Code: http.StatusInternalServerError, Message: `selected-resource authorization returned an unknown decision ""`}, + }, + { + name: "unknown decision", + result: authorization.Result{Decision: "Unknown"}, + want: outcome{Code: http.StatusInternalServerError, Message: `selected-resource authorization returned an unknown decision "Unknown"`}, + }, + { + name: "indeterminate without error", + result: authorization.Result{Decision: authorization.DecisionIndeterminate}, + want: outcome{Code: http.StatusInternalServerError, Message: "selected-resource authorization returned an indeterminate decision without an error"}, + }, + { + name: "denied without reason", + result: authorization.Result{Decision: authorization.DecisionDenied}, + want: outcome{Code: http.StatusInternalServerError, Message: "selected-resource authorization returned a denial without a reason"}, + }, + { + name: "allowed with error", + result: authorization.Result{Decision: authorization.DecisionAllowed, Err: reviewErr}, + want: outcome{Code: http.StatusInternalServerError, Message: "selected-resource authorization returned an allowed decision with denial or error details"}, + }, + { + name: "allowed with denial reason", + result: authorization.Result{Decision: authorization.DecisionAllowed, DenialReason: denialReason}, + want: outcome{Code: http.StatusInternalServerError, Message: "selected-resource authorization returned an allowed decision with denial or error details"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + response := HandlePlacementValidation( + context.Background(), + admission.Request{AdmissionRequest: admissionv1.AdmissionRequest{Operation: admissionv1.Create}}, + nil, + "RP", + func(admission.Request, webhook.AdmissionDecoder) (placementv1beta1.PlacementObj, error) { + return &placementv1beta1.ResourcePlacement{}, nil + }, + nil, + func(placementv1beta1.PlacementObj) error { return nil }, + &stubPlacementAuthorizer{result: tt.result}, + ) + got := outcome{Allowed: response.Allowed, Code: response.Result.Code, Message: response.Result.Message} + if diff := cmp.Diff(tt.want, got); diff != "" { + t.Errorf("HandlePlacementValidation() mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/pkg/v1/controllers/annotationplacement/controller.go b/pkg/v1/controllers/annotationplacement/controller.go new file mode 100644 index 000000000..a8818a3c6 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/controller.go @@ -0,0 +1,428 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "errors" + "fmt" + "time" + + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/tools/events" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/utils/resourceeligibility" +) + +// The reasons of the events this controller records on the annotated resource. Each reason is the +// same whether the generated object is a PlacementPolicy or a ClusterPlacementPolicy -- the reason +// names the action, and the event message names the concrete kind. +// +// The events are recorded on the resource the user annotated rather than on the policy generated +// from it, because the resource is where a user who has just run kubectl annotate is looking, and +// because a rejected annotation generates no policy for an event to be attached to. +const ( + // EventReasonPolicyCreated is recorded when a generated policy is created for a resource. + EventReasonPolicyCreated = "PlacementPolicyCreated" + // EventReasonPolicyUpdated is recorded when an annotation change reaches its generated policy. + EventReasonPolicyUpdated = "PlacementPolicyUpdated" + // EventReasonPolicyDeleted is recorded when the policy generated for a resource is deleted -- + // because the annotation was removed, or because the resource stopped being eligible for + // placement. The event's message names which. + EventReasonPolicyDeleted = "PlacementPolicyDeleted" + // EventReasonInvalidAnnotation is recorded when an annotation cannot be parsed. It is a warning + // rather than an error on the queue: no amount of retrying makes a malformed annotation parse, + // so the event is the only way the user learns that the placement they asked for is not running. + EventReasonInvalidAnnotation = "InvalidClusterSelectorsAnnotation" + // EventReasonPolicyConflict is recorded when a policy already exists at a resource's generated + // name but is not one this controller generated. It is a warning: the controller neither + // overwrites nor deletes the pre-existing policy, so the placement the annotation asks for is not + // running, and the event is how the user learns why. + EventReasonPolicyConflict = "PlacementPolicyConflict" + // EventReasonPlacementBlocked is recorded when a kind cannot be placed through annotations. + EventReasonPlacementBlocked = "AnnotationPlacementBlocked" +) + +// errForeignPolicy reports that a policy occupies a resource's generated name but is not one this +// controller generated. +// +// It is returned as an error rather than swallowed with a fixed requeue so that the source is +// retried with the queue's exponential backoff: nothing else brings it back, since the blocking +// policy carries no owner reference to the source and the annotation does not change, yet a +// conflict left in place must not be re-examined at a fixed, busy pace forever. +var errForeignPolicy = errors.New("a policy at the generated name was not generated from the annotation") + +// errRetiredVersion reports that the version a request names is no longer served while its kind +// still is. The request cannot be retried as it stands -- the resource is re-reported under the +// served version as a different request, which is what reconciles it -- so it is dropped rather +// than requeued, and the generated policy is left alone. +var errRetiredVersion = errors.New("the version the resource was reported under is no longer served") + +// Request identifies the resource a reconciliation is for. +// +// A plain namespaced name would not do: this controller reconciles resources of any kind the hub +// agent watches, and the name alone does not say which API to read the resource from. +type Request struct { + schema.GroupVersionKind + types.NamespacedName +} + +// RequestFor returns the request that reconciles the given resource, which must carry its kind. +func RequestFor(object client.Object) Request { + return Request{ + GroupVersionKind: object.GetObjectKind().GroupVersionKind(), + NamespacedName: client.ObjectKeyFromObject(object), + } +} + +// String formats the request for logs. +func (r Request) String() string { + return fmt.Sprintf("%s %s", r.GroupVersionKind.String(), r.NamespacedName.String()) +} + +// Reconciler keeps the placement policy generated from a resource's cluster-selectors annotation in +// sync with that annotation. +// +// Its queue holds requests for resources of any kind the hub agent watches, not for the generated +// policies themselves; an event on a generated policy is mapped back to the resource it came from. +type Reconciler struct { + // Client reads and writes the generated placement policies. + // + // It may be backed by the manager's cache: the generated policies are watched through the same + // cache, so a policy event is never delivered ahead of the state this reads, and a write that + // races a change the cache has yet to see fails with a conflict and is retried. The one thing + // the cache can lag is this controller's own writes, which is why an already-exists error on a + // create is treated as transient rather than as a failure. + client.Client + + // APIReader reads the annotated resources straight from the API server. + // + // The resources are of any kind the hub agent watches, and a cache-backed read of a kind the + // cache has not seen would start an informer for it -- for a kind that cannot be listed or + // watched, one that never syncs. Only annotated resources and the owners of generated policies + // reach this controller's queue, so the direct read is not on the hot path of every watched + // resource in the cluster. + APIReader client.Reader + + // RESTMapper resolves the kind a request names into the resource it is read through, and the + // scope that decides whether the read is namespaced. + RESTMapper meta.RESTMapper + + // Recorder records the outcome of a reconciliation on the annotated resource. + Recorder events.EventRecorder + + // ResourceEligibilityChecker reports whether a resource can be placed via KubeFleet's placement + // capabilities. + // + // Note that when using the annotation-based placement experience, additional constraints might apply. + ResourceEligibilityChecker resourceeligibility.Checker +} + +// Reconcile brings the generated policy for one resource in line with that resource's annotation. +func (r *Reconciler) Reconcile(ctx context.Context, req Request) (ctrl.Result, error) { + startTime := time.Now() + klog.V(2).InfoS("Reconciling annotation-based placement", "obj", req) + defer func() { + klog.V(2).InfoS("Annotation-based placement reconciliation loop ends", "obj", req, "latency", time.Since(startTime).Milliseconds()) + }() + + source, err := r.sourceObject(ctx, req) + switch { + case errors.Is(err, errRetiredVersion): + // Dropped rather than retried: this request names a version that is gone, and no number of + // retries brings it back. API discovery asks for server-preferred versions, so the resource + // is re-reported under the served one as a fresh request, and that is what reconciles it. + // The generated policy stands untouched in the meantime, since its kind still places. + klog.V(2).InfoS("Dropped a request naming a version that is no longer served", "obj", req) + return ctrl.Result{}, nil + case apierrors.IsNotFound(err), meta.IsNoMatchError(err): + // The resource is gone, or its whole kind is (the CRD was removed; the generated policy + // watch enqueues a stale policy's owner). The generated policy is KubeFleet's own object, + // so it is always deleted here rather than left to garbage collection, which would wait on + // every owner reference -- including the foreign ones the merge deliberately preserves. + _, deleted, err := r.deleteGeneratedPolicy(ctx, req.GroupVersionKind, req.Namespace, req.Name) + switch { + case err != nil: + klog.ErrorS(err, "Failed to delete the policy generated for a resource that is gone", kferrors.Args(err, "obj", req)...) + case deleted: + klog.V(2).InfoS("Deleted the policy generated for a resource that is gone", "obj", req) + } + return ctrl.Result{}, err + case err != nil: + klog.ErrorS(err, "Failed to get the annotated resource", kferrors.Args(err)...) + return ctrl.Result{}, err + } + + if isObjectBlockedForAnnotationBasedPlacement(source) { + if HasClusterSelectorsAnnotation(source) { + r.Recorder.Eventf(source, nil, corev1.EventTypeWarning, EventReasonPlacementBlocked, "CheckEligibility", + "The %s annotation cannot place %s resources; use an explicitly authored placement policy instead", + kfplacementv1alpha1.ClusterSelectorsAnnotation, source.GetKind()) + } + return ctrl.Result{}, r.deletePolicy(ctx, source, "the resource kind is blocked for annotation-based placement") + } + + if r.ResourceEligibilityChecker != nil { + if !r.ResourceEligibilityChecker.AllowsGVK(source.GroupVersionKind()) { + // The resource's GVK is not allowed for placement. + return ctrl.Result{}, r.deletePolicy(ctx, source, "the resource is not eligible for placement (blocked GVK)") + } + // A Namespace object is checked against its own name, as it carries no namespace of its own. + if !r.ResourceEligibilityChecker.AllowsNamespace(SourceNamespace(source)) { + return ctrl.Result{}, r.deletePolicy(ctx, source, "the resource is not eligible for placement (blocked namespace)") + } + eligible, cause, err := r.ResourceEligibilityChecker.AllowsObject(source) + if err != nil { + wrappedErr := kferrors.Wraps(err, "", "obj", req) + klog.ErrorS(wrappedErr, "Failed to check resource eligibility", kferrors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + if !eligible { + return ctrl.Result{}, r.deletePolicy(ctx, source, fmt.Sprintf("the resource is not eligible for placement (%s)", cause)) + } + } + + value, annotated := source.GetAnnotations()[kfplacementv1alpha1.ClusterSelectorsAnnotation] + if !annotated { + return ctrl.Result{}, r.deletePolicy(ctx, source, "the "+kfplacementv1alpha1.ClusterSelectorsAnnotation+" annotation was removed") + } + + selectors, err := parseClusterSelectors(value) + if err != nil { + // The annotation is the user's to fix, so the failure is reported to the user and the + // request is dropped. Any policy generated from an earlier, valid annotation is deliberately + // left standing: the desired state is now unknown, and tearing down a running placement is + // a worse answer to a typo than leaving the last one the user did express. + klog.V(2).InfoS("The annotation cannot be parsed", "obj", req, "err", err) + r.Recorder.Eventf(source, nil, corev1.EventTypeWarning, EventReasonInvalidAnnotation, "ParseAnnotation", + "The %s annotation is not valid and no placement policy was generated from it: %s", kfplacementv1alpha1.ClusterSelectorsAnnotation, err) + return ctrl.Result{}, nil + } + return ctrl.Result{}, r.syncPolicy(ctx, source, selectors) +} + +// sourceObject reads the annotated resource a request refers to. +// +// A not-found error and a no-match error (the kind itself is unknown) are returned as they are, so +// that the caller can tell the two apart from a read that failed. +func (r *Reconciler) sourceObject(ctx context.Context, req Request) (*unstructured.Unstructured, error) { + // The resource is read under the version the request names and no other. Falling back to + // whatever version is served now would quietly retarget placement at a version the member + // clusters may not serve, turning a clear failure here into a confusing one later; and it + // would buy nothing, since API discovery asks for server-preferred versions, so a resource + // seen under a retired version is re-reported under the current one and reconciled again. + restMapping, err := r.RESTMapper.RESTMapping(req.GroupKind(), req.Version) + if err != nil { + if meta.IsNoMatchError(err) { + // A no-match means either that the kind is gone or that only this version is. The two + // call for opposite answers -- a gone kind should take its generated policy with it, + // while a retired version should not, since the kind still places -- so they are told + // apart by asking for the kind without a version. Only the truly gone kind is reported + // as a no-match; a retired version is transient, and the re-report that discovery + // produces under the served version reconciles it. + if _, kindErr := r.RESTMapper.RESTMapping(req.GroupKind()); kindErr == nil { + return nil, errRetiredVersion + } + return nil, err + } + return nil, kferrors.NewUnexpectedError(err, "failed to resolve the resource of the annotated object", "obj", req) + } + + source := &unstructured.Unstructured{} + source.SetGroupVersionKind(restMapping.GroupVersionKind) + // Scope comes from the mapping rather than from the request, since the mapping is what the read + // actually goes through. A cluster-scoped read must not name a namespace. + key := client.ObjectKey{Name: req.Name} + if restMapping.Scope.Name() != meta.RESTScopeNameRoot { + key.Namespace = req.Namespace + } + if err := r.APIReader.Get(ctx, key, source); err != nil { + if apierrors.IsNotFound(err) { + return nil, err + } + return nil, kferrors.NewAPIServerError(err, "failed to get the annotated resource", false, "obj", req) + } + return source, nil +} + +// syncPolicy creates or updates the policy generated from a resource's annotation. +func (r *Reconciler) syncPolicy(ctx context.Context, source *unstructured.Unstructured, selectors []kfplacementv1alpha1.ClusterSelector) error { + desired := desiredPolicy(source, selectors) + policy := emptyPolicyForScope(source.GetNamespace()) + policy.SetName(desired.GetName()) + policy.SetNamespace(desired.GetNamespace()) + + // The mutation runs only once the read behind it has succeeded, so whether it ran tells a failed + // read apart from a failed write. + read := false + result, err := controllerutil.CreateOrUpdate(ctx, r.Client, policy, func() error { + read = true + if policy.GetResourceVersion() != "" && !isGeneratedFor(policy, source.GroupVersionKind(), source.GetName(), r.Scheme()) { + // A policy already occupies this resource's generated name, but it is not one this + // controller produced -- a user or another tool authored a policy that happens to collide + // with the deterministic name. Overwriting its spec would silently commandeer it, so it is + // left exactly as found and the conflict is surfaced to the user instead. The name mixes in + // a hash of the resource's identity, so a genuine collision is near impossible and almost + // always means the name was chosen deliberately. + return errForeignPolicy + } + return applyDesiredPolicy(policy, desired, source, r.Scheme()) + }) + switch { + case errors.Is(err, errForeignPolicy): + klog.V(2).InfoS("A policy at the generated name was not generated by this controller; leaving it untouched", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + r.Recorder.Eventf(source, nil, corev1.EventTypeWarning, EventReasonPolicyConflict, "GeneratePolicy", + "A %s named %s already exists and was not generated from the %s annotation; it was left unchanged", generatedPolicyKind(source.GetNamespace()), policy.GetName(), kfplacementv1alpha1.ClusterSelectorsAnnotation) + return kferrors.NewUserError(err, "the generated name is taken", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + case apierrors.IsAlreadyExists(err): + // The cache has yet to see a policy this controller created moments ago, and an unrelated + // second event for the source arrived in that window. The next pass, after the queue's + // backoff, reads the policy the watch has delivered by then. + klog.V(2).InfoS("The generated placement policy was created but is not in the cache yet; retrying", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + return kferrors.NewTransientError(err, "the generated placement policy is not in the cache yet", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + case err != nil: + err = kferrors.NewAPIServerError(err, "failed to create or update the generated placement policy", !read, "obj", klog.KObj(source), "policy", klog.KObj(policy)) + klog.ErrorS(err, "Failed to sync the generated placement policy", kferrors.Args(err)...) + return err + } + + switch result { + case controllerutil.OperationResultCreated: + klog.V(2).InfoS("Created the generated placement policy", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + r.Recorder.Eventf(source, nil, corev1.EventTypeNormal, EventReasonPolicyCreated, "GeneratePolicy", + "Created the %s %s from the %s annotation", generatedPolicyKind(source.GetNamespace()), policy.GetName(), kfplacementv1alpha1.ClusterSelectorsAnnotation) + case controllerutil.OperationResultUpdated: + klog.V(2).InfoS("Updated the generated placement policy", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + r.Recorder.Eventf(source, nil, corev1.EventTypeNormal, EventReasonPolicyUpdated, "GeneratePolicy", + "Updated the %s %s from the %s annotation", generatedPolicyKind(source.GetNamespace()), policy.GetName(), kfplacementv1alpha1.ClusterSelectorsAnnotation) + default: + klog.V(3).InfoS("The generated placement policy is already up to date", "obj", klog.KObj(source), "policy", klog.KObj(policy)) + } + return nil +} + +// deletePolicy removes the policy generated for a resource that should not have one -- because the +// annotation was removed, or because the resource is not eligible for placement -- and tells the +// user which through an event carrying the given cause. +// +// The cause is a plain string, never a format: keeping the only format string in the Eventf call +// below constant is what lets go vet check it. +func (r *Reconciler) deletePolicy(ctx context.Context, source *unstructured.Unstructured, cause string) error { + name, deleted, err := r.deleteGeneratedPolicy(ctx, source.GroupVersionKind(), source.GetNamespace(), source.GetName()) + if err != nil { + klog.ErrorS(err, "Failed to delete the generated placement policy", kferrors.Args(err, "obj", klog.KObj(source), "policy", klog.KRef(source.GetNamespace(), name))...) + return err + } + if !deleted { + // The common case by far: a resource nobody annotated. + return nil + } + klog.V(2).InfoS("Deleted the generated placement policy", "obj", klog.KObj(source), "policy", klog.KRef(source.GetNamespace(), name)) + r.Recorder.Eventf(source, nil, corev1.EventTypeNormal, EventReasonPolicyDeleted, "DeletePolicy", "Deleted the %s %s because %s", generatedPolicyKind(source.GetNamespace()), name, cause) + return nil +} + +// deleteGeneratedPolicy deletes the policy generated for the given resource identity, reporting the +// policy's name and whether this pass performed the deletion. The name is returned so a caller that +// logs or records an event about the deletion need not derive it a second time. +func (r *Reconciler) deleteGeneratedPolicy(ctx context.Context, gvk schema.GroupVersionKind, namespace, name string) (string, bool, error) { + actual := emptyPolicyForScope(namespace) + policyName := generatedPolicyName(gvk, namespace, name) + + err := r.Get(ctx, client.ObjectKey{Namespace: namespace, Name: policyName}, actual) + switch { + case apierrors.IsNotFound(err): + return policyName, false, nil + case err != nil: + return policyName, false, kferrors.NewAPIServerError(err, "failed to get the generated placement policy", true) + } + + if !isGeneratedFor(actual, gvk, name, r.Scheme()) { + // A policy occupies the generated name but this controller did not create it, so deleting it + // would destroy a user's or another tool's object. It is never this controller's to remove; + // the deletion is declined and reported as though nothing was there to delete. + klog.V(2).InfoS("A policy at the generated name was not generated by this controller; declining to delete it", "policy", klog.KRef(namespace, policyName)) + return policyName, false, nil + } + + // The delete carries the resource version the read returned as a precondition, so it removes only + // the exact object this pass read and confirmed was one it generated. Between that read and here + // the policy could be replaced -- deleted and a hand-authored one created at the same name, or + // overwritten in place with its provenance stripped -- and an unconditioned delete, which targets + // the name alone, would then remove whatever now sits there. The precondition turns that into a + // conflict instead; the conflict requeues, and the next pass reads the current object and declines + // it if it is no longer one this controller generated. + resourceVersion := actual.GetResourceVersion() + if err := r.Delete(ctx, actual, client.Preconditions{ResourceVersion: &resourceVersion}); err != nil { + if apierrors.IsNotFound(err) { + // The read above raced a deletion that already happened -- typically this controller's + // own, re-entered through the generated policy watch moments later. Nothing was deleted + // here, and reporting otherwise would log, and on some paths announce to the user, a + // deletion that this pass did not perform. + return policyName, false, nil + } + return policyName, false, kferrors.NewAPIServerError(err, "failed to delete the generated placement policy", false) + } + return policyName, true, nil +} + +// applyDesiredPolicy brings a live generated policy in line with the desired one. +// +// Only what this controller generates is overwritten: the spec, the provenance labels, and the owner +// reference to the annotated resource. Labels and annotations that something else added are left +// alone, so that a generated policy can be labelled by an operator or a GitOps tool without this +// controller and that tool taking turns undoing each other. Whether anything changed is for the +// caller to judge by comparing the object before and after. +func applyDesiredPolicy(actual, desired kfplacementv1alpha1.PlacementPolicyAccessor, source *unstructured.Unstructured, scheme *runtime.Scheme) error { + desired.GetSpec().DeepCopyInto(actual.GetSpec()) + + labels := actual.GetLabels() + if labels == nil { + labels = make(map[string]string, len(desired.GetLabels())) + } + for key, value := range desired.GetLabels() { + labels[key] = value + } + actual.SetLabels(labels) + + // The owner reference is matched on the source's group, kind, and name and replaced in place, + // so a reference left by a source that was deleted and recreated under the same name (a new UID) + // or written under a since-retired version is corrected rather than joined by a second one. + return controllerutil.SetOwnerReference(source, actual, scheme) +} + +// HasClusterSelectorsAnnotation reports whether an object carries the annotation this controller +// acts on. It is what keeps every unrelated resource in the cluster out of this controller's queue: +// whichever event source feeds the controller is expected to call it on every event for every +// watched resource, and to enqueue only the objects it reports true for. +func HasClusterSelectorsAnnotation(object metav1.Object) bool { + _, found := object.GetAnnotations()[kfplacementv1alpha1.ClusterSelectorsAnnotation] + return found +} diff --git a/pkg/v1/controllers/annotationplacement/controller_integration_test.go b/pkg/v1/controllers/annotationplacement/controller_integration_test.go new file mode 100644 index 000000000..ecc9b81d6 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/controller_integration_test.go @@ -0,0 +1,358 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "fmt" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + eventuallyTimeout = "10s" + eventuallyTimeoutDuration = 10 * time.Second + eventuallyInterval = "250ms" +) + +var configMapCount int + +// annotate sets, changes, or (with an empty value) removes the annotation on an object. +func annotate(object client.Object, value string) { + Expect(hubClient.Get(ctx, client.ObjectKeyFromObject(object), object)).Should(Succeed()) + annotations := object.GetAnnotations() + if annotations == nil { + annotations = map[string]string{} + } + if value == "" { + delete(annotations, kfplacementv1alpha1.ClusterSelectorsAnnotation) + } else { + annotations[kfplacementv1alpha1.ClusterSelectorsAnnotation] = value + } + object.SetAnnotations(annotations) + Expect(hubClient.Update(ctx, object)).Should(Succeed()) +} + +// reconcile runs one pass of the reconciler over an object, as the resource watcher would. +func reconcile(gvk schema.GroupVersionKind, object client.Object) error { + _, err := reconciler.Reconcile(ctx, requestFor(gvk, object.GetNamespace(), object.GetName())) + return err +} + +// generatedPolicyFor reads back the policy generated for an object, whichever scope it has. +func generatedPolicyFor(gvk schema.GroupVersionKind, object client.Object) (kfplacementv1alpha1.PlacementPolicyAccessor, error) { + namespace := object.GetNamespace() + policy := emptyPolicyForScope(namespace) + name := generatedPolicyName(gvk, namespace, object.GetName()) + err := hubClient.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, policy) + return policy, err +} + +// drainEvents returns the reasons of every event recorded so far, clearing the recorder. +func drainEvents() []string { + reasons := []string{} + for { + select { + case event := <-eventRecorder.Events: + fields := strings.SplitN(event, " ", 3) + if len(fields) >= 2 { + reasons = append(reasons, fields[1]) + } + default: + return reasons + } + } +} + +// The API server itself enforces the count bounds on both forms of the int-or-string: the Pattern +// marker covers the string form, and the CEL rule covers the integer form, which a pattern alone +// leaves unbounded. The parser mirrors the same bounds for annotations; these specs pin the API +// side against a real API server, where a plain schema reading cannot. +var _ = Describe("the count bounds of a hand-authored policy", func() { + newPolicy := func(count intstr.IntOrString) *kfplacementv1alpha1.PlacementPolicy { + configMapCount++ + return &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf("hand-authored-%d", configMapCount), + Namespace: "default", + }, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{{Count: &count}}, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIVersion: "v1", Kind: "ConfigMap", Name: "app", + }}, + }, + } + } + + DescribeTable("integer and string forms share the same bounds", + func(count intstr.IntOrString, wantAccepted bool) { + policy := newPolicy(count) + err := hubClient.Create(ctx, policy) + if err == nil { + // Whatever the verdict was meant to be, an object that made it in must not + // outlive the spec; a regression that admits an out-of-range count would + // otherwise leave its evidence lying around for the rest of the suite. + DeferCleanup(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, policy))).Should(Succeed()) + }) + } + if wantAccepted { + Expect(err).Should(Succeed()) + return + } + Expect(apierrors.IsInvalid(err)).Should(BeTrue(), "got %v, want an invalid error", err) + }, + Entry("count 1 as an integer is accepted", intstr.FromInt32(1), true), + Entry("count 999 as an integer is accepted", intstr.FromInt32(999), true), + Entry("count 999 as a string is accepted", intstr.FromString("999"), true), + Entry("count All is accepted", intstr.FromString("All"), true), + // The integer entries below are the reason the CEL rule exists: before it, they were + // accepted while their quoted twins were rejected. + Entry("count 1000 as an integer is rejected", intstr.FromInt32(1000), false), + Entry("count 0 as an integer is rejected", intstr.FromInt32(0), false), + Entry("count -1 as an integer is rejected", intstr.FromInt32(-1), false), + Entry("count 1000 as a string is rejected", intstr.FromString("1000"), false), + Entry("count 0 as a string is rejected", intstr.FromString("0"), false), + ) + + // A digit string too long for an int64 used to reach the int() conversion inside the + // minCount<=count rule, adding an opaque evaluation error beside the length violation. The + // count field's own validation must be the whole story now: MaxLength rejects the string + // first, and the rule's digit guard keeps int() out of reach regardless. + It("should report an over-long count with the count field's own message alone", func() { + policy := newPolicy(intstr.FromString("99999999999999999999")) + policy.GetSpec().ClusterSelectors[0].MinCount = ptr.To(int32(1)) + + err := hubClient.Create(ctx, policy) + Expect(apierrors.IsInvalid(err)).Should(BeTrue(), "got %v, want an invalid error", err) + Expect(err.Error()).Should(ContainSubstring("Too long"), "the count field's length bound must report the malformed count") + Expect(err.Error()).ShouldNot(ContainSubstring("minCount"), "the minCount rule must stay out of a problem that is not its own") + }) +}) + +var _ = Describe("annotation based placement", func() { + Context("a namespaced resource", Ordered, func() { + var configMap *corev1.ConfigMap + + BeforeAll(func() { + configMapCount++ + configMap = &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf("app-%d", configMapCount), + Namespace: "default", + }, + } + Expect(hubClient.Create(ctx, configMap)).Should(Succeed()) + drainEvents() + }) + + AfterAll(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, configMap))).Should(Succeed()) + }) + + It("should generate no policy while the resource carries no annotation", func() { + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + _, err := generatedPolicyFor(configMapGVK, configMap) + Expect(apierrors.IsNotFound(err)).Should(BeTrue(), "got %v, want a not found error", err) + Expect(drainEvents()).Should(BeEmpty()) + }) + + It("should generate a placement policy when the annotation is set", func() { + annotate(configMap, "env=staging,count=All") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + policy, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + + spec := policy.GetSpec() + Expect(spec.ClusterSelectors).Should(HaveLen(1)) + Expect(spec.ClusterSelectors[0].Count).Should(Equal(ptrIntOrString(countAll))) + Expect(spec.ClusterSelectors[0].Terms[0].MatchLabels).Should(Equal(map[string]string{"env": "staging"})) + Expect(spec.ResourceSelectors).Should(Equal([]kfplacementv1alpha1.ResourceSelector{{ + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMap.Name, + }})) + + Expect(policy.GetNamespace()).Should(Equal(configMap.Namespace), "a namespaced resource must generate a policy in its own namespace") + Expect(policy.GetLabels()).Should(HaveKeyWithValue(kfplacementv1alpha1.ParentKindLabel, "ConfigMap")) + Expect(policy.GetLabels()).Should(HaveKeyWithValue(kfplacementv1alpha1.ParentNameLabel, configMap.Name)) + Expect(policy.GetLabels()).Should(HaveKeyWithValue(kfplacementv1alpha1.ParentAPIGroupLabel, "")) + + Expect(policy.GetOwnerReferences()).Should(HaveLen(1)) + owner := policy.GetOwnerReferences()[0] + Expect(owner.UID).Should(Equal(configMap.UID)) + Expect(owner.Controller).Should(BeNil(), "a generated policy must not claim controller ownership") + Expect(owner.BlockOwnerDeletion).Should(BeNil(), "a generated policy must not block deletion of the resource it came from") + + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyCreated})) + }) + + It("should not touch the policy when nothing changed", func() { + before, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + after, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + // A resource version that moved means the reconciler wrote to the API server without + // anything having changed, which every pass would then repeat. + Expect(after.GetResourceVersion()).Should(Equal(before.GetResourceVersion())) + Expect(drainEvents()).Should(BeEmpty()) + }) + + It("should update the policy when the annotation changes", func() { + annotate(configMap, "env=canary,region=eastus,count=3") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + policy, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + spec := policy.GetSpec() + Expect(spec.ClusterSelectors).Should(HaveLen(1)) + Expect(spec.ClusterSelectors[0].Count).Should(Equal(ptrIntOrString("3"))) + Expect(spec.ClusterSelectors[0].Terms[0].MatchLabels).Should(Equal(map[string]string{ + "env": "canary", + corev1.LabelTopologyRegion: "eastus", + })) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyUpdated})) + }) + + It("should restore the policy when someone edits it", func() { + policy, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + policy.GetSpec().ClusterSelectors = nil + Expect(hubClient.Update(ctx, policy)).Should(Succeed()) + + // In the running agent this reconcile is triggered by the watch on the generated + // policies themselves; an edit produces no event on the ConfigMap. + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + restored, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + Expect(restored.GetSpec().ClusterSelectors).Should(HaveLen(1)) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyUpdated})) + }) + + It("should recreate the policy when someone deletes it", func() { + policy, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + Expect(hubClient.Delete(ctx, policy)).Should(Succeed()) + + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + _, err = generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed(), "the deleted policy must be generated again") + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyCreated})) + }) + + It("should keep the policy and warn when the annotation becomes invalid", func() { + annotate(configMap, "env") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed(), "a malformed annotation must not be retried") + + _, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed(), "the policy from the last valid annotation must be left in place") + Expect(drainEvents()).Should(Equal([]string{EventReasonInvalidAnnotation})) + }) + + It("should delete the policy when the annotation is removed", func() { + annotate(configMap, "") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + + _, err := generatedPolicyFor(configMapGVK, configMap) + Expect(apierrors.IsNotFound(err)).Should(BeTrue(), "got %v, want a not found error", err) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyDeleted})) + }) + + It("should delete the policy once the resource itself is gone", func() { + // Deleting explicitly, rather than leaving the policy to garbage collection, is what + // keeps an owner reference some other party added from holding the policy up forever. + // It also means envtest, which runs no garbage collector, can observe the cleanup. + annotate(configMap, "env=staging") + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + _, err := generatedPolicyFor(configMapGVK, configMap) + Expect(err).Should(Succeed()) + drainEvents() + + Expect(hubClient.Delete(ctx, configMap)).Should(Succeed()) + + Expect(reconcile(configMapGVK, configMap)).Should(Succeed()) + _, err = generatedPolicyFor(configMapGVK, configMap) + Expect(apierrors.IsNotFound(err)).Should(BeTrue(), "got %v, want a not found error", err) + // No event: the resource an event would be recorded on no longer exists. + Expect(drainEvents()).Should(BeEmpty()) + }) + }) + + Context("a cluster scoped resource", Ordered, func() { + var namespace *corev1.Namespace + + BeforeAll(func() { + configMapCount++ + namespace = &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf("team-%d", configMapCount)}, + } + Expect(hubClient.Create(ctx, namespace)).Should(Succeed()) + drainEvents() + }) + + AfterAll(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, namespace))).Should(Succeed()) + }) + + It("should generate a cluster scoped policy", func() { + annotate(namespace, "env=staging") + Expect(reconcile(namespaceGVK, namespace)).Should(Succeed()) + + policy, err := generatedPolicyFor(namespaceGVK, namespace) + Expect(err).Should(Succeed()) + Expect(policy).Should(BeAssignableToTypeOf(&kfplacementv1alpha1.ClusterPlacementPolicy{}), + "a cluster scoped resource must generate a ClusterPlacementPolicy, since a namespaced policy owned by it would never be collected") + Expect(policy.GetNamespace()).Should(BeEmpty()) + Expect(policy.GetSpec().ClusterSelectors).Should(HaveLen(1)) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyCreated})) + }) + + It("should delete the cluster scoped policy when the annotation is removed", func() { + annotate(namespace, "") + Expect(reconcile(namespaceGVK, namespace)).Should(Succeed()) + + _, err := generatedPolicyFor(namespaceGVK, namespace) + Expect(apierrors.IsNotFound(err)).Should(BeTrue(), "got %v, want a not found error", err) + Expect(drainEvents()).Should(Equal([]string{EventReasonPolicyDeleted})) + }) + }) +}) + +func ptrIntOrString(value string) *intstr.IntOrString { + parsed := intstr.Parse(value) + return &parsed +} diff --git a/pkg/v1/controllers/annotationplacement/controller_test.go b/pkg/v1/controllers/annotationplacement/controller_test.go new file mode 100644 index 000000000..d78f1ed9e --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/controller_test.go @@ -0,0 +1,1140 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "errors" + "slices" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + "k8s.io/apimachinery/pkg/api/equality" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/tools/events" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/resourceeligibility" +) + +var ( + deploymentGVR = schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} + namespaceGVR = schema.GroupVersionResource{Group: "", Version: "v1", Resource: "namespaces"} +) + +const ( + testNamespace = "prod" + testName = "web" + + oneSelector = "env=staging" + twoSelectors = "env=staging,count=All;env=canary,region=eastus" +) + +// newSource builds the annotated resource as the API server would return it. +func newSource(gvk schema.GroupVersionKind, namespace, name string, annotations map[string]string) *unstructured.Unstructured { + source := sourceObject(gvk, namespace, name) + if annotations != nil { + source.SetAnnotations(annotations) + } + return source +} + +// requestFor builds the request the resource watcher would enqueue for a resource. +func requestFor(gvk schema.GroupVersionKind, namespace, name string) Request { + return Request{ + GroupVersionKind: gvk, + NamespacedName: client.ObjectKey{Namespace: namespace, Name: name}, + } +} + +func newScheme(t *testing.T) *runtime.Scheme { + t.Helper() + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + return scheme +} + +// newRESTMapper knows the served kinds these tests read: a namespaced Deployment and a cluster +// scoped Namespace. A kind or a version outside it resolves to a no-match error, as a kind whose +// CRD was removed would at the API server. +func newRESTMapper() meta.RESTMapper { + mapper := meta.NewDefaultRESTMapper([]schema.GroupVersion{deploymentGVK.GroupVersion(), namespaceGVK.GroupVersion()}) + mapper.AddSpecific(deploymentGVK, deploymentGVR, deploymentGVR, meta.RESTScopeNamespace) + mapper.AddSpecific(namespaceGVK, namespaceGVR, namespaceGVR, meta.RESTScopeRoot) + return mapper +} + +// newReconciler wires a reconciler whose API server holds the given annotated resources and +// policies. One fake client backs both the policy client and the API reader, so the two are always +// consistent, which is what every test here wants. +func newReconciler(t *testing.T, funcs interceptor.Funcs, objects ...client.Object) (*Reconciler, *events.FakeRecorder) { + t.Helper() + recorder := events.NewFakeRecorder(10) + c := fake.NewClientBuilder().WithScheme(newScheme(t)).WithObjects(objects...).WithInterceptorFuncs(funcs).Build() + return &Reconciler{ + Client: c, + APIReader: c, + RESTMapper: newRESTMapper(), + Recorder: recorder, + }, recorder +} + +// recordedReasons drains the recorder and returns the reason of every event it holds. +func recordedReasons(recorder *events.FakeRecorder) []string { + reasons := make([]string, 0, len(recorder.Events)) + for { + select { + case event := <-recorder.Events: + // A recorded event reads "Normal ". + fields := strings.SplitN(event, " ", 3) + if len(fields) < 2 { + reasons = append(reasons, event) + continue + } + reasons = append(reasons, fields[1]) + default: + return reasons + } + } +} + +// policyFrom reads the generated policy for a resource back out of the client, or reports that none +// exists. +func policyFrom(ctx context.Context, t *testing.T, r *Reconciler, source *unstructured.Unstructured) (kfplacementv1alpha1.PlacementPolicyAccessor, bool) { + t.Helper() + namespace := source.GetNamespace() + policy := emptyPolicyForScope(namespace) + name := generatedPolicyName(source.GroupVersionKind(), namespace, source.GetName()) + err := r.Get(ctx, client.ObjectKey{Namespace: namespace, Name: name}, policy) + switch { + case apierrors.IsNotFound(err): + return nil, false + case err != nil: + t.Fatalf("Get(%s) = %v, want no error", name, err) + } + return policy, true +} + +// policyIgnoreOpts drop the fields the API server owns, which no expectation here can predict. +var policyIgnoreOpts = cmp.Options{ + cmpopts.IgnoreFields(metav1.TypeMeta{}, "Kind", "APIVersion"), + cmpopts.IgnoreFields(metav1.ObjectMeta{}, "ResourceVersion", "Generation", "CreationTimestamp", "ManagedFields"), +} + +func mustParse(t *testing.T, value string) []kfplacementv1alpha1.ClusterSelector { + t.Helper() + selectors, err := parseClusterSelectors(value) + if err != nil { + t.Fatalf("parseClusterSelectors(%q) = %v, want no error", value, err) + } + return selectors +} + +func TestReconcile(t *testing.T) { + // The desired policy is built with desiredPolicy, which policy_test.go covers on its own; what + // is under test here is whether the reconciler puts that object in the cluster, takes it away + // again, and says so. + annotatedSource := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + bareSource := newSource(deploymentGVK, testNamespace, testName, nil) + clusterScopedSource := newSource(namespaceGVK, "", testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + + testCases := []struct { + name string + // source is the resource the API server holds, and the resource the request points at. + source *unstructured.Unstructured + // existing is the annotation value a policy was already generated from, if any. + existing string + wantPolicy bool + wantValue string + wantReasons []string + }{ + { + name: "an annotated resource with no policy yet gets one", + source: annotatedSource, + wantPolicy: true, + wantValue: oneSelector, + wantReasons: []string{EventReasonPolicyCreated}, + }, + { + name: "a policy that already matches the annotation is left alone", + source: annotatedSource, + existing: oneSelector, + wantPolicy: true, + wantValue: oneSelector, + wantReasons: nil, + }, + { + name: "a changed annotation reaches the policy", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: twoSelectors}), + existing: oneSelector, + wantPolicy: true, + wantValue: twoSelectors, + wantReasons: []string{EventReasonPolicyUpdated}, + }, + { + name: "removing the annotation deletes the policy", + source: bareSource, + existing: oneSelector, + wantPolicy: false, + wantReasons: []string{EventReasonPolicyDeleted}, + }, + { + name: "a resource that was never annotated is left alone", + source: bareSource, + wantPolicy: false, + wantReasons: nil, + }, + { + name: "a cluster scoped resource generates a cluster scoped policy", + source: clusterScopedSource, + wantPolicy: true, + wantValue: oneSelector, + wantReasons: []string{EventReasonPolicyCreated}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + objects := []client.Object{tc.source} + if tc.existing != "" { + objects = append(objects, desiredPolicy(tc.source, mustParse(t, tc.existing))) + } + r, recorder := newReconciler(t, interceptor.Funcs{}, objects...) + + req := RequestFor(tc.source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + + got, found := policyFrom(ctx, t, r, tc.source) + if found != tc.wantPolicy { + t.Fatalf("Reconcile(%v) left a generated policy = %v, want %v", req, found, tc.wantPolicy) + } + if tc.wantPolicy { + want := desiredPolicy(tc.source, mustParse(t, tc.wantValue)) + if diff := cmp.Diff(got, want, policyIgnoreOpts); diff != "" { + t.Errorf("Reconcile(%v) generated policy mismatch (-got, +want):\n%s", req, diff) + } + } + if diff := cmp.Diff(recordedReasons(recorder), tc.wantReasons, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + }) + } +} + +// TestReconcileInvalidAnnotation covers the case the parser rejects: the user hears about it, the +// request is not retried, and a policy generated from an earlier valid annotation stays up. +func TestReconcileInvalidAnnotation(t *testing.T) { + ctx := context.Background() + valid := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + existing := desiredPolicy(valid, mustParse(t, oneSelector)) + + broken := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: "env"}) + r, recorder := newReconciler(t, interceptor.Funcs{}, broken, existing) + + req := RequestFor(broken) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error, so that a malformed annotation is not retried", req, err) + } + + got, found := policyFrom(ctx, t, r, broken) + if !found { + t.Fatalf("Reconcile(%v) removed the policy generated from the previous annotation, want it left in place", req) + } + if diff := cmp.Diff(got, existing, policyIgnoreOpts); diff != "" { + t.Errorf("Reconcile(%v) changed the policy generated from the previous annotation (-got, +want):\n%s", req, diff) + } + if diff := cmp.Diff(recordedReasons(recorder), []string{EventReasonInvalidAnnotation}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } +} + +// TestReconcileDeletedResource covers a request for a resource that is already gone. The reconciler +// deletes the generated policy itself rather than deferring to garbage collection, which removes a +// dependent only once every owner reference on it is gone -- and the merge deliberately preserves +// owner references that other parties added, any live one of which would keep the policy standing. +// No event is recorded, since the resource an event would be recorded on no longer exists. +func TestReconcileDeletedResource(t *testing.T) { + ctx := context.Background() + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + existing := desiredPolicy(source, mustParse(t, oneSelector)) + // The other party whose owner reference would hold the policy back from garbage collection. + otherOwner := metav1.OwnerReference{APIVersion: "example.com/v1", Kind: "Widget", Name: "unrelated", UID: "00000000-0000-0000-0000-0000000000ff"} + existing.SetOwnerReferences(append(existing.GetOwnerReferences(), otherOwner)) + + r, recorder := newReconciler(t, interceptor.Funcs{}, existing) + + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if _, found := policyFrom(ctx, t, r, source); found { + t.Errorf("Reconcile(%v) left the generated policy standing, want it deleted", req) + } + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none", req, got) + } +} + +// TestReconcileIneligibleResource covers a resource that exists but fails the placement eligibility +// test -- for instance one in a skipped namespace, or a ReplicaSet that a Deployment has adopted. +// Its generated policy is deleted: the resource watcher reports such a resource as deleted and then +// never again, so this reconciliation is the last chance to clean up. +func TestReconcileIneligibleResource(t *testing.T) { + for _, check := range []string{"GVK", "namespace", "namespace object", "object"} { + t.Run(check, func(t *testing.T) { + ctx := context.Background() + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + checker := resourceeligibility.AllowAll(newRESTMapper()) + switch check { + case "GVK": + if err := checker.SetGVKCheckList(resourceeligibility.GVKEligibilityModeDenyList, []schema.GroupVersionKind{deploymentGVK}); err != nil { + t.Fatalf("SetGVKCheckList() = %v, want no error", err) + } + case "namespace": + checker.SetNamespaceCheckList(resourceeligibility.NamespaceEligibilityModeDenyList, []string{testNamespace}, nil) + case "namespace object": + // A Namespace carries no namespace of its own, so it must be checked against its name; + // checking its empty namespace field would let a skipped namespace through. + source = newSource(namespaceGVK, "", testNamespace, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + checker.SetNamespaceCheckList(resourceeligibility.NamespaceEligibilityModeDenyList, []string{testNamespace}, nil) + case "object": + controller := true + source.SetOwnerReferences([]metav1.OwnerReference{{ + APIVersion: "apps/v1", + Kind: "Deployment", + Name: "owner", + UID: "owner-uid", + Controller: &controller, + }}) + } + existing := desiredPolicy(source, mustParse(t, oneSelector)) + r, recorder := newReconciler(t, interceptor.Funcs{}, source, existing) + r.ResourceEligibilityChecker = checker + + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if _, found := policyFrom(ctx, t, r, source); found { + t.Errorf("Reconcile(%v) left the generated policy standing, want it deleted", req) + } + if diff := cmp.Diff(recordedReasons(recorder), []string{EventReasonPolicyDeleted}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + + // A second pass over a resource that never generated anything must stay silent. + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none on the second pass", req, got) + } + }) + } +} + +// TestReconcileBlockedAdmissionPolicy covers the admission policy kinds, which annotation-based +// placement refuses to place even though an explicitly authored placement policy can still place +// them. An annotated one is warned about rather than given a policy, and a policy generated for it +// before the kind was blocked is deleted. +func TestReconcileBlockedAdmissionPolicy(t *testing.T) { + for _, kind := range []string{ + "ValidatingAdmissionPolicy", + "ValidatingAdmissionPolicyBinding", + "MutatingAdmissionPolicy", + "MutatingAdmissionPolicyBinding", + } { + for _, existingPolicy := range []bool{false, true} { + name := kind + "/new" + if existingPolicy { + name = kind + "/existing" + } + t.Run(name, func(t *testing.T) { + gvk := schema.GroupVersionKind{Group: "admissionregistration.k8s.io", Version: "v1", Kind: kind} + source := newSource(gvk, "", testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + objects := []client.Object{source} + wantReasons := []string{EventReasonPlacementBlocked} + if existingPolicy { + objects = append(objects, desiredPolicy(source, mustParse(t, oneSelector))) + wantReasons = append(wantReasons, EventReasonPolicyDeleted) + } + r, recorder := newReconciler(t, interceptor.Funcs{}, objects...) + mapper := meta.NewDefaultRESTMapper([]schema.GroupVersion{gvk.GroupVersion()}) + mapper.Add(gvk, meta.RESTScopeRoot) + r.RESTMapper = mapper + + ctx := context.Background() + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if _, found := policyFrom(ctx, t, r, source); found { + t.Errorf("Reconcile(%v) left a generated policy, want none", req) + } + if diff := cmp.Diff(recordedReasons(recorder), wantReasons); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + }) + } + } +} + +type failingEligibilityChecker struct { + resourceeligibility.Checker + err error +} + +func (c *failingEligibilityChecker) AllowsObject(*unstructured.Unstructured) (bool, string, error) { + return false, "", c.err +} + +// TestReconcileEligibilityError covers the eligibility test itself failing, which must surface as a +// retryable error rather than as either verdict. +func TestReconcileEligibilityError(t *testing.T) { + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + r, _ := newReconciler(t, interceptor.Funcs{}, source) + wantErr := errors.New("the eligibility test is unwell") + r.ResourceEligibilityChecker = &failingEligibilityChecker{ + Checker: resourceeligibility.AllowAll(r.RESTMapper), + err: wantErr, + } + + req := RequestFor(source) + if _, err := r.Reconcile(context.Background(), req); !errors.Is(err, wantErr) { + t.Errorf("Reconcile(%v) = %v, want an error wrapping %v", req, err, wantErr) + } + if _, found := policyFrom(context.Background(), t, r, source); found { + t.Errorf("Reconcile(%v) acted on the policy despite the eligibility error", req) + } +} + +// TestReconcileAPIServerErrors covers the paths where a read or a write fails. Each has to come +// back as a retryable error rather than as a silent success, because the annotation and the cluster +// have disagreed at that point and only another pass can settle it. +func TestReconcileAPIServerErrors(t *testing.T) { + annotated := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + bare := newSource(deploymentGVK, testNamespace, testName, nil) + existing := desiredPolicy(annotated, mustParse(t, oneSelector)) + failure := apierrors.NewInternalError(errors.New("the api server is unwell")) + + testCases := []struct { + name string + source *unstructured.Unstructured + existing []client.Object + interceptor interceptor.Funcs + }{ + { + name: "the annotated resource cannot be read", + source: annotated, + interceptor: interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, isSource := obj.(*unstructured.Unstructured); isSource { + return failure + } + return c.Get(ctx, key, obj, opts...) + }, + }, + }, + { + name: "the policy cannot be read", + source: annotated, + interceptor: interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, isSource := obj.(*unstructured.Unstructured); isSource { + return c.Get(ctx, key, obj, opts...) + } + return failure + }, + }, + }, + { + name: "the policy cannot be created", + source: annotated, + interceptor: interceptor.Funcs{ + Create: func(context.Context, client.WithWatch, client.Object, ...client.CreateOption) error { + return failure + }, + }, + }, + { + name: "the policy cannot be updated", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: twoSelectors}), + existing: []client.Object{existing}, + interceptor: interceptor.Funcs{ + Update: func(context.Context, client.WithWatch, client.Object, ...client.UpdateOption) error { + return failure + }, + }, + }, + { + name: "the policy cannot be deleted", + source: bare, + existing: []client.Object{existing}, + interceptor: interceptor.Funcs{ + Delete: func(context.Context, client.WithWatch, client.Object, ...client.DeleteOption) error { + return failure + }, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + r, recorder := newReconciler(t, tc.interceptor, append([]client.Object{tc.source}, tc.existing...)...) + + req := RequestFor(tc.source) + _, err := r.Reconcile(context.Background(), req) + if !errors.Is(err, failure) { + t.Errorf("Reconcile(%v) = %v, want an error wrapping %v", req, err, failure) + } + // A write that failed must not be announced as though it had happened. + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none", req, got) + } + }) + } +} + +// TestReconcileRetriesWhenCreateRacesTheCache covers a create that finds the policy already there: +// the cache has yet to see the policy this controller created a moment ago, and a second, unrelated +// event for the source arrived in that window. The pass must come back as a retryable error, not +// as a failure, and must not announce a creation it did not perform. +func TestReconcileRetriesWhenCreateRacesTheCache(t *testing.T) { + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + alreadyExists := apierrors.NewAlreadyExists(schema.GroupResource{Group: kfplacementv1alpha1.GroupVersion.Group, Resource: "placementpolicies"}, generatedPolicyName(deploymentGVK, testNamespace, testName)) + raced := interceptor.Funcs{ + Create: func(context.Context, client.WithWatch, client.Object, ...client.CreateOption) error { + return alreadyExists + }, + } + r, recorder := newReconciler(t, raced, source) + + req := RequestFor(source) + if _, err := r.Reconcile(context.Background(), req); !errors.Is(err, alreadyExists) { + t.Errorf("Reconcile(%v) = %v, want an error wrapping %v so the source is retried", req, err, alreadyExists) + } + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none for a creation this pass did not perform", req, got) + } +} + +// TestDeleteRacesAnotherDeletion covers a cached read handing the reconciler a policy that is +// already gone by the time it deletes -- typically its own deletion re-entered through the +// generated policy watch. The pass must not claim, in logs or events, a deletion it did not do. +func TestDeleteRacesAnotherDeletion(t *testing.T) { + ctx := context.Background() + source := newSource(deploymentGVK, testNamespace, testName, nil) + annotated := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + existing := desiredPolicy(annotated, mustParse(t, oneSelector)) + + // The interceptor stands in for the stale cache: the Get finds the policy, the Delete + // discovers someone else got there first. + raced := interceptor.Funcs{ + Delete: func(context.Context, client.WithWatch, client.Object, ...client.DeleteOption) error { + return apierrors.NewNotFound(schema.GroupResource{Group: kfplacementv1alpha1.GroupVersion.Group, Resource: "placementpolicies"}, existing.GetName()) + }, + } + r, recorder := newReconciler(t, raced, source, existing) + + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none for a deletion this pass did not perform", req, got) + } +} + +// TestReconcileDoesNotDeleteAReplacementPolicy covers the window between the read that confirms a +// policy is this controller's and the delete that removes it. If the policy is replaced in that +// window -- overwritten with its provenance stripped, or deleted and a hand-authored one created at +// the same name -- an unconditioned delete would remove the replacement, since it targets the name +// alone. The resource-version precondition must turn that into a conflict, and the retry must then +// read the current object and decline it. +func TestReconcileDoesNotDeleteAReplacementPolicy(t *testing.T) { + ctx := context.Background() + bare := newSource(deploymentGVK, testNamespace, testName, nil) + annotated := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + ours := desiredPolicy(annotated, mustParse(t, oneSelector)) + + // A spec no generated policy would carry, marking the object that takes the name. + foreignSpec := kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{APIGroup: "example.com", APIVersion: "v1", Kind: "Widget", Name: "hand-authored"}}, + } + + replaced := false + raced := interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + if !replaced { + replaced = true + // Stand in for the replacement: the live object loses its provenance and its resource + // version moves on, so it is no longer the one the precondition names. + live := &kfplacementv1alpha1.PlacementPolicy{} + if err := c.Get(ctx, client.ObjectKeyFromObject(obj), live); err != nil { + return err + } + live.SetLabels(nil) + live.SetOwnerReferences(nil) + live.Spec = foreignSpec + if err := c.Update(ctx, live); err != nil { + return err + } + } + return c.Delete(ctx, obj, opts...) + }, + } + r, recorder := newReconciler(t, raced, bare, ours) + + req := RequestFor(bare) + // First pass: the policy is replaced after it is read, so the guarded delete conflicts and the + // pass returns a retryable error rather than removing the replacement. + if _, err := r.Reconcile(ctx, req); err == nil { + t.Fatalf("Reconcile(%v) = nil, want a conflict error so the delete of a replaced policy is retried", req) + } + got, found := policyFrom(ctx, t, r, bare) + if !found { + t.Fatalf("Reconcile(%v) deleted the replacement policy, want it left in place", req) + } + if diff := cmp.Diff(got.(*kfplacementv1alpha1.PlacementPolicy).Spec, foreignSpec); diff != "" { + t.Errorf("Reconcile(%v) changed the replacement policy (-got, +want):\n%s", req, diff) + } + + // Second pass: the reconciler reads the current, now foreign object, declines it, and records no + // deletion it did not perform. + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error once the replacement is recognized as foreign", req, err) + } + if _, found := policyFrom(ctx, t, r, bare); !found { + t.Errorf("Reconcile(%v) deleted the foreign replacement on retry, want it left in place", req) + } + if got := recordedReasons(recorder); slices.Contains(got, EventReasonPolicyDeleted) { + t.Errorf("Reconcile(%v) recorded events = %v, want no deletion of a policy it did not generate", req, got) + } +} + +func TestApplyDesiredPolicy(t *testing.T) { + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + desired := desiredPolicy(source, mustParse(t, oneSelector)) + scheme := newScheme(t) + + otherOwner := metav1.OwnerReference{ + APIVersion: "example.com/v1", + Kind: "Widget", + Name: "unrelated", + UID: "00000000-0000-0000-0000-0000000000ff", + } + + testCases := []struct { + name string + // mutate turns the desired policy into the live one the reconciler would read back. + mutate func(kfplacementv1alpha1.PlacementPolicyAccessor) + wantChanged bool + // check asserts on whatever the merge is supposed to preserve. + check func(*testing.T, kfplacementv1alpha1.PlacementPolicyAccessor) + }{ + { + name: "an untouched policy needs no update", + mutate: func(kfplacementv1alpha1.PlacementPolicyAccessor) {}, + wantChanged: false, + }, + { + name: "a drifted spec is restored", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ClusterSelectors = nil + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + if diff := cmp.Diff(policy.GetSpec(), desired.GetSpec()); diff != "" { + t.Errorf("applyDesiredPolicy() spec mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + name: "a missing provenance label is restored", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + labels := policy.GetLabels() + delete(labels, kfplacementv1alpha1.ParentKindLabel) + policy.SetLabels(labels) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + if diff := cmp.Diff(policy.GetLabels(), desired.GetLabels()); diff != "" { + t.Errorf("applyDesiredPolicy() labels mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + name: "a policy stripped of every label gets the provenance labels back", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.SetLabels(nil) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + if diff := cmp.Diff(policy.GetLabels(), desired.GetLabels()); diff != "" { + t.Errorf("applyDesiredPolicy() labels mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + name: "labels added by something else are kept", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + labels := policy.GetLabels() + labels["example.com/managed-by"] = "gitops" + delete(labels, kfplacementv1alpha1.ParentKindLabel) + policy.SetLabels(labels) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + if got := policy.GetLabels()["example.com/managed-by"]; got != "gitops" { + t.Errorf("applyDesiredPolicy() label example.com/managed-by = %q, want %q", got, "gitops") + } + }, + }, + { + name: "an owner reference that drifted is corrected in place", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + drifted := parentOwnerReference(source) + drifted.APIVersion = "apps/v1beta1" + policy.SetOwnerReferences([]metav1.OwnerReference{drifted}) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + want := []metav1.OwnerReference{parentOwnerReference(source)} + if diff := cmp.Diff(policy.GetOwnerReferences(), want); diff != "" { + t.Errorf("applyDesiredPolicy() owner references mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + name: "a missing owner reference is restored alongside any other", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.SetOwnerReferences([]metav1.OwnerReference{otherOwner}) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + want := []metav1.OwnerReference{otherOwner, parentOwnerReference(source)} + if diff := cmp.Diff(policy.GetOwnerReferences(), want); diff != "" { + t.Errorf("applyDesiredPolicy() owner references mismatch (-got, +want):\n%s", diff) + } + }, + }, + { + // The source was deleted and recreated under the same name, so its reference carries the + // old UID. It must be updated in place, not left dangling while a second one is appended -- + // otherwise the list grows by one on every such cycle. + name: "an owner reference left by a recreated source is replaced, not appended", + mutate: func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + stale := parentOwnerReference(source) + stale.UID = "00000000-0000-0000-0000-00000000dead" + policy.SetOwnerReferences([]metav1.OwnerReference{stale}) + }, + wantChanged: true, + check: func(t *testing.T, policy kfplacementv1alpha1.PlacementPolicyAccessor) { + want := []metav1.OwnerReference{parentOwnerReference(source)} + if diff := cmp.Diff(policy.GetOwnerReferences(), want); diff != "" { + t.Errorf("applyDesiredPolicy() owner references mismatch (-got, +want):\n%s", diff) + } + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + actual := desired.DeepCopyObject().(kfplacementv1alpha1.PlacementPolicyAccessor) + tc.mutate(actual) + before := actual.DeepCopyObject() + + if err := applyDesiredPolicy(actual, desired, source, scheme); err != nil { + t.Fatalf("applyDesiredPolicy() = %v, want no error", err) + } + if got := !equality.Semantic.DeepEqual(before, actual); got != tc.wantChanged { + t.Errorf("applyDesiredPolicy() changed the policy = %v, want %v", got, tc.wantChanged) + } + if tc.check != nil { + tc.check(t, actual) + } + // Whatever the merge did, a second pass over its own output must find nothing left to + // do; otherwise the reconciler would issue an update on every single pass. + settled := actual.DeepCopyObject() + if err := applyDesiredPolicy(actual, desired, source, scheme); err != nil { + t.Fatalf("applyDesiredPolicy() = %v on the second pass, want no error", err) + } + if !equality.Semantic.DeepEqual(settled, actual) { + t.Errorf("applyDesiredPolicy() changed the policy on the second pass, want it settled") + } + }) + } +} + +// TestEventMessagesNameTheGeneratedKind pins that an event's message names the kind actually +// generated: the two scopes generate different kinds, and a message claiming a PlacementPolicy for +// what is really a ClusterPlacementPolicy sends the user's kubectl to a resource that is not there. +func TestEventMessagesNameTheGeneratedKind(t *testing.T) { + testCases := []struct { + name string + source *unstructured.Unstructured + wantKind string + }{ + { + name: "a namespaced source names PlacementPolicy", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + wantKind: "PlacementPolicy", + }, + { + name: "a cluster scoped source names ClusterPlacementPolicy", + source: newSource(namespaceGVK, "", testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + wantKind: "ClusterPlacementPolicy", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.source) + + req := RequestFor(tc.source) + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + + select { + case event := <-recorder.Events: + // The kind is matched with its surrounding spaces: "ClusterPlacementPolicy" + // contains "PlacementPolicy" as a bare substring, so an unanchored check would + // pass the namespaced case even if the wrong kind were named. + if !strings.Contains(event, " the "+tc.wantKind+" ") { + t.Errorf("Reconcile(%v) recorded event %q, want it to name the kind %q", req, event, tc.wantKind) + } + default: + t.Fatalf("Reconcile(%v) recorded no event, want one naming the kind %q", req, tc.wantKind) + } + }) + } +} + +// TestReconcileUnknownKind covers a request whose kind the API server does not know, which the +// generated policy watch can produce by enqueuing whatever a policy names as its owner. The kind +// being gone means the source's own CRD was removed, so any policy generated from it is stale and is +// deleted; when none exists the request is simply dropped, since no retry can make the kind exist +// and an error would keep it backing off forever. +func TestReconcileUnknownKind(t *testing.T) { + widgetGVK := schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: "Widget"} + // The resource whose kind is gone. It is only used to derive the generated policy the reconciler + // must find and delete; the REST mapper deliberately does not know its kind. + widget := newSource(widgetGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + stale := desiredPolicy(widget, mustParse(t, oneSelector)) + + testCases := []struct { + name string + existing []client.Object + wantPolicy bool + }{ + {name: "no policy to clean up, request is dropped", existing: nil, wantPolicy: false}, + {name: "a stale policy left by the gone kind is deleted", existing: []client.Object{stale}, wantPolicy: false}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.existing...) + req := RequestFor(widget) + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Errorf("Reconcile(%v) = %v, want no error, so that a kind that does not exist is not retried", req, err) + } + if _, found := policyFrom(context.Background(), t, r, widget); found != tc.wantPolicy { + t.Errorf("Reconcile(%v) left a generated policy = %v, want %v", req, found, tc.wantPolicy) + } + // No event is recorded either way: the resource an event would attach to is gone. + if got := recordedReasons(recorder); len(got) != 0 { + t.Errorf("Reconcile(%v) recorded events = %v, want none", req, got) + } + }) + } +} + +// TestReconcileKeepsPolicyWhenOnlyVersionRetired covers a request whose recorded version is no +// longer served while the kind lives on under another. The generated policy watch can enqueue such +// a request, and treating the retired version as a gone kind would delete a policy that is still +// wanted. The resource is deliberately not re-read under the served version -- that version may not +// exist on the member clusters -- so the round is retried instead, and API discovery re-reports the +// resource under the served version. +func TestReconcileKeepsPolicyWhenOnlyVersionRetired(t *testing.T) { + testCases := []struct { + name string + source *unstructured.Unstructured + // req names the source under a version the REST mapper no longer knows. + req Request + }{ + { + name: "a namespaced source served under apps/v1 is reached through apps/v2", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + req: requestFor(schema.GroupVersionKind{Group: "apps", Version: "v2", Kind: "Deployment"}, testNamespace, testName), + }, + { + name: "a cluster scoped source served under v1 is reached through v2", + source: newSource(namespaceGVK, "", testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + req: requestFor(schema.GroupVersionKind{Group: "", Version: "v2", Kind: "Namespace"}, "", testName), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.source) + + // An existing policy stands in for one generated before the version was retired; the + // round must leave it alone rather than take it for the policy of a gone kind. + existing := desiredPolicy(tc.source, mustParse(t, oneSelector)) + if err := r.Create(ctx, existing); err != nil { + t.Fatalf("Create(%v) = %v, want no error", klog.KObj(existing), err) + } + + // Dropped, not retried: no error and no requeue, since the resource is re-reported + // under the served version as a different request. + res, err := r.Reconcile(ctx, tc.req) + if err != nil { + t.Errorf("Reconcile(%v) = %v, want the retired version dropped rather than retried", tc.req, err) + } + if res.RequeueAfter != 0 { + t.Errorf("Reconcile(%v) = %+v, want no requeue for a version that will never come back", tc.req, res) + } + if _, found := policyFrom(ctx, t, r, tc.source); !found { + t.Errorf("Reconcile(%v) deleted the generated policy, want it kept while the kind is still served", tc.req) + } + if diff := cmp.Diff(recordedReasons(recorder), []string(nil), cmpopts.EquateEmpty()); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", tc.req, diff) + } + }) + } +} + +// TestReconcileForeignPolicyAtGeneratedName covers a policy that already occupies a resource's +// generated name but was authored by someone else -- it carries none of this controller's provenance +// labels. The controller must neither overwrite it when the annotation asks for a policy nor delete +// it when the annotation is removed; a bare name match would do both. +func TestReconcileForeignPolicyAtGeneratedName(t *testing.T) { + // A policy sitting at the generated name, distinguishable by a spec this controller would never + // produce and by the absence of the provenance labels. + foreignSpec := func() kfplacementv1alpha1.PlacementPolicySpec { + return kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{APIGroup: "example.com", APIVersion: "v1", Kind: "Widget", Name: "hand-authored"}}, + } + } + newForeign := func() *kfplacementv1alpha1.PlacementPolicy { + return &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: generatedPolicyName(deploymentGVK, testNamespace, testName), + Namespace: testNamespace, + }, + Spec: foreignSpec(), + } + } + + testCases := []struct { + name string + source *unstructured.Unstructured + wantErr error + wantReasons []string + }{ + { + name: "the annotation asks for a policy, the foreign one is not overwritten", + source: newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}), + // The source is retried with backoff -- nothing else brings it back once the conflict is + // cleared -- and the user hears why the placement they asked for is not running. + wantErr: errForeignPolicy, + wantReasons: []string{EventReasonPolicyConflict}, + }, + { + name: "the annotation is absent, the foreign one is not deleted", + // Nothing asked for a policy here, so declining to delete a policy that was never this + // controller's is silent -- the same as a resource that never generated anything. + source: newSource(deploymentGVK, testNamespace, testName, nil), + wantReasons: nil, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.source, newForeign()) + + req := RequestFor(tc.source) + if _, err := r.Reconcile(ctx, req); !errors.Is(err, tc.wantErr) { + t.Fatalf("Reconcile(%v) = %v, want %v", req, err, tc.wantErr) + } + + got, found := policyFrom(ctx, t, r, tc.source) + if !found { + t.Fatalf("Reconcile(%v) removed the foreign policy, want it left in place", req) + } + gotSpec := got.(*kfplacementv1alpha1.PlacementPolicy).Spec + if diff := cmp.Diff(gotSpec, foreignSpec()); diff != "" { + t.Errorf("Reconcile(%v) changed the foreign policy's spec (-got, +want):\n%s", req, diff) + } + if labels := got.GetLabels(); len(labels) != 0 { + t.Errorf("Reconcile(%v) added labels %v to the foreign policy, want it left untouched", req, labels) + } + if diff := cmp.Diff(recordedReasons(recorder), tc.wantReasons, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + }) + } +} + +// TestReconcileRepairsDriftedProvenance covers a policy this controller generated whose provenance +// labels were later edited away. Its owner reference still identifies it as ours, so it must be +// repaired while the annotation stands and deleted once the annotation is removed -- never stranded +// as though it were foreign, which would leave its placement running with nothing able to reconcile +// or remove it. +func TestReconcileRepairsDriftedProvenance(t *testing.T) { + annotated := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + selectors := mustParse(t, oneSelector) + // A generated policy whose provenance labels have drifted -- one label is gone -- while the owner + // reference this controller set is untouched, so the policy is still recognizable as ours. + drifted := func() client.Object { + policy := desiredPolicy(annotated, selectors) + labels := policy.GetLabels() + delete(labels, kfplacementv1alpha1.ParentKindLabel) + policy.SetLabels(labels) + return policy + } + + testCases := []struct { + name string + source *unstructured.Unstructured + wantPolicy bool + wantReason string + }{ + { + name: "the annotation stands, so the drifted labels are repaired", + source: annotated, + wantPolicy: true, + wantReason: EventReasonPolicyUpdated, + }, + { + name: "the annotation is gone, so the policy is deleted", + source: newSource(deploymentGVK, testNamespace, testName, nil), + wantPolicy: false, + wantReason: EventReasonPolicyDeleted, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + r, recorder := newReconciler(t, interceptor.Funcs{}, tc.source, drifted()) + + req := RequestFor(tc.source) + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error", req, err) + } + + got, found := policyFrom(ctx, t, r, annotated) + if found != tc.wantPolicy { + t.Fatalf("Reconcile(%v) left a generated policy = %v, want %v", req, found, tc.wantPolicy) + } + if tc.wantPolicy { + if diff := cmp.Diff(got.GetLabels(), desiredPolicy(annotated, selectors).GetLabels()); diff != "" { + t.Errorf("Reconcile(%v) did not restore the drifted provenance labels (-got, +want):\n%s", req, diff) + } + } + if diff := cmp.Diff(recordedReasons(recorder), []string{tc.wantReason}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + }) + } +} + +// TestReconcileResumesOnceForeignPolicyClears covers a source whose generated name is occupied by a +// foreign policy. Removing that policy fires no event that reaches the source -- it carries no owner +// reference to it -- and the annotation does not change, so the reconciler reports the conflict as +// an error to be retried with backoff; the pass after the conflict clears must then create the +// requested policy. +func TestReconcileResumesOnceForeignPolicyClears(t *testing.T) { + ctx := context.Background() + source := newSource(deploymentGVK, testNamespace, testName, map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}) + foreign := &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: generatedPolicyName(deploymentGVK, testNamespace, testName), + Namespace: testNamespace, + }, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{APIGroup: "example.com", APIVersion: "v1", Kind: "Widget", Name: "hand-authored"}}, + }, + } + r, recorder := newReconciler(t, interceptor.Funcs{}, source, foreign) + + req := RequestFor(source) + if _, err := r.Reconcile(ctx, req); !errors.Is(err, errForeignPolicy) { + t.Fatalf("Reconcile(%v) = %v, want %v so the blocked source is retried", req, err, errForeignPolicy) + } + if diff := cmp.Diff(recordedReasons(recorder), []string{EventReasonPolicyConflict}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } + + // The user removes the blocking policy. The next pass creates the requested policy. + if err := r.Delete(ctx, foreign); err != nil { + t.Fatalf("Delete(foreign) = %v, want no error", err) + } + if _, err := r.Reconcile(ctx, req); err != nil { + t.Fatalf("Reconcile(%v) = %v, want no error after the conflict cleared", req, err) + } + if _, found := policyFrom(ctx, t, r, source); !found { + t.Errorf("Reconcile(%v) did not create the policy after the conflict cleared", req) + } + if diff := cmp.Diff(recordedReasons(recorder), []string{EventReasonPolicyCreated}); diff != "" { + t.Errorf("Reconcile(%v) recorded events mismatch (-got, +want):\n%s", req, diff) + } +} + +func TestHasClusterSelectorsAnnotation(t *testing.T) { + testCases := []struct { + name string + annotations map[string]string + want bool + }{ + {name: "no annotations at all", annotations: nil, want: false}, + {name: "other annotations only", annotations: map[string]string{"example.com/other": "value"}, want: false}, + {name: "the annotation is present", annotations: map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: oneSelector}, want: true}, + // An empty value is still a request for annotation-based placement; the parser is what + // rejects it, and it can only do that if the resource reaches the queue at all. + {name: "the annotation is present but empty", annotations: map[string]string{kfplacementv1alpha1.ClusterSelectorsAnnotation: ""}, want: true}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + source := newSource(deploymentGVK, testNamespace, testName, tc.annotations) + if got := HasClusterSelectorsAnnotation(source); got != tc.want { + t.Errorf("HasClusterSelectorsAnnotation(%v) = %v, want %v", tc.annotations, got, tc.want) + } + }) + } +} diff --git a/pkg/v1/controllers/annotationplacement/denylist.go b/pkg/v1/controllers/annotationplacement/denylist.go new file mode 100644 index 000000000..0b4373e1a --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/denylist.go @@ -0,0 +1,38 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + admissionregistrationv1 "k8s.io/api/admissionregistration/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" +) + +// isObjectBlockedForAnnotationBasedPlacement blocks admission policies and their bindings in +// every API version. They can still be placed through explicitly authored placement policies. +func isObjectBlockedForAnnotationBasedPlacement(obj *unstructured.Unstructured) bool { + gvk := obj.GroupVersionKind() + if gvk.Group != admissionregistrationv1.GroupName { + return false + } + switch gvk.Kind { + case "ValidatingAdmissionPolicy", "ValidatingAdmissionPolicyBinding", + "MutatingAdmissionPolicy", "MutatingAdmissionPolicyBinding": + return true + default: + return false + } +} diff --git a/pkg/v1/controllers/annotationplacement/denylist_test.go b/pkg/v1/controllers/annotationplacement/denylist_test.go new file mode 100644 index 000000000..70e5e5388 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/denylist_test.go @@ -0,0 +1,53 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "testing" + + admissionregistrationv1 "k8s.io/api/admissionregistration/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" +) + +func TestIsObjectBlockedForAnnotationBasedPlacement(t *testing.T) { + for _, kind := range []string{ + "ValidatingAdmissionPolicy", + "ValidatingAdmissionPolicyBinding", + "MutatingAdmissionPolicy", + "MutatingAdmissionPolicyBinding", + "ValidatingWebhookConfiguration", + "MutatingWebhookConfiguration", + "Deployment", + } { + for _, group := range []string{admissionregistrationv1.GroupName, "example.com", ""} { + for _, version := range []string{"v1alpha1", "v1beta1", "v1"} { + gvk := schema.GroupVersionKind{Group: group, Version: version, Kind: kind} + t.Run(gvk.String(), func(t *testing.T) { + obj := &unstructured.Unstructured{} + obj.SetGroupVersionKind(gvk) + want := group == admissionregistrationv1.GroupName && + (kind == "ValidatingAdmissionPolicy" || kind == "ValidatingAdmissionPolicyBinding" || + kind == "MutatingAdmissionPolicy" || kind == "MutatingAdmissionPolicyBinding") + if got := isObjectBlockedForAnnotationBasedPlacement(obj); got != want { + t.Errorf("isObjectBlockedForAnnotationBasedPlacement(%v) = %v, want %v", gvk, got, want) + } + }) + } + } + } +} diff --git a/pkg/v1/controllers/annotationplacement/manager_integration_test.go b/pkg/v1/controllers/annotationplacement/manager_integration_test.go new file mode 100644 index 000000000..56ccb7bd0 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/manager_integration_test.go @@ -0,0 +1,163 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/tools/events" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// These specs run the controller the way the hub agent would: registered with a manager, fed +// annotated resources through the channel the resource watcher writes to, and reading generated +// policies from the manager's cache. They are what proves the watch on the generated policies -- +// the reconciler's own tests call Reconcile directly and never see a policy event. +var _ = Describe("annotation based placement under a manager", Ordered, func() { + var ( + managerCtx context.Context + stopManager context.CancelFunc + managerDone chan struct{} + sourceQueue *SourceQueue + configMap *corev1.ConfigMap + managedEvents *events.FakeRecorder + ) + + // report hands the resource watcher's view of a resource to the controller, as the hub agent's + // change detector would on every event for an annotated resource. + report := func(object client.Object) { + source := &unstructured.Unstructured{} + content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(object) + Expect(err).Should(Succeed()) + source.SetUnstructuredContent(content) + source.SetGroupVersionKind(configMapGVK) + sourceQueue.Add(source) + } + + managedPolicy := func() (kfplacementv1alpha1.PlacementPolicyAccessor, error) { + policy := &kfplacementv1alpha1.PlacementPolicy{} + key := client.ObjectKey{Namespace: configMap.Namespace, Name: generatedPolicyName(configMapGVK, configMap.Namespace, configMap.Name)} + return policy, hubClient.Get(ctx, key, policy) + } + + BeforeAll(func() { + managerCtx, stopManager = context.WithCancel(ctx) + mgr, err := ctrl.NewManager(restConfig, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{BindAddress: "0"}, + }) + Expect(err).Should(Succeed()) + + sourceQueue = &SourceQueue{} + managedEvents = events.NewFakeRecorder(100) + managed := &Reconciler{ + Client: mgr.GetClient(), + APIReader: mgr.GetAPIReader(), + RESTMapper: mgr.GetRESTMapper(), + Recorder: managedEvents, + } + Expect(managed.SetupWithManager(mgr, sourceQueue)).Should(Succeed()) + managerDone = make(chan struct{}) + go func() { + defer GinkgoRecover() + defer close(managerDone) + Expect(mgr.Start(managerCtx)).Should(Succeed()) + }() + + configMapCount++ + configMap = &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf("managed-%d", configMapCount), + Namespace: "default", + Annotations: map[string]string{ + kfplacementv1alpha1.ClusterSelectorsAnnotation: "env=staging,count=All", + }, + }, + } + Expect(hubClient.Create(ctx, configMap)).Should(Succeed()) + }) + + AfterAll(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, configMap))).Should(Succeed()) + stopManager() + // Start's own assertion must land while this spec tree is still running. + Eventually(managerDone, eventuallyTimeout).Should(BeClosed()) + }) + + It("should generate a policy for a resource the watcher reports", func() { + report(configMap) + Eventually(func() error { + _, err := managedPolicy() + return err + }, eventuallyTimeout, eventuallyInterval).Should(Succeed()) + }) + + It("should restore a generated policy someone edits, with no event on the resource", func() { + policy, err := managedPolicy() + Expect(err).Should(Succeed()) + policy.GetSpec().ClusterSelectors = nil + Expect(hubClient.Update(ctx, policy)).Should(Succeed()) + + Eventually(func() ([]kfplacementv1alpha1.ClusterSelector, error) { + policy, err := managedPolicy() + if err != nil { + return nil, err + } + return policy.GetSpec().ClusterSelectors, nil + }, eventuallyTimeout, eventuallyInterval).Should(HaveLen(1), "the watch on generated policies must bring an edited policy back") + }) + + It("should recreate a generated policy someone deletes, with no event on the resource", func() { + policy, err := managedPolicy() + Expect(err).Should(Succeed()) + deletedUID := policy.GetUID() + Expect(hubClient.Delete(ctx, policy)).Should(Succeed()) + + Eventually(func() (bool, error) { + policy, err := managedPolicy() + if err != nil { + return false, err + } + return policy.GetUID() != deletedUID, nil + }, eventuallyTimeout, eventuallyInterval).Should(BeTrue(), "the watch on generated policies must bring a deleted policy back") + }) + + It("should delete the policy once the watcher reports the annotation gone", func() { + Expect(hubClient.Get(ctx, client.ObjectKeyFromObject(configMap), configMap)).Should(Succeed()) + configMap.Annotations = nil + Expect(hubClient.Update(ctx, configMap)).Should(Succeed()) + report(configMap) + + Eventually(func() bool { + _, err := managedPolicy() + return apierrors.IsNotFound(err) + }, eventuallyTimeout, eventuallyInterval).Should(BeTrue()) + }) +}) diff --git a/pkg/v1/controllers/annotationplacement/parser.go b/pkg/v1/controllers/annotationplacement/parser.go new file mode 100644 index 000000000..98085f1e6 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/parser.go @@ -0,0 +1,230 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package annotationplacement implements FEP-0001 annotation-based placement: it keeps a placement +// policy object in sync with the kubefleet.dev/cluster-selectors annotation set on a Kubernetes +// resource, so that simple placement scenarios can be expressed without authoring a placement +// policy by hand. +package annotationplacement + +import ( + "errors" + "fmt" + "strconv" + "strings" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/validation" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + kferrors "go.goms.io/fleet/pkg/utils/errors" +) + +const ( + // selectorSeparator separates the cluster selectors within the annotation value. + selectorSeparator = ";" + // matcherSeparator separates the label matchers, and the count directive, within one selector. + matcherSeparator = "," + // keyValueSeparator separates a label matcher's key from its value. + keyValueSeparator = "=" + + // countKey is the reserved matcher key that sets the desired cluster count of a selector. + countKey = "count" + // countAll is the count value that selects every cluster matching the selector's terms. + countAll = "All" + + // regionShorthand and aliasShorthand are the label key shorthands FEP-0001 reserves. + regionShorthand = "region" + aliasShorthand = "alias" +) + +// The limits below mirror the validation markers on the placement policy API, so that a rejected +// annotation is reported against the annotation itself rather than surfacing later as an opaque +// failure to create the policy object. +const ( + // maxSelectors mirrors MaxItems on PlacementPolicySpec.ClusterSelectors. + maxSelectors = 10 + // maxMatchLabels mirrors MaxProperties on ClusterLabelAndPropertySelectorTerm.MatchLabels. + maxMatchLabels = 10 + // maxCount mirrors the ceiling on ClusterSelector.Count, which the API enforces once per form: + // the Pattern marker bounds the string form, and a CEL rule bounds the integer form, since a + // pattern does not constrain the integer side of an int-or-string. + maxCount = 999 +) + +// defaultCount is the desired cluster count of a selector that does not carry a count directive. It +// matches the API's own default for the field, and is set explicitly rather than left to the API +// server so that the policy object built from an annotation compares equal to the one the API +// server stores; otherwise every reconciliation would observe a difference and issue an update. +const defaultCount = 1 + +// shorthandLabelKeys maps the label key shorthands reserved by FEP-0001 to the keys they expand to. +var shorthandLabelKeys = map[string]string{ + regionShorthand: corev1.LabelTopologyRegion, + aliasShorthand: placementv1beta1.ClusterAliasLabel, +} + +// parseClusterSelectors converts the value of the kubefleet.dev/cluster-selectors annotation into +// the cluster selectors of a placement policy. +// +// Errors returned by this function are always caused by the annotation's own contents, so they are +// categorized as user errors: they are never transient, and a caller should surface them to the +// user rather than retry. The message stays human-readable, since it is what the event recorded on +// the annotated resource shows: the category is for callers, the text is for whoever wrote the +// annotation. +func parseClusterSelectors(value string) ([]kfplacementv1alpha1.ClusterSelector, error) { + if strings.TrimSpace(value) == "" { + return nil, kferrors.NewUserError(nil, "the annotation value is empty; it must list at least one cluster selector") + } + + segments := strings.Split(value, selectorSeparator) + if len(segments) > maxSelectors { + return nil, kferrors.NewUserError(nil, fmt.Sprintf("the annotation lists %d cluster selectors, more than the supported maximum of %d", len(segments), maxSelectors)) + } + + selectors := make([]kfplacementv1alpha1.ClusterSelector, 0, len(segments)) + for idx, segment := range segments { + selector, err := parseSelector(segment) + if err != nil { + // The index is 1-based: it is read by whoever wrote the annotation, who counts the + // selectors in it by eye. + return nil, kferrors.NewUserError(err, fmt.Sprintf("cluster selector %d (%q) is invalid", idx+1, strings.TrimSpace(segment))) + } + selectors = append(selectors, selector) + } + return selectors, nil +} + +// parseSelector converts one semicolon-delimited segment of the annotation into a cluster selector. +func parseSelector(segment string) (kfplacementv1alpha1.ClusterSelector, error) { + var selector kfplacementv1alpha1.ClusterSelector + + matchLabels := make(map[string]string) + var count *intstr.IntOrString + + for _, matcher := range strings.Split(segment, matcherSeparator) { + matcher = strings.TrimSpace(matcher) + if matcher == "" { + return selector, fmt.Errorf("it has an empty label matcher") + } + + rawKey, rawValue, found := strings.Cut(matcher, keyValueSeparator) + if !found { + return selector, fmt.Errorf("the label matcher %q is not in the KEY=VALUE format", matcher) + } + rawKey, rawValue = strings.TrimSpace(rawKey), strings.TrimSpace(rawValue) + + if rawKey == countKey { + if count != nil { + return selector, fmt.Errorf("it sets %s more than once", countKey) + } + parsed, err := parseCount(rawValue) + if err != nil { + return selector, err + } + count = parsed + continue + } + + key, err := parseLabelKey(rawKey) + if err != nil { + return selector, err + } + if _, duplicate := matchLabels[key]; duplicate { + return selector, fmt.Errorf("it matches on the label key %q more than once", key) + } + if err := validateLabelValue(key, rawValue); err != nil { + return selector, err + } + matchLabels[key] = rawValue + } + + if len(matchLabels) > maxMatchLabels { + return selector, fmt.Errorf("it matches on %d label keys, more than the supported maximum of %d", len(matchLabels), maxMatchLabels) + } + + if count == nil { + fallback := intstr.FromInt32(defaultCount) + count = &fallback + } + selector.Count = count + // A selector that carries only a count directive has no terms, which the API reads as a match on + // every cluster; this is deliberate, as it lets `count=All` on its own select the whole fleet. + if len(matchLabels) > 0 { + selector.Terms = []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: matchLabels}, + } + } + return selector, nil +} + +// parseCount converts the value of a count directive into the desired cluster count of a selector. +func parseCount(value string) (*intstr.IntOrString, error) { + if value == countAll { + all := intstr.FromString(countAll) + return &all, nil + } + if strings.EqualFold(value, countAll) { + return nil, fmt.Errorf("the %s value %q is not recognized; it must be spelled exactly %q", countKey, value, countAll) + } + + // ParseUint accepts exactly a run of decimal digits: no sign, no whitespace, no exponent. + parsed, err := strconv.ParseUint(value, 10, 32) + switch { + case errors.Is(err, strconv.ErrRange): + return nil, fmt.Errorf("the %s value %q is out of the supported range (1-%d)", countKey, value, maxCount) + case err != nil: + return nil, fmt.Errorf("the %s value %q is neither a positive integer nor %q", countKey, value, countAll) + } + if parsed < 1 || parsed > maxCount { + return nil, fmt.Errorf("the %s value %d is out of the supported range (1-%d)", countKey, parsed, maxCount) + } + + count := intstr.FromInt32(int32(parsed)) + return &count, nil +} + +// parseLabelKey expands a label key shorthand, if one is used, and verifies that the result is a +// key that Kubernetes accepts. +func parseLabelKey(key string) (string, error) { + if expanded, isShorthand := shorthandLabelKeys[key]; isShorthand { + return expanded, nil + } + if key == "" { + return "", fmt.Errorf("it has a label matcher with an empty label key") + } + if errs := validation.IsQualifiedName(key); len(errs) > 0 { + return "", fmt.Errorf("the label key %q is not valid: %s", key, strings.Join(errs, "; ")) + } + return key, nil +} + +// validateLabelValue verifies that a label matcher's value is one that Kubernetes accepts. +// +// Kubernetes considers the empty string a valid label value, but this surface rejects it: an empty +// value in a shorthand annotation is far more often a typo than a deliberate match on a label set +// to the empty string, and the placement policy API remains available for the latter. +func validateLabelValue(key, value string) error { + if value == "" { + return fmt.Errorf("the label key %q is matched against an empty value; use a placement policy object to match on an empty label value", key) + } + if errs := validation.IsValidLabelValue(value); len(errs) > 0 { + return fmt.Errorf("the value %q of the label key %q is not valid: %s", value, key, strings.Join(errs, "; ")) + } + return nil +} diff --git a/pkg/v1/controllers/annotationplacement/parser_test.go b/pkg/v1/controllers/annotationplacement/parser_test.go new file mode 100644 index 000000000..16a83f14c --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/parser_test.go @@ -0,0 +1,485 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "fmt" + "os" + "regexp" + "strconv" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/validation" + "sigs.k8s.io/yaml" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + regionLabel = "topology.kubernetes.io/region" + aliasLabel = "kubefleet.dev/cluster-alias" + + // qualifiedNameMaxLength is the longest name segment of a label key that Kubernetes accepts. + // It mirrors the unexported limit that validation.IsQualifiedName enforces. + qualifiedNameMaxLength = 63 + + // crdPath is the placement policy CRD, read by TestLimitsMatchAPIValidation. + crdPath = "../../../../config/crd/bases/placement.kubefleet.dev_placementpolicies.yaml" +) + +// selectorOf builds the cluster selector that the parser is expected to produce for one segment of +// the annotation; a nil matchLabels stands for a selector that carries no terms. +func selectorOf(count intstr.IntOrString, matchLabels map[string]string) kfplacementv1alpha1.ClusterSelector { + selector := kfplacementv1alpha1.ClusterSelector{Count: &count} + if matchLabels != nil { + selector.Terms = []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: matchLabels}, + } + } + return selector +} + +func TestParseClusterSelectors(t *testing.T) { + one, three, all := intstr.FromInt32(1), intstr.FromInt32(3), intstr.FromString("All") + + testCases := []struct { + name string + value string + want []kfplacementv1alpha1.ClusterSelector + }{ + { + name: "single label matcher with the region shorthand", + value: "region=eastus", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{regionLabel: "eastus"}), + }, + }, + { + // The first example given in FEP-0001. + name: "one cluster per region", + value: "region=eastus;region=westus;region=centralus", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{regionLabel: "eastus"}), + selectorOf(one, map[string]string{regionLabel: "westus"}), + selectorOf(one, map[string]string{regionLabel: "centralus"}), + }, + }, + { + // The second example given in FEP-0001. + name: "named clusters through the alias shorthand", + value: "alias=bravelion;alias=smartfish", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{aliasLabel: "bravelion"}), + selectorOf(one, map[string]string{aliasLabel: "smartfish"}), + }, + }, + { + // The third example given in FEP-0001. + name: "mixed counts and multiple matchers per selector", + value: "env=staging,count=All;env=canary,region=eastus,count=1", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(all, map[string]string{"env": "staging"}), + selectorOf(one, map[string]string{"env": "canary", regionLabel: "eastus"}), + }, + }, + { + name: "count on its own selects the whole fleet", + value: "count=All", + want: []kfplacementv1alpha1.ClusterSelector{selectorOf(all, nil)}, + }, + { + name: "count directive may precede the label matchers", + value: "count=3,env=prod", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(three, map[string]string{"env": "prod"}), + }, + }, + { + name: "fully qualified label keys are passed through", + value: fmt.Sprintf("%s=eastus", regionLabel), + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{regionLabel: "eastus"}), + }, + }, + { + // Shorthands are expanded on an exact match only; Kubernetes label keys are + // case-sensitive, so a differently cased key is a different label. + name: "shorthand expansion is case-sensitive", + value: "Region=eastus", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{"Region": "eastus"}), + }, + }, + { + name: "surrounding whitespace is ignored", + value: " region = eastus , count = 3 ; env=prod ", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(three, map[string]string{regionLabel: "eastus"}), + selectorOf(one, map[string]string{"env": "prod"}), + }, + }, + { + name: "count at the top of the supported range", + value: fmt.Sprintf("env=prod,count=%d", maxCount), + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(intstr.FromInt32(maxCount), map[string]string{"env": "prod"}), + }, + }, + { + name: "label value at the longest length Kubernetes accepts", + value: "env=" + strings.Repeat("a", validation.LabelValueMaxLength), + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{"env": strings.Repeat("a", validation.LabelValueMaxLength)}), + }, + }, + { + name: "label matchers at the supported maximum", + value: func() string { + matchers := make([]string, 0, maxMatchLabels) + for i := 0; i < maxMatchLabels; i++ { + matchers = append(matchers, fmt.Sprintf("key%d=value", i)) + } + return strings.Join(matchers, matcherSeparator) + }(), + want: func() []kfplacementv1alpha1.ClusterSelector { + matchLabels := make(map[string]string, maxMatchLabels) + for i := 0; i < maxMatchLabels; i++ { + matchLabels[fmt.Sprintf("key%d", i)] = "value" + } + return []kfplacementv1alpha1.ClusterSelector{selectorOf(one, matchLabels)} + }(), + }, + { + // The count directive is reserved on an exact match only, like the shorthands above. + name: "the count keyword is case-sensitive", + value: "Count=5", + want: []kfplacementv1alpha1.ClusterSelector{ + selectorOf(one, map[string]string{"Count": "5"}), + }, + }, + { + name: "selector count at the supported maximum", + value: strings.TrimSuffix(strings.Repeat("region=eastus;", maxSelectors), ";"), + want: func() []kfplacementv1alpha1.ClusterSelector { + selectors := make([]kfplacementv1alpha1.ClusterSelector, 0, maxSelectors) + for i := 0; i < maxSelectors; i++ { + selectors = append(selectors, selectorOf(one, map[string]string{regionLabel: "eastus"})) + } + return selectors + }(), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := parseClusterSelectors(tc.value) + if err != nil { + t.Fatalf("parseClusterSelectors(%q) = %v, want no error", tc.value, err) + } + if diff := cmp.Diff(got, tc.want); diff != "" { + t.Errorf("parseClusterSelectors(%q) mismatch (-got, +want):\n%s", tc.value, diff) + } + }) + } +} + +// formatLimit renders a CRD size limit, which is absent as often as it is set. +func formatLimit(limit *int64) string { + if limit == nil { + return "no limit" + } + return strconv.FormatInt(*limit, 10) +} + +// TestLimitsMatchAPIValidation ties the parser's limits to the validation markers on the placement +// policy CRD. +// +// The parser turns an annotation that exceeds a limit into an error reported against the annotation +// itself. Were the API to tighten a limit on its own, the parser would instead keep building policy +// objects that the API server rejects, and the failure would surface as an endless retry far away +// from the annotation that caused it. +func TestLimitsMatchAPIValidation(t *testing.T) { + rawCRD, err := os.ReadFile(crdPath) + if err != nil { + t.Fatalf("ReadFile(%q) = %v, want no error", crdPath, err) + } + var crd apiextensionsv1.CustomResourceDefinition + if err := yaml.Unmarshal(rawCRD, &crd); err != nil { + t.Fatalf("Unmarshal(%q) = %v, want no error", crdPath, err) + } + + var schema *apiextensionsv1.JSONSchemaProps + for idx := range crd.Spec.Versions { + if crd.Spec.Versions[idx].Name == kfplacementv1alpha1.GroupVersion.Version { + schema = crd.Spec.Versions[idx].Schema.OpenAPIV3Schema + break + } + } + if schema == nil { + t.Fatalf("the %s CRD has no %s version", crd.Name, kfplacementv1alpha1.GroupVersion.Version) + } + + clusterSelectors := schema.Properties["spec"].Properties["clusterSelectors"] + if got := clusterSelectors.MaxItems; got == nil || *got != maxSelectors { + t.Errorf("the CRD caps clusterSelectors at %s, want maxSelectors (%d)", formatLimit(got), maxSelectors) + } + // Every step below walks into the schema, so a renamed or restructured field must stop the test + // with a message that names the missing step rather than panic on a nil pointer. + if clusterSelectors.Items == nil || clusterSelectors.Items.Schema == nil { + t.Fatalf("the CRD schema of clusterSelectors carries no item schema") + } + + selector := clusterSelectors.Items.Schema.Properties + terms := selector["terms"] + if terms.Items == nil || terms.Items.Schema == nil { + t.Fatalf("the CRD schema of the selector terms carries no item schema") + } + matchLabels := terms.Items.Schema.Properties["matchLabels"] + if got := matchLabels.MaxProperties; got == nil || *got != maxMatchLabels { + t.Errorf("the CRD caps matchLabels at %s, want maxMatchLabels (%d)", formatLimit(got), maxMatchLabels) + } + + // The count field is an int-or-string, and the API bounds each form separately: a pattern for + // the string form, checked here by exercising it, and a CEL rule for the integer form, checked + // here by asserting the rule spells out the same bounds the parser enforces. A pattern alone + // would leave the integer form unbounded. + count := selector["count"] + countPattern, err := regexp.Compile(count.Pattern) + if err != nil { + t.Fatalf("Compile(%q) = %v, want no error", count.Pattern, err) + } + if got := strconv.Itoa(maxCount); !countPattern.MatchString(got) { + t.Errorf("the CRD pattern %q rejects maxCount (%s), want it accepted", count.Pattern, got) + } + if got := strconv.Itoa(maxCount + 1); countPattern.MatchString(got) { + t.Errorf("the CRD pattern %q accepts %s, want maxCount (%d) to be the ceiling", count.Pattern, got, maxCount) + } + if !countPattern.MatchString(countAll) { + t.Errorf("the CRD pattern %q rejects %q, want it accepted", count.Pattern, countAll) + } + + // The rule's spelling is free to change; what must not drift is the ceiling it encodes, in + // both the rule and the message a user is shown. The behavior itself is pinned separately by + // the integration suite, which submits out-of-range counts to a real API server. + wantCeiling := fmt.Sprintf("%d", maxCount) + integerRuleFound := false + for _, validation := range count.XValidations { + if !strings.Contains(validation.Rule, wantCeiling) { + continue + } + integerRuleFound = true + if !strings.Contains(validation.Message, wantCeiling) { + t.Errorf("the CRD rejects an out-of-range count with the message %q, want it to name the ceiling (%s)", validation.Message, wantCeiling) + } + } + if !integerRuleFound { + rules := make([]string, 0, len(count.XValidations)) + for _, validation := range count.XValidations { + rules = append(rules, validation.Rule) + } + t.Errorf("the CRD validates count with the rules %q, want one to bound the integer form at maxCount (%s)", rules, wantCeiling) + } + + if count.Default == nil { + t.Fatalf("the CRD schema of the selector count carries no default") + } + wantDefault := strconv.Itoa(defaultCount) + if got := string(count.Default.Raw); got != wantDefault { + t.Errorf("the CRD defaults count to %s, want defaultCount (%s)", got, wantDefault) + } +} + +func TestParseClusterSelectorsInvalid(t *testing.T) { + testCases := []struct { + name string + // value is the annotation value under test. + value string + // wantErrContains is a fragment the error message must carry, so that the annotation's + // author can tell which part of the value was rejected and why. + wantErrContains string + }{ + { + name: "empty value", + value: "", + wantErrContains: "empty", + }, + { + name: "whitespace-only value", + value: " ", + wantErrContains: "empty", + }, + { + name: "more selectors than supported", + value: strings.TrimSuffix(strings.Repeat("region=eastus;", maxSelectors+1), ";"), + wantErrContains: "more than the supported maximum", + }, + { + // Selectors are counted from one in the error, since whoever wrote the annotation + // counts them by eye. + name: "empty selector between two others", + value: "region=eastus;;region=westus", + wantErrContains: "cluster selector 2", + }, + { + name: "empty selector reports the reason as well as the position", + value: "region=eastus;;region=westus", + wantErrContains: "empty label matcher", + }, + { + name: "trailing separator leaves an empty selector", + value: "region=eastus;", + wantErrContains: "empty label matcher", + }, + { + name: "trailing matcher separator", + value: "region=eastus,", + wantErrContains: "empty label matcher", + }, + { + name: "matcher without a value", + value: "region", + wantErrContains: "KEY=VALUE", + }, + { + name: "matcher with an empty key", + value: "=eastus", + wantErrContains: "empty label key", + }, + { + name: "label key that Kubernetes rejects", + value: "not a key=eastus", + wantErrContains: "is not valid", + }, + { + name: "label value that Kubernetes rejects", + value: "env=not a value", + wantErrContains: "is not valid", + }, + { + name: "label matched against an empty value", + value: "env=", + wantErrContains: "empty value", + }, + { + // A value one byte past what Kubernetes accepts. Left unchecked, this would become a + // policy object that the API server rejects, turning a typo into an endless retry. + name: "label value one byte too long", + value: "env=" + strings.Repeat("a", validation.LabelValueMaxLength+1), + wantErrContains: "is not valid", + }, + { + name: "label key one byte too long", + value: strings.Repeat("k", qualifiedNameMaxLength+1) + "=prod", + wantErrContains: "is not valid", + }, + { + // strings.Cut splits on the first separator only, so the value keeps the rest; it is + // then rejected, since a label value cannot contain a separator. + name: "value containing the key-value separator", + value: "env=prod=east", + wantErrContains: "is not valid", + }, + { + name: "same label key matched twice", + value: "env=staging,env=canary", + wantErrContains: "more than once", + }, + { + // The shorthand and the key it expands to are the same label, which would otherwise + // silently drop one of the two matchers. + name: "shorthand collides with its expanded key", + value: fmt.Sprintf("region=eastus,%s=westus", regionLabel), + wantErrContains: "more than once", + }, + { + name: "count set twice", + value: "env=prod,count=1,count=2", + wantErrContains: "more than once", + }, + { + name: "count of zero", + value: "env=prod,count=0", + wantErrContains: "out of the supported range", + }, + { + name: "count above the supported range", + value: fmt.Sprintf("env=prod,count=%d", maxCount+1), + wantErrContains: "out of the supported range", + }, + { + // All digits, but too large to hold; this must be reported like any other out-of-range + // count rather than escaping as a conversion failure. + name: "count too large to be represented", + value: "env=prod,count=99999999999999999999", + wantErrContains: "out of the supported range", + }, + { + name: "negative count", + value: "env=prod,count=-1", + wantErrContains: "neither a positive integer", + }, + { + name: "explicitly signed count", + value: "env=prod,count=+5", + wantErrContains: "neither a positive integer", + }, + { + name: "count that is not a number", + value: "env=prod,count=many", + wantErrContains: "neither a positive integer", + }, + { + // A near miss on the All sentinel is worth its own message; the API accepts the exact + // spelling only. + name: "lowercased All sentinel", + value: "env=prod,count=all", + wantErrContains: `spelled exactly "All"`, + }, + { + name: "uppercased All sentinel", + value: "env=prod,count=ALL", + wantErrContains: `spelled exactly "All"`, + }, + { + name: "more label matchers than supported in one selector", + value: func() string { + matchers := make([]string, 0, maxMatchLabels+1) + for i := 0; i <= maxMatchLabels; i++ { + matchers = append(matchers, fmt.Sprintf("key%d=value", i)) + } + return strings.Join(matchers, matcherSeparator) + }(), + wantErrContains: "more than the supported maximum", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := parseClusterSelectors(tc.value) + if err == nil { + t.Fatalf("parseClusterSelectors(%q) = %v, want an error", tc.value, got) + } + if !strings.Contains(err.Error(), tc.wantErrContains) { + t.Errorf("parseClusterSelectors(%q) = %v, want an error containing %q", tc.value, err, tc.wantErrContains) + } + }) + } +} diff --git a/pkg/v1/controllers/annotationplacement/policy.go b/pkg/v1/controllers/annotationplacement/policy.go new file mode 100644 index 000000000..9a5fc0bf4 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/policy.go @@ -0,0 +1,258 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "fmt" + "strings" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/naming" +) + +const ( + // maxKindSegmentLength caps how much of the kind appears in a generated name. Every kind a user + // is likely to annotate is far shorter; the cap exists so that a pathologically long custom + // kind cannot crowd out the part of the name that identifies the resource. + maxKindSegmentLength = 40 + + // separatorCount is the number of separators a generated name spends joining its three parts. + separatorCount = 2 +) + +// The values the placement policy API defaults these fields to. +// +// A generated policy sets them explicitly, rather than letting the API server fill them in, so +// that the object built here equals the one the API server stores. Left unset, every field the API +// defaults would read as a difference on every pass and provoke an update that changes nothing. +// +// whenUnfulfilled deliberately follows the API default of requesting a cluster: asking the platform +// for a cluster that does not exist yet is the point of these APIs, and the annotation is meant to +// be the lowest-friction way to do so. The grammar has no slot for the field, so guarding against +// a selector nothing can ever satisfy (a mistyped region, say) is the claim fulfillment layer's +// job -- quota, approval, expiry -- not a softer default here. +const ( + defaultResourceRevisionHistoryLimit int32 = 3 + + defaultWhenUnfulfilled = kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim +) + +// generatedPolicyName derives the name of the placement policy generated for a resource. +// +// The name is deterministic, so that reconciling the same resource twice converges on one policy +// instead of accumulating them, and it is readable, so that the policy can be recognized in a +// listing. Neither property may come at the cost of the third: the hash is taken over the +// resource's whole identity, including the parts that do not appear in the name at all, so two +// resources cannot generate one name even when truncation erases what distinguishes them. +func generatedPolicyName(gvk schema.GroupVersionKind, namespace, name string) string { + // The kind and the name are sanitized before they appear in the name: a resource legal for its + // own API can carry characters a DNS-1123 object name forbids (an RBAC name like + // system:aggregate-to-admin), and copying them through would produce a policy the API server + // rejects, hot-looping the reconciler. The hash below is taken over the unsanitized identity, so + // sanitizing the visible parts costs no uniqueness. + kindSegment := naming.Truncate(naming.Sanitize(gvk.Kind), maxKindSegmentLength) + + // Whatever the kind and the hash do not use is available to the resource's own name. + nameBudget := validation.DNS1123SubdomainMaxLength - len(kindSegment) - naming.HashLength - separatorCount + nameSegment := naming.Truncate(naming.Sanitize(name), nameBudget) + + // Empty segments are dropped rather than joined. An object always has a name, and one read + // from the API server always has a kind, but joining an empty leading segment would put a + // separator at the front of the name, where the API server does not allow one. + segments := make([]string, 0, 3) + for _, segment := range []string{kindSegment, nameSegment} { + if segment != "" { + segments = append(segments, segment) + } + } + identity := fmt.Sprintf("%s/%s/%s/%s", gvk.Group, gvk.Kind, namespace, name) + segments = append(segments, naming.Hash(identity)) + + return strings.Join(segments, "-") +} + +// parentLabels returns the labels recording which resource a generated policy came from. +// +// Every value is shortened to fit a label value, including the API group: a custom resource's +// group is validated as a DNS-1123 subdomain and so may run to 253 bytes, well past what a label +// value holds. A kind cannot exceed the limit today, but it is shortened alongside the others +// rather than relying on a bound that belongs to a different validation rule than this one. +func parentLabels(gvk schema.GroupVersionKind, name string) map[string]string { + return map[string]string{ + kfplacementv1alpha1.ParentAPIGroupLabel: naming.LabelValue(gvk.Group), + kfplacementv1alpha1.ParentKindLabel: naming.LabelValue(gvk.Kind), + kfplacementv1alpha1.ParentNameLabel: naming.LabelValue(name), + } +} + +// isGeneratedFor reports whether a live policy is one this controller generated for the given +// resource identity. It is what stops the controller from overwriting or deleting a policy a user or +// another tool authored at the deterministic name this resource happens to generate. +// +// Ownership is recognized from either marker this controller stamps -- the owner reference back to +// the source, or the provenance labels -- and does not require both. Both are things applyDesiredPolicy +// repairs, so demanding both be intact would let a single edited label or stripped owner reference +// classify one of this controller's own policies as foreign: it would then be denied the very repair +// that would restore the marker and, worse, denied deletion when its annotation is removed, leaving a +// placement running with no way to reconcile or clean it up. Recognizing either marker keeps the +// policy repairable as long as one survives; a foreign policy that merely collides with the name +// carries neither and is still left untouched. Both markers are derived from the resource's stable +// identity rather than its UID, so the judgment holds across a delete and recreate of the resource. +// +// The one residual gap is a policy that loses both markers at once, which is then indistinguishable +// from a foreign object and abandoned. It is reachable without malice -- a tool that rewrites the +// whole object, such as a plain kubectl apply of a hand-kept manifest or a non-server-side-apply +// Update, drops metadata.labels and metadata.ownerReferences together -- but a policy this controller +// generated is not one a user is expected to manage that way, and requiring both markers to survive +// is the accepted price of never touching a policy that is genuinely someone else's. +func isGeneratedFor(policy client.Object, gvk schema.GroupVersionKind, name string, scheme *runtime.Scheme) bool { + // The owner reference is matched on the source's group, kind, and name -- deliberately not on + // its version or its UID, both of which can change while the resource keeps its name: a source + // deleted and recreated comes back with a new UID, and a served version can be retired. + source := &unstructured.Unstructured{} + source.SetGroupVersionKind(gvk) + source.SetName(name) + if owned, err := controllerutil.HasOwnerReference(policy.GetOwnerReferences(), source, scheme); err == nil && owned { + return true + } + + labels := policy.GetLabels() + for key, want := range parentLabels(gvk, name) { + if labels[key] != want { + return false + } + } + return true +} + +// parentOwnerReference returns the owner reference that ties a generated policy to the resource it +// was generated from, so that deleting the resource collects the policy with it. +// +// The reference deliberately claims neither controller nor blocking ownership: setting either +// requires permission to update the finalizers subresource of the owner, and the owner here can be +// a resource of any kind at all. Cascading deletion, the only property this needs, works the same +// without them. +func parentOwnerReference(source *unstructured.Unstructured) metav1.OwnerReference { + gvk := source.GroupVersionKind() + return metav1.OwnerReference{ + APIVersion: gvk.GroupVersion().String(), + Kind: gvk.Kind, + Name: source.GetName(), + UID: source.GetUID(), + } +} + +// parentResourceSelector returns the selector that places the annotated resource itself. +// +// It is the only resource selector a generated policy has: an annotation asks for the resource it is +// written on to be placed, and nothing else. Should that ever stop being true, note that the +// reconciler replaces the whole spec of a generated policy, so a selector added from elsewhere would +// have to be reconciled here rather than merely appended to the live object. +func parentResourceSelector(source *unstructured.Unstructured) kfplacementv1alpha1.ResourceSelector { + gvk := source.GroupVersionKind() + // The namespace is deliberately left unset. A generated policy for a namespaced resource lives + // in that resource's own namespace, where the API requires the selector's namespace to be + // empty or identical; a generated policy for a cluster-scoped resource has no namespace to name. + return kfplacementv1alpha1.ResourceSelector{ + APIGroup: gvk.Group, + APIVersion: gvk.Version, + Kind: gvk.Kind, + Name: source.GetName(), + } +} + +// withAPIDefaults returns the selectors with every field the API defaults set explicitly, leaving +// the caller's own slice untouched. +func withAPIDefaults(selectors []kfplacementv1alpha1.ClusterSelector) []kfplacementv1alpha1.ClusterSelector { + if selectors == nil { + return nil + } + defaulted := make([]kfplacementv1alpha1.ClusterSelector, len(selectors)) + for i, selector := range selectors { + selector.DeepCopyInto(&defaulted[i]) + if defaulted[i].WhenUnfulfilled == "" { + defaulted[i].WhenUnfulfilled = defaultWhenUnfulfilled + } + } + return defaulted +} + +// desiredPolicy builds the placement policy that should exist for an annotated resource. +// +// The policy's scope follows the resource's own: a namespaced resource yields a PlacementPolicy in +// its namespace, and a cluster-scoped resource yields a ClusterPlacementPolicy. That is what keeps +// the owner reference valid, since a cluster-scoped object owned by a namespaced one is accepted +// on creation and then never collected at all. +// +// Scope is read from the resource's own namespace, which the API server has already reconciled +// with the scope its kind is registered under; the source must therefore be an object read from +// the API server rather than one assembled in memory, which is also what guarantees it carries the +// UID and the kind that the owner reference and the generated name are built from. Taking an +// unstructured object rather than a client.Object is deliberate for the same reason: a typed +// object routinely arrives with an empty kind, which would silently change the generated name. +func desiredPolicy(source *unstructured.Unstructured, selectors []kfplacementv1alpha1.ClusterSelector) kfplacementv1alpha1.PlacementPolicyAccessor { + gvk := source.GroupVersionKind() + namespace := source.GetNamespace() + + policy := emptyPolicyForScope(namespace) + policy.SetName(generatedPolicyName(gvk, namespace, source.GetName())) + policy.SetNamespace(namespace) + policy.SetLabels(parentLabels(gvk, source.GetName())) + policy.SetOwnerReferences([]metav1.OwnerReference{parentOwnerReference(source)}) + *policy.GetSpec() = kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: withAPIDefaults(selectors), + // Only the annotated resource is selected; nothing it depends on. Selecting dependencies + // is tracked in #927, where the questions are which references count, whether following + // them should be opt-in, and how the closure stays fresh as the source changes. + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{parentResourceSelector(source)}, + ResourceRevisionHistoryLimit: ptr.To(defaultResourceRevisionHistoryLimit), + } + return policy +} + +// emptyPolicyForScope returns an empty generated policy of the scope that a resource in the given +// namespace generates: a namespaced resource yields a PlacementPolicy in its own namespace, and a +// cluster-scoped resource, whose namespace is empty, yields a ClusterPlacementPolicy. +// +// This is the single place the scope is decided. The reconciler needs the same answer to read and to +// delete a generated policy as it does to build one, and a disagreement between those would leave a +// policy behind rather than fail. +func emptyPolicyForScope(namespace string) kfplacementv1alpha1.PlacementPolicyAccessor { + if namespace == "" { + return &kfplacementv1alpha1.ClusterPlacementPolicy{} + } + return &kfplacementv1alpha1.PlacementPolicy{} +} + +// generatedPolicyKind returns the kind of the policy generated for a resource in the given +// namespace, spelled as the kind itself is, so that a message carrying it can be pasted straight +// into a kubectl command against the right resource. +func generatedPolicyKind(namespace string) string { + if namespace == "" { + return "ClusterPlacementPolicy" + } + return "PlacementPolicy" +} diff --git a/pkg/v1/controllers/annotationplacement/policy_test.go b/pkg/v1/controllers/annotationplacement/policy_test.go new file mode 100644 index 000000000..f2359751e --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/policy_test.go @@ -0,0 +1,463 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "fmt" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/naming" +) + +var ( + deploymentGVK = schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "Deployment"} + namespaceGVK = schema.GroupVersionKind{Group: "", Version: "v1", Kind: "Namespace"} +) + +// sourceObject builds the annotated resource a policy is generated from. +func sourceObject(gvk schema.GroupVersionKind, namespace, name string) *unstructured.Unstructured { + source := &unstructured.Unstructured{} + source.SetGroupVersionKind(gvk) + source.SetNamespace(namespace) + source.SetName(name) + source.SetUID(types.UID("uid-" + name)) + return source +} + +func TestGeneratedPolicyName(t *testing.T) { + testCases := []struct { + name string + gvk schema.GroupVersionKind + namespace string + objName string + want string + }{ + { + name: "namespaced resource", + gvk: deploymentGVK, + namespace: "prod", + objName: "app", + want: "deployment-app-" + hashOfIdentity("apps", "Deployment", "prod", "app"), + }, + { + name: "cluster-scoped resource", + gvk: namespaceGVK, + objName: "work", + want: "namespace-work-" + hashOfIdentity("", "Namespace", "", "work"), + }, + { + // The kind is lowercased so that the name is a legal DNS-1123 subdomain; the hash is + // taken over the kind as it is really spelled, so two kinds that differ only in case + // cannot generate the same name. + name: "kind is lowercased in the name but not in the identity", + gvk: schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: "MyWidget"}, + namespace: "prod", + objName: "gadget", + want: "mywidget-gadget-" + hashOfIdentity("example.com", "MyWidget", "prod", "gadget"), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := generatedPolicyName(tc.gvk, tc.namespace, tc.objName) + if got != tc.want { + t.Errorf("generatedPolicyName(%v, %q, %q) = %v, want %v", tc.gvk, tc.namespace, tc.objName, got, tc.want) + } + }) + } +} + +func hashOfIdentity(group, kind, namespace, name string) string { + return naming.Hash(fmt.Sprintf("%s/%s/%s/%s", group, kind, namespace, name)) +} + +// TestGeneratedPolicyNameValidity covers the inputs where the name has to be shortened to fit, +// which is where a generated name has twice gone wrong in this project: the result must stay a +// legal object name, and it must still tell two resources apart after the part that distinguishes +// them has been cut away. +func TestGeneratedPolicyNameValidity(t *testing.T) { + longKind := "My" + strings.Repeat("Widget", 40) + longName := strings.Repeat("a", validation.DNS1123SubdomainMaxLength) + + testCases := []struct { + name string + gvk schema.GroupVersionKind + namespace string + objName string + }{ + {name: "short everything", gvk: deploymentGVK, namespace: "prod", objName: "app"}, + { + // An RBAC name carries a colon, legal for the resource but not for a Kubernetes object + // name; without sanitizing it the generated name is rejected and the reconciler hot-loops. + name: "rbac name with a colon", + gvk: schema.GroupVersionKind{Group: "rbac.authorization.k8s.io", Version: "v1", Kind: "ClusterRole"}, + objName: "system:aggregate-to-admin", + }, + { + // A dot beside an illegal character is the case that a naive sanitizer keeping dots gets + // wrong: it would leave a label ending in a dash, which the API server rejects. Resource + // names commonly carry dots (domain-like custom resource names), so this is reachable. + name: "name with a dot beside an illegal character", + gvk: deploymentGVK, + objName: "my.:app", + }, + {name: "name at the object limit", gvk: deploymentGVK, namespace: "prod", objName: longName}, + { + name: "name truncated onto a separator", + gvk: deploymentGVK, + namespace: "prod", + objName: strings.Repeat("a", 194) + "-" + strings.Repeat("b", 40), + }, + { + name: "name truncated onto a dot", + gvk: deploymentGVK, + namespace: "prod", + objName: strings.Repeat("a", 194) + "." + strings.Repeat("b", 40), + }, + { + name: "kind longer than its budget", + gvk: schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: longKind}, + namespace: "prod", + objName: longName, + }, + { + // An object read from the API server always carries its kind, but an empty one must + // not put a separator at the front of the generated name, where it would be rejected. + name: "empty kind", + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1"}, + namespace: "prod", + objName: "app", + }, + { + name: "empty kind and empty name", + gvk: schema.GroupVersionKind{Group: "apps", Version: "v1"}, + namespace: "prod", + }, + } + + seen := make(map[string]string, len(testCases)) + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := generatedPolicyName(tc.gvk, tc.namespace, tc.objName) + if errs := validation.IsDNS1123Subdomain(got); len(errs) > 0 { + t.Errorf("generatedPolicyName(%v, %q, %q) = %v, want a valid object name: %s", + tc.gvk, tc.namespace, tc.objName, got, strings.Join(errs, "; ")) + } + if len(got) > validation.DNS1123SubdomainMaxLength { + t.Errorf("len(generatedPolicyName(%v, %q, %q)) = %d, want at most %d", + tc.gvk, tc.namespace, tc.objName, len(got), validation.DNS1123SubdomainMaxLength) + } + if previous, collided := seen[got]; collided { + t.Errorf("generatedPolicyName(%v, %q, %q) = %v, want a name distinct from that of %s", + tc.gvk, tc.namespace, tc.objName, got, previous) + } + seen[got] = tc.name + }) + } +} + +// TestGeneratedPolicyNameDistinguishesIdentities pins the property the hash exists for: resources +// that differ anywhere in their identity, including in parts the readable name never shows, must +// not generate the same policy name. +func TestGeneratedPolicyNameDistinguishesIdentities(t *testing.T) { + longName := strings.Repeat("a", 300) + + identities := []struct { + name string + gvk schema.GroupVersionKind + namespace string + objName string + }{ + {name: "baseline", gvk: deploymentGVK, namespace: "prod", objName: "app"}, + {name: "different namespace", gvk: deploymentGVK, namespace: "staging", objName: "app"}, + {name: "different api group, same kind", gvk: schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: "Deployment"}, namespace: "prod", objName: "app"}, + {name: "kind differing only in case", gvk: schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "DeploymenT"}, namespace: "prod", objName: "app"}, + // Two names that sanitize to the same visible segment -- a dot and a dash both render as a + // dash -- must still generate distinct policy names, since the hash is over the raw identity. + {name: "name with a dot", gvk: deploymentGVK, namespace: "prod", objName: "my.app"}, + {name: "name with a dash where the other has a dot", gvk: deploymentGVK, namespace: "prod", objName: "my-app"}, + // Two names identical up to well past the truncation point. + {name: "long name", gvk: deploymentGVK, namespace: "prod", objName: longName + "one"}, + {name: "long name sharing a prefix", gvk: deploymentGVK, namespace: "prod", objName: longName + "two"}, + } + + seen := make(map[string]string, len(identities)) + for _, identity := range identities { + got := generatedPolicyName(identity.gvk, identity.namespace, identity.objName) + if previous, collided := seen[got]; collided { + t.Errorf("generatedPolicyName for %s = %v, want a name distinct from that of %s", identity.name, got, previous) + } + seen[got] = identity.name + } +} + +// TestIsGeneratedFor pins the check that keeps this controller from commandeering or deleting a +// policy someone else authored at a resource's generated name, while still recognizing its own policy +// after either provenance marker has drifted so that the policy stays repairable and removable. +func TestIsGeneratedFor(t *testing.T) { + source := sourceObject(deploymentGVK, "prod", "app") + mine := desiredPolicy(source, nil) + + // The same policy with its provenance labels edited away; the owner reference still identifies it. + labelsStripped := mine.DeepCopyObject().(client.Object) + labelsStripped.SetLabels(nil) + + // The same policy with its owner reference removed; the provenance labels still identify it. + ownerStripped := mine.DeepCopyObject().(client.Object) + ownerStripped.SetOwnerReferences(nil) + + // The same policy stripped of both markers; nothing is left to tell it from a foreign object. + bothStripped := mine.DeepCopyObject().(client.Object) + bothStripped.SetLabels(nil) + bothStripped.SetOwnerReferences(nil) + + // A policy at the same name authored by someone else, carrying neither marker. + foreign := emptyPolicyForScope("prod") + foreign.SetName(mine.GetName()) + foreign.SetNamespace("prod") + + // A policy that carries another resource's provenance -- the collision a bare name match would + // miss -- and no owner reference to this source. + otherSource := sourceObject(deploymentGVK, "prod", "other") + otherLabelled := emptyPolicyForScope("prod") + otherLabelled.SetName(mine.GetName()) + otherLabelled.SetNamespace("prod") + otherLabelled.SetLabels(parentLabels(otherSource.GroupVersionKind(), otherSource.GetName())) + + testCases := []struct { + name string + policy client.Object + want bool + }{ + {name: "the policy this controller generated", policy: mine, want: true}, + {name: "our policy with its labels stripped is still recognized by its owner reference", policy: labelsStripped, want: true}, + {name: "our policy with its owner reference stripped is still recognized by its labels", policy: ownerStripped, want: true}, + {name: "our policy with both markers gone is indistinguishable from foreign", policy: bothStripped, want: false}, + {name: "a foreign policy with neither marker", policy: foreign, want: false}, + {name: "a policy carrying another resource's provenance", policy: otherLabelled, want: false}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := isGeneratedFor(tc.policy, source.GroupVersionKind(), source.GetName(), newScheme(t)); got != tc.want { + t.Errorf("isGeneratedFor(%q) = %v, want %v", tc.policy.GetName(), got, tc.want) + } + }) + } +} + +func TestDesiredPolicy(t *testing.T) { + // The parser leaves whenUnfulfilled unset, since the annotation grammar cannot express it; the + // generated policy carries the API's default explicitly so that it does not read as drift. + selectors := []kfplacementv1alpha1.ClusterSelector{ + { + Count: ptr.To(intstr.FromInt32(1)), + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{"env": "prod"}}, + }, + }, + } + wantSelectors := []kfplacementv1alpha1.ClusterSelector{ + { + Count: ptr.To(intstr.FromInt32(1)), + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{"env": "prod"}}, + }, + WhenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + } + + testCases := []struct { + name string + source *unstructured.Unstructured + want client.Object + }{ + { + name: "namespaced resource yields a namespaced policy", + source: sourceObject(deploymentGVK, "prod", "app"), + want: &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: "deployment-app-" + hashOfIdentity("apps", "Deployment", "prod", "app"), + Namespace: "prod", + Labels: map[string]string{ + kfplacementv1alpha1.ParentAPIGroupLabel: "apps", + kfplacementv1alpha1.ParentKindLabel: "Deployment", + kfplacementv1alpha1.ParentNameLabel: "app", + }, + OwnerReferences: []metav1.OwnerReference{{ + APIVersion: "apps/v1", + Kind: "Deployment", + Name: "app", + UID: types.UID("uid-app"), + }}, + }, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: wantSelectors, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Name: "app", + }}, + ResourceRevisionHistoryLimit: ptr.To(defaultResourceRevisionHistoryLimit), + }, + }, + }, + { + name: "cluster-scoped resource yields a cluster-scoped policy", + source: sourceObject(namespaceGVK, "", "work"), + want: &kfplacementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: "namespace-work-" + hashOfIdentity("", "Namespace", "", "work"), + Labels: map[string]string{ + // The core API group is the empty string, and the label is present all the + // same, so that core-group resources can be selected like any other. + kfplacementv1alpha1.ParentAPIGroupLabel: "", + kfplacementv1alpha1.ParentKindLabel: "Namespace", + kfplacementv1alpha1.ParentNameLabel: "work", + }, + OwnerReferences: []metav1.OwnerReference{{ + APIVersion: "v1", + Kind: "Namespace", + Name: "work", + UID: types.UID("uid-work"), + }}, + }, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: wantSelectors, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIVersion: "v1", + Kind: "Namespace", + Name: "work", + }}, + ResourceRevisionHistoryLimit: ptr.To(defaultResourceRevisionHistoryLimit), + }, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := desiredPolicy(tc.source, selectors) + if diff := cmp.Diff(got, tc.want); diff != "" { + t.Errorf("desiredPolicy() mismatch (-got, +want):\n%s", diff) + } + }) + } +} + +// TestParentOwnerReferenceIsNotControlling pins the choice not to claim controller or blocking +// ownership: either would require permission to update the finalizers subresource of a resource of +// any kind the user cares to annotate. +func TestParentOwnerReferenceIsNotControlling(t *testing.T) { + got := parentOwnerReference(sourceObject(deploymentGVK, "prod", "app")) + if got.Controller != nil { + t.Errorf("parentOwnerReference().Controller = %v, want nil", *got.Controller) + } + if got.BlockOwnerDeletion != nil { + t.Errorf("parentOwnerReference().BlockOwnerDeletion = %v, want nil", *got.BlockOwnerDeletion) + } +} + +// TestParentNameLabelIsShortenedWhenTooLong covers a resource whose name cannot fit in a label +// value, where the label is knowingly lossy and the owner reference is what remains exact. +func TestParentNameLabelIsShortenedWhenTooLong(t *testing.T) { + longName := strings.Repeat("a", 200) + source := sourceObject(deploymentGVK, "prod", longName) + + policy := desiredPolicy(source, nil) + labels := policy.GetLabels() + got := labels[kfplacementv1alpha1.ParentNameLabel] + + if got == longName { + t.Errorf("parent name label = the full name, want it shortened to fit a label value") + } + if errs := validation.IsValidLabelValue(got); len(errs) > 0 { + t.Errorf("parent name label = %v, want a valid label value: %s", got, strings.Join(errs, "; ")) + } + if wantOwner := longName; policy.GetOwnerReferences()[0].Name != wantOwner { + t.Errorf("owner reference name = %v, want the unshortened %v", policy.GetOwnerReferences()[0].Name, wantOwner) + } +} + +// TestParentLabelsAreValid covers the label values that have to be shortened to fit. A custom +// resource's API group is validated as a DNS-1123 subdomain and so runs to 253 bytes, four times +// what a label value holds; left unshortened it would make every generated policy for that +// resource fail validation on creation, with nothing but a retrying reconciler to show for it. +func TestParentLabelsAreValid(t *testing.T) { + testCases := []struct { + name string + gvk schema.GroupVersionKind + objName string + }{ + {name: "ordinary resource", gvk: deploymentGVK, objName: "app"}, + {name: "core group resource", gvk: namespaceGVK, objName: "work"}, + { + name: "api group too long for a label value", + gvk: schema.GroupVersionKind{Group: strings.Repeat("g", 90) + ".example.com", Version: "v1", Kind: "Widget"}, + objName: "gadget", + }, + { + name: "kind at the longest a kind can be", + gvk: schema.GroupVersionKind{Group: "example.com", Version: "v1", Kind: strings.Repeat("W", 63)}, + objName: "gadget", + }, + {name: "name too long for a label value", gvk: deploymentGVK, objName: strings.Repeat("a", 200)}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + for key, value := range parentLabels(tc.gvk, tc.objName) { + if errs := validation.IsValidLabelValue(value); len(errs) > 0 { + t.Errorf("parentLabels()[%s] = %v, want a valid label value: %s", key, value, strings.Join(errs, "; ")) + } + } + }) + } +} + +// TestDesiredPolicySetsDefaultedFields pins that the generated policy carries a value for every +// field the API server would otherwise default. A field left unset is stored with the API's +// default and then reads as a difference on the next pass, which would have the reconciler issue +// an update that changes nothing, forever. +func TestDesiredPolicySetsDefaultedFields(t *testing.T) { + selectors := []kfplacementv1alpha1.ClusterSelector{{Count: ptr.To(intstr.FromInt32(1))}} + spec := desiredPolicy(sourceObject(deploymentGVK, "prod", "app"), selectors).GetSpec() + + if got := spec.ResourceRevisionHistoryLimit; got == nil || *got != defaultResourceRevisionHistoryLimit { + t.Errorf("resourceRevisionHistoryLimit = %v, want %d", got, defaultResourceRevisionHistoryLimit) + } + if got := spec.ClusterSelectors[0].WhenUnfulfilled; got != defaultWhenUnfulfilled { + t.Errorf("whenUnfulfilled = %v, want %v", got, defaultWhenUnfulfilled) + } + // The caller's own selectors must not have been defaulted in place. + if got := selectors[0].WhenUnfulfilled; got != "" { + t.Errorf("the caller's selector whenUnfulfilled = %v, want it left empty", got) + } +} diff --git a/pkg/v1/controllers/annotationplacement/suite_test.go b/pkg/v1/controllers/annotationplacement/suite_test.go new file mode 100644 index 000000000..ed0068ca2 --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/suite_test.go @@ -0,0 +1,96 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "path/filepath" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/discovery" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/client-go/restmapper" + "k8s.io/client-go/tools/events" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +var configMapGVK = schema.GroupVersionKind{Group: "", Version: "v1", Kind: "ConfigMap"} + +var ( + testEnv *envtest.Environment + restConfig *rest.Config + hubClient client.Client + reconciler *Reconciler + eventRecorder *events.FakeRecorder + ctx context.Context + cancel context.CancelFunc +) + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + RunSpecs(t, "Annotation Based Placement Controller Suite") +} + +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("bootstrapping the test environment") + testEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("../../../../", "config", "crd", "bases")}, + ErrorIfCRDPathMissing: true, + } + var err error + restConfig, err = testEnv.Start() + Expect(err).Should(Succeed()) + Expect(restConfig).NotTo(BeNil()) + + Expect(kfplacementv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + hubClient, err = client.New(restConfig, client.Options{Scheme: scheme.Scheme}) + Expect(err).Should(Succeed()) + + discoveryClient, err := discovery.NewDiscoveryClientForConfig(restConfig) + Expect(err).Should(Succeed()) + groupResources, err := restmapper.GetAPIGroupResources(discoveryClient) + Expect(err).Should(Succeed()) + + // The recorder is buffered generously: an unread event blocks the reconciler that records it, + // which would surface as a timeout somewhere unrelated rather than as a failed expectation. + eventRecorder = events.NewFakeRecorder(100) + reconciler = &Reconciler{ + // The envtest client reads straight from the API server, so it serves as the API reader + // too; there is no cache to fall behind here. + Client: hubClient, + APIReader: hubClient, + RESTMapper: restmapper.NewDiscoveryRESTMapper(groupResources), + Recorder: eventRecorder, + } +}) + +var _ = AfterSuite(func() { + defer func() { + Expect(testEnv.Stop()).Should(Succeed()) + }() + cancel() +}) diff --git a/pkg/v1/controllers/annotationplacement/watch.go b/pkg/v1/controllers/annotationplacement/watch.go new file mode 100644 index 000000000..ba478e7aa --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/watch.go @@ -0,0 +1,157 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + + "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/handler" + "sigs.k8s.io/controller-runtime/pkg/predicate" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/sourcequeue" +) + +const controllerName = "annotation-placement-controller" + +// SourceQueue is how the resource watcher hands annotated resources to this controller. +// +// The annotated resources are of any kind the hub agent watches, which the manager's cache does not +// know ahead of time, so they cannot be watched through the manager and must be fed in instead, +// through a sourcequeue.Queue: it never blocks the watcher, collapses repeated reports of one +// resource, and holds reports made before the manager starts until it does. +// +// The zero value is ready to use. +type SourceQueue struct { + sourcequeue.Queue[Request] +} + +// Add reports an annotated resource. It never blocks, and it is safe to call from any goroutine +// and before the manager has started. +func (q *SourceQueue) Add(object client.Object) { + req := RequestFor(object) + if req.Kind == "" { + klog.ErrorS(nil, "Skipped a resource that names no kind", "obj", klog.KObj(object)) + return + } + q.Queue.Add(req) +} + +// SetupWithManager registers the controller with the manager. +// +// Annotated resources arrive through the given SourceQueue -- fed by the resource watcher, in the +// hub agent -- rather than from a watch of their own. The generated policies are watched through +// the manager, and an event on one enqueues the resource it was generated from: without that, an +// edit or a deletion of a generated policy produces no event on its resource, and for a resource +// that never changes again the policy would stay missing or wrong forever. +// +// The caller owns what the controller cannot decide for itself: the placement.kubefleet.dev types +// must be registered in the manager's scheme, and the hub agent must hold RBAC to read, create, +// update, and delete the two policy kinds. +func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, sources *SourceQueue) error { + if sources == nil { + // Reported here rather than left to panic once the manager starts the source: the caller + // already checks this error, and a controller with nothing feeding it places nothing. + return kferrors.NewUnexpectedError(nil, "the annotated resource source is nil", "controller", controllerName) + } + return builder.TypedControllerManagedBy[Request](mgr). + Named(controllerName). + WatchesRawSource(sources). + Watches( + &kfplacementv1alpha1.PlacementPolicy{}, + handler.TypedEnqueueRequestsFromMapFunc(mapGeneratedPolicyToSource), + builder.WithPredicates(generatedPolicyDrift()), + ). + Watches( + &kfplacementv1alpha1.ClusterPlacementPolicy{}, + handler.TypedEnqueueRequestsFromMapFunc(mapGeneratedPolicyToSource), + builder.WithPredicates(generatedPolicyDrift()), + ). + Complete(r) +} + +// generatedPolicyDrift admits the policy events that can mean a generated policy needs repair: a +// creation or deletion, and an update to the spec (which bumps the generation), the labels, or the +// owner references -- exactly what applyDesiredPolicy writes. Status updates are filtered out: the +// controllers that schedule a policy write its status on every change of the fleet, and each event +// let through here costs an uncached read of the annotated resource to learn that nothing drifted. +func generatedPolicyDrift() predicate.Predicate { + ownerReferencesChanged := predicate.Funcs{ + UpdateFunc: func(e event.UpdateEvent) bool { + return !equality.Semantic.DeepEqual(e.ObjectOld.GetOwnerReferences(), e.ObjectNew.GetOwnerReferences()) + }, + } + return predicate.Or(predicate.GenerationChangedPredicate{}, predicate.LabelChangedPredicate{}, ownerReferencesChanged) +} + +// mapGeneratedPolicyToSource enqueues the resource that generated a policy, identified from the +// policy's owner references. +// +// Only the owner whose identity reproduces this policy's own generated name is enqueued. A policy +// may carry more than one owner reference -- a foreign one that applyDesiredPolicy deliberately +// preserves, or any owner on a hand-authored policy that shares these watches -- and following +// those would enqueue a request for a kind the resource watcher does not track. Matching the +// generated name is exact, so only the true source passes. +// +// The owner reference is used rather than the parent labels because the labels are lossy (a long +// name is shortened to a prefix and a hash) and, being labels, can be stripped -- which is itself +// drift this watch exists to repair. On an update both the old and the new object are mapped, so an +// owner reference present only on the old side still enqueues the resource whose policy the update +// just took away from it. +func mapGeneratedPolicyToSource(_ context.Context, policy client.Object) []Request { + // A generated policy always lives in the namespace of the resource it came from, and a + // cluster-scoped one has none, matching a cluster-scoped owner. + policyName, policyNamespace := policy.GetName(), policy.GetNamespace() + for _, owner := range policy.GetOwnerReferences() { + gv, err := schema.ParseGroupVersion(owner.APIVersion) + if err != nil { + klog.ErrorS(err, "Skipped an owner with an unparsable API version", "policy", klog.KObj(policy), "apiVersion", owner.APIVersion) + continue + } + gvk := gv.WithKind(owner.Kind) + if generatedPolicyName(gvk, policyNamespace, owner.Name) != policyName { + // Not the owner this policy was generated from; following it would reach a resource + // the watcher never selected. + continue + } + return []Request{{ + GroupVersionKind: gvk, + NamespacedName: client.ObjectKey{Namespace: policyNamespace, Name: owner.Name}, + }} + } + return nil +} + +// SourceNamespace returns the namespace whose skip-status governs whether a source resource is +// placed. For an ordinary namespaced resource that is its metadata.namespace; a core Namespace +// object carries none and is itself the namespace, so its own name is returned. The skip check +// would otherwise read an empty namespace for a Namespace source and place one KubeFleet excludes. +func SourceNamespace(source *unstructured.Unstructured) string { + if gvk := source.GroupVersionKind(); gvk.Group == "" && gvk.Kind == "Namespace" { + return source.GetName() + } + return source.GetNamespace() +} diff --git a/pkg/v1/controllers/annotationplacement/watch_test.go b/pkg/v1/controllers/annotationplacement/watch_test.go new file mode 100644 index 000000000..56507894a --- /dev/null +++ b/pkg/v1/controllers/annotationplacement/watch_test.go @@ -0,0 +1,287 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package annotationplacement + +import ( + "context" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/util/workqueue" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +var configMapSourceGVK = schema.GroupVersionKind{Group: "", Version: "v1", Kind: "ConfigMap"} + +// generatedPolicyOwnedBy builds a policy as the cache would deliver it: correctly named for the +// resource that generated it, owned by that resource, plus any extra owner references a caller +// wants to plant. A policy whose name does not derive from an owner is that owner's, which is what +// lets the map function tell the true source from a foreign owner or a hand-authored policy. +func generatedPolicyOwnedBy(source *unstructured.Unstructured, extraOwners ...metav1.OwnerReference) client.Object { + namespace := source.GetNamespace() + policy := emptyPolicyForScope(namespace) + policy.SetNamespace(namespace) + policy.SetName(generatedPolicyName(source.GroupVersionKind(), namespace, source.GetName())) + policy.SetOwnerReferences(append([]metav1.OwnerReference{parentOwnerReference(source)}, extraOwners...)) + return policy +} + +// namedPolicy builds a policy with an explicit name (not derived from any owner), for the cases a +// generating source is not supposed to be found -- a hand-authored policy, or one whose only owners +// are foreign. +func namedPolicy(namespace, name string, owners ...metav1.OwnerReference) client.Object { + policy := &kfplacementv1alpha1.PlacementPolicy{} + policy.SetNamespace(namespace) + policy.SetName(name) + policy.SetOwnerReferences(owners) + return policy +} + +func ownerRef(apiVersion, kind, name string) metav1.OwnerReference { + return metav1.OwnerReference{APIVersion: apiVersion, Kind: kind, Name: name, UID: "00000000-0000-0000-0000-000000000001"} +} + +func TestMapGeneratedPolicyToSource(t *testing.T) { + deploymentSource := sourceObject(deploymentGVK, "prod", "web") + configMapSource := sourceObject(configMapSourceGVK, "prod", "app") + namespaceSource := sourceObject(namespaceGVK, "", "team") + + deploymentRequest := Request{GroupVersionKind: deploymentGVK, NamespacedName: client.ObjectKey{Namespace: "prod", Name: "web"}} + + testCases := []struct { + name string + policy client.Object + want []Request + }{ + { + name: "a generated policy enqueues the resource that generated it", + policy: generatedPolicyOwnedBy(deploymentSource), + want: []Request{deploymentRequest}, + }, + { + name: "a core group generating resource keeps its empty group", + policy: generatedPolicyOwnedBy(configMapSource), + want: []Request{{GroupVersionKind: configMapSourceGVK, NamespacedName: client.ObjectKey{Namespace: "prod", Name: "app"}}}, + }, + { + // A foreign owner reference -- one applyDesiredPolicy preserves, or any owner on a policy + // that merely shares these watches -- must not be followed, or the queue would hold a + // request for a kind the resource watcher never selected. + name: "a foreign owner reference is not followed", + policy: generatedPolicyOwnedBy(deploymentSource, ownerRef("v1", "Pod", "some-pod"), ownerRef("apps/v1", "Deployment", "unrelated")), + want: []Request{deploymentRequest}, + }, + { + // A hand-authored policy is owned by nothing this controller generated: its name does + // not derive from its owner, so no owner is followed. + name: "a policy whose name does not derive from its owner enqueues nothing", + policy: namedPolicy("prod", "hand-authored", ownerRef("apps/v1", "Deployment", "web")), + want: nil, + }, + { + name: "a policy with no owners enqueues nothing", + policy: namedPolicy("prod", "hand-authored"), + want: nil, + }, + { + // ParseGroupVersion rejects a second slash; the owner is skipped rather than turned + // into a bogus kind on the queue. + name: "an owner with an unparsable API version is skipped", + policy: generatedPolicyOwnedBy(deploymentSource, ownerRef("apps/v1/extra", "Deployment", "web")), + want: []Request{deploymentRequest}, + }, + { + name: "a cluster scoped policy enqueues a cluster scoped source", + policy: generatedPolicyOwnedBy(namespaceSource), + want: []Request{{GroupVersionKind: namespaceGVK, NamespacedName: client.ObjectKey{Name: "team"}}}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := mapGeneratedPolicyToSource(context.Background(), tc.policy) + if diff := cmp.Diff(got, tc.want, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("mapGeneratedPolicyToSource() mismatch (-got, +want):\n%s", diff) + } + }) + } +} + +// newTestQueue returns the controller workqueue a source is started with, so that a test can see +// exactly what a SourceQueue enqueued. +func newTestQueue() *testQueue { + return &testQueue{TypedRateLimitingInterface: workqueue.NewTypedRateLimitingQueue[Request](workqueue.DefaultTypedControllerRateLimiter[Request]())} +} + +type testQueue struct { + workqueue.TypedRateLimitingInterface[Request] +} + +// drain returns every queued request in order, leaving the queue empty. +func (q *testQueue) drain() []Request { + var got []Request + for q.Len() > 0 { + req, shutdown := q.Get() + if shutdown { + break + } + got = append(got, req) + q.Done(req) + } + return got +} + +func TestSourceQueueAdd(t *testing.T) { + testCases := []struct { + name string + source client.Object + want []Request + }{ + { + name: "an annotated resource is enqueued under its kind and namespaced name", + source: sourceObject(deploymentGVK, "prod", "web"), + want: []Request{{GroupVersionKind: deploymentGVK, NamespacedName: client.ObjectKey{Namespace: "prod", Name: "web"}}}, + }, + { + name: "a cluster scoped resource is enqueued with no namespace", + source: sourceObject(namespaceGVK, "", "team"), + want: []Request{{GroupVersionKind: namespaceGVK, NamespacedName: client.ObjectKey{Name: "team"}}}, + }, + { + // A typed object routinely arrives with an empty kind; a request built from it would + // name no API to read the resource from, so it is dropped rather than enqueued. + name: "an object that names no kind is dropped", + source: &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Namespace: "prod", Name: "web"}}, + want: nil, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + q := &SourceQueue{} + queue := newTestQueue() + if err := q.Start(context.Background(), queue); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + q.Add(tc.source) + + if diff := cmp.Diff(queue.drain(), tc.want, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("SourceQueue.Add() enqueued (-got, +want):\n%s", diff) + } + }) + } +} + +// TestSourceNamespace covers the skip-check namespace: a namespaced resource uses its own +// namespace, but a Namespace object -- which carries none -- is itself the namespace. +func TestSourceNamespace(t *testing.T) { + testCases := []struct { + name string + source *unstructured.Unstructured + want string + }{ + {name: "a namespaced resource uses its namespace", source: sourceObject(deploymentGVK, "prod", "web"), want: "prod"}, + {name: "a Namespace object is itself the namespace", source: sourceObject(namespaceGVK, "", "team"), want: "team"}, + {name: "another cluster scoped resource has no namespace", source: sourceObject(schema.GroupVersionKind{Group: "rbac.authorization.k8s.io", Version: "v1", Kind: "ClusterRole"}, "", "admin"), want: ""}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := SourceNamespace(tc.source); got != tc.want { + t.Errorf("SourceNamespace(%s %s) = %q, want %q", tc.source.GetKind(), tc.source.GetName(), got, tc.want) + } + }) + } +} + +// TestGeneratedPolicyDrift pins which policy updates reach the reconciler: what applyDesiredPolicy +// writes must, a status write must not. +func TestGeneratedPolicyDrift(t *testing.T) { + base := func() *kfplacementv1alpha1.PlacementPolicy { + policy := generatedPolicyOwnedBy(sourceObject(deploymentGVK, "prod", "web")).(*kfplacementv1alpha1.PlacementPolicy) + policy.Generation = 1 + return policy + } + + testCases := []struct { + name string + mutate func(*kfplacementv1alpha1.PlacementPolicy) + want bool + }{ + {name: "status only", mutate: func(p *kfplacementv1alpha1.PlacementPolicy) { + p.ResourceVersion = "2" + p.Status.Conditions = []metav1.Condition{{Type: "Scheduled", Status: metav1.ConditionTrue}} + }}, + {name: "spec edit bumps the generation", mutate: func(p *kfplacementv1alpha1.PlacementPolicy) { p.Generation = 2 }, want: true}, + {name: "label change", mutate: func(p *kfplacementv1alpha1.PlacementPolicy) { p.Labels = map[string]string{"drift": "yes"} }, want: true}, + {name: "owner reference stripped", mutate: func(p *kfplacementv1alpha1.PlacementPolicy) { p.OwnerReferences = nil }, want: true}, + } + + pred := generatedPolicyDrift() + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + updated := base() + tc.mutate(updated) + if got := pred.Update(event.UpdateEvent{ObjectOld: base(), ObjectNew: updated}); got != tc.want { + t.Errorf("generatedPolicyDrift().Update() = %v, want %v", got, tc.want) + } + }) + } +} + +// TestSourceQueueHoldsUntilStarted pins that a resource reported before the manager started is not +// lost: the watcher does not have to know when this controller became ready. +func TestSourceQueueHoldsUntilStarted(t *testing.T) { + q := &SourceQueue{} + q.Add(sourceObject(deploymentGVK, "prod", "web")) + + queue := newTestQueue() + if err := q.Start(context.Background(), queue); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + + want := []Request{{GroupVersionKind: deploymentGVK, NamespacedName: client.ObjectKey{Namespace: "prod", Name: "web"}}} + if diff := cmp.Diff(queue.drain(), want); diff != "" { + t.Errorf("SourceQueue.Add() before Start enqueued (-got, +want):\n%s", diff) + } +} + +// TestSourceQueueDeduplicates pins the reason this is a queue rather than a channel: a burst of +// reports for one resource costs one reconcile, not one per report. +func TestSourceQueueDeduplicates(t *testing.T) { + q := &SourceQueue{} + queue := newTestQueue() + if err := q.Start(context.Background(), queue); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + + source := sourceObject(deploymentGVK, "prod", "web") + for i := 0; i < 5; i++ { + q.Add(source) + } + + if got := queue.Len(); got != 1 { + t.Errorf("SourceQueue.Add() x5 queued %d items, want 1", got) + } +} diff --git a/pkg/v1/controllers/autorollout/claim.go b/pkg/v1/controllers/autorollout/claim.go new file mode 100644 index 000000000..07a95a347 --- /dev/null +++ b/pkg/v1/controllers/autorollout/claim.go @@ -0,0 +1,83 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package autorollout + +import ( + "context" + + "k8s.io/klog/v2" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/bindingmanager" +) + +func holdsBindingManagerRole(placementPolicy placementv1alpha1.PlacementPolicyAccessor) bool { + bindingManager := placementPolicy.GetStatus().BindingManager + return bindingManager != nil && bindingManager.ControllerName == controllerName +} + +// placementPolicyObjRef returns an object reference to the placement policy, which is the object that +// the auto-rollout controller manages bindings on behalf of. +func placementPolicyObjRef(placementPolicy placementv1alpha1.PlacementPolicyAccessor) placementv1alpha1.ObjectReference { + var kind string + if placementPolicy.GetNamespace() == "" { + kind = placementv1alpha1.ClusterPlacementPolicyKind + } else { + kind = placementv1alpha1.PlacementPolicyKind + } + return placementv1alpha1.ObjectReference{ + APIGroup: placementv1alpha1.GroupVersion.Group, + APIVersion: placementv1alpha1.GroupVersion.Version, + Kind: kind, + Namespace: placementPolicy.GetNamespace(), + Name: placementPolicy.GetName(), + } +} + +// claimAsBindingManagerFor claims the binding manager role for the placement policy on behalf of the +// auto-rollout controller. +// +// This method returns (true, nil) if the role has been claimed; (false, nil) if another source currently holds the +// role and the claim should be retried later; and (false, err) if an error occurs during the attempt. +func (r *Reconciler) claimAsBindingManagerFor(ctx context.Context, placementPolicy placementv1alpha1.PlacementPolicyAccessor) (bool, error) { + claimed, err := bindingmanager.ClaimRoleAs(ctx, r.hubClient, placementPolicy, controllerName, placementPolicyObjRef(placementPolicy)) + if err != nil { + wrappedErr := errors.Wraps(err, "", "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to claim binding manager role", errors.Args(wrappedErr)...) + return false, wrappedErr + } + if !claimed { + klog.V(2).InfoS("Cannot claim the binding manager role for now; will retry later", + "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + return false, nil + } + klog.V(2).InfoS("Successfully claimed the binding manager role for the placement policy", + "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + return true, nil +} + +// relinquishBindingManagerRole gives up the binding manager role for the placement policy on behalf of the +// auto-rollout controller, if the role is currently held. +func (r *Reconciler) relinquishBindingManagerRole(ctx context.Context, placementPolicy placementv1alpha1.PlacementPolicyAccessor) error { + if err := bindingmanager.RelinquishRoleFor(ctx, r.hubClient, placementPolicy, controllerName, placementPolicyObjRef(placementPolicy)); err != nil { + wrappedErr := errors.Wraps(err, "", "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to relinquish binding manager role", errors.Args(wrappedErr)...) + return wrappedErr + } + return nil +} diff --git a/pkg/v1/controllers/autorollout/controller.go b/pkg/v1/controllers/autorollout/controller.go new file mode 100644 index 000000000..7d0f99f22 --- /dev/null +++ b/pkg/v1/controllers/autorollout/controller.go @@ -0,0 +1,233 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package autorollout features the KubeFleet auto-rollout controller for automatically rolling out resource +// changes to placement policies. +package autorollout + +import ( + "context" + "time" + + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/client-go/tools/events" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + parallelizerutil "go.goms.io/fleet/pkg/utils/parallelizer" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot" +) + +const ( + controllerName = "auto-rollout" + + requeueImmediatelyAfterPeriod = time.Millisecond * 200 +) + +const ( + autoRolloutMinimumWaitPeriodLowerBound = time.Minute + autoRolloutMinimumWaitPeriodUpperBound = 1 * time.Hour + autoRolloutDefaultMinimumWaitPeriod = 3 * time.Minute +) + +const ( + bindingManagerClaimRateLimiterBaseDelay = time.Second * 5 + bindingManagerClaimRateLimiterMaxDelay = time.Second * 180 +) + +// Reconciler helps reconcile placement policies with auto-rollout enabled. +type Reconciler struct { + hubClient client.Client + + placementResourceSnapshotManager *placementresourcesnapshot.Manager + + parallelizer parallelizerutil.Parallelizer + + bindingManagerClaimRateLimiter workqueue.TypedRateLimiter[ctrl.Request] + + eventRecorder events.EventRecorder +} + +// New creates a Reconciler. +// +// If no parallelizer or binding manager claim rate limiter is provided, the defaults are used. The hub client, +// the placement resource snapshot manager, and the event recorder are required. +func New( + hubClient client.Client, + placementResourceSnapshotManager *placementresourcesnapshot.Manager, + parallelizer parallelizerutil.Parallelizer, + bindingManagerClaimRateLimiter workqueue.TypedRateLimiter[ctrl.Request], + eventRecorder events.EventRecorder, +) (*Reconciler, error) { + if hubClient == nil { + return nil, errors.NewUnexpectedError(nil, "hub client is nil") + } + if placementResourceSnapshotManager == nil { + return nil, errors.NewUnexpectedError(nil, "placement resource snapshot manager is nil") + } + if eventRecorder == nil { + return nil, errors.NewUnexpectedError(nil, "event recorder is nil") + } + if parallelizer == nil { + parallelizer = parallelizerutil.NewParallelizer(parallelizerutil.DefaultNumOfWorkers) + } + if bindingManagerClaimRateLimiter == nil { + bindingManagerClaimRateLimiter = workqueue.NewTypedItemExponentialFailureRateLimiter[ctrl.Request](bindingManagerClaimRateLimiterBaseDelay, bindingManagerClaimRateLimiterMaxDelay) + } + + return &Reconciler{ + hubClient: hubClient, + placementResourceSnapshotManager: placementResourceSnapshotManager, + parallelizer: parallelizer, + bindingManagerClaimRateLimiter: bindingManagerClaimRateLimiter, + eventRecorder: eventRecorder, + }, nil +} + +func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + startTime := time.Now() + klog.V(2).InfoS("Reconciliation starts", "placementPolicy", req.NamespacedName, "controller", controllerName) + defer func() { + latency := time.Since(startTime).Milliseconds() + klog.V(2).InfoS("Reconciliation ends", "placementPolicy", req.NamespacedName, "latency", latency, "controller", controllerName) + }() + + // Retrieve the placement policy (or cluster placement policy) resource and return its accessor interface. + placementPolicy, err := r.retrievePlacementPolicy(ctx, req.NamespacedName) + if err != nil { + if apierrors.IsNotFound(err) { + // The placement policy cannot be found; there is nothing to roll out. + r.bindingManagerClaimRateLimiter.Forget(req) + return ctrl.Result{}, nil + } + + wrappedErr := errors.Wraps(err, "", "placementPolicy", req.NamespacedName, "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to retrieve placement policy object", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + klog.V(2).InfoS("Retrieved placement policy", "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + + // Give up the binding manager role (if currently held) and skip reconciliation if the placement policy is + // marked for deletion. + if !placementPolicy.GetDeletionTimestamp().IsZero() { + r.bindingManagerClaimRateLimiter.Forget(req) + klog.V(2).InfoS("Placement policy is marked for deletion, relinquishing the binding manager role and skipping reconciliation", "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + // Note that the event handler of this controller guarantees that the controller will always see + // any binding manager role it has claimed. + if holdsBindingManagerRole(placementPolicy) { + if err := r.relinquishBindingManagerRole(ctx, placementPolicy); err != nil { + // The error has been wrapped and logged in the relinquish method. + return ctrl.Result{}, err + } + } + return ctrl.Result{}, nil + } + + // Skip processing if auto-rollout is not enabled, no rollout is needed, or the minimum wait period since the + // last rollout has not passed. Give up the binding manager role (if currently held) in this case. + minWaitPeriod := retrieveMinimumWaitPeriod(placementPolicy) + shouldSkip, requeueAfter := shouldSkipProcessing(placementPolicy, minWaitPeriod) + if shouldSkip { + r.bindingManagerClaimRateLimiter.Forget(req) + if holdsBindingManagerRole(placementPolicy) { + // Note that the event handler of this controller guarantees that the controller will always see + // any binding manager role it has claimed. + if err := r.relinquishBindingManagerRole(ctx, placementPolicy); err != nil { + // The error has been wrapped and logged in the relinquish method. + return ctrl.Result{}, err + } + } + return ctrl.Result{RequeueAfter: requeueAfter}, nil + } + + // Claim the binding manager role for the placement policy. + // This ensures that no other controllers (e.g., the scheduling process, the migrations, etc.) can + // interfere with the rollout process. + claimed, err := r.claimAsBindingManagerFor(ctx, placementPolicy) + if err != nil { + // The error has been wrapped and logged in the claim method. + return ctrl.Result{}, err + } + if !claimed { + backoffDuration := r.bindingManagerClaimRateLimiter.When(req) + return ctrl.Result{RequeueAfter: backoffDuration}, nil + } + // Reset the binding manager claim rate limiter for the placement policy. + r.bindingManagerClaimRateLimiter.Forget(req) + + if err := r.rollout(ctx, placementPolicy); err != nil { + wrappedErr := errors.Wraps(err, "", "placementPolicy", klog.KObj(placementPolicy)) + klog.ErrorS(wrappedErr, "Failed to rollout for the placement policy", errors.Args(wrappedErr)...) + + if errors.Category(wrappedErr) == errors.ErrCategoryUser { + // The snapshotting failed due to a user error, which retries cannot fix; try again at the next rollout + // attempt, i.e., once the minimum wait period passes. + r.eventRecorder.Eventf(placementPolicy, nil, corev1.EventTypeWarning, "FailedToSnapshotResourcesDuringAutoRollout", "SnapshotResources", + "Failed to complete auto-rollout for the placement policy due to a snapshotting error: %v", wrappedErr) + + // Bookkeep the last rollout attempt timestamp for the failed attempt as well. The claim of the binding + // manager role above is a status update, which makes the event handler enqueue the placement policy again; + // without the timestamp, the minimum wait period would not hold back that reconciliation, and the controller + // would claim the role, fail, and relinquish it over and over. + // + // The timestamp is written before the role is relinquished, so that the wait period holds back retries + // even if relinquishing fails. + if err := r.bookkeepLastRolloutAttemptTimestamp(ctx, placementPolicy, time.Now()); err != nil { + wrappedErr := errors.Wraps(err, "", "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to bookkeep the last rollout attempt timestamp on the placement policy for a failed rollout", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + if err := r.relinquishBindingManagerRole(ctx, placementPolicy); err != nil { + // The error has been wrapped and logged in the relinquish method. + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: minWaitPeriod}, nil + } + return ctrl.Result{}, wrappedErr + } + + // Bookkeep the last rollout attempt timestamp on the placement policy. + if err := r.bookkeepLastRolloutAttemptTimestamp(ctx, placementPolicy, time.Now()); err != nil { + wrappedErr := errors.Wraps(err, "", "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to bookkeep the last rollout attempt timestamp on the placement policy", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + klog.V(2).InfoS("Successfully rolled out changes to all bindings of the placement policy", + "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + // The auto rollout has been completed. Requeue it right away to perform the cleanup steps. + return ctrl.Result{RequeueAfter: requeueImmediatelyAfterPeriod}, nil +} + +// SetupWithManager sets up the controller with the manager. +// +// The controller watches both ClusterPlacementPolicy (cluster-scoped) and PlacementPolicy (namespace-scoped) +// objects, funneling events for either kind into the same reconcile queue; Reconcile (via retrievePlacementPolicy) +// tells the two apart by whether the incoming request carries a namespace. +func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, maxConcurrentReconciles int) error { + return ctrl.NewControllerManagedBy(mgr). + Named(controllerName). + WithOptions(controller.Options{MaxConcurrentReconciles: maxConcurrentReconciles}). + Watches(&placementv1alpha1.ClusterPlacementPolicy{}, placementPolicyEventHandlerFuncs). + Watches(&placementv1alpha1.PlacementPolicy{}, placementPolicyEventHandlerFuncs). + Complete(r) +} diff --git a/pkg/v1/controllers/autorollout/eventhandler.go b/pkg/v1/controllers/autorollout/eventhandler.go new file mode 100644 index 000000000..d780c17f9 --- /dev/null +++ b/pkg/v1/controllers/autorollout/eventhandler.go @@ -0,0 +1,146 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package autorollout + +import ( + "context" + + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/handler" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +var ( + // placementPolicyEventHandlerFuncs handles events for both PlacementPolicy and ClusterPlacementPolicy objects. + // + // Policies whose binding manager is this controller are always enqueued so outstanding claims can be + // cleaned up. Otherwise, updates must affect whether an enabled auto-rollout is needed or when it can happen. + placementPolicyEventHandlerFuncs = &handler.TypedFuncs[client.Object, ctrl.Request]{ + // Handle create events to recover enabled policies and outstanding binding manager claims after a restart. + CreateFunc: func( + ctx context.Context, + e event.TypedCreateEvent[client.Object], + q workqueue.TypedRateLimitingInterface[ctrl.Request]) { + obj := e.Object + policy, err := castToPlacementPolicy(obj) + if err != nil { + klog.ErrorS(err, "Failed to enqueue a placement policy create event", errors.Args(err)...) + return + } + + if !isAutoRolloutEnabled(policy) && !holdsBindingManagerRole(policy) { + return + } + q.Add(ctrl.Request{ + NamespacedName: types.NamespacedName{Namespace: policy.GetNamespace(), Name: policy.GetName()}, + }) + }, + // Handle updates for outstanding claims, or relevant changes when auto-rollout is or was enabled. + UpdateFunc: func( + ctx context.Context, + e event.TypedUpdateEvent[client.Object], + q workqueue.TypedRateLimitingInterface[ctrl.Request]) { + oldPolicy, err := castToPlacementPolicy(e.ObjectOld) + if err != nil { + klog.ErrorS(err, "Failed to cast old placement policy in update event", errors.Args(err)...) + return + } + newPolicy, err := castToPlacementPolicy(e.ObjectNew) + if err != nil { + klog.ErrorS(err, "Failed to cast new placement policy in update event", errors.Args(err)...) + return + } + + if !holdsBindingManagerRole(newPolicy) { + if !isAutoRolloutEnabled(oldPolicy) && !isAutoRolloutEnabled(newPolicy) { + return + } + if !autoRolloutRelevantChange(oldPolicy, newPolicy) { + return + } + } + + q.Add(ctrl.Request{ + NamespacedName: types.NamespacedName{Namespace: newPolicy.GetNamespace(), Name: newPolicy.GetName()}, + }) + }, + // Deletions need no handling; the controller does not add finalizers to placement policies. + } +) + +// autoRolloutRelevantChange returns true if the update to the placement policy might change the outcome of an +// auto-rollout evaluation. +func autoRolloutRelevantChange(oldPolicy, newPolicy placementv1alpha1.PlacementPolicyAccessor) bool { + if !oldPolicy.GetDeletionTimestamp().Equal(newPolicy.GetDeletionTimestamp()) { + return true + } + + // Check the annotations that control auto-rollout. + for _, key := range []string{ + rolloutv1alpha1.AutoRolloutEnabledAnnotationKey, + rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey, + } { + if oldPolicy.GetAnnotations()[key] != newPolicy.GetAnnotations()[key] { + return true + } + } + + // Check the conditions that determine whether a rollout is needed: enqueue if the new object has them in + // the states that call for a rollout, and at least one of them differs from its older version. + if !isRolloutNeeded(newPolicy) { + return false + } + for _, condType := range []string{ + placementv1alpha1.PlacementPolicyCondTypeResourceCollected, + placementv1alpha1.PlacementPolicyCondTypeSynchronized, + } { + oldCond := meta.FindStatusCondition(oldPolicy.GetStatus().Conditions, condType) + // The new condition is guaranteed to exist, as a rollout is needed on the new object. + newCond := meta.FindStatusCondition(newPolicy.GetStatus().Conditions, condType) + if oldCond == nil || + oldCond.Status != newCond.Status || + oldCond.Reason != newCond.Reason || + oldCond.ObservedGeneration != newCond.ObservedGeneration { + return true + } + } + return false +} + +func castToPlacementPolicy(obj client.Object) (placementv1alpha1.PlacementPolicyAccessor, error) { + if obj.GetNamespace() == "" { + policy, ok := obj.(*placementv1alpha1.ClusterPlacementPolicy) + if !ok { + return nil, errors.NewUnexpectedError(nil, "failed to cast object as a cluster placement policy", "gvk", obj.GetObjectKind().GroupVersionKind(), "obj", klog.KObj(obj)) + } + return policy, nil + } + policy, ok := obj.(*placementv1alpha1.PlacementPolicy) + if !ok { + return nil, errors.NewUnexpectedError(nil, "failed to cast object as a placement policy", "gvk", obj.GetObjectKind().GroupVersionKind(), "obj", klog.KObj(obj)) + } + return policy, nil +} diff --git a/pkg/v1/controllers/autorollout/lifecycle_integration_test.go b/pkg/v1/controllers/autorollout/lifecycle_integration_test.go new file mode 100644 index 000000000..9919af44b --- /dev/null +++ b/pkg/v1/controllers/autorollout/lifecycle_integration_test.go @@ -0,0 +1,1798 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package autorollout + +import ( + "fmt" + "slices" + "time" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +// The number of placement bindings that each placement policy in the lifecycle test cases has. +const lifecycleBindingCount = 3 + +// The binding manager claim details of another (fictional) controller. +const ( + otherBindingManagerControllerName = "other-controller" + otherBindingManagerAPIGroup = "example.kubefleet.dev" + otherBindingManagerAPIVersion = "v1" + otherBindingManagerKind = "OtherRollout" + otherBindingManagerNameTemplate = "other-rollout-%s" +) + +// otherBindingManager returns the binding manager claim of another (fictional) controller on a placement policy. +func otherBindingManager(randStr string) *placementv1alpha1.BindingManager { + return &placementv1alpha1.BindingManager{ + ControllerName: otherBindingManagerControllerName, + ObjectRefs: []placementv1alpha1.ObjectReference{ + { + APIGroup: otherBindingManagerAPIGroup, + APIVersion: otherBindingManagerAPIVersion, + Kind: otherBindingManagerKind, + Name: fmt.Sprintf(otherBindingManagerNameTemplate, randStr), + }, + }, + } +} + +// markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim marks a placement policy as having resources pending rollout +// (with the conditions that pendingRolloutConds returns for ResourcesChangedPendingRollout) and leaves a binding +// manager claim of the auto-rollout controller on it, in one status write; this simulates a claim that the +// controller has left behind, e.g., after a restart in the middle of a rollout. +// +// A single write makes sure that the controller never observes one change without the other. +func markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim(policyNSName, policyName string) { + updatePlacementPolicyStatus(policyNSName, policyName, func(policy placementv1alpha1.PlacementPolicyAccessor) { + setConditionsOnPlacementPolicy(policy, pendingRolloutConds(placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout)...) + policy.GetStatus().BindingManager = wantAutoRolloutBindingManager(policyNSName, policyName) + }) +} + +// placementPolicySpecAndAnnotationsUntouchedActual verifies that a placement policy is still at its first generation +// (i.e., its spec has not been changed) and that, apart from the last rollout attempt timestamp, its annotations are +// exactly the given ones. +func placementPolicySpecAndAnnotationsUntouchedActual(policyNSName, policyName string, wantAnnotations map[string]string) func() error { + return func() error { + policy, err := getPlacementPolicy(policyNSName, policyName) + if err != nil { + return err + } + if gotGeneration := policy.GetGeneration(); gotGeneration != 1 { + return fmt.Errorf("the placement policy is at generation %d, want 1", gotGeneration) + } + gotAnnotations := make(map[string]string, len(policy.GetAnnotations())) + for k, v := range policy.GetAnnotations() { + if k != rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey { + gotAnnotations[k] = v + } + } + if diff := cmp.Diff(gotAnnotations, wantAnnotations); diff != "" { + return fmt.Errorf("placement policy annotations (other than the last rollout attempt timestamp) diff (-got +want):\n%s", diff) + } + return nil + } +} + +var _ = Describe("gating auto-rollouts of placement policies", func() { + Context("auto-rollouts of placement policies with resources pending rollout", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + var wantLastRolloutAttemptTimestamp string + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + It("should keep the spec and the other annotations of the placement policy as they are", func() { + untouched := placementPolicySpecAndAnnotationsUntouchedActual(appNamespaceName, policyName, autoRolloutEnabledAnnotations()) + Expect(untouched()).To(Succeed(), "Changed the placement policy unexpectedly") + }) + + It("can mark the placement policy as synchronized", func() { + wantLastRolloutAttemptTimestamp = lastRolloutAttemptTimestamp(appNamespaceName, policyName) + markPlacementPolicyAsSynchronized(appNamespaceName, policyName) + forcePlacementPolicyReconciliation(appNamespaceName, policyName) + }) + + It("should not roll out the placement policy again", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotNames[1]), + placementBindingsActual(appNamespaceName, bindingNames, snapshotNames[0], 2), + placementPolicyAtRestActual(appNamespaceName, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[0]), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[1]), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies with auto-rollout disabled", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + noRolloutActual := func() func() error { + return allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, ""), + ) + } + + BeforeAll(func() { + createConfigMap(configMapName) + // The placement policy has no auto-rollout enabled annotation. + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, nil) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the placement policy", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can set the auto-rollout enabled annotation to a value other than the enabling one", func() { + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutEnabledAnnotationKey] = "True" + }) + }) + + It("should still not roll out the placement policy", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can enable auto-rollout for the placement policy", func() { + notBefore = time.Now() + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutEnabledAnnotationKey] = rolloutv1alpha1.AutoRolloutEnabledAnnotationValue + }) + }) + + It("should roll out the latest resources to all the placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies with auto-rollout disabled after a rollout", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + // A last rollout attempt timestamp long enough ago that the minimum wait period never holds a rollout back. + backdatedLastRolloutAttemptTimestamp := backdatedTimestamp(time.Hour * 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + It("can mark the placement policy as synchronized", func() { + markPlacementPolicyAsSynchronized(appNamespaceName, policyName) + }) + + It("can disable auto-rollout for the placement policy", func() { + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + delete(annotations, rolloutv1alpha1.AutoRolloutEnabledAnnotationKey) + }) + }) + + It("can change the selected resources and backdate the last rollout attempt timestamp", func() { + updateConfigMapData(configMapName, map[string]string{configMapDataKey: altConfigMapDataValue}) + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedLastRolloutAttemptTimestamp + }) + }) + + It("can mark the placement policy as having resources pending rollout again", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the placement policy again", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotNames[1]), + placementBindingsActual(appNamespaceName, bindingNames, snapshotNames[0], 2), + placementPolicyAtRestActual(appNamespaceName, policyName, backdatedLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[0]), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[1]), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policy conditions that do or do not call for a rollout", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + noRolloutActual := func() func() error { + return allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, ""), + ) + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can force a reconciliation of the placement policy with no conditions", func() { + forcePlacementPolicyReconciliation(appNamespaceName, policyName) + }) + + It("should not roll out the placement policy with no conditions", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can mark the placement policy as having failed to collect some resources", func() { + setPlacementPolicyConditions(appNamespaceName, policyName, + resourceCollectedCond(metav1.ConditionFalse, placementv1alpha1.PlacementPolicyResourceCollectedCondReasonFailedToCollectSomeResources), + synchronizedCond(metav1.ConditionFalse, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout), + ) + forcePlacementPolicyReconciliation(appNamespaceName, policyName) + }) + + It("should not roll out the placement policy with resources not collected", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can mark the placement policy as synchronized", func() { + markPlacementPolicyAsSynchronized(appNamespaceName, policyName) + forcePlacementPolicyReconciliation(appNamespaceName, policyName) + }) + + It("should not roll out the placement policy with resources synchronized", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can mark the placement policy as having failed to synchronize resources on some clusters", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters) + forcePlacementPolicyReconciliation(appNamespaceName, policyName) + }) + + It("should not roll out the placement policy with resources failed to synchronize", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can mark the placement policy as having resources pending rollout at a stale generation", func() { + // Bump the generation first; writing the conditions at the current generation first would call for a + // rollout right away. + generation := bumpPlacementPolicyGeneration(appNamespaceName, policyName) + staleResourceCollectedCond := resourceCollectedCond(metav1.ConditionTrue, placementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected) + staleResourceCollectedCond.ObservedGeneration = generation - 1 + staleSynchronizedCond := synchronizedCond(metav1.ConditionFalse, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + staleSynchronizedCond.ObservedGeneration = generation - 1 + setPlacementPolicyConditions(appNamespaceName, policyName, staleResourceCollectedCond, staleSynchronizedCond) + forcePlacementPolicyReconciliation(appNamespaceName, policyName) + }) + + It("should not roll out the placement policy with stale conditions", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can mark the placement policy as having some clusters on an older resource revision", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision) + }) + + It("should roll out the latest resources to all the placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies marked for deletion", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var bindingNames []string + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy for deletion", func() { + markObjectForDeletion(newPlacementPolicyObj(appNamespaceName, policyName)) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the placement policy", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, ""), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can leave a binding manager claim of the auto-rollout controller on the placement policy", func() { + updatePlacementPolicyBindingManager(appNamespaceName, policyName, wantAutoRolloutBindingManager(appNamespaceName, policyName)) + }) + + It("should relinquish the binding manager role", func() { + released := bindingManagerReleasedOnAPIServerActual(appNamespaceName, policyName) + Eventually(released, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to relinquish the binding manager role") + }) + + It("can remove the test finalizer from the placement policy", func() { + policy := newPlacementPolicyObj(appNamespaceName, policyName) + removeTestFinalizer(policy) + Eventually(objectRemovedActual(policy), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement policy") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementPolicyObj(appNamespaceName, policyName)) + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("gating auto-rollouts of cluster placement policies", func() { + Context("auto-rollouts of cluster placement policies with resources pending rollout", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + var wantLastRolloutAttemptTimestamp string + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the cluster placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + It("should keep the spec and the other annotations of the cluster placement policy as they are", func() { + untouched := placementPolicySpecAndAnnotationsUntouchedActual(clusterScopedNS, policyName, autoRolloutEnabledAnnotations()) + Expect(untouched()).To(Succeed(), "Changed the cluster placement policy unexpectedly") + }) + + It("can mark the cluster placement policy as synchronized", func() { + wantLastRolloutAttemptTimestamp = lastRolloutAttemptTimestamp(clusterScopedNS, policyName) + markPlacementPolicyAsSynchronized(clusterScopedNS, policyName) + forcePlacementPolicyReconciliation(clusterScopedNS, policyName) + }) + + It("should not roll out the cluster placement policy again", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotNames[1]), + placementBindingsActual(clusterScopedNS, bindingNames, snapshotNames[0], 2), + placementPolicyAtRestActual(clusterScopedNS, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[0]), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[1]), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies with auto-rollout disabled", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + noRolloutActual := func() func() error { + return allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, ""), + ) + } + + BeforeAll(func() { + createConfigMap(configMapName) + // The cluster placement policy has no auto-rollout enabled annotation. + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, nil) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the cluster placement policy", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can set the auto-rollout enabled annotation to a value other than the enabling one", func() { + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutEnabledAnnotationKey] = "True" + }) + }) + + It("should still not roll out the cluster placement policy", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can enable auto-rollout for the cluster placement policy", func() { + notBefore = time.Now() + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutEnabledAnnotationKey] = rolloutv1alpha1.AutoRolloutEnabledAnnotationValue + }) + }) + + It("should roll out the latest resources to all the cluster placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies with auto-rollout disabled after a rollout", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + // A last rollout attempt timestamp long enough ago that the minimum wait period never holds a rollout back. + backdatedLastRolloutAttemptTimestamp := backdatedTimestamp(time.Hour * 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the cluster placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + It("can mark the cluster placement policy as synchronized", func() { + markPlacementPolicyAsSynchronized(clusterScopedNS, policyName) + }) + + It("can disable auto-rollout for the cluster placement policy", func() { + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + delete(annotations, rolloutv1alpha1.AutoRolloutEnabledAnnotationKey) + }) + }) + + It("can change the selected resources and backdate the last rollout attempt timestamp", func() { + updateConfigMapData(configMapName, map[string]string{configMapDataKey: altConfigMapDataValue}) + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedLastRolloutAttemptTimestamp + }) + }) + + It("can mark the cluster placement policy as having resources pending rollout again", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the cluster placement policy again", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotNames[1]), + placementBindingsActual(clusterScopedNS, bindingNames, snapshotNames[0], 2), + placementPolicyAtRestActual(clusterScopedNS, policyName, backdatedLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[0]), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[1]), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policy conditions that do or do not call for a rollout", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + noRolloutActual := func() func() error { + return allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, ""), + ) + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can force a reconciliation of the cluster placement policy with no conditions", func() { + forcePlacementPolicyReconciliation(clusterScopedNS, policyName) + }) + + It("should not roll out the cluster placement policy with no conditions", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can mark the cluster placement policy as having failed to collect some resources", func() { + setPlacementPolicyConditions(clusterScopedNS, policyName, + resourceCollectedCond(metav1.ConditionFalse, placementv1alpha1.PlacementPolicyResourceCollectedCondReasonFailedToCollectSomeResources), + synchronizedCond(metav1.ConditionFalse, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout), + ) + forcePlacementPolicyReconciliation(clusterScopedNS, policyName) + }) + + It("should not roll out the cluster placement policy with resources not collected", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can mark the cluster placement policy as synchronized", func() { + markPlacementPolicyAsSynchronized(clusterScopedNS, policyName) + forcePlacementPolicyReconciliation(clusterScopedNS, policyName) + }) + + It("should not roll out the cluster placement policy with resources synchronized", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can mark the cluster placement policy as having failed to synchronize resources on some clusters", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters) + forcePlacementPolicyReconciliation(clusterScopedNS, policyName) + }) + + It("should not roll out the cluster placement policy with resources failed to synchronize", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can mark the cluster placement policy as having resources pending rollout at a stale generation", func() { + // Bump the generation first; writing the conditions at the current generation first would call for a + // rollout right away. + generation := bumpPlacementPolicyGeneration(clusterScopedNS, policyName) + staleResourceCollectedCond := resourceCollectedCond(metav1.ConditionTrue, placementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected) + staleResourceCollectedCond.ObservedGeneration = generation - 1 + staleSynchronizedCond := synchronizedCond(metav1.ConditionFalse, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + staleSynchronizedCond.ObservedGeneration = generation - 1 + setPlacementPolicyConditions(clusterScopedNS, policyName, staleResourceCollectedCond, staleSynchronizedCond) + forcePlacementPolicyReconciliation(clusterScopedNS, policyName) + }) + + It("should not roll out the cluster placement policy with stale conditions", func() { + Consistently(noRolloutActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can mark the cluster placement policy as having some clusters on an older resource revision", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision) + }) + + It("should roll out the latest resources to all the cluster placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies marked for deletion", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var bindingNames []string + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy for deletion", func() { + markObjectForDeletion(newPlacementPolicyObj(clusterScopedNS, policyName)) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the cluster placement policy", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, ""), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can leave a binding manager claim of the auto-rollout controller on the cluster placement policy", func() { + updatePlacementPolicyBindingManager(clusterScopedNS, policyName, wantAutoRolloutBindingManager(clusterScopedNS, policyName)) + }) + + It("should relinquish the binding manager role", func() { + released := bindingManagerReleasedOnAPIServerActual(clusterScopedNS, policyName) + Eventually(released, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to relinquish the binding manager role") + }) + + It("can remove the test finalizer from the cluster placement policy", func() { + policy := newPlacementPolicyObj(clusterScopedNS, policyName) + removeTestFinalizer(policy) + Eventually(objectRemovedActual(policy), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement policy") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementPolicyObj(clusterScopedNS, policyName)) + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("managing the binding manager role of placement policies", func() { + Context("placement policies whose binding manager role another controller holds", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + otherClaim := otherBindingManager(randStr) + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + updatePlacementPolicyBindingManager(appNamespaceName, policyName, otherClaim) + // Wait until the cache (which the controller shares) observes the claim of the other controller, so that the + // controller never sees the role as free. + otherClaimObserved := placementPolicyAutoRolloutStateActual(appNamespaceName, policyName, otherClaim, "") + Eventually(otherClaimObserved, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the claim of the other controller") + waitForObjectsToBeCached(placementBindingObjs(appNamespaceName, bindingNames)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the placement policy while another controller holds the binding manager role", func() { + // The controller keeps retrying the claim (with backoff); in the meantime, the claim of the other + // controller stays, no timestamp is written (the controller writes it only after a rollout), and nothing + // is rolled out. + noRollout := allActual( + placementPolicyAutoRolloutStateActual(appNamespaceName, policyName, otherClaim, ""), + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can release the binding manager role held by the other controller", func() { + notBefore = time.Now() + updatePlacementPolicyBindingManager(appNamespaceName, policyName, nil) + }) + + It("should roll out the latest resources to all the placement bindings", func() { + // The release does not make the controller process the placement policy; the next claim retry does. + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies with auto-rollout disabled and a leftover claim of the auto-rollout controller", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var bindingNames []string + + BeforeAll(func() { + createConfigMap(configMapName) + // The placement policy has no auto-rollout enabled annotation. + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName, nil) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout, with a claim of the auto-rollout controller left on it", func() { + markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim(appNamespaceName, policyName) + }) + + It("should relinquish the binding manager role", func() { + released := bindingManagerReleasedOnAPIServerActual(appNamespaceName, policyName) + Eventually(released, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to relinquish the binding manager role") + }) + + It("should not roll out the placement policy", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, ""), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies within the minimum wait period and a leftover claim of the auto-rollout controller", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var bindingNames []string + var recentLastRolloutAttemptTimestamp string + + BeforeAll(func() { + createConfigMap(configMapName) + // The placement policy was rolled out just now; the (default) minimum wait period holds the next + // rollout back for longer than the test case runs. + recentLastRolloutAttemptTimestamp = backdatedTimestamp(0) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = recentLastRolloutAttemptTimestamp + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout, with a claim of the auto-rollout controller left on it", func() { + markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim(appNamespaceName, policyName) + }) + + It("should relinquish the binding manager role", func() { + released := bindingManagerReleasedOnAPIServerActual(appNamespaceName, policyName) + Eventually(released, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to relinquish the binding manager role") + }) + + It("should not roll out the placement policy within the minimum wait period", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, recentLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies with resources pending rollout and a leftover claim of the auto-rollout controller", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout, with a claim of the auto-rollout controller left on it", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim(appNamespaceName, policyName) + }) + + It("should roll out the latest resources to all the placement bindings, then relinquish the binding manager role", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("managing the binding manager role of cluster placement policies", func() { + Context("cluster placement policies whose binding manager role another controller holds", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + otherClaim := otherBindingManager(randStr) + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + updatePlacementPolicyBindingManager(clusterScopedNS, policyName, otherClaim) + // Wait until the cache (which the controller shares) observes the claim of the other controller, so that the + // controller never sees the role as free. + otherClaimObserved := placementPolicyAutoRolloutStateActual(clusterScopedNS, policyName, otherClaim, "") + Eventually(otherClaimObserved, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the claim of the other controller") + waitForObjectsToBeCached(placementBindingObjs(clusterScopedNS, bindingNames)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the cluster placement policy while another controller holds the binding manager role", func() { + // The controller keeps retrying the claim (with backoff); in the meantime, the claim of the other + // controller stays, no timestamp is written (the controller writes it only after a rollout), and nothing + // is rolled out. + noRollout := allActual( + placementPolicyAutoRolloutStateActual(clusterScopedNS, policyName, otherClaim, ""), + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can release the binding manager role held by the other controller", func() { + notBefore = time.Now() + updatePlacementPolicyBindingManager(clusterScopedNS, policyName, nil) + }) + + It("should roll out the latest resources to all the cluster placement bindings", func() { + // The release does not make the controller process the cluster placement policy; the next claim retry + // does. + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies with auto-rollout disabled and a leftover claim of the auto-rollout controller", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var bindingNames []string + + BeforeAll(func() { + createConfigMap(configMapName) + // The cluster placement policy has no auto-rollout enabled annotation. + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName, nil) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout, with a claim of the auto-rollout controller left on it", func() { + markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim(clusterScopedNS, policyName) + }) + + It("should relinquish the binding manager role", func() { + released := bindingManagerReleasedOnAPIServerActual(clusterScopedNS, policyName) + Eventually(released, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to relinquish the binding manager role") + }) + + It("should not roll out the cluster placement policy", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, ""), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies within the minimum wait period and a leftover claim of the auto-rollout controller", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var bindingNames []string + var recentLastRolloutAttemptTimestamp string + + BeforeAll(func() { + createConfigMap(configMapName) + // The cluster placement policy was rolled out just now; the (default) minimum wait period holds the next + // rollout back for longer than the test case runs. + recentLastRolloutAttemptTimestamp = backdatedTimestamp(0) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = recentLastRolloutAttemptTimestamp + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout, with a claim of the auto-rollout controller left on it", func() { + markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim(clusterScopedNS, policyName) + }) + + It("should relinquish the binding manager role", func() { + released := bindingManagerReleasedOnAPIServerActual(clusterScopedNS, policyName) + Eventually(released, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to relinquish the binding manager role") + }) + + It("should not roll out the cluster placement policy within the minimum wait period", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, recentLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies with resources pending rollout and a leftover claim of the auto-rollout controller", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout, with a claim of the auto-rollout controller left on it", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRolloutWithAutoRolloutClaim(clusterScopedNS, policyName) + }) + + It("should roll out the latest resources to all the cluster placement bindings, then relinquish the binding manager role", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("enforcing the minimum wait period between auto-rollouts of placement policies", func() { + Context("placement policies rolled out within the minimum wait period", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var wantLastRolloutAttemptTimestamp string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + // The placement policy was rolled out two minutes ago; the default minimum wait period (3 minutes) holds + // the next rollout back for about another minute. + wantLastRolloutAttemptTimestamp = backdatedTimestamp(time.Minute * 2) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = wantLastRolloutAttemptTimestamp + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the placement policy within the minimum wait period", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can shorten the minimum wait period of the placement policy", func() { + notBefore = time.Now() + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "1m" + }) + }) + + It("should roll out the latest resources to all the placement bindings right away", func() { + // The controller would otherwise come back to the placement policy only once the default minimum wait + // period has passed, i.e., in about a minute, which is longer than the check waits. + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies waiting for the minimum wait period to pass", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var wantLastRolloutAttemptTimestamp string + + BeforeAll(func() { + createConfigMap(configMapName) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "1m" + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as rolled out 45 seconds ago and as having resources pending rollout", func() { + // Write the timestamp only now (rather than at creation), so that the setup does not eat into the + // remaining 15 seconds of the minimum wait period; the timestamp change alone does not make the controller + // process the placement policy. + wantLastRolloutAttemptTimestamp = backdatedTimestamp(time.Second * 45) + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = wantLastRolloutAttemptTimestamp + }) + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the placement policy before the minimum wait period passes", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("should roll out the latest resources to all the placement bindings once the minimum wait period passes, with no further changes", func() { + lastRolloutAttempt, err := time.Parse(time.RFC3339, wantLastRolloutAttemptTimestamp) + Expect(err).To(Succeed(), "Failed to parse the last rollout attempt timestamp") + // The new rollout must not happen before the minimum wait period (1 minute) has passed since the last one. + rolledOut := rolledOutActual(policy, bindingNames, 0, lastRolloutAttempt.Add(time.Minute)) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies with minimum wait periods out of the allowed range", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var wantLastRolloutAttemptTimestamp string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + // A minimum wait period below the lower bound (1 minute), which the controller clamps up to the bound; the + // placement policy was rolled out 10 seconds ago. + wantLastRolloutAttemptTimestamp = backdatedTimestamp(time.Second * 10) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "1s" + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = wantLastRolloutAttemptTimestamp + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the placement policy within the lower bound of the minimum wait period", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can set a minimum wait period above the upper bound and backdate the last rollout attempt timestamp", func() { + // A minimum wait period above the upper bound (1 hour), which the controller clamps down to the bound; the + // last rollout attempt of the placement policy was 2 hours ago. Both annotations change in one write. + notBefore = time.Now() + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "5h" + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedTimestamp(time.Hour * 2) + }) + }) + + It("should roll out the latest resources to all the placement bindings right away", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement policies with malformed auto-rollout annotations", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var wantLastRolloutAttemptTimestamp string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + // A malformed minimum wait period, for which the controller uses the default (3 minutes); the placement + // policy was rolled out two minutes ago, i.e., within the default period but not within the lower bound + // (1 minute). + wantLastRolloutAttemptTimestamp = backdatedTimestamp(time.Minute * 2) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "soon" + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = wantLastRolloutAttemptTimestamp + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the placement policy within the default minimum wait period", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(appNamespaceName, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the placement policy unexpectedly") + }) + + It("can set a malformed last rollout attempt timestamp", func() { + // The controller treats a placement policy with a malformed last rollout attempt timestamp as never rolled + // out. A timestamp change alone does not make the controller process the placement policy; change the + // (still malformed) minimum wait period in the same write as well. + notBefore = time.Now() + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = "not-a-timestamp" + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "later" + }) + }) + + It("should roll out the latest resources to all the placement bindings right away, replacing the malformed timestamp", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("enforcing the minimum wait period between auto-rollouts of cluster placement policies", func() { + Context("cluster placement policies rolled out within the minimum wait period", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var wantLastRolloutAttemptTimestamp string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + // The cluster placement policy was rolled out two minutes ago; the default minimum wait period (3 minutes) + // holds the next rollout back for about another minute. + wantLastRolloutAttemptTimestamp = backdatedTimestamp(time.Minute * 2) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = wantLastRolloutAttemptTimestamp + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the cluster placement policy within the minimum wait period", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can shorten the minimum wait period of the cluster placement policy", func() { + notBefore = time.Now() + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "1m" + }) + }) + + It("should roll out the latest resources to all the cluster placement bindings right away", func() { + // The controller would otherwise come back to the cluster placement policy only once the default minimum + // wait period has passed, i.e., in about a minute, which is longer than the check waits. + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies waiting for the minimum wait period to pass", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var wantLastRolloutAttemptTimestamp string + + BeforeAll(func() { + createConfigMap(configMapName) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "1m" + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as rolled out 45 seconds ago and as having resources pending rollout", func() { + // Write the timestamp only now (rather than at creation), so that the setup does not eat into the + // remaining 15 seconds of the minimum wait period; the timestamp change alone does not make the controller + // process the cluster placement policy. + wantLastRolloutAttemptTimestamp = backdatedTimestamp(time.Second * 45) + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = wantLastRolloutAttemptTimestamp + }) + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the cluster placement policy before the minimum wait period passes", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("should roll out the latest resources to all the cluster placement bindings once the minimum wait period passes, with no further changes", func() { + lastRolloutAttempt, err := time.Parse(time.RFC3339, wantLastRolloutAttemptTimestamp) + Expect(err).To(Succeed(), "Failed to parse the last rollout attempt timestamp") + // The new rollout must not happen before the minimum wait period (1 minute) has passed since the last one. + rolledOut := rolledOutActual(policy, bindingNames, 0, lastRolloutAttempt.Add(time.Minute)) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies with minimum wait periods out of the allowed range", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var wantLastRolloutAttemptTimestamp string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + // A minimum wait period below the lower bound (1 minute), which the controller clamps up to the bound; the + // cluster placement policy was rolled out 10 seconds ago. + wantLastRolloutAttemptTimestamp = backdatedTimestamp(time.Second * 10) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "1s" + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = wantLastRolloutAttemptTimestamp + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the cluster placement policy within the lower bound of the minimum wait period", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can set a minimum wait period above the upper bound and backdate the last rollout attempt timestamp", func() { + // A minimum wait period above the upper bound (1 hour), which the controller clamps down to the bound; the + // last rollout attempt of the cluster placement policy was 2 hours ago. Both annotations change in one + // write. + notBefore = time.Now() + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "5h" + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedTimestamp(time.Hour * 2) + }) + }) + + It("should roll out the latest resources to all the cluster placement bindings right away", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement policies with malformed auto-rollout annotations", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var wantLastRolloutAttemptTimestamp string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + // A malformed minimum wait period, for which the controller uses the default (3 minutes); the placement + // policy was rolled out two minutes ago, i.e., within the default period but not within the lower bound + // (1 minute). + wantLastRolloutAttemptTimestamp = backdatedTimestamp(time.Minute * 2) + annotations := autoRolloutEnabledAnnotations() + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "soon" + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = wantLastRolloutAttemptTimestamp + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, annotations) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, lifecycleBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should not roll out the cluster placement policy within the default minimum wait period", func() { + noRollout := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementPolicyAtRestActual(clusterScopedNS, policyName, wantLastRolloutAttemptTimestamp), + ) + Consistently(noRollout, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out the cluster placement policy unexpectedly") + }) + + It("can set a malformed last rollout attempt timestamp", func() { + // The controller treats a cluster placement policy with a malformed last rollout attempt timestamp as never + // rolled out. A timestamp change alone does not make the controller process the cluster placement policy; + // change the (still malformed) minimum wait period in the same write as well. + notBefore = time.Now() + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = "not-a-timestamp" + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = "later" + }) + }) + + It("should roll out the latest resources to all the cluster placement bindings right away, replacing the malformed timestamp", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) +}) diff --git a/pkg/v1/controllers/autorollout/retrieval.go b/pkg/v1/controllers/autorollout/retrieval.go new file mode 100644 index 000000000..029cd3a94 --- /dev/null +++ b/pkg/v1/controllers/autorollout/retrieval.go @@ -0,0 +1,101 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package autorollout + +import ( + "context" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +// retrievePlacementPolicy retrieves the PlacementPolicy or ClusterPlacementPolicy resource that +// the key points to and returns it as a PlacementPolicyAccessor interface. +// +// An empty namespace in the key means that the resource is a ClusterPlacementPolicy. +func (r *Reconciler) retrievePlacementPolicy(ctx context.Context, key types.NamespacedName) (placementv1alpha1.PlacementPolicyAccessor, error) { + var placementPolicyAccessor placementv1alpha1.PlacementPolicyAccessor + var gvk schema.GroupVersionKind + if key.Namespace == "" { + // The object to retrieve is a ClusterPlacementPolicy. + placementPolicyAccessor = &placementv1alpha1.ClusterPlacementPolicy{} + gvk = placementv1alpha1.GroupVersion.WithKind(placementv1alpha1.ClusterPlacementPolicyKind) + } else { + // The object to retrieve is a PlacementPolicy. + placementPolicyAccessor = &placementv1alpha1.PlacementPolicy{} + gvk = placementv1alpha1.GroupVersion.WithKind(placementv1alpha1.PlacementPolicyKind) + } + + if err := r.hubClient.Get(ctx, key, placementPolicyAccessor); err != nil { + return nil, errors.NewAPIServerError(err, "failed to get placement policy object", true) + } + // The controller-runtime client (and the informer cache backing it) does not populate TypeMeta + // (apiVersion/kind) on typed Get calls, so the GVK is set explicitly here. + placementPolicyAccessor.GetObjectKind().SetGroupVersionKind(gvk) + return placementPolicyAccessor, nil +} + +// listAllPlacementBindings lists all the placement bindings (or cluster placement bindings) owned by the +// placement policy and returns them as PlacementBindingAccessor interfaces. Bindings that have been marked for +// deletion are excluded. +func (r *Reconciler) listAllPlacementBindings( + ctx context.Context, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, +) ([]placementv1alpha1.PlacementBindingAccessor, error) { + // List candidates by the policy name, then verify controller ownership to exclude bindings from an older policy instance. + ownerFieldSelector := client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByCustomFieldName: placementPolicy.GetName(), + } + + var placementBindings []placementv1alpha1.PlacementBindingAccessor + if placementPolicy.GetNamespace() == "" { + // The placement policy is cluster-scoped; list all cluster placement bindings owned by it. + cpbs := &placementv1alpha1.ClusterPlacementBindingList{} + if err := r.hubClient.List(ctx, cpbs, ownerFieldSelector); err != nil { + return nil, errors.NewAPIServerError(err, "failed to list cluster placement bindings", true, + "placementPolicy", klog.KObj(placementPolicy)) + } + for i := range cpbs.Items { + cpb := &cpbs.Items[i] + if cpb.DeletionTimestamp.IsZero() && metav1.IsControlledBy(cpb, placementPolicy) { + placementBindings = append(placementBindings, cpb) + } + } + } else { + // The placement policy is namespaced; list all placement bindings owned by it in its namespace. + namespaceSelector := client.InNamespace(placementPolicy.GetNamespace()) + pbs := &placementv1alpha1.PlacementBindingList{} + if err := r.hubClient.List(ctx, pbs, namespaceSelector, ownerFieldSelector); err != nil { + return nil, errors.NewAPIServerError(err, "failed to list placement bindings", true, + "placementPolicy", klog.KObj(placementPolicy)) + } + for i := range pbs.Items { + pb := &pbs.Items[i] + if pb.DeletionTimestamp.IsZero() && metav1.IsControlledBy(pb, placementPolicy) { + placementBindings = append(placementBindings, pb) + } + } + } + return placementBindings, nil +} diff --git a/pkg/v1/controllers/autorollout/rollout.go b/pkg/v1/controllers/autorollout/rollout.go new file mode 100644 index 000000000..292e2f7ec --- /dev/null +++ b/pkg/v1/controllers/autorollout/rollout.go @@ -0,0 +1,222 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package autorollout + +import ( + "context" + "time" + + "k8s.io/apimachinery/pkg/api/meta" + utilerrors "k8s.io/apimachinery/pkg/util/errors" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/condition" + "go.goms.io/fleet/pkg/utils/errors" +) + +// retrieveMinimumWaitPeriod returns the minimum wait period between consecutive auto-rollouts for the placement policy. +// +// The default period is used if the annotation is absent or cannot be parsed; a period outside of the +// allowed range is clamped to the closest bound. +func retrieveMinimumWaitPeriod(placementPolicy placementv1alpha1.PlacementPolicyAccessor) time.Duration { + val, found := placementPolicy.GetAnnotations()[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] + if !found { + return autoRolloutDefaultMinimumWaitPeriod + } + period, err := time.ParseDuration(val) + if err != nil { + klog.V(2).InfoS("Failed to parse the minimum wait period annotation; using the default", + "placementPolicy", klog.KObj(placementPolicy), + "observedValue", val, "defaultValue", autoRolloutDefaultMinimumWaitPeriod, "err", err, + "controller", controllerName) + return autoRolloutDefaultMinimumWaitPeriod + } + return min(max(period, autoRolloutMinimumWaitPeriodLowerBound), autoRolloutMinimumWaitPeriodUpperBound) +} + +// remainingWaitPeriod returns how much longer the controller must wait before it can roll out the placement policy +// again; zero or a negative value means that the minimum wait period has passed since the last rollout. +// +// A missing or malformed last rollout attempt timestamp is treated as if the placement policy has never been rolled +// out. A timestamp in the future (e.g., set by hand, or written by a controller instance with a skewed clock) is +// treated as if the placement policy has just been rolled out, i.e., the remaining wait period never exceeds the +// minimum wait period. +func remainingWaitPeriod(placementPolicy placementv1alpha1.PlacementPolicyAccessor, minWaitPeriod time.Duration, now time.Time) time.Duration { + val, found := placementPolicy.GetAnnotations()[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] + if !found { + return 0 + } + lastRolloutAttempt, err := time.Parse(time.RFC3339, val) + if err != nil { + klog.V(2).InfoS("Failed to parse the last rollout attempt timestamp annotation; treating the placement policy as never rolled out", + "placementPolicy", klog.KObj(placementPolicy), + "observedValue", val, "err", err, + "controller", controllerName) + return 0 + } + return min(minWaitPeriod-now.Sub(lastRolloutAttempt), minWaitPeriod) +} + +// isAutoRolloutEnabled returns true if auto-rollout is enabled for the placement policy. +func isAutoRolloutEnabled(placementPolicy placementv1alpha1.PlacementPolicyAccessor) bool { + return placementPolicy.GetAnnotations()[rolloutv1alpha1.AutoRolloutEnabledAnnotationKey] == rolloutv1alpha1.AutoRolloutEnabledAnnotationValue +} + +// shouldSkipProcessing checks if the placement policy should not be processed for an auto-rollout right now, which +// is the case if auto-rollout is not enabled, no rollout is needed, or the minimum wait period since the last +// rollout has not passed. +// +// If the processing should be skipped, a non-zero requeue-after duration is returned when the controller should come +// back to the placement policy after that duration (i.e., when only the minimum wait period is holding processing back). +func shouldSkipProcessing(placementPolicy placementv1alpha1.PlacementPolicyAccessor, minWaitPeriod time.Duration) (skip bool, requeueAfter time.Duration) { + if !isAutoRolloutEnabled(placementPolicy) { + klog.V(2).InfoS("Auto-rollout is not enabled for the placement policy, skipping processing", + "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + return true, 0 + } + + if !isRolloutNeeded(placementPolicy) { + klog.V(2).InfoS("No rollout is needed for the placement policy, skipping processing", + "placementPolicy", klog.KObj(placementPolicy), "controller", controllerName) + return true, 0 + } + + if remaining := remainingWaitPeriod(placementPolicy, minWaitPeriod, time.Now()); remaining > 0 { + klog.V(2).InfoS("The minimum wait period since the last rollout has not passed, skipping processing", + "placementPolicy", klog.KObj(placementPolicy), "minimumWaitPeriod", minWaitPeriod, "remaining", remaining, "controller", controllerName) + return true, remaining + } + return false, 0 +} + +// isRolloutNeeded returns true if the placement policy has collected all the resources for its latest +// generation and has changed resources pending rollout or clusters still on an older resource revision. +// ResourceCollected must be true and Synchronized must be false for the latest generation. +func isRolloutNeeded(placementPolicy placementv1alpha1.PlacementPolicyAccessor) bool { + conditions := placementPolicy.GetStatus().Conditions + generation := placementPolicy.GetGeneration() + + resourceCollectedCond := meta.FindStatusCondition(conditions, placementv1alpha1.PlacementPolicyCondTypeResourceCollected) + synchronizedCond := meta.FindStatusCondition(conditions, placementv1alpha1.PlacementPolicyCondTypeSynchronized) + return condition.IsConditionStatusTrue(resourceCollectedCond, generation) && + condition.IsConditionStatusFalse(synchronizedCond, generation) && + (synchronizedCond.Reason == placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout || + synchronizedCond.Reason == placementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision) +} + +// bookkeepLastRolloutAttemptTimestamp writes the current time to the last rollout attempt timestamp annotation of the +// placement policy, after a successful rollout or one that failed due to a user error. +// +// The annotation is written with a JSON merge patch that carries only this annotation (and no resource version), +// so that the write does not conflict with, or overwrite, changes made to the placement policy since it was +// retrieved; in particular, the status updates that the placement policy controller makes in reaction to the +// binding refreshes of the rollout. +func (r *Reconciler) bookkeepLastRolloutAttemptTimestamp(ctx context.Context, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, now time.Time) error { + // Take the copy before the annotations are modified, as the annotation map is shared with the object. + patch := client.MergeFrom(placementPolicy.DeepCopyObject().(placementv1alpha1.PlacementPolicyAccessor)) + + annotations := placementPolicy.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string, 1) + } + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = now.UTC().Format(time.RFC3339Nano) + placementPolicy.SetAnnotations(annotations) + + if err := r.hubClient.Patch(ctx, placementPolicy, patch); err != nil { + return errors.NewAPIServerError(err, "failed to patch the last rollout attempt timestamp annotation", false, + "placementPolicy", klog.KObj(placementPolicy)) + } + return nil +} + +func (r *Reconciler) rollout(ctx context.Context, placementPolicy placementv1alpha1.PlacementPolicyAccessor) error { + // List all the bindings created for the placement policy. + placementBindings, err := r.listAllPlacementBindings(ctx, placementPolicy) + if err != nil { + return err + } + klog.V(2).InfoS("Listed placement bindings", + "placementPolicy", klog.KObj(placementPolicy), "bindingCount", len(placementBindings), "controller", controllerName) + + // Request a new placement resource snapshot for the placement policy. + // + // The snapshot manager returns the latest snapshot as it is if it is still up-to-date, or creates a new one + // if the latest snapshot has become stale. + resourceSnapshots, _, err := r.placementResourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + if err != nil { + return errors.Wraps(err, "failed to retrieve or create the latest resource snapshot") + } + if len(resourceSnapshots) == 0 { + return errors.NewUnexpectedError(nil, "no resource snapshots returned") + } + // The first snapshot returned is always the primary snapshot. + latestPrimarySnapshot := resourceSnapshots[0] + klog.V(2).InfoS("Retrieved the latest placement resource snapshot", + "placementPolicy", klog.KObj(placementPolicy), "resourceSnapshot", klog.KObj(latestPrimarySnapshot), "controller", controllerName) + + // Refresh the bindings in parallel, so that they all point to the latest primary resource snapshot. + if err := r.refreshBindings(ctx, placementBindings, latestPrimarySnapshot.GetName()); err != nil { + return errors.Wraps(err, "failed to refresh the placement bindings with the latest resource snapshot") + } + return nil +} + +// refreshBindings updates, in parallel, all the placement bindings that do not point to the given primary resource +// snapshot so that they do. +func (r *Reconciler) refreshBindings( + ctx context.Context, + placementBindings []placementv1alpha1.PlacementBindingAccessor, + wantResourceSnapshotName string, +) error { + // Prepare a child context. + childCtx, childCancel := context.WithCancel(ctx) + defer childCancel() + + // Pre-allocate the slice to avoid contention. + errs := make([]error, len(placementBindings)) + doWork := func(piece int) { + binding := placementBindings[piece] + if binding.GetSpec().ResourceSnapshotName == wantResourceSnapshotName { + // The binding is already up-to-date; skip it. + return + } + + binding.GetSpec().ResourceSnapshotName = wantResourceSnapshotName + if err := r.hubClient.Update(childCtx, binding); err != nil { + errs[piece] = errors.NewAPIServerError(err, + "failed to update the placement binding with the latest resource snapshot", false, + "placementBinding", klog.KObj(binding), "resourceSnapshotName", wantResourceSnapshotName) + // Fail fast. + childCancel() + return + } + klog.V(2).InfoS("Refreshed the placement binding with the latest resource snapshot", + "placementBinding", klog.KObj(binding), "resourceSnapshotName", wantResourceSnapshotName, "controller", controllerName) + } + r.parallelizer.ParallelizeUntil(childCtx, len(placementBindings), doWork, "refreshBindings") + + if err := utilerrors.NewAggregate(errs); err != nil { + return errors.Wraps(err, "aggregated errors occurred when refreshing placement bindings") + } + // Return the context error (nil if the context has not been cancelled) here to avoid silently skipping + // bindings when the parent context is cancelled. + return ctx.Err() +} diff --git a/pkg/v1/controllers/autorollout/rollout_integration_test.go b/pkg/v1/controllers/autorollout/rollout_integration_test.go new file mode 100644 index 000000000..1faf5e973 --- /dev/null +++ b/pkg/v1/controllers/autorollout/rollout_integration_test.go @@ -0,0 +1,1370 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package autorollout + +import ( + "fmt" + "slices" + "strings" + "time" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + admissionregistrationv1 "k8s.io/api/admissionregistration/v1" + corev1 "k8s.io/api/core/v1" + eventsv1 "k8s.io/api/events/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +const ( + // The number of placement bindings that each placement policy in the rollout test cases has, unless noted + // otherwise. + rolloutBindingCount = 3 + // A number of placement bindings larger than the number of workers in the parallelizer that the auto-rollout + // controller uses to refresh placement bindings. + manyBindingCount = 12 + + // How long ago test cases backdate the last rollout attempt timestamp to, so that the (default) minimum wait period + // never holds a rollout back. + backdatedLastRolloutAttemptTimestampAge = time.Minute * 10 +) + +// The UID of a (fictional) older instance of a placement policy, i.e., one that has been deleted and re-created +// under the same name since it created a placement binding. +const olderPlacementPolicyInstanceUID = types.UID("00000000-0000-0000-0000-000000000001") + +// The details of the validating admission policy (and its binding) that test cases use to make the updates that the +// auto-rollout controller makes to a placement binding fail. +const ( + denySnapshotChangeNameTemplate = "deny-snapshot-change-%s" + // The label that selects the placement bindings that the validating admission policy applies to; the value is the + // random string of the test case, so that each test case only affects its own placement bindings. + denySnapshotChangeLabelKey = "kubefleet.dev/integration-test-deny-snapshot-change" + denySnapshotChangeMsg = "changes to the resource snapshot of this placement binding are denied for testing" + // The resource snapshot name that test cases write (as a dry run) to check whether the denial is in force. + dryRunResourceSnapshotName = "dry-run-resource-snapshot" +) + +// The details of the event that the auto-rollout controller reports when it fails to snapshot the selected resources +// of a placement policy due to a user error. +const ( + snapshotFailedEventReason = "FailedToSnapshotResourcesDuringAutoRollout" + snapshotFailedEventAction = "SnapshotResources" + // A part of the event note that identifies the user error that test cases cause: a selected resource that is too + // large to fit in a placement resource snapshot. + resourceTooLargeErrMsg = "too large to fit in a placement resource snapshot" +) + +// oversizedConfigMapData returns ConfigMap data that is too large to fit in a placement resource snapshot (see +// maxPerSnapshotResourceDataSizeBytes in the suite); a placement policy that selects a ConfigMap with this data cannot +// have its resources snapshotted, which is a user error. +func oversizedConfigMapData() map[string]string { + return map[string]string{ + configMapDataKey: strings.Repeat("x", 2*maxPerSnapshotResourceDataSizeBytes), + } +} + +// snapshotFailedEvent is the part of an event (reported on a snapshotting failure) that test cases check; the other +// fields (e.g., the name, the timestamps, and the reporting instance) vary. +type snapshotFailedEvent struct { + Type string + Reason string + Action string + ReportingController string + Regarding corev1.ObjectReference +} + +// snapshotFailedEventActual verifies that the auto-rollout controller has reported, with a Warning event regarding +// a placement policy, that it failed to snapshot the selected resources because one of them is too large. +func snapshotFailedEventActual(policy placementv1alpha1.PlacementPolicyAccessor) func() error { + return func() error { + // Events regarding cluster-scoped objects are recorded in the default namespace. + eventNS := policy.GetNamespace() + if eventNS == clusterScopedNS { + eventNS = metav1.NamespaceDefault + } + // Read the events from the API server directly, so that the cache does not start watching events. + events := &eventsv1.EventList{} + if err := hubAPIReader.List(ctx, events, client.InNamespace(eventNS)); err != nil { + return fmt.Errorf("failed to list events: %w", err) + } + + wantEvent := snapshotFailedEvent{ + Type: corev1.EventTypeWarning, + Reason: snapshotFailedEventReason, + Action: snapshotFailedEventAction, + ReportingController: controllerName, + Regarding: corev1.ObjectReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementPolicyKind(policy.GetNamespace()), + Namespace: policy.GetNamespace(), + Name: policy.GetName(), + UID: policy.GetUID(), + }, + } + for i := range events.Items { + event := &events.Items[i] + if event.Regarding.UID != policy.GetUID() || event.Reason != snapshotFailedEventReason { + continue + } + gotEvent := snapshotFailedEvent{ + Type: event.Type, + Reason: event.Reason, + Action: event.Action, + ReportingController: event.ReportingController, + Regarding: corev1.ObjectReference{ + APIVersion: event.Regarding.APIVersion, + Kind: event.Regarding.Kind, + Namespace: event.Regarding.Namespace, + Name: event.Regarding.Name, + UID: event.Regarding.UID, + }, + } + if diff := cmp.Diff(gotEvent, wantEvent); diff != "" { + return fmt.Errorf("snapshotting failure event diff (-got +want):\n%s", diff) + } + if !strings.Contains(event.Note, resourceTooLargeErrMsg) { + return fmt.Errorf("the snapshotting failure event note %q does not mention the user error (%q)", event.Note, resourceTooLargeErrMsg) + } + return nil + } + return fmt.Errorf("no snapshotting failure event regarding the placement policy has been found in namespace %q", eventNS) + } +} + +// variantPlacementBindingNames returns the name of a placement binding of the given variant (e.g., one that the +// auto-rollout controller should leave alone), along with the name of the (fictional) member cluster it binds to. +func variantPlacementBindingNames(randStr, variant string) (bindingName, clusterName string) { + nameSeed := fmt.Sprintf("%s-%s", randStr, variant) + return fmt.Sprintf(placementBindingNameTemplate, nameSeed, 0), fmt.Sprintf(memberClusterNameTemplate, nameSeed, 0) +} + +// newSnapshotChangeDenyPolicyObjs returns the validating admission policy (and its binding) that +// createSnapshotChangeDenyPolicy creates for the given random string, with only the names set, e.g., for cleanup. +func newSnapshotChangeDenyPolicyObjs(randStr string) (*admissionregistrationv1.ValidatingAdmissionPolicy, *admissionregistrationv1.ValidatingAdmissionPolicyBinding) { + name := fmt.Sprintf(denySnapshotChangeNameTemplate, randStr) + return &admissionregistrationv1.ValidatingAdmissionPolicy{ObjectMeta: metav1.ObjectMeta{Name: name}}, + &admissionregistrationv1.ValidatingAdmissionPolicyBinding{ObjectMeta: metav1.ObjectMeta{Name: name}} +} + +// createSnapshotChangeDenyPolicy creates a validating admission policy that denies changes to the resource snapshot of +// placement bindings (of both scopes), along with a binding that applies it to the placement bindings with the +// denySnapshotChangeLabelKey label set to the given random string. +// +// Other updates (e.g., to the labels of a placement binding) and status updates are not affected. +func createSnapshotChangeDenyPolicy(randStr string) { + policy, binding := newSnapshotChangeDenyPolicyObjs(randStr) + policy.Spec = admissionregistrationv1.ValidatingAdmissionPolicySpec{ + FailurePolicy: ptr.To(admissionregistrationv1.Fail), + MatchConstraints: &admissionregistrationv1.MatchResources{ + ResourceRules: []admissionregistrationv1.NamedRuleWithOperations{ + { + RuleWithOperations: admissionregistrationv1.RuleWithOperations{ + Operations: []admissionregistrationv1.OperationType{admissionregistrationv1.Update}, + Rule: admissionregistrationv1.Rule{ + APIGroups: []string{placementv1alpha1.GroupVersion.Group}, + APIVersions: []string{placementv1alpha1.GroupVersion.Version}, + Resources: []string{"placementbindings", "clusterplacementbindings"}, + }, + }, + }, + }, + }, + Validations: []admissionregistrationv1.Validation{ + { + Expression: "object.spec.resourceSnapshotName == oldObject.spec.resourceSnapshotName", + Message: denySnapshotChangeMsg, + }, + }, + } + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create the validating admission policy") + + binding.Spec = admissionregistrationv1.ValidatingAdmissionPolicyBindingSpec{ + PolicyName: policy.Name, + ValidationActions: []admissionregistrationv1.ValidationAction{admissionregistrationv1.Deny}, + MatchResources: &admissionregistrationv1.MatchResources{ + ObjectSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{denySnapshotChangeLabelKey: randStr}, + }, + }, + } + Expect(hubClient.Create(ctx, binding)).To(Succeed(), "Failed to create the validating admission policy binding") +} + +// placementBindingSnapshotChangeActual verifies whether the API server denies (wantDenied) or accepts changes to the +// resource snapshot of a placement binding, by making such a change as a dry run. +// +// The validating admission policy machinery of the API server picks up policy (binding) changes asynchronously; test +// cases use this check to wait until a change is in force. +func placementBindingSnapshotChangeActual(bindingNS, bindingName string, wantDenied bool) func() error { + return func() error { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); err != nil { + return fmt.Errorf("failed to retrieve placement binding %s: %w", bindingName, err) + } + binding.GetSpec().ResourceSnapshotName = dryRunResourceSnapshotName + err := hubClient.Update(ctx, binding, client.DryRunAll) + switch { + case wantDenied && err == nil: + return fmt.Errorf("the API server accepted the resource snapshot change on placement binding %s, want it denied", bindingName) + case wantDenied && !strings.Contains(err.Error(), denySnapshotChangeMsg): + return fmt.Errorf("the resource snapshot change on placement binding %s failed for an unexpected reason: %w", bindingName, err) + case !wantDenied && err != nil: + return fmt.Errorf("the API server did not accept the resource snapshot change on placement binding %s: %w", bindingName, err) + } + return nil + } +} + +// placementPolicyRolloutRetryingActual verifies that the auto-rollout controller is (still) in the middle of a rollout +// of a placement policy: it holds the binding manager role of the placement policy, and has not written the last +// rollout attempt timestamp (both checked on the same read of the placement policy). +func placementPolicyRolloutRetryingActual(policyNSName, policyName string) func() error { + return placementPolicyAutoRolloutStateActual(policyNSName, policyName, wantAutoRolloutBindingManager(policyNSName, policyName), "") +} + +var _ = Describe("rolling out placement policies", func() { + Context("placement policies with many placement bindings", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, manyBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the placement bindings, updating each of them once", func() { + rolledOut := allActual( + rolledOutActual(policy, bindingNames, 0, notBefore), + placementBindingsActual(appNamespaceName, bindingNames, snapshotName, 2), + ) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement bindings already pointing to the latest resource snapshot", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var outdatedBindingNames, upToDateBindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + outdatedBindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount) + // These placement bindings point to the resource snapshot that the rollout is about to create (snapshot + // names are deterministic). + upToDateBindingNames = createPlacementBindingsPointingTo(policy, randStr+"-up-to-date", rolloutBindingCount, snapshotName) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, slices.Concat(outdatedBindingNames, upToDateBindingNames)), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources, updating only the outdated placement bindings", func() { + rolledOut := allActual( + rolledOutActual(policy, slices.Concat(outdatedBindingNames, upToDateBindingNames), 0, notBefore), + placementBindingsActual(appNamespaceName, outdatedBindingNames, snapshotName, 2), + placementBindingsActual(appNamespaceName, upToDateBindingNames, snapshotName, 1), + ) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, slices.Concat(outdatedBindingNames, upToDateBindingNames)), + )...) + }) + }) + + Context("placement bindings that do not count towards a rollout", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + otherPolicyName := fmt.Sprintf(placementPolicyNameTemplate, randStr+"-other") + var snapshotName string + + deletingBindingName, deletingClusterName := variantPlacementBindingNames(randStr, "deleting") + noOwnerBindingName, noOwnerClusterName := variantPlacementBindingNames(randStr, "no-owner") + olderOwnerBindingName, olderOwnerClusterName := variantPlacementBindingNames(randStr, "older-owner") + nonControllerOwnerBindingName, nonControllerOwnerClusterName := variantPlacementBindingNames(randStr, "non-controller-owner") + otherPolicyBindingName, otherPolicyClusterName := variantPlacementBindingNames(randStr, "other-policy") + excludedBindingNames := []string{ + deletingBindingName, + noOwnerBindingName, + olderOwnerBindingName, + nonControllerOwnerBindingName, + otherPolicyBindingName, + } + + var policy placementv1alpha1.PlacementPolicyAccessor + var regularBindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + // Another placement policy, with auto-rollout disabled. + otherPolicy := createPlacementPolicy(appNamespaceName, otherPolicyName, configMapName, nil) + + // A regular placement binding, as a control. + regularBindingNames = createPlacementBindings(policy, randStr, 1) + + // A placement binding that is marked for deletion (see below). + createPlacementBinding(newOwnedPlacementBinding(policy, deletingBindingName, deletingClusterName, previousResourceSnapshotName)) + + // A placement binding that names the placement policy, but has no owner. + noOwnerBinding := newOwnedPlacementBinding(policy, noOwnerBindingName, noOwnerClusterName, previousResourceSnapshotName) + noOwnerBinding.SetOwnerReferences(nil) + createPlacementBinding(noOwnerBinding) + + // A placement binding that an older instance of the placement policy (of the same name) controls. + olderOwnerBinding := newOwnedPlacementBinding(policy, olderOwnerBindingName, olderOwnerClusterName, previousResourceSnapshotName) + olderOwnerRef := controllerOwnerRefTo(policy) + olderOwnerRef.UID = olderPlacementPolicyInstanceUID + olderOwnerBinding.SetOwnerReferences([]metav1.OwnerReference{olderOwnerRef}) + createPlacementBinding(olderOwnerBinding) + + // A placement binding that the placement policy owns, but does not control. + nonControllerOwnerBinding := newOwnedPlacementBinding(policy, nonControllerOwnerBindingName, nonControllerOwnerClusterName, previousResourceSnapshotName) + nonControllerOwnerRef := controllerOwnerRefTo(policy) + nonControllerOwnerRef.Controller = ptr.To(false) + nonControllerOwnerBinding.SetOwnerReferences([]metav1.OwnerReference{nonControllerOwnerRef}) + createPlacementBinding(nonControllerOwnerBinding) + + // A placement binding of the other placement policy. + createPlacementBinding(newOwnedPlacementBinding(otherPolicy, otherPolicyBindingName, otherPolicyClusterName, previousResourceSnapshotName)) + + markObjectForDeletion(newPlacementBindingObj(appNamespaceName, deletingBindingName)) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, slices.Concat(regularBindingNames, excludedBindingNames)), policy, otherPolicy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to the regular placement binding", func() { + rolledOut := rolledOutActual(policy, regularBindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + It("should leave the placement bindings that do not count untouched", func() { + untouched := placementBindingsResourceSnapshotActual(appNamespaceName, excludedBindingNames, previousResourceSnapshotName) + Consistently(untouched, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Refreshed a placement binding that does not count unexpectedly") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementBindingObj(appNamespaceName, deletingBindingName)) + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newPlacementPolicyObj(appNamespaceName, otherPolicyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, slices.Concat(regularBindingNames, excludedBindingNames)), + )...) + }) + }) + + Context("placement bindings of same-named placement policies in other scopes", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + crossNSOwnerBindingName, crossNSOwnerClusterName := variantPlacementBindingNames(randStr, "cross-namespace-owner") + + var policy placementv1alpha1.PlacementPolicyAccessor + // The placement bindings in all scopes share the same names. + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // Placement policies of the same name in another namespace and in the cluster scope, with auto-rollout + // disabled. + otherNSPolicy := createPlacementPolicy(otherAppNamespaceName, policyName, configMapName, nil) + clusterPolicy := createPlacementPolicy(clusterScopedNS, policyName, configMapName, nil) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(clusterPolicy, 0) + + bindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount) + createPlacementBindings(otherNSPolicy, randStr, rolloutBindingCount) + createPlacementBindings(clusterPolicy, randStr, rolloutBindingCount) + + // A placement binding in the other namespace with the placement policy (in the app namespace) as its + // controller owner. Kubernetes treats cross-namespace owner references as absent; this binding is excluded + // only by the namespace the auto-rollout controller lists placement bindings in, unlike the other bindings + // in the other namespace, which the controller ownership check excludes as well. + crossNSOwnerBinding := newOwnedPlacementBinding(policy, crossNSOwnerBindingName, crossNSOwnerClusterName, previousResourceSnapshotName) + crossNSOwnerBinding.SetNamespace(otherAppNamespaceName) + createPlacementBinding(crossNSOwnerBinding) + + waitForObjectsToBeCached(slices.Concat( + []client.Object{policy, otherNSPolicy, clusterPolicy}, + placementBindingObjs(appNamespaceName, bindingNames), + placementBindingObjs(otherAppNamespaceName, slices.Concat(bindingNames, []string{crossNSOwnerBindingName})), + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to the placement bindings of the placement policy", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + It("should leave the placement bindings in the other namespace and in the cluster scope untouched", func() { + untouched := allActual( + placementBindingsActual(otherAppNamespaceName, slices.Concat(bindingNames, []string{crossNSOwnerBindingName}), previousResourceSnapshotName, 1), + placementBindingsActual(clusterScopedNS, bindingNames, previousResourceSnapshotName, 1), + placementResourceSnapshotAbsentActual(otherAppNamespaceName, snapshotName), + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotName), + ) + Consistently(untouched, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to placement bindings in another scope unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newPlacementPolicyObj(otherAppNamespaceName, policyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + placementBindingObjs(otherAppNamespaceName, slices.Concat(bindingNames, []string{crossNSOwnerBindingName})), + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("placement policies with no placement bindings", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + waitForObjectsToBeCached(policy) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should snapshot the latest resources and complete the rollout", func() { + rolledOut := rolledOutActual(policy, nil, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + AfterAll(func() { + removeObjects( + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + ) + }) + }) + + Context("successive rollouts of placement policies with changed resources", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + bindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + It("can mark the placement policy as synchronized", func() { + markPlacementPolicyAsSynchronized(appNamespaceName, policyName) + }) + + It("can make the selected resources too large to snapshot and backdate the last rollout attempt timestamp", func() { + updateConfigMapData(configMapName, oversizedConfigMapData()) + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedTimestamp(backdatedLastRolloutAttemptTimestampAge) + }) + }) + + It("can mark the placement policy as having resources pending rollout again", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should report the snapshotting failure, write the last rollout attempt timestamp, and release the binding manager role", func() { + // The controller writes the timestamp for a rollout that fails due to a user error too, so that the minimum + // wait period spaces out the retries. + failed := allActual( + snapshotFailedEventActual(policy), + placementPolicyRolledOutActual(appNamespaceName, policyName, notBefore), + ) + Eventually(failed, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the snapshotting failure") + }) + + It("should not retry the rollout within the minimum wait period, even when it processes the placement policy again", func() { + forcePlacementPolicyReconciliation(appNamespaceName, policyName) + // A retry would claim (and then release) the binding manager role again, i.e., write to the placement policy. + resourceVersion := placementPolicyResourceVersionOnAPIServer(appNamespaceName, policyName) + noRetry := allActual( + placementPolicyUnwrittenOnAPIServerActual(appNamespaceName, policyName, resourceVersion), + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotNames[1]), + placementBindingsActual(appNamespaceName, bindingNames, snapshotNames[0], 2), + ) + Consistently(noRetry, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Retried the rollout unexpectedly") + }) + + It("can change the selected resources to a size that fits, backdate the last rollout attempt timestamp, and make the controller retry the rollout", func() { + updateConfigMapData(configMapName, map[string]string{configMapDataKey: altConfigMapDataValue}) + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedTimestamp(backdatedLastRolloutAttemptTimestampAge) + }) + notBefore = time.Now() + // The timestamp change alone does not make the controller process the placement policy. + forcePlacementPolicyReconciliation(appNamespaceName, policyName) + }) + + It("should roll out the changed resources, as a new resource snapshot, to all the placement bindings", func() { + rolledOut := allActual( + rolledOutActual(policy, bindingNames, 1, notBefore), + placementBindingsActual(appNamespaceName, bindingNames, snapshotNames[1], 3), + ) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy again") + }) + + It("should keep the previous resource snapshot", func() { + Expect(primaryPlacementResourceSnapshotCreatedActual(policy, 0)()).To(Succeed(), "Failed to keep the previous resource snapshot") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[0]), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[1]), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("successive rollouts of placement policies with unchanged resources", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + bindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, bindingNames), policy)...) + }) + + It("can mark the placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy") + }) + + It("can mark the placement policy as synchronized", func() { + markPlacementPolicyAsSynchronized(appNamespaceName, policyName) + }) + + It("can backdate the last rollout attempt timestamp", func() { + updatePlacementPolicyAnnotations(appNamespaceName, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedTimestamp(backdatedLastRolloutAttemptTimestampAge) + }) + }) + + It("can mark the placement policy as having some clusters on an older resource revision", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision) + }) + + It("should complete the rollout again", func() { + rolledOut := placementPolicyRolledOutActual(appNamespaceName, policyName, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the placement policy again") + }) + + It("should reuse the latest resource snapshot and leave the placement bindings as they are", func() { + untouched := allActual( + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotNames[1]), + placementBindingsActual(appNamespaceName, bindingNames, snapshotNames[0], 2), + ) + Consistently(untouched, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Changed the resource snapshot or the placement bindings unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[0]), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[1]), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("placement binding updates that fail", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + deniedBindingName, deniedClusterName := variantPlacementBindingNames(randStr, "denied") + denyPolicy, denyPolicyBinding := newSnapshotChangeDenyPolicyObjs(randStr) + + var policy placementv1alpha1.PlacementPolicyAccessor + var regularBindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + regularBindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount-1) + // The API server denies changes to the resource snapshot of this placement binding (see below). + deniedBinding := newOwnedPlacementBinding(policy, deniedBindingName, deniedClusterName, previousResourceSnapshotName) + deniedBinding.SetLabels(map[string]string{denySnapshotChangeLabelKey: randStr}) + createPlacementBinding(deniedBinding) + + createSnapshotChangeDenyPolicy(randStr) + waitForObjectsToBeCached(append(placementBindingObjs(appNamespaceName, slices.Concat(regularBindingNames, []string{deniedBindingName})), policy)...) + }) + + It("should deny changes to the resource snapshot of the labeled placement binding", func() { + denied := placementBindingSnapshotChangeActual(appNamespaceName, deniedBindingName, true) + Eventually(denied, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to put the denial in force") + }) + + It("can mark the placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(appNamespaceName, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should snapshot the latest resources and claim the binding manager role", func() { + started := allActual( + primaryPlacementResourceSnapshotCreatedActual(policy, 0), + placementPolicyRolloutRetryingActual(appNamespaceName, policyName), + ) + Eventually(started, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start the rollout") + }) + + It("should keep retrying the rollout, holding the binding manager role", func() { + // The other placement bindings may or may not have been refreshed, as the controller stops refreshing + // placement bindings at the first failure. + retrying := allActual( + placementPolicyRolloutRetryingActual(appNamespaceName, policyName), + placementBindingsResourceSnapshotActual(appNamespaceName, []string{deniedBindingName}, previousResourceSnapshotName), + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotNames[1]), + ) + Consistently(retrying, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Failed to keep retrying the rollout") + }) + + It("can lift the denial", func() { + notBefore = time.Now() + removeObjects(denyPolicyBinding) + allowed := placementBindingSnapshotChangeActual(appNamespaceName, deniedBindingName, false) + Eventually(allowed, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to lift the denial") + }) + + It("can make the controller retry the rollout right away", func() { + // The controller backs off further after each failed attempt; as it holds the binding manager role, any + // update to the placement policy makes it retry right away instead. + touchPlacementPolicy(appNamespaceName, policyName) + }) + + It("should complete the rollout", func() { + rolledOut := rolledOutActual(policy, slices.Concat(regularBindingNames, []string{deniedBindingName}), 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the rollout") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + denyPolicyBinding, + denyPolicy, + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[0]), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotNames[1]), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(appNamespaceName, slices.Concat(regularBindingNames, []string{deniedBindingName})), + )...) + }) + }) +}) + +var _ = Describe("rolling out cluster placement policies", func() { + Context("cluster placement policies with many cluster placement bindings", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + bindingNames = createPlacementBindings(policy, randStr, manyBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the cluster placement bindings, updating each of them once", func() { + rolledOut := allActual( + rolledOutActual(policy, bindingNames, 0, notBefore), + placementBindingsActual(clusterScopedNS, bindingNames, snapshotName, 2), + ) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement bindings already pointing to the latest resource snapshot", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var outdatedBindingNames, upToDateBindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + outdatedBindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount) + // These cluster placement bindings point to the resource snapshot that the rollout is about to create + // (snapshot names are deterministic). + upToDateBindingNames = createPlacementBindingsPointingTo(policy, randStr+"-up-to-date", rolloutBindingCount, snapshotName) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, slices.Concat(outdatedBindingNames, upToDateBindingNames)), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources, updating only the outdated cluster placement bindings", func() { + rolledOut := allActual( + rolledOutActual(policy, slices.Concat(outdatedBindingNames, upToDateBindingNames), 0, notBefore), + placementBindingsActual(clusterScopedNS, outdatedBindingNames, snapshotName, 2), + placementBindingsActual(clusterScopedNS, upToDateBindingNames, snapshotName, 1), + ) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, slices.Concat(outdatedBindingNames, upToDateBindingNames)), + )...) + }) + }) + + Context("cluster placement bindings that do not count towards a rollout", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + otherPolicyName := fmt.Sprintf(placementPolicyNameTemplate, randStr+"-other") + var snapshotName string + + deletingBindingName, deletingClusterName := variantPlacementBindingNames(randStr, "deleting") + noOwnerBindingName, noOwnerClusterName := variantPlacementBindingNames(randStr, "no-owner") + olderOwnerBindingName, olderOwnerClusterName := variantPlacementBindingNames(randStr, "older-owner") + nonControllerOwnerBindingName, nonControllerOwnerClusterName := variantPlacementBindingNames(randStr, "non-controller-owner") + otherPolicyBindingName, otherPolicyClusterName := variantPlacementBindingNames(randStr, "other-policy") + excludedBindingNames := []string{ + deletingBindingName, + noOwnerBindingName, + olderOwnerBindingName, + nonControllerOwnerBindingName, + otherPolicyBindingName, + } + + var policy placementv1alpha1.PlacementPolicyAccessor + var regularBindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + // Another cluster placement policy, with auto-rollout disabled. + otherPolicy := createPlacementPolicy(clusterScopedNS, otherPolicyName, configMapName, nil) + + // A regular cluster placement binding, as a control. + regularBindingNames = createPlacementBindings(policy, randStr, 1) + + // A cluster placement binding that is marked for deletion (see below). + createPlacementBinding(newOwnedPlacementBinding(policy, deletingBindingName, deletingClusterName, previousResourceSnapshotName)) + + // A cluster placement binding that names the cluster placement policy, but has no owner. + noOwnerBinding := newOwnedPlacementBinding(policy, noOwnerBindingName, noOwnerClusterName, previousResourceSnapshotName) + noOwnerBinding.SetOwnerReferences(nil) + createPlacementBinding(noOwnerBinding) + + // A cluster placement binding that an older instance of the cluster placement policy (of the same name) + // controls. + olderOwnerBinding := newOwnedPlacementBinding(policy, olderOwnerBindingName, olderOwnerClusterName, previousResourceSnapshotName) + olderOwnerRef := controllerOwnerRefTo(policy) + olderOwnerRef.UID = olderPlacementPolicyInstanceUID + olderOwnerBinding.SetOwnerReferences([]metav1.OwnerReference{olderOwnerRef}) + createPlacementBinding(olderOwnerBinding) + + // A cluster placement binding that the cluster placement policy owns, but does not control. + nonControllerOwnerBinding := newOwnedPlacementBinding(policy, nonControllerOwnerBindingName, nonControllerOwnerClusterName, previousResourceSnapshotName) + nonControllerOwnerRef := controllerOwnerRefTo(policy) + nonControllerOwnerRef.Controller = ptr.To(false) + nonControllerOwnerBinding.SetOwnerReferences([]metav1.OwnerReference{nonControllerOwnerRef}) + createPlacementBinding(nonControllerOwnerBinding) + + // A cluster placement binding of the other cluster placement policy. + createPlacementBinding(newOwnedPlacementBinding(otherPolicy, otherPolicyBindingName, otherPolicyClusterName, previousResourceSnapshotName)) + + markObjectForDeletion(newPlacementBindingObj(clusterScopedNS, deletingBindingName)) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, slices.Concat(regularBindingNames, excludedBindingNames)), policy, otherPolicy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to the regular cluster placement binding", func() { + rolledOut := rolledOutActual(policy, regularBindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + It("should leave the cluster placement bindings that do not count untouched", func() { + untouched := placementBindingsResourceSnapshotActual(clusterScopedNS, excludedBindingNames, previousResourceSnapshotName) + Consistently(untouched, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Refreshed a cluster placement binding that does not count unexpectedly") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementBindingObj(clusterScopedNS, deletingBindingName)) + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newPlacementPolicyObj(clusterScopedNS, otherPolicyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, slices.Concat(regularBindingNames, excludedBindingNames)), + )...) + }) + }) + + Context("cluster placement bindings and the placement bindings of a same-named placement policy", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + // The placement bindings in both scopes share the same names. + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // A placement policy of the same name in the app namespace, with auto-rollout disabled. + nsPolicy := createPlacementPolicy(appNamespaceName, policyName, configMapName, nil) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(nsPolicy, 0) + + bindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount) + createPlacementBindings(nsPolicy, randStr, rolloutBindingCount) + waitForObjectsToBeCached(slices.Concat( + []client.Object{policy, nsPolicy}, + placementBindingObjs(clusterScopedNS, bindingNames), + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to the cluster placement bindings of the cluster placement policy", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + It("should leave the placement bindings of the same-named placement policy untouched", func() { + untouched := allActual( + placementBindingsActual(appNamespaceName, bindingNames, previousResourceSnapshotName, 1), + placementResourceSnapshotAbsentActual(appNamespaceName, snapshotName), + ) + Consistently(untouched, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to placement bindings in another scope unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + placementBindingObjs(appNamespaceName, bindingNames), + )...) + }) + }) + + Context("cluster placement policies with no cluster placement bindings", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotName string + + var policy placementv1alpha1.PlacementPolicyAccessor + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + waitForObjectsToBeCached(policy) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should snapshot the latest resources and complete the rollout", func() { + rolledOut := rolledOutActual(policy, nil, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + AfterAll(func() { + removeObjects( + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + ) + }) + }) + + Context("successive rollouts of cluster placement policies with changed resources", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + bindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the cluster placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + It("can mark the cluster placement policy as synchronized", func() { + markPlacementPolicyAsSynchronized(clusterScopedNS, policyName) + }) + + It("can make the selected resources too large to snapshot and backdate the last rollout attempt timestamp", func() { + updateConfigMapData(configMapName, oversizedConfigMapData()) + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedTimestamp(backdatedLastRolloutAttemptTimestampAge) + }) + }) + + It("can mark the cluster placement policy as having resources pending rollout again", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should report the snapshotting failure, write the last rollout attempt timestamp, and release the binding manager role", func() { + // The controller writes the timestamp for a rollout that fails due to a user error too, so that the minimum + // wait period spaces out the retries. + failed := allActual( + snapshotFailedEventActual(policy), + placementPolicyRolledOutActual(clusterScopedNS, policyName, notBefore), + ) + Eventually(failed, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the snapshotting failure") + }) + + It("should not retry the rollout within the minimum wait period, even when it processes the cluster placement policy again", func() { + forcePlacementPolicyReconciliation(clusterScopedNS, policyName) + // A retry would claim (and then release) the binding manager role again, i.e., write to the cluster placement + // policy. + resourceVersion := placementPolicyResourceVersionOnAPIServer(clusterScopedNS, policyName) + noRetry := allActual( + placementPolicyUnwrittenOnAPIServerActual(clusterScopedNS, policyName, resourceVersion), + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotNames[1]), + placementBindingsActual(clusterScopedNS, bindingNames, snapshotNames[0], 2), + ) + Consistently(noRetry, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Retried the rollout unexpectedly") + }) + + It("can change the selected resources to a size that fits, backdate the last rollout attempt timestamp, and make the controller retry the rollout", func() { + updateConfigMapData(configMapName, map[string]string{configMapDataKey: altConfigMapDataValue}) + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedTimestamp(backdatedLastRolloutAttemptTimestampAge) + }) + notBefore = time.Now() + // The timestamp change alone does not make the controller process the cluster placement policy. + forcePlacementPolicyReconciliation(clusterScopedNS, policyName) + }) + + It("should roll out the changed resources, as a new resource snapshot, to all the cluster placement bindings", func() { + rolledOut := allActual( + rolledOutActual(policy, bindingNames, 1, notBefore), + placementBindingsActual(clusterScopedNS, bindingNames, snapshotNames[1], 3), + ) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy again") + }) + + It("should keep the previous resource snapshot", func() { + Expect(primaryPlacementResourceSnapshotCreatedActual(policy, 0)()).To(Succeed(), "Failed to keep the previous resource snapshot") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[0]), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[1]), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("successive rollouts of cluster placement policies with unchanged resources", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + + var policy placementv1alpha1.PlacementPolicyAccessor + var bindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + bindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, bindingNames), policy)...) + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should roll out the latest resources to all the cluster placement bindings", func() { + rolledOut := rolledOutActual(policy, bindingNames, 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy") + }) + + It("can mark the cluster placement policy as synchronized", func() { + markPlacementPolicyAsSynchronized(clusterScopedNS, policyName) + }) + + It("can backdate the last rollout attempt timestamp", func() { + updatePlacementPolicyAnnotations(clusterScopedNS, policyName, func(annotations map[string]string) { + annotations[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = backdatedTimestamp(backdatedLastRolloutAttemptTimestampAge) + }) + }) + + It("can mark the cluster placement policy as having some clusters on an older resource revision", func() { + notBefore = time.Now() + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision) + }) + + It("should complete the rollout again", func() { + rolledOut := placementPolicyRolledOutActual(clusterScopedNS, policyName, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out the cluster placement policy again") + }) + + It("should reuse the latest resource snapshot and leave the cluster placement bindings as they are", func() { + untouched := allActual( + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotNames[1]), + placementBindingsActual(clusterScopedNS, bindingNames, snapshotNames[0], 2), + ) + Consistently(untouched, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Changed the resource snapshot or the cluster placement bindings unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[0]), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[1]), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, bindingNames), + )...) + }) + }) + + Context("cluster placement binding updates that fail", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + var snapshotNames []string + deniedBindingName, deniedClusterName := variantPlacementBindingNames(randStr, "denied") + denyPolicy, denyPolicyBinding := newSnapshotChangeDenyPolicyObjs(randStr) + + var policy placementv1alpha1.PlacementPolicyAccessor + var regularBindingNames []string + var notBefore time.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName, autoRolloutEnabledAnnotations()) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotNames = []string{ + primaryPlacementResourceSnapshotName(policy, 0), + primaryPlacementResourceSnapshotName(policy, 1), + } + regularBindingNames = createPlacementBindings(policy, randStr, rolloutBindingCount-1) + // The API server denies changes to the resource snapshot of this cluster placement binding (see below). + deniedBinding := newOwnedPlacementBinding(policy, deniedBindingName, deniedClusterName, previousResourceSnapshotName) + deniedBinding.SetLabels(map[string]string{denySnapshotChangeLabelKey: randStr}) + createPlacementBinding(deniedBinding) + + createSnapshotChangeDenyPolicy(randStr) + waitForObjectsToBeCached(append(placementBindingObjs(clusterScopedNS, slices.Concat(regularBindingNames, []string{deniedBindingName})), policy)...) + }) + + It("should deny changes to the resource snapshot of the labeled cluster placement binding", func() { + denied := placementBindingSnapshotChangeActual(clusterScopedNS, deniedBindingName, true) + Eventually(denied, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to put the denial in force") + }) + + It("can mark the cluster placement policy as having resources pending rollout", func() { + markPlacementPolicyAsPendingRollout(clusterScopedNS, policyName, placementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout) + }) + + It("should snapshot the latest resources and claim the binding manager role", func() { + started := allActual( + primaryPlacementResourceSnapshotCreatedActual(policy, 0), + placementPolicyRolloutRetryingActual(clusterScopedNS, policyName), + ) + Eventually(started, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start the rollout") + }) + + It("should keep retrying the rollout, holding the binding manager role", func() { + // The other cluster placement bindings may or may not have been refreshed, as the controller stops + // refreshing cluster placement bindings at the first failure. + retrying := allActual( + placementPolicyRolloutRetryingActual(clusterScopedNS, policyName), + placementBindingsResourceSnapshotActual(clusterScopedNS, []string{deniedBindingName}, previousResourceSnapshotName), + placementResourceSnapshotAbsentActual(clusterScopedNS, snapshotNames[1]), + ) + Consistently(retrying, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Failed to keep retrying the rollout") + }) + + It("can lift the denial", func() { + notBefore = time.Now() + removeObjects(denyPolicyBinding) + allowed := placementBindingSnapshotChangeActual(clusterScopedNS, deniedBindingName, false) + Eventually(allowed, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to lift the denial") + }) + + It("can make the controller retry the rollout right away", func() { + // The controller backs off further after each failed attempt; as it holds the binding manager role, any + // update to the cluster placement policy makes it retry right away instead. + touchPlacementPolicy(clusterScopedNS, policyName) + }) + + It("should complete the rollout", func() { + rolledOut := rolledOutActual(policy, slices.Concat(regularBindingNames, []string{deniedBindingName}), 0, notBefore) + Eventually(rolledOut, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the rollout") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + denyPolicyBinding, + denyPolicy, + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[0]), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotNames[1]), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + placementBindingObjs(clusterScopedNS, slices.Concat(regularBindingNames, []string{deniedBindingName})), + )...) + }) + }) +}) diff --git a/pkg/v1/controllers/autorollout/suite_test.go b/pkg/v1/controllers/autorollout/suite_test.go new file mode 100644 index 000000000..c76654742 --- /dev/null +++ b/pkg/v1/controllers/autorollout/suite_test.go @@ -0,0 +1,216 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package autorollout + +import ( + "context" + "flag" + "path/filepath" + "sync" + "testing" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + "sigs.k8s.io/controller-runtime/pkg/manager" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/parallelizer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + // The namespace in which namespace-scoped objects (e.g., PlacementPolicy and PlacementBinding objects, and the + // ConfigMap objects that placement policies select) are created. + appNamespaceName = "app" + // Another namespace, for verifying that namespace-scoped objects are not looked up across namespaces. + // + // Note that the namespaces are created once per (Ginkgo) process, as the envtest environment does not + // support namespace deletion. + otherAppNamespaceName = "other-app" + + // The number of max. concurrent reconciliations for the auto-rollout controller. + maxConcurrentReconciles = 4 + // The number of workers in the parallelizer that the auto-rollout controller uses to refresh placement bindings + // in parallel. + workerCount = 4 + + // The number of mutex slots used by the placement resource snapshot manager. + placementResourceSnapshotMuSlotCnt = 256 + // The max. size (in bytes) of the resource data in a single placement resource snapshot. + // + // The limit is far lower than the default (~1.2 MiB), so that test cases can select a resource too large to + // snapshot (a user error) without writing megabytes of data; the resources that test cases select otherwise are + // a few hundred bytes each. + maxPerSnapshotResourceDataSizeBytes = 16 * 1024 + + // The base and max. delays of the rate limiter for binding manager claims used by the auto-rollout controller. + // + // The defaults (5 seconds and 180 seconds respectively) are too long for the integration tests; + // shorter delays are used here to keep the tests fast. + rateLimiterBaseDelay = time.Millisecond * 250 + rateLimiterMaxDelay = time.Second * 2 +) + +var ( + hubCfg *rest.Config + hubEnv *envtest.Environment + hubMgr manager.Manager + // hubClient is the client from the controller manager; reads are served from its cache, which + // matches how the auto-rollout controller reads objects. + hubClient client.Client + // hubAPIReader reads objects from the API server directly (i.e., bypassing the cache); test cases use it only + // when a read must reflect a write that the test case has just made (e.g., to tell that the controller has + // undone the write, rather than that the cache has yet to observe it). + hubAPIReader client.Reader + + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup +) + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Auto-Rollout Controller Integration Test Suite") +} + +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + logger := zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)) + klog.SetLogger(logger) + ctrl.SetLogger(logger) + + By("Bootstrapping the test environment") + hubEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("../../../../", "config", "crd", "bases")}, + ErrorIfCRDPathMissing: true, + } + + var err error + hubCfg, err = hubEnv.Start() + Expect(err).Should(Succeed()) + Expect(hubCfg).NotTo(BeNil()) + + By("Setting up the scheme") + Expect(placementv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + By("Setting up the controller manager") + hubMgr, err = ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{ + BindAddress: "0", + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).Should(Succeed()) + + // Use the same (cache-backed) client as the controller, so that test cases read objects the same way the + // controller does; a test case that has waited for the cache to observe a change can be sure that the controller + // observes it too. + By("Setting the hub client to the one used by the controller manager") + hubClient = hubMgr.GetClient() + hubAPIReader = hubMgr.GetAPIReader() + + // The auto-rollout controller (and the placement resource snapshot manager) list objects via custom field + // indexes; the indexes must be set up before the manager starts. + By("Setting up the field indexes") + Expect(fieldindexers.SetupWithHubAgentControllerManager(ctx, hubMgr)).Should(Succeed()) + + By("Setting up the placement resource snapshot manager") + hubDynamicClient, err := dynamic.NewForConfig(hubCfg) + Expect(err).Should(Succeed()) + // No resource change detector runs in the test environment, so no informers are registered with the + // dynamic informer manager; the placement resource snapshot manager reads selected resources directly + // from the API server in this case. + hubDynamicInformerManager := informer.NewInformerManager(hubDynamicClient, 0, ctx.Done()) + placementResourceSnapshotManager, err := placementresourcesnapshot.New( + hubMgr, + hubDynamicClient, + hubDynamicInformerManager, + resourceeligibility.Baseline(hubMgr.GetRESTMapper()), + hubMgr.GetRESTMapper(), + placementResourceSnapshotMuSlotCnt, + maxPerSnapshotResourceDataSizeBytes, + placementresourcesnapshot.DefaultMaxPerSnapshotResourceCnt, + ) + Expect(err).Should(Succeed()) + // Add the placement resource snapshot manager to the controller manager so that its snapshot GC runs. + Expect(hubMgr.Add(placementResourceSnapshotManager)).Should(Succeed()) + + By("Setting up the auto-rollout controller") + reconciler, err := New( + hubMgr.GetClient(), + placementResourceSnapshotManager, + parallelizer.NewParallelizer(workerCount), + workqueue.NewTypedItemExponentialFailureRateLimiter[ctrl.Request](rateLimiterBaseDelay, rateLimiterMaxDelay), + hubMgr.GetEventRecorder(controllerName), + ) + Expect(err).Should(Succeed()) + Expect(reconciler.SetupWithManager(hubMgr, maxConcurrentReconciles)).Should(Succeed()) + + By("Starting the controller manager") + wg = sync.WaitGroup{} + wg.Add(1) + go func() { + defer GinkgoRecover() + defer wg.Done() + Expect(hubMgr.Start(ctx)).Should(Succeed(), "Failed to run the controller manager") + }() + Expect(hubMgr.GetCache().WaitForCacheSync(ctx)).Should(BeTrue(), "Failed to sync the controller manager cache") + + By("Creating the application namespaces") + for _, nsName := range []string{appNamespaceName, otherAppNamespaceName} { + Expect(hubClient.Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: nsName}, + })).Should(Succeed(), "Failed to create the application namespace %s", nsName) + } +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + + cancel() + // Wait for the controller manager to stop first; otherwise the API server might not shut down in time + // as the manager's watches are still open. + wg.Wait() + + By("Tearing down the test environment") + Expect(hubEnv.Stop()).Should(Succeed()) +}) diff --git a/pkg/v1/controllers/autorollout/utils_integration_test.go b/pkg/v1/controllers/autorollout/utils_integration_test.go new file mode 100644 index 000000000..d89b2c27c --- /dev/null +++ b/pkg/v1/controllers/autorollout/utils_integration_test.go @@ -0,0 +1,803 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License");f +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package autorollout + +import ( + "fmt" + "strings" + "time" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" +) + +const ( + configMapNameTemplate = "configmap-%s" + placementPolicyNameTemplate = "policy-%s" + // Placement bindings (and the member clusters they bind to) are numbered from 0 (zero-padded). + placementBindingNameTemplate = "binding-%s-%02d" + memberClusterNameTemplate = "cluster-%s-%02d" + + // The resource snapshot name that placement bindings point to before an auto-rollout; no snapshot of this name + // needs to exist, as the auto-rollout controller never reads it. + previousResourceSnapshotName = "previous-resource-snapshot" + + configMapDataKey = "foo" + configMapDataValue = "bar" + // An alternative ConfigMap data value, used to simulate changes to the selected resources. + altConfigMapDataValue = "baz" +) + +// The namespace of cluster-scoped objects (i.e., none); the test helpers use it to tell cluster-scoped objects +// (e.g., ClusterPlacementPolicy objects) apart from their namespace-scoped counterparts. +const clusterScopedNS = "" + +const ( + eventuallyDuration = time.Second * 20 + eventuallyInterval = time.Millisecond * 500 + consistentlyDuration = time.Second * 5 + consistentlyInterval = time.Millisecond * 500 +) + +// The minimum wait period annotation values that forcePlacementPolicyReconciliation alternates between. Both are +// within the allowed range; they only hold an auto-rollout back if the placement policy has been rolled out recently. +const ( + minWaitPeriodForcingValue = "1m" + altMinWaitPeriodForcingValue = "2m" +) + +// A finalizer that test cases add to objects, so that the objects stay around (with a deletion timestamp set) +// after being deleted. +const testFinalizer = "kubefleet.dev/integration-test" + +// An annotation that test cases set on placement policies only to make the auto-rollout controller process them +// (see touchPlacementPolicy). +const touchedAnnotationKey = "kubefleet.dev/integration-test-touched" + +// placementPolicyKind returns the kind of the placement policy of the scope that the given namespace implies. +func placementPolicyKind(policyNSName string) string { + if policyNSName == clusterScopedNS { + return placementv1alpha1.ClusterPlacementPolicyKind + } + return placementv1alpha1.PlacementPolicyKind +} + +// newPlacementPolicyObj returns a placement policy object of the appropriate scope with only the namespace and name +// set: a ClusterPlacementPolicy object if the namespace is empty, or a PlacementPolicy object otherwise. +func newPlacementPolicyObj(policyNSName, policyName string) placementv1alpha1.PlacementPolicyAccessor { + if policyNSName == clusterScopedNS { + return &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: policyName, + }, + } + } + return &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: policyNSName, + Name: policyName, + }, + } +} + +// newPlacementBindingObj returns a placement binding object of the appropriate scope with only the namespace and name +// set: a ClusterPlacementBinding object if the namespace is empty, or a PlacementBinding object otherwise. +func newPlacementBindingObj(bindingNS, bindingName string) placementv1alpha1.PlacementBindingAccessor { + if bindingNS == clusterScopedNS { + return &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: bindingName, + }, + } + } + return &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: bindingNS, + Name: bindingName, + }, + } +} + +// newPlacementResourceSnapshotObj returns a placement resource snapshot object of the appropriate scope with only the +// namespace and name set: a ClusterPlacementResourceSnapshot object if the namespace is empty, or a +// PlacementResourceSnapshot object otherwise. +func newPlacementResourceSnapshotObj(snapshotNS, snapshotName string) placementv1alpha1.PlacementResourceSnapshotAccessor { + if snapshotNS == clusterScopedNS { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + }, + } + } + return &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: snapshotNS, + Name: snapshotName, + }, + } +} + +// newConfigMapObj returns a ConfigMap object in the app namespace with only the namespace and name set. +func newConfigMapObj(configMapName string) *corev1.ConfigMap { + return &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: appNamespaceName, + Name: configMapName, + }, + } +} + +// createConfigMap creates a ConfigMap object of the given name in the app namespace, for placement policies to select. +func createConfigMap(configMapName string) *corev1.ConfigMap { + configMap := newConfigMapObj(configMapName) + configMap.Data = map[string]string{ + configMapDataKey: configMapDataValue, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the ConfigMap") + return configMap +} + +// autoRolloutEnabledAnnotations returns the annotations that enable auto-rollout for a placement policy. +func autoRolloutEnabledAnnotations() map[string]string { + return map[string]string{ + rolloutv1alpha1.AutoRolloutEnabledAnnotationKey: rolloutv1alpha1.AutoRolloutEnabledAnnotationValue, + } +} + +// createPlacementPolicy creates a placement policy with the given annotations that selects a ConfigMap (by name) in +// the app namespace; it returns the policy as written to the API server. +// +// If the namespace is empty, a ClusterPlacementPolicy object is created; otherwise, a PlacementPolicy object is +// created, which must be in the app namespace to select the ConfigMap. +func createPlacementPolicy(policyNSName, policyName, configMapName string, annotations map[string]string) placementv1alpha1.PlacementPolicyAccessor { + resourceSelector := placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMapName, + } + if policyNSName == clusterScopedNS { + // A cluster placement policy must specify the namespace of the namespaced resources it selects. + resourceSelector.Namespace = appNamespaceName + } + + policy := newPlacementPolicyObj(policyNSName, policyName) + policy.SetAnnotations(annotations) + policy.SetSpec(placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{resourceSelector}, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create the placement policy") + return policy +} + +// controllerOwnerRefTo returns the controller owner reference to a placement policy that the placement policy +// controller sets on the placement bindings it creates. +func controllerOwnerRefTo(policy placementv1alpha1.PlacementPolicyAccessor) metav1.OwnerReference { + return *metav1.NewControllerRef(policy, placementv1alpha1.GroupVersion.WithKind(placementPolicyKind(policy.GetNamespace()))) +} + +// newOwnedPlacementBinding returns (without creating it) a placement binding that binds a placement policy to a +// (fictional) member cluster and points to the given resource snapshot. +// +// As the placement policy controller would do, the binding is in the namespace of the placement policy (i.e., it is a +// ClusterPlacementBinding object if the placement policy is cluster-scoped), names the placement policy in its spec, +// and has the placement policy as its controller owner. Test cases may change it before creating it with +// createPlacementBinding, e.g., to simulate bindings that the auto-rollout controller should leave alone. +func newOwnedPlacementBinding( + policy placementv1alpha1.PlacementPolicyAccessor, + bindingName, clusterName, snapshotName string, +) placementv1alpha1.PlacementBindingAccessor { + binding := newPlacementBindingObj(policy.GetNamespace(), bindingName) + binding.SetOwnerReferences([]metav1.OwnerReference{controllerOwnerRefTo(policy)}) + binding.SetSpec(placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: policy.GetName(), + ClusterName: clusterName, + ResourceSnapshotName: snapshotName, + }) + return binding +} + +// createPlacementBinding creates the given placement binding. +func createPlacementBinding(binding placementv1alpha1.PlacementBindingAccessor) { + Expect(hubClient.Create(ctx, binding)).To(Succeed(), "Failed to create placement binding %s", binding.GetName()) +} + +// createPlacementBindings creates the given number of placement bindings (see newOwnedPlacementBinding) of a placement +// policy, all pointing to the previous resource snapshot; it returns the names of the bindings. +// +// The auto-rollout controller never reads member clusters, so no MemberCluster objects are created. +func createPlacementBindings(policy placementv1alpha1.PlacementPolicyAccessor, randStr string, count int) []string { + return createPlacementBindingsPointingTo(policy, randStr, count, previousResourceSnapshotName) +} + +// createPlacementBindingsPointingTo creates the given number of placement bindings (see newOwnedPlacementBinding) of a +// placement policy, all pointing to the given resource snapshot; it returns the names of the bindings. +// +// The names of the bindings (and of the member clusters they bind to) are derived from the given name seed; use +// different seeds for different sets of bindings in the same scope. +func createPlacementBindingsPointingTo( + policy placementv1alpha1.PlacementPolicyAccessor, + nameSeed string, + count int, + snapshotName string, +) []string { + bindingNames := make([]string, 0, count) + for idx := 0; idx < count; idx++ { + bindingName := fmt.Sprintf(placementBindingNameTemplate, nameSeed, idx) + createPlacementBinding(newOwnedPlacementBinding(policy, bindingName, fmt.Sprintf(memberClusterNameTemplate, nameSeed, idx), snapshotName)) + bindingNames = append(bindingNames, bindingName) + } + return bindingNames +} + +// placementBindingObjs returns objects (with only the namespace and name set) for the given placement bindings, +// e.g., for cleanup. +func placementBindingObjs(bindingNS string, bindingNames []string) []client.Object { + objs := make([]client.Object, 0, len(bindingNames)) + for _, bindingName := range bindingNames { + objs = append(objs, newPlacementBindingObj(bindingNS, bindingName)) + } + return objs +} + +// updateConfigMapData updates the data of a ConfigMap object in the app namespace. +func updateConfigMapData(configMapName string, data map[string]string) { + Eventually(func() error { + configMap := newConfigMapObj(configMapName) + if err := hubClient.Get(ctx, client.ObjectKeyFromObject(configMap), configMap); err != nil { + return fmt.Errorf("failed to retrieve the ConfigMap: %w", err) + } + configMap.Data = data + return hubClient.Update(ctx, configMap) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the ConfigMap") +} + +// getPlacementPolicy retrieves a placement policy (cluster-scoped if the namespace is empty). +func getPlacementPolicy(policyNSName, policyName string) (placementv1alpha1.PlacementPolicyAccessor, error) { + policy := newPlacementPolicyObj(policyNSName, policyName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: policyNSName, Name: policyName}, policy); err != nil { + return nil, fmt.Errorf("failed to retrieve the placement policy: %w", err) + } + return policy, nil +} + +// updatePlacementPolicyAnnotations updates the annotations of a placement policy with the given function, which +// always receives a non-nil map. +func updatePlacementPolicyAnnotations(policyNSName, policyName string, mutate func(annotations map[string]string)) { + Eventually(func() error { + policy, err := getPlacementPolicy(policyNSName, policyName) + if err != nil { + return err + } + annotations := policy.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string) + } + mutate(annotations) + policy.SetAnnotations(annotations) + return hubClient.Update(ctx, policy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the annotations of the placement policy") +} + +// forcePlacementPolicyReconciliation makes the auto-rollout controller process a placement policy (with auto-rollout +// enabled) again, by switching its minimum wait period annotation between two values, as the controller is not +// triggered by changes that cannot affect the outcome of an auto-rollout evaluation (e.g., condition changes that +// do not call for a rollout). +// +// Test cases use it to make sure that a check for the absence of an auto-rollout covers what the controller does +// when it processes the placement policy, rather than only that it has not processed the placement policy at all. +func forcePlacementPolicyReconciliation(policyNSName, policyName string) { + updatePlacementPolicyAnnotations(policyNSName, policyName, func(annotations map[string]string) { + if annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] == minWaitPeriodForcingValue { + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = altMinWaitPeriodForcingValue + return + } + annotations[rolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = minWaitPeriodForcingValue + }) +} + +// touchPlacementPolicy sets a test annotation on a placement policy to a fresh value; the change has no bearing on +// auto-rollouts, but the auto-rollout controller processes every update to a placement policy whose binding manager +// role it holds, so the change makes the controller process such a placement policy right away (e.g., instead of +// after a retry backoff). +func touchPlacementPolicy(policyNSName, policyName string) { + updatePlacementPolicyAnnotations(policyNSName, policyName, func(annotations map[string]string) { + annotations[touchedAnnotationKey] = time.Now().UTC().Format(time.RFC3339Nano) + }) +} + +// backdatedTimestamp returns a timestamp (in the format of the last rollout attempt timestamp annotation) of the given +// duration ago, e.g., to simulate a placement policy whose last rollout attempt was a while ago. +func backdatedTimestamp(ago time.Duration) string { + return time.Now().Add(-ago).UTC().Format(time.RFC3339) +} + +// lastRolloutAttemptTimestamp returns the value of the last rollout attempt timestamp annotation of a placement policy +// (empty if the annotation is absent). +func lastRolloutAttemptTimestamp(policyNSName, policyName string) string { + policy, err := getPlacementPolicy(policyNSName, policyName) + Expect(err).To(Succeed()) + return policy.GetAnnotations()[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] +} + +// resourceCollectedCond returns a ResourceCollected condition of a placement policy, as the placement policy controller +// would report; setPlacementPolicyConditions fills in the observed generation. +func resourceCollectedCond(status metav1.ConditionStatus, reason string) metav1.Condition { + return metav1.Condition{ + Type: placementv1alpha1.PlacementPolicyCondTypeResourceCollected, + Status: status, + Reason: reason, + } +} + +// synchronizedCond returns a Synchronized condition of a placement policy, as the placement policy controller would +// report; setPlacementPolicyConditions fills in the observed generation. +func synchronizedCond(status metav1.ConditionStatus, reason string) metav1.Condition { + return metav1.Condition{ + Type: placementv1alpha1.PlacementPolicyCondTypeSynchronized, + Status: status, + Reason: reason, + } +} + +// updatePlacementPolicyStatus updates the status of a placement policy with the given function, in one write. +// +// Note that the function does not wait for the cache-backed client (which the controller shares) to observe the +// change, as the controller may act on it right away (e.g., relinquish a binding manager claim of its own). +func updatePlacementPolicyStatus(policyNSName, policyName string, mutate func(policy placementv1alpha1.PlacementPolicyAccessor)) { + Eventually(func() error { + policy, err := getPlacementPolicy(policyNSName, policyName) + if err != nil { + return err + } + mutate(policy) + return hubClient.Status().Update(ctx, policy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the placement policy") +} + +// setConditionsOnPlacementPolicy sets the given conditions on the status of a placement policy object (without writing +// it); conditions with no observed generation set are stamped with the current generation of the placement policy. +func setConditionsOnPlacementPolicy(policy placementv1alpha1.PlacementPolicyAccessor, conds ...metav1.Condition) { + status := policy.GetStatus() + for _, cond := range conds { + if cond.ObservedGeneration == 0 { + cond.ObservedGeneration = policy.GetGeneration() + } + meta.SetStatusCondition(&status.Conditions, cond) + } +} + +// setPlacementPolicyConditions sets the given conditions on the status of a placement policy, as the placement policy +// controller would do; conditions with no observed generation set are stamped with the current generation of the +// placement policy. +func setPlacementPolicyConditions(policyNSName, policyName string, conds ...metav1.Condition) { + updatePlacementPolicyStatus(policyNSName, policyName, func(policy placementv1alpha1.PlacementPolicyAccessor) { + setConditionsOnPlacementPolicy(policy, conds...) + }) +} + +// pendingRolloutConds returns the conditions of a placement policy that has collected its resources but does not +// have them synchronized for the given reason, which calls for a rollout if the reason is +// ResourcesChangedPendingRollout or SomeClustersOnOlderResourceRevision. +func pendingRolloutConds(synchronizedReason string) []metav1.Condition { + return []metav1.Condition{ + resourceCollectedCond(metav1.ConditionTrue, placementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected), + synchronizedCond(metav1.ConditionFalse, synchronizedReason), + } +} + +// markPlacementPolicyAsPendingRollout marks a placement policy with the conditions that pendingRolloutConds returns. +func markPlacementPolicyAsPendingRollout(policyNSName, policyName, synchronizedReason string) { + setPlacementPolicyConditions(policyNSName, policyName, pendingRolloutConds(synchronizedReason)...) +} + +// markPlacementPolicyAsSynchronized marks a placement policy as having collected its resources and having them +// synchronized on all target clusters, i.e., as the placement policy controller would do once a rollout completes. +func markPlacementPolicyAsSynchronized(policyNSName, policyName string) { + setPlacementPolicyConditions(policyNSName, policyName, + resourceCollectedCond(metav1.ConditionTrue, placementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected), + synchronizedCond(metav1.ConditionTrue, placementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized), + ) +} + +// bumpPlacementPolicyGeneration makes a (harmless) change to the spec of a placement policy, which bumps its +// generation; it returns the new generation. +func bumpPlacementPolicyGeneration(policyNSName, policyName string) int64 { + var generation int64 + Eventually(func() error { + policy, err := getPlacementPolicy(policyNSName, policyName) + if err != nil { + return err + } + revisionHistoryLimit := ptr.Deref(policy.GetSpec().ResourceRevisionHistoryLimit, 0) + 5 + policy.GetSpec().ResourceRevisionHistoryLimit = ptr.To(revisionHistoryLimit) + if err := hubClient.Update(ctx, policy); err != nil { + return err + } + generation = policy.GetGeneration() + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bump the generation of the placement policy") + return generation +} + +// wantAutoRolloutObjectRef returns the reference to a placement policy (cluster-scoped if the namespace is empty) +// that the auto-rollout controller registers in the binding manager claim of the placement policy itself. +func wantAutoRolloutObjectRef(policyNSName, policyName string) placementv1alpha1.ObjectReference { + return placementv1alpha1.ObjectReference{ + APIGroup: placementv1alpha1.GroupVersion.Group, + APIVersion: placementv1alpha1.GroupVersion.Version, + Kind: placementPolicyKind(policyNSName), + Namespace: policyNSName, + Name: policyName, + } +} + +// wantAutoRolloutBindingManager returns the binding manager claim that the auto-rollout controller makes on a +// placement policy (cluster-scoped if the namespace is empty) while it rolls the placement policy out. +func wantAutoRolloutBindingManager(policyNSName, policyName string) *placementv1alpha1.BindingManager { + return &placementv1alpha1.BindingManager{ + ControllerName: controllerName, + ObjectRefs: []placementv1alpha1.ObjectReference{ + wantAutoRolloutObjectRef(policyNSName, policyName), + }, + } +} + +// updatePlacementPolicyBindingManager sets the binding manager of a placement policy to the given one (nil to clear +// it), e.g., to simulate a claim that the auto-rollout controller (or another controller) has left behind. +// +// Note that the function does not wait for the cache-backed client (which the controller shares) to observe the +// change, as the controller may act on it right away (e.g., relinquish a claim of its own). +func updatePlacementPolicyBindingManager(policyNSName, policyName string, bindingManager *placementv1alpha1.BindingManager) { + updatePlacementPolicyStatus(policyNSName, policyName, func(policy placementv1alpha1.PlacementPolicyAccessor) { + policy.GetStatus().BindingManager = bindingManager.DeepCopy() + }) +} + +// markObjectForDeletion adds the test finalizer to an object and then deletes it, so that the object stays around +// with a deletion timestamp set; it waits until the cache-backed client (which the controller shares) observes the +// deletion timestamp. +// +// Test cases must call removeTestFinalizer on the object as part of the cleanup. +func markObjectForDeletion(obj client.Object) { + key := client.ObjectKeyFromObject(obj) + Eventually(func() error { + if err := hubClient.Get(ctx, key, obj); err != nil { + return fmt.Errorf("failed to retrieve object %T %s: %w", obj, key, err) + } + controllerutil.AddFinalizer(obj, testFinalizer) + return hubClient.Update(ctx, obj) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the test finalizer to object %T %s", obj, key) + + Expect(hubClient.Delete(ctx, obj)).To(Succeed(), "Failed to delete object %T %s", obj, key) + + Eventually(func() error { + objCopy := obj.DeepCopyObject().(client.Object) + if err := hubClient.Get(ctx, key, objCopy); err != nil { + return fmt.Errorf("failed to retrieve object %T %s: %w", obj, key, err) + } + if objCopy.GetDeletionTimestamp() == nil { + return fmt.Errorf("object %T %s has not been marked for deletion yet", obj, key) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the deletion timestamp on object %T %s", obj, key) +} + +// removeTestFinalizer removes the test finalizer from an object (if the object still exists), so that the deletion +// of the object can complete. +func removeTestFinalizer(obj client.Object) { + key := client.ObjectKeyFromObject(obj) + Eventually(func() error { + objCopy := obj.DeepCopyObject().(client.Object) + if err := hubClient.Get(ctx, key, objCopy); err != nil { + if apierrors.IsNotFound(err) { + return nil + } + return fmt.Errorf("failed to retrieve object %T %s: %w", obj, key, err) + } + if !controllerutil.ContainsFinalizer(objCopy, testFinalizer) { + return nil + } + controllerutil.RemoveFinalizer(objCopy, testFinalizer) + return hubClient.Update(ctx, objCopy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the test finalizer from object %T %s", obj, key) +} + +// waitForObjectsToBeCached waits until the cache-backed client (which the controller shares) observes all the given +// objects. +// +// The controller reads placement bindings from the cache, and silently skips the ones that the cache has yet to +// observe; test cases should call this function on their fixtures before they write the conditions that call for a +// rollout, so that a rollout never runs against a cache that has yet to catch up. +func waitForObjectsToBeCached(objs ...client.Object) { + for _, obj := range objs { + Eventually(func() error { + // Read into a copy so that the object passed in is not modified. + objCopy := obj.DeepCopyObject().(client.Object) + return hubClient.Get(ctx, client.ObjectKeyFromObject(obj), objCopy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe object %T %s in the cache", obj, client.ObjectKeyFromObject(obj)) + } +} + +// primaryPlacementResourceSnapshotName returns the name of the primary placement resource snapshot of the given +// index for a placement policy. +func primaryPlacementResourceSnapshotName(policy placementv1alpha1.PlacementPolicyAccessor, index int) string { + return uniquename.PrimaryPlacementResourceSnapshotName(policy.GetName(), policy.GetUID(), index) +} + +// allActual combines the given checks into one, which fails with the error of the first failing check. +func allActual(actuals ...func() error) func() error { + return func() error { + for _, actual := range actuals { + if err := actual(); err != nil { + return err + } + } + return nil + } +} + +// primaryPlacementResourceSnapshotCreatedActual verifies that a primary placement resource snapshot of the given +// index has been created for a placement policy, as the only snapshot of its index, and is owned by the policy. +func primaryPlacementResourceSnapshotCreatedActual(policy placementv1alpha1.PlacementPolicyAccessor, index int) func() error { + return func() error { + // Retrieve the placement resource snapshot. + snapshotName := primaryPlacementResourceSnapshotName(policy, index) + snapshot := newPlacementResourceSnapshotObj(policy.GetNamespace(), snapshotName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: policy.GetNamespace(), Name: snapshotName}, snapshot); err != nil { + return fmt.Errorf("failed to retrieve the placement resource snapshot: %w", err) + } + + // Check the labels. + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: policy.GetName(), + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: fmt.Sprintf("%d", index), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(snapshot.GetLabels(), wantLabels); diff != "" { + return fmt.Errorf("placement resource snapshot labels diff (-got +want):\n%s", diff) + } + + // Check the owner references. + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementPolicyKind(policy.GetNamespace()), + Name: policy.GetName(), + UID: policy.GetUID(), + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + if diff := cmp.Diff(snapshot.GetOwnerReferences(), wantOwnerRefs); diff != "" { + return fmt.Errorf("placement resource snapshot owner references diff (-got +want):\n%s", diff) + } + return nil + } +} + +// placementResourceSnapshotAbsentActual verifies that a placement resource snapshot (cluster-scoped if the namespace +// is empty) does not exist. +func placementResourceSnapshotAbsentActual(snapshotNS, snapshotName string) func() error { + return objectRemovedActual(newPlacementResourceSnapshotObj(snapshotNS, snapshotName)) +} + +// placementBindingsResourceSnapshotActual verifies that all the given placement bindings point to the given resource +// snapshot. +func placementBindingsResourceSnapshotActual(bindingNS string, bindingNames []string, wantSnapshotName string) func() error { + return func() error { + for _, bindingName := range bindingNames { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); err != nil { + return fmt.Errorf("failed to retrieve placement binding %s: %w", bindingName, err) + } + if gotSnapshotName := binding.GetSpec().ResourceSnapshotName; gotSnapshotName != wantSnapshotName { + return fmt.Errorf("placement binding %s points to resource snapshot %s, want %s", bindingName, gotSnapshotName, wantSnapshotName) + } + } + return nil + } +} + +// placementBindingsActual verifies that all the given placement bindings point to the given resource snapshot, at the +// given generation (i.e., their specs have been updated exactly the expected number of times since creation). +func placementBindingsActual(bindingNS string, bindingNames []string, wantSnapshotName string, wantGeneration int64) func() error { + return func() error { + for _, bindingName := range bindingNames { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); err != nil { + return fmt.Errorf("failed to retrieve placement binding %s: %w", bindingName, err) + } + if gotSnapshotName := binding.GetSpec().ResourceSnapshotName; gotSnapshotName != wantSnapshotName { + return fmt.Errorf("placement binding %s points to resource snapshot %s, want %s", bindingName, gotSnapshotName, wantSnapshotName) + } + if gotGeneration := binding.GetGeneration(); gotGeneration != wantGeneration { + return fmt.Errorf("placement binding %s is at generation %d, want %d", bindingName, gotGeneration, wantGeneration) + } + } + return nil + } +} + +// placementPolicyRolledOutActual verifies that a placement policy has been rolled out (or has had a rollout fail due +// to a user error) at or after the given time (per its last rollout attempt timestamp annotation, which must be in +// RFC 3339 format and in UTC), and that no one holds its binding manager role afterwards. +// +// Both are checked on the same read of the placement policy: the controller writes the timestamp after it claims the +// role, so a read that has the timestamp but no binding manager reflects a released role, rather than a cache that +// has yet to observe the claim. +func placementPolicyRolledOutActual(policyNSName, policyName string, notBefore time.Time) func() error { + return func() error { + policy, err := getPlacementPolicy(policyNSName, policyName) + if err != nil { + return err + } + + val, found := policy.GetAnnotations()[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] + if !found { + return fmt.Errorf("the placement policy has no last rollout attempt timestamp annotation") + } + if !strings.HasSuffix(val, "Z") { + return fmt.Errorf("the last rollout attempt timestamp %q is not in UTC", val) + } + lastRolloutAttempt, err := time.Parse(time.RFC3339Nano, val) + if err != nil { + return fmt.Errorf("failed to parse the last rollout attempt timestamp %q: %w", val, err) + } + if now := time.Now(); lastRolloutAttempt.Before(notBefore) || lastRolloutAttempt.After(now) { + return fmt.Errorf("the last rollout attempt timestamp is %s, want a time within [%s, %s]", + lastRolloutAttempt.Format(time.RFC3339Nano), notBefore.UTC().Format(time.RFC3339Nano), now.UTC().Format(time.RFC3339Nano)) + } + + if bindingManager := policy.GetStatus().BindingManager; bindingManager != nil { + return fmt.Errorf("the binding manager role is still held: %+v", *bindingManager) + } + return nil + } +} + +// rolledOutActual verifies that an auto-rollout of a placement policy has completed at or after the given time: +// the primary placement resource snapshot of the given index has been created, all the given placement bindings +// point to it, the last rollout attempt timestamp has been written, and the binding manager role has been released. +func rolledOutActual( + policy placementv1alpha1.PlacementPolicyAccessor, + bindingNames []string, + snapshotIndex int, + notBefore time.Time, +) func() error { + return allActual( + primaryPlacementResourceSnapshotCreatedActual(policy, snapshotIndex), + placementBindingsResourceSnapshotActual(policy.GetNamespace(), bindingNames, primaryPlacementResourceSnapshotName(policy, snapshotIndex)), + placementPolicyRolledOutActual(policy.GetNamespace(), policy.GetName(), notBefore), + ) +} + +// placementPolicyAtRestActual verifies that the auto-rollout controller has left a placement policy at rest: no one +// holds its binding manager role, and its last rollout attempt timestamp annotation has the given value (empty if the +// annotation should be absent). +func placementPolicyAtRestActual(policyNSName, policyName, wantLastRolloutAttemptTimestamp string) func() error { + return placementPolicyAutoRolloutStateActual(policyNSName, policyName, nil, wantLastRolloutAttemptTimestamp) +} + +// placementPolicyAutoRolloutStateActual verifies, on one read of a placement policy, that its binding manager is the +// given one (nil if no one should hold the role) and that its last rollout attempt timestamp annotation has the given +// value (empty if the annotation should be absent). +func placementPolicyAutoRolloutStateActual( + policyNSName, policyName string, + wantBindingManager *placementv1alpha1.BindingManager, + wantLastRolloutAttemptTimestamp string, +) func() error { + return func() error { + policy, err := getPlacementPolicy(policyNSName, policyName) + if err != nil { + return err + } + if diff := cmp.Diff(policy.GetStatus().BindingManager, wantBindingManager); diff != "" { + return fmt.Errorf("binding manager diff (-got +want):\n%s", diff) + } + val, found := policy.GetAnnotations()[rolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] + switch { + case wantLastRolloutAttemptTimestamp == "" && found: + return fmt.Errorf("the placement policy has a last rollout attempt timestamp annotation (%q), want none", val) + case val != wantLastRolloutAttemptTimestamp: + return fmt.Errorf("the last rollout attempt timestamp is %q, want %q", val, wantLastRolloutAttemptTimestamp) + } + return nil + } +} + +// bindingManagerReleasedOnAPIServerActual verifies that no one holds the binding manager role of a placement policy, +// reading the placement policy from the API server directly. +// +// Test cases use it after they write a binding manager claim themselves: a read from the cache might predate the +// claim, and would then pass even if the controller never released it. +func bindingManagerReleasedOnAPIServerActual(policyNSName, policyName string) func() error { + return func() error { + policy := newPlacementPolicyObj(policyNSName, policyName) + if err := hubAPIReader.Get(ctx, client.ObjectKey{Namespace: policyNSName, Name: policyName}, policy); err != nil { + return fmt.Errorf("failed to retrieve the placement policy from the API server: %w", err) + } + if bindingManager := policy.GetStatus().BindingManager; bindingManager != nil { + return fmt.Errorf("the binding manager role is still held: %+v", *bindingManager) + } + return nil + } +} + +// placementPolicyResourceVersionOnAPIServer returns the resource version of a placement policy, reading the placement +// policy from the API server directly. +func placementPolicyResourceVersionOnAPIServer(policyNSName, policyName string) string { + policy := newPlacementPolicyObj(policyNSName, policyName) + Expect(hubAPIReader.Get(ctx, client.ObjectKey{Namespace: policyNSName, Name: policyName}, policy)).To(Succeed(), "Failed to retrieve the placement policy from the API server") + return policy.GetResourceVersion() +} + +// placementPolicyUnwrittenOnAPIServerActual verifies that no one has written to a placement policy since it was at the +// given resource version (e.g., that the controller has not claimed and released its binding manager role again), +// reading the placement policy from the API server directly. +func placementPolicyUnwrittenOnAPIServerActual(policyNSName, policyName, wantResourceVersion string) func() error { + return func() error { + policy := newPlacementPolicyObj(policyNSName, policyName) + if err := hubAPIReader.Get(ctx, client.ObjectKey{Namespace: policyNSName, Name: policyName}, policy); err != nil { + return fmt.Errorf("failed to retrieve the placement policy from the API server: %w", err) + } + if gotResourceVersion := policy.GetResourceVersion(); gotResourceVersion != wantResourceVersion { + return fmt.Errorf("the placement policy is at resource version %s, want %s (it has been written to)", gotResourceVersion, wantResourceVersion) + } + return nil + } +} + +// objectRemovedActual verifies that an object is gone. +func objectRemovedActual(obj client.Object) func() error { + return func() error { + objCopy := obj.DeepCopyObject().(client.Object) + err := hubClient.Get(ctx, client.ObjectKeyFromObject(obj), objCopy) + switch { + case err == nil: + return fmt.Errorf("object %T %s still exists", obj, client.ObjectKeyFromObject(obj)) + case !apierrors.IsNotFound(err): + return fmt.Errorf("failed to retrieve object %T %s: %w", obj, client.ObjectKeyFromObject(obj), err) + } + return nil + } +} + +// removeObjects deletes the given objects, and waits until all of them are gone. All the deletions are issued before +// any wait, so that the cleanup does not take one polling interval per object. +// +// The envtest environment does not run the garbage collector; test cases must remove all the objects they create +// (including those created by the controller on their behalf, e.g., placement resource snapshots) explicitly. +func removeObjects(objs ...client.Object) { + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T %s", obj, client.ObjectKeyFromObject(obj)) + } + for _, obj := range objs { + Eventually(objectRemovedActual(obj), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove object %T %s", obj, client.ObjectKeyFromObject(obj)) + } +} diff --git a/pkg/v1/controllers/placementpolicy/IMPLEMENTATION-NOTES.md b/pkg/v1/controllers/placementpolicy/IMPLEMENTATION-NOTES.md new file mode 100644 index 000000000..331e3c9e4 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/IMPLEMENTATION-NOTES.md @@ -0,0 +1,289 @@ +# Implementation notes: FEP-0001 placement policy controller (#786, #788) + +Observations collected while implementing the controller against the +`placement.kubefleet.dev/v1alpha1` APIs. Items in "API gaps" are candidate +follow-ups for the API definition (#781/#803); they are recorded here rather +than silently worked around, and the ones marked *pinned* have a test in this +package demonstrating the current behavior. + +## API gaps found while implementing + +1. **`ClusterSelector.count` accepts non-positive integers** (*closed*). The + field is `XIntOrString` with `Pattern="^([1-9][0-9]{0,2}|All)$"`, and a + pattern only constrains the string form, so `count: 0` and `count: -1` used + to bypass CRD validation and be rejected by the controller at evaluation + time instead. #829 added the CEL rule bounding the integer form, so they are + now rejected at admission; the controller's own check remains as a backstop + for objects admitted before the rule shipped. + +2. **Numeric operators are accepted in `matchLabelExpressions` at admission** + (*pinned*). `MatchLabelExpressions` reuses the + `LabelClusterPropertyExpression` type, whose enum includes `Gt`/`Lt`/etc.; + the API doc comment defers misuse to "an error at the scheduling phase". + Admission could reject it instead: + `self.all(e, e.operator in ['In', 'NotIn', 'Exists', 'DoesNotExist'])` on + the `matchLabelExpressions` field. Until then, the controller surfaces + `Scheduled=False/InvalidClusterSelectors`. + +3. **Aggregation across multiple selectors follows the FEP's overlapping-selectors + note.** A cluster satisfying two selectors of the same policy counts toward + both in `desiredClusters`/`scheduledClusters` (per-selector sums) — the + FEP's "About overlapping cluster selectors" section explicitly allows + totals to deviate from the sum of counts when selectors overlap. For a + `count: All` selector the desired count floors at `minCount`, so an + unfulfilled All selector shows a gap instead of trivially reporting + `desired == scheduled`. + +4. **`Scheduled` condition semantics with `minCount`.** The API's reason + constants encode a binary contract (`FoundAllRequiredClusters` / + `FailedToFindSomeRequiredClusters`), so reaching the floor (minCount) but + not the desired count reports `False` here, with the floor state surfaced + in the condition message. Whether the floor deserves a dedicated reason + (or a `True` polarity) is an FEP-level question worth settling before + beta; the claim lifecycle also needs a defined stance on whether a + floor-satisfied selector still warrants a claim. + +5. **Status count fields lack `+kubebuilder:validation:Optional` markers** + (`DesiredClusters` through `ActiveClusterClaims`) — cosmetic, but the + package convention annotates every optional field. + +## Deliberate implementation decisions (for reviewers) + +- **Fulfillment is judged against the scheduler's eligibility gate** + (`clustereligibilitychecker`) plus taint/toleration filtering, not raw label + matching — a provisioner-created or newly registered cluster does not count + until its member agent is online, heartbeating, and joined. See the + discussion on #791. +- **Clusters are ranked per the FEP's preferences** (`scoring.go`): on node + count (fewer is better), available CPU, and available memory (more is + better), each normalized to 0–100 against the candidates' own min and max, + summed, with the name as the tie-breaker. A cluster that does not report a + metric scores 0 on it, the least preferred position either way; a malformed + value is treated the same. The candidates are the schedulable clusters that + match the selector and are not already chosen, so the normalization is + per selector per round — the FEP's "among all clusters that match the + cluster selector". +- **Preferences are sticky, and stickiness survives an outage** + (`chooseClusters`): a bound cluster keeps its binding while it exists and + still matches a selector's terms, even when it is not schedulable (agent + dark, or an untolerated taint appeared). Unbinding on a transient outage + would tear resources down on a cluster that may be back in minutes, which + neither the current scheduler nor the FEP's "sticky once made" ask for. + An unschedulable bound cluster does not count toward the selector, so the + gap is filled from the schedulable candidates; when the cluster returns the + policy is briefly overpicked (`scheduledClusters` above `desiredClusters`), which the + count then trims: the bindings of clusters that no longer satisfy the + selector's terms go first, then the newest ones (creation timestamp, to the + second, then name), so a scale-down undoes the latest picks and a cluster + back from an outage, being the older binding, wins over its replacement. + The same trim also absorbs a duplicate pick made from a stale binding list + (the cache lagging a create by one round). +- **A scheduling decision sticks until the selector itself changes** + (`evaluateSelectors`), as it does for ClusterResourcePlacement and + ResourcePlacement: a bound cluster whose labels or properties stop matching + keeps its binding and still counts, because a cluster hovering around a + property threshold (a node count moving with autoscaling) would otherwise + have its resources removed and placed again on every swing, and the people + who label clusters are not always the ones who own the placements. The + binding's `spec.clusterSelectors` is the record: a cluster stays selected by + every selector whose terms its binding still lists. Changing a selector's + terms re-evaluates its clusters from scratch, as a new scheduling policy does + for placements; a cluster leaving the fleet, or its binding being deleted, + releases it too. The record is the binding itself, so anyone who rewrites a + binding's `spec.clusterSelectors` has that cluster re-evaluated from scratch + on the next round. A cluster whose reported data cannot be evaluated is named + on the `Scheduled` condition message, since it keeps its binding uncounted. +- **One binding per cluster per policy, counting toward every selector it + matches**: selectors are processed in order, and a cluster chosen for an + earlier selector satisfies each later selector whose terms it matches + before any new cluster is ranked for it, per the FEP's overlapping-selectors + note. The binding's informational `spec.clusterSelectors` lists the terms of + every selector the cluster currently serves and is kept up to date; the + selector *hashes* the FEP mentions keeping in annotations have no key in the + API yet, so nothing writes them. +- **Binding changes run under the binding manager role** (`bindingmanager`), + the FEP's mutual exclusion between scheduling decision changes and rollouts: + the role is claimed only when a create, update, or delete is actually due, + and released once every change is through. A failed write returns with the + role still held, so the retry resumes under the role it already has (the + FEP's restart semantics); a role held by another process backs the policy + off exponentially (5s up to 5m, reset by a round that completes) rather + than blocking the rest of the reconcile, which still refreshes claims and + status. +- **A binding needs a resource snapshot to roll out**, so the controller takes + the placement resource snapshot manager as a seam (`snapshotter`) and asks it + every round for the current snapshot — creating the first one, per the FEP's + "when a placement is first created" — and whether it still matches the + selected resources. The seam exists for the same reason as + `eligibilityChecker`: the real manager needs a dynamic client and informer + manager the envtest suite has no use for. Scheduling changes apply + independently of resources, per the FEP: a round that cannot get the + snapshot still unbinds, updates, and refreshes claims and the `Scheduled` + status, holds back only the new bindings, and is retried. +- **The status reports how far the latest revision has reached** + (`status.go`): `latestResourceRevisionName` is the latest snapshot; + `synchronizedClusters` counts the target clusters whose binding names that + revision, has had it processed by the work generator, and reports + `Synchronized` for its current generation; `resourcesAvailableClusters` + counts, of those, the ones reporting `Available` (availability is judged + only for the latest revision, as placements judge it only once + synchronized). A binding being deleted, or one not yet updated to the + policy's sync strategy, counts as neither: its conditions describe what is + going away. Target clusters include a bound cluster kept while + unschedulable, since it still holds the resources. The `Synchronized` + condition is true only + when every target cluster is synchronized *and* the snapshot still matches + the hub — per the FEP a resource change alone takes no snapshot, so until a + rollout does, the condition says the resources changed and a rollout is + pending (`ResourcesChangedPendingRollout`). Otherwise it reports + `SomeClustersOnOlderResourceRevision` while a target cluster is still on an + older revision and `FailedToSynchronizeSomeClusters` while the rest have yet + to apply the latest one, the message counting both. `ResourceCollected` is + true once the selected resources are collected and the snapshot taken or + found, false when they cannot be collected for good (a resource selected by + name is gone, or the selection is invalid), and left as it was on a + transient failure or while the cluster selectors are invalid, since no + snapshot is attempted then. When the resources cannot be collected, no + target cluster counts as synchronized, but `resourcesAvailableClusters` and + `Available` are still reported from the bindings, whatever revision each + holds, the way a Deployment reports its ready pods; the latest revision is + left as it was. + Rolling a new revision out is left to the rollout APIs; this + controller only reports. Binding updates wake the policy when their spec or + this progress changes (`bindingRelevantChanges`). +- **A change to a selected resource re-queues the policy** + (`EnqueuePoliciesSelecting`): the hub's resource-change controller hands + every changed resource's key to the policy controller, which re-queues the + policies that may select it — namespaced policies in that namespace, and + cluster-scoped ones whose selector names that namespace or none — through a + `sourcequeue.Queue` per controller, the never-blocking source the annotation + placement controller already uses. Matching is on the key alone (group, + kind, name, namespace), so deletions work the same way; label selectors are + not evaluated and the version is not compared, both of which can only + over-trigger a reconcile, never miss one. The hook is best effort: a failure + is logged and the change still reaches the placements. Resources the hub + agent's resource watcher drops produce no change events, so their policies + notice only on the periodic requeue: skipped namespaces (including + `default`, `kube-*`, and `fleet-*`), disabled kinds, and the objects + `ShouldPropagateObj` excludes (owned ReplicaSets and ControllerRevisions, + `kube-root-ca.crt`, default service accounts and their token secrets, and + the like), all of which the snapshot manager can still select. +- **Bindings are owned by the policy** through a controller reference — + policy and binding share a scope, so unlike claims they can be garbage + collected — and the controller `Owns()` them: a binding deleted out of band + is recreated, while binding status writes by the work generator are filtered + out by the generation predicate. +- **A taken binding name is settled, not ignored**: binding names are + deterministic, so a policy deleted without cascading and recreated under + the same name finds its old bindings in the way. Such an orphan (no + controller, same `spec.placementPolicyName` and cluster) is adopted and + brought up to date — the policy's desired state wins, lifting a suspension + the orphan carried — the way a ReplicaSet adopts an orphaned pod; deleting + it would tear down the workloads the orphaning kept running. A binding still + controlled by an earlier policy of the same name is reported as transient + (the garbage collector removes it), and one any other object controls as a + user error. Either way only that cluster waits: the round's other changes go + through and the role is released before the error is returned. +- **Invalid selectors release the binding manager role**: a role a failed + binding write left held is normally kept for the retry to resume under, but + once the selectors are invalid no binding change can follow until the spec + is fixed, so the role is released rather than blocking rollouts. +- **Heartbeat-noise suppression**: member cluster watch events pass through a + projection predicate that ignores heartbeat/observation timestamps; + time-driven eligibility transitions (e.g., heartbeats going stale) are + covered by periodic requeues instead (fulfilled policies re-check at half + the eligibility timeout, bounding worst-case staleness detection at about + 1.5x the timeout). +- **Malformed per-cluster data never fails a policy**: selector specs are + validated structurally up front, so any evaluation-time error is by + construction caused by data one cluster self-reported (e.g., a property + value that is not a quantity); that cluster is skipped and logged rather + than aborting evaluation for the whole policy. +- **Claims are issued while a selector is below its desired count** (not just + below the minCount floor), consistent with the binary Scheduled contract; + whether floor-satisfied selectors should stop claiming is part of the same + FEP follow-up as item 4 above. +- **Claim names are deterministic and namespace-qualified**: the name embeds + the selector index and a hash of the policy's namespaced name, so + same-named policies in different namespaces cannot collide on the + cluster-scoped claim namespace, and a restarted reconciler converges on + get-or-create instead of duplicating claims. +- **A Terminating claim occupies its budget slot**: a claim held by a + provisioner finalizer is not replaced, so a slow teardown can starve the + policy of its single claim slot but can never cause double-provisioning — + the safer failure mode for a provisioner acting on the claim. +- **Cleanup is finalizer-driven**: cross-scope owner references (namespaced + policy owning a cluster-scoped claim) are invalid in garbage collection — + and, trap-like, accepted at admission — so claims carry ownership labels, + the policy carries a cleanup finalizer (added before the first claim is + created, so a crash between the two writes cannot orphan a claim), and + deletion withdraws claims before the policy goes away. Claims are deleted + one by one rather than via DeleteAllOf, which keeps the RBAC surface free + of deletecollection. +- **Withdrawal is eligibility-gated**: a claim is withdrawn only when its + selector is fulfilled by schedulable clusters, so a provisioned cluster + that has registered but not yet joined does not withdraw the claim (the + join-window race raised on the verification issue). +- **Selector-term changes replace the claim under the same name**: the stale + claim is withdrawn and the same deterministic name is re-created with the + new terms. While the old claim is held Terminating by a provisioner + finalizer, `activeClusterClaims` keeps counting it: the withdraw round + treats the just-deleted claim as still occupying its slot, and the + replacement create waits for the slot to free rather than standing beside + it, so the budget holds through the swap. +- **A policy with no cluster selectors claims like any other**: the + synthesized "all clusters" selector inherits the API's `AddClusterClaim` + default, so a bare policy facing an empty fleet still signals that a + cluster is needed rather than silently waiting forever. +- **The FEP's eligible-keys allowlist is not implemented yet** (same status + as the per-fleet concurrency limit): the only opt-out today is the + per-selector `whenUnfulfilled: KeepSearching`. Both belong to the config + surface the FEP describes in prose only. + +## How a claim records the policy that owns it + +`spec.placementPolicyRef` is the authoritative identity: it is required, +immutable, and has no length limit, so the claim watch maps an event back to +its policy through that field. The ownership **labels** exist only so that a +policy's claims can be selected server-side with a `List`, which no spec field +allows. + +That distinction matters because label values stop at 63 bytes while object +names run to 253. When a policy's name does not fit, the label carries a +prefix plus a hash of the full name instead — so selecting on the label with +the policy's own name matches nothing, and consumers that must handle such +names should list claims and match on `spec.placementPolicyRef`. Generated +names are trimmed of trailing separators at every truncation point: a `.` or +`-` landing exactly on the boundary would otherwise produce a name the API +server rejects. Both flaws were found by manual testing with a 250-character +policy name, and each made claim creation fail validation and the reconciler +retry forever, with the policy silently never getting a claim. + +## Operational notes + +- **Stopping the controller with claims outstanding leaves policies + undeletable.** The cleanup finalizer is only removed by this controller, so + running the hub agent without it while a policy holds cluster claims parks + that policy in `Terminating` and leaves its claims in place until the + controller runs again, at which point reconciliation resumes and completes + the cleanup. This mirrors the existing behavior of the staged update run + APIs and their finalizer; the integration PR that wires the controller in + should document it on the enabling flag and in the chart README rather than + work around it, since removing a finalizer without a controller to withdraw + the claims would orphan them instead. +- **Not wired into the hub agent yet.** The controller is dormant code until + the integration PR adds the flag, scheme registration, RBAC, and the chart + CRDs; its kind e2e is parked at `parked/fep0001-placement-policy-e2e`. +- **Restarting the hub agent mid-flow is safe**: claim creation is + get-or-create on a deterministic name, and the finalizer is added before the + first claim exists, so no window can orphan a claim. + +## Cleanup lists every claim + +Releasing the claim-cleanup finalizer is gated on an uncached list of ALL +cluster claims, filtered by each claim's immutable `spec.placementPolicyRef` +-- the ownership labels are mutable, and a label-stripped claim vanishing +from a label-selected list would be orphaned permanently. This trades a +full-collection read per deleting policy for that guarantee; cheap while the +per-policy claim budget is 1, worth revisiting together with the +configurable concurrency limits. diff --git a/pkg/v1/controllers/placementpolicy/bindings.go b/pkg/v1/controllers/placementpolicy/bindings.go new file mode 100644 index 000000000..af97245f0 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/bindings.go @@ -0,0 +1,401 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "cmp" + "context" + "fmt" + "slices" + + apiequality "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/utils/naming" + "go.goms.io/fleet/pkg/v1/utils/bindingmanager" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + // controllerName identifies this controller, to the manager and in the binding manager + // role it holds on a policy while it changes the policy's bindings. + controllerName = "placement-policy-controller" + + // bindingNameSegmentMaxLength bounds each of the two name segments of a generated binding + // name; two segments, two separators, and the hash stay well within the 253-character + // object name limit. + bindingNameSegmentMaxLength = 100 +) + +// snapshotter is the slice of the placement resource snapshot manager this controller depends +// on. A binding names the resource snapshot it rolls out, and the FEP has the first snapshot +// taken when the placement is created, so the controller asks every round for the current +// snapshot -- creating it if there is none yet -- and whether it still matches the resources. +type snapshotter interface { + SnapshotResourcesIfNoSnapshotExists(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor) ([]kfplacementv1alpha1.PlacementResourceSnapshotAccessor, bool, error) +} + +// bindingName derives the deterministic name of the binding between a policy and a cluster. A +// binding lives in its policy's scope, so the policy name alone tells bindings of different +// policies apart; the hash covers the full identity so that two long names shortened to the +// same prefixes still get distinct bindings. Determinism makes creation get-or-create: a +// reconciler restarted between a create and the next list converges on the same object. +func bindingName(policy kfplacementv1alpha1.PlacementPolicyAccessor, clusterName string) string { + return fmt.Sprintf("%s-%s-%s", + naming.Truncate(policy.GetName(), bindingNameSegmentMaxLength), + naming.Truncate(clusterName, bindingNameSegmentMaxLength), + naming.Hash(policy.GetNamespace()+"/"+policy.GetName()+"/"+clusterName)) +} + +// chooseClusters decides which clusters each selector selects, records the decision on the +// outcomes, and returns, for every cluster that should hold a binding, the indices of the +// selectors it satisfies, in ascending order. bound maps each currently bound cluster to the +// time it was bound. +// +// Preferences are sticky, per the FEP: a bound cluster keeps its place ahead of any better +// ranked newcomer, and ranking is never re-applied to bound clusters, since re-ranking them on +// live metrics would move workloads around as the metrics drift. When a selector's count is +// lowered, the bindings of schedulable clusters that no longer satisfy its terms go first, then +// the newest ones, oldest and name breaking ties, so a scale-down undoes the latest picks and +// nothing else. +// +// Stickiness survives an outage: a bound cluster that still matches a selector's terms keeps +// its binding even while it is not schedulable (its agent has gone dark, or it carries a taint +// the policy stopped tolerating), because unbinding on an outage would tear resources down on +// a cluster that may be back in minutes. Such a cluster does not count toward the selector, +// so the gap is filled from the schedulable candidates, best-ranked first; once the cluster is +// back it is the older binding, and the replacement is what the count trims. A cluster no +// selector selects anymore -- evaluateSelectors keeps a bound cluster selected until the +// selector's terms change -- or that has left the fleet, is unbound. +// +// Selectors are processed in order, and a cluster chosen for an earlier selector counts toward +// every later selector it also matches, which is how one cluster may satisfy several selectors. +func chooseClusters(outcomes []selectorOutcome, bound map[string]metav1.Time, clusters map[string]*clusterv1beta1.MemberCluster) map[string][]int { + desired := make(map[string][]int) + // seniority orders the sticky phase, oldest binding first; a cluster picked earlier this + // round has no binding yet and is the newest of all, so that a scale-down drops it first. + seniority := func(a, b string) int { + boundA, isBoundA := bound[a] + boundB, isBoundB := bound[b] + switch { + case isBoundA != isBoundB: + return cmp.Compare(btoi(isBoundB), btoi(isBoundA)) + case isBoundA: + return cmp.Or(boundA.Time.Compare(boundB.Time), cmp.Compare(a, b)) + default: + return cmp.Compare(a, b) + } + } + + for i := range outcomes { + o := &outcomes[i] + schedulable := sets.New(o.matched...) + stale := sets.New(o.stale...) + + var kept []string + for _, name := range slices.Concat(o.matched, o.sticky) { + if _, isBound := bound[name]; isBound || desired[name] != nil { + kept = append(kept, name) + } + } + slices.SortFunc(kept, func(a, b string) int { + return cmp.Or(cmp.Compare(btoi(stale.Has(a)), btoi(stale.Has(b))), seniority(a, b)) + }) + + chosen := sets.New[string]() + for _, name := range kept { + if !schedulable.Has(name) { + desired[name] = append(desired[name], i) + continue + } + if !o.counts.selectAll && chosen.Len() >= int(o.counts.desired) { + // Beyond the count: this is the scale-down. A later selector may still want it. + continue + } + chosen.Insert(name) + desired[name] = append(desired[name], i) + } + + candidates := make([]*clusterv1beta1.MemberCluster, 0, schedulable.Len()) + for name := range schedulable.Difference(chosen) { + candidates = append(candidates, clusters[name]) + } + ranked := rankClusters(candidates) + if !o.counts.selectAll { + ranked = ranked[:max(0, min(len(ranked), int(o.counts.desired)-chosen.Len()))] + } + for _, name := range ranked { + chosen.Insert(name) + desired[name] = append(desired[name], i) + } + o.chosen = sets.List(chosen) + } + return desired +} + +func btoi(b bool) int { + if b { + return 1 + } + return 0 +} + +// listBindings returns the bindings the policy controls. The list is served from the informer +// cache through the binding index on spec.placementPolicyName (registered by +// fieldindexers.SetupWithHubAgentControllerManager), within the policy's own scope: a +// ClusterPlacementPolicy only ever lists cluster-scoped bindings and a PlacementPolicy only the +// bindings in its namespace. Ownership is still decided by the controller reference, because the +// name does not tell a policy apart from an earlier, deleted one of the same name whose bindings +// were orphaned rather than collected. A binding the cache has yet to observe is caught by the +// AlreadyExists tolerance on create and by the re-queue its own create event fires. +func (r *Reconciler) listBindings(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor) ([]kfplacementv1alpha1.PlacementBindingAccessor, error) { + byPolicy := client.MatchingFields{fieldindexers.PlacementBindingOwnedByCustomFieldName: policy.GetName()} + var all []kfplacementv1alpha1.PlacementBindingAccessor + if policy.GetNamespace() == "" { + list := &kfplacementv1alpha1.ClusterPlacementBindingList{} + if err := r.List(ctx, list, byPolicy); err != nil { + return nil, kferrors.NewAPIServerError(err, "failed to list the cluster placement bindings of the policy", true, "placementPolicy", klog.KObj(policy)) + } + for i := range list.Items { + all = append(all, &list.Items[i]) + } + } else { + list := &kfplacementv1alpha1.PlacementBindingList{} + if err := r.List(ctx, list, client.InNamespace(policy.GetNamespace()), byPolicy); err != nil { + return nil, kferrors.NewAPIServerError(err, "failed to list the placement bindings of the policy", true, "placementPolicy", klog.KObj(policy)) + } + for i := range list.Items { + all = append(all, &list.Items[i]) + } + } + owned := make([]kfplacementv1alpha1.PlacementBindingAccessor, 0, len(all)) + for _, binding := range all { + if metav1.IsControlledBy(binding, policy) { + owned = append(owned, binding) + } + } + return owned, nil +} + +// holdsBindingManagerRole reports whether this controller currently holds the policy's binding +// manager role. +func holdsBindingManagerRole(policy kfplacementv1alpha1.PlacementPolicyAccessor) bool { + manager := policy.GetStatus().BindingManager + return manager != nil && manager.ControllerName == controllerName +} + +// reconcileBindings creates, updates, and deletes the policy's bindings to match the decided +// cluster set. It reports roleHeld when the policy's binding manager role is held by another +// process, in which case nothing was changed and the caller should retry later. +// +// Binding changes run under the policy's binding manager role, the FEP's mutual exclusion between +// scheduling decision changes and rollouts. The role is released only once every change has gone +// through: a failed write returns with the role still held, so the retry resumes the same +// manipulation under the role it already holds, as the FEP prescribes for a process restarted +// mid-flow. A round that finds nothing left to do still releases a role a previous round left +// held, so a change that became moot before its retry cannot keep the role forever. +// +// A new binding rolls out snapshotName, the policy's latest resource snapshot; with no snapshot at +// hand (snapshotName empty), the clusters still to bind wait for a later round while every other +// change goes through. +func (r *Reconciler) reconcileBindings(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor, outcomes []selectorOutcome, desired map[string][]int, existing []kfplacementv1alpha1.PlacementBindingAccessor, snapshotName string) (roleHeld bool, err error) { + selectorsFor := func(clusterName string) []kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + selectors := make([]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, 0, len(desired[clusterName])) + for _, i := range desired[clusterName] { + selectors = append(selectors, kfplacementv1alpha1.ClusterSelectorWithTermsOnly{Terms: outcomes[i].terms}) + } + return selectors + } + + existingByCluster := make(map[string]kfplacementv1alpha1.PlacementBindingAccessor, len(existing)) + for _, binding := range existing { + existingByCluster[binding.GetSpec().ClusterName] = binding + } + var toCreate []string + var toUpdate, toDelete []kfplacementv1alpha1.PlacementBindingAccessor + for clusterName := range desired { + binding, bound := existingByCluster[clusterName] + switch { + case !bound: + toCreate = append(toCreate, clusterName) + case !apiequality.Semantic.DeepEqual(binding.GetSpec().ClusterSelectors, selectorsFor(clusterName)), + !apiequality.Semantic.DeepEqual(binding.GetSpec().SyncStrategy, policy.GetSpec().SyncStrategy): + toUpdate = append(toUpdate, binding) + } + } + if snapshotName == "" { + // Without a snapshot to roll out the clusters still to bind wait; dropping them before the + // role is claimed keeps a round with nothing else to do from claiming it at all. + toCreate = nil + } + slices.Sort(toCreate) + for clusterName, binding := range existingByCluster { + if _, wanted := desired[clusterName]; !wanted && binding.GetDeletionTimestamp().IsZero() { + toDelete = append(toDelete, binding) + } + } + + ref := *policyReference(policy) + if len(toCreate)+len(toUpdate)+len(toDelete) == 0 { + if !holdsBindingManagerRole(policy) { + return false, nil + } + return false, bindingmanager.RelinquishRoleFor(ctx, r.Client, policy, controllerName, ref) + } + + claimed, err := bindingmanager.ClaimRoleAs(ctx, r.Client, policy, controllerName, ref) + if err != nil { + return false, err + } + if !claimed { + return true, nil + } + + // A taken binding name blocks only its own cluster: the occupant cannot be dealt with here, so + // the rest of the changes go through, the role is released -- only a person or the garbage + // collector can clear the way, and holding the role meanwhile would block the policy's rollouts + // -- and the first such error is returned for a retry. + var takenName error + for _, clusterName := range toCreate { + binding, err := r.newBinding(policy, clusterName, snapshotName, selectorsFor(clusterName)) + if err != nil { + return false, err + } + klog.V(2).InfoS("Binding a cluster", "placementPolicy", klog.KObj(policy), "memberCluster", clusterName, "placementBinding", binding.GetName()) + if err := r.Create(ctx, binding); err != nil { + if !errors.IsAlreadyExists(err) { + return false, kferrors.NewAPIServerError(err, "failed to create the placement binding", false, "placementBinding", klog.KObj(binding)) + } + if err := r.adoptOrReportBindingOccupant(ctx, policy, binding); err != nil { + if category := kferrors.Category(err); category != kferrors.ErrCategoryUser && category != kferrors.ErrCategoryTransient { + return false, err + } + klog.ErrorS(err, "Cannot bind a cluster: the binding name is taken", kferrors.Args(err)...) + if takenName == nil { + takenName = err + } + } + } + } + for _, binding := range toUpdate { + binding.GetSpec().ClusterSelectors = selectorsFor(binding.GetSpec().ClusterName) + binding.GetSpec().SyncStrategy = policy.GetSpec().SyncStrategy.DeepCopy() + if err := r.Update(ctx, binding); err != nil { + return false, kferrors.NewAPIServerError(err, "failed to update the placement binding", false, "placementBinding", klog.KObj(binding)) + } + } + for _, binding := range toDelete { + klog.V(2).InfoS("Unbinding a cluster", "placementPolicy", klog.KObj(policy), "memberCluster", binding.GetSpec().ClusterName, "placementBinding", binding.GetName()) + uid := binding.GetUID() + if err := r.Delete(ctx, binding, client.Preconditions{UID: &uid}); err != nil && !errors.IsNotFound(err) { + return false, kferrors.NewAPIServerError(err, "failed to delete the placement binding", false, "placementBinding", klog.KObj(binding)) + } + } + + if err := bindingmanager.RelinquishRoleFor(ctx, r.Client, policy, controllerName, ref); err != nil { + return false, err + } + return false, takenName +} + +// adoptOrReportBindingOccupant settles a binding name the policy failed to create because it is +// taken. The occupant is read uncached, since the cache is what failed to show it: +// - one the policy already controls is a create the cache has yet to show, and needs nothing; +// - one being deleted frees the name shortly, and a later round creates the binding; +// - an orphan of the policy's name for the same cluster -- left by an earlier policy of the same +// name deleted without cascading -- is adopted and brought up to date, the way a ReplicaSet +// adopts an orphaned pod, rather than leaving the cluster unbound behind a name the policy cannot +// take; deleting it instead would tear down the workloads the orphaning kept running. Adoption +// makes the policy's desired state win, so a suspension the orphan carried is lifted; +// - one still controlled by an earlier policy of the same name is a transient error: the garbage +// collector removes it shortly; +// - one any other object controls is a user error: without it the cluster would stay unbound with +// nothing saying why. +func (r *Reconciler) adoptOrReportBindingOccupant(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor, want kfplacementv1alpha1.PlacementBindingAccessor) error { + var occupant kfplacementv1alpha1.PlacementBindingAccessor = &kfplacementv1alpha1.PlacementBinding{} + if policy.GetNamespace() == "" { + occupant = &kfplacementv1alpha1.ClusterPlacementBinding{} + } + if err := r.uncachedReader.Get(ctx, client.ObjectKeyFromObject(want), occupant); err != nil { + if errors.IsNotFound(err) { + return nil + } + return kferrors.NewAPIServerError(err, "failed to get the placement binding occupying the name", false, "placementBinding", klog.KObj(want)) + } + switch { + case metav1.IsControlledBy(occupant, policy), !occupant.GetDeletionTimestamp().IsZero(): + return nil + case metav1.GetControllerOf(occupant) == nil && occupant.GetSpec().PlacementPolicyName == policy.GetName() && + occupant.GetSpec().ClusterName == want.GetSpec().ClusterName: + klog.V(2).InfoS("Adopting an orphaned placement binding", "placementPolicy", klog.KObj(policy), "placementBinding", klog.KObj(occupant)) + if err := controllerutil.SetControllerReference(policy, occupant, r.Scheme()); err != nil { + return kferrors.NewUnexpectedError(err, "failed to set the controller reference on the placement binding", "placementBinding", klog.KObj(occupant)) + } + occupant.SetSpec(*want.GetSpec()) + if err := r.Update(ctx, occupant); err != nil { + return kferrors.NewAPIServerError(err, "failed to adopt the orphaned placement binding", false, "placementBinding", klog.KObj(occupant)) + } + return nil + case isEarlierPolicyOfSameName(metav1.GetControllerOf(occupant), policy): + return kferrors.NewTransientError(nil, "the placement binding name is taken by a binding of an earlier policy of the same name, awaiting garbage collection", + "placementBinding", klog.KObj(occupant), "placementPolicy", klog.KObj(policy)) + default: + return kferrors.NewUserError(nil, "the placement binding name is taken by a binding the policy does not control", + "placementBinding", klog.KObj(occupant), "placementPolicy", klog.KObj(policy), "controller", metav1.GetControllerOf(occupant)) + } +} + +// isEarlierPolicyOfSameName reports whether a controller reference names a policy of the same kind +// and name as the given one, but another incarnation of it. +func isEarlierPolicyOfSameName(ref *metav1.OwnerReference, policy kfplacementv1alpha1.PlacementPolicyAccessor) bool { + kind := kfplacementv1alpha1.PlacementPolicyKind + if policy.GetNamespace() == "" { + kind = kfplacementv1alpha1.ClusterPlacementPolicyKind + } + return ref != nil && ref.Kind == kind && ref.Name == policy.GetName() && ref.UID != policy.GetUID() +} + +// newBinding builds the binding between a policy and a cluster, owned by the policy so that +// deleting the policy takes its bindings with it. +func (r *Reconciler) newBinding(policy kfplacementv1alpha1.PlacementPolicyAccessor, clusterName, snapshotName string, selectors []kfplacementv1alpha1.ClusterSelectorWithTermsOnly) (kfplacementv1alpha1.PlacementBindingAccessor, error) { + spec := kfplacementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: policy.GetName(), + ClusterSelectors: selectors, + ClusterName: clusterName, + ResourceSnapshotName: snapshotName, + SyncStrategy: policy.GetSpec().SyncStrategy.DeepCopy(), + } + objectMeta := metav1.ObjectMeta{Name: bindingName(policy, clusterName), Namespace: policy.GetNamespace()} + var binding kfplacementv1alpha1.PlacementBindingAccessor + if policy.GetNamespace() == "" { + binding = &kfplacementv1alpha1.ClusterPlacementBinding{ObjectMeta: objectMeta, Spec: spec} + } else { + binding = &kfplacementv1alpha1.PlacementBinding{ObjectMeta: objectMeta, Spec: spec} + } + if err := controllerutil.SetControllerReference(policy, binding, r.Scheme()); err != nil { + return nil, kferrors.NewUnexpectedError(err, "failed to set the controller reference on the placement binding", "placementBinding", klog.KObj(binding)) + } + return binding, nil +} diff --git a/pkg/v1/controllers/placementpolicy/bindings_integration_test.go b/pkg/v1/controllers/placementpolicy/bindings_integration_test.go new file mode 100644 index 000000000..6a9ac89a4 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/bindings_integration_test.go @@ -0,0 +1,344 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "cmp" + "slices" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/propertyprovider" +) + +// setNodeCount reports a node count on a member cluster, which is what the ranking prefers +// fewer of. +func setNodeCount(name, count string) { + mc := &clusterv1beta1.MemberCluster{} + Expect(k8sClient.Get(ctx, types.NamespacedName{Name: name}, mc)).Should(Succeed()) + mc.Status.Properties = map[clusterv1beta1.PropertyName]clusterv1beta1.PropertyValue{ + propertyprovider.NodeCountProperty: {Value: count, ObservationTime: metav1.Now()}, + } + Expect(k8sClient.Status().Update(ctx, mc)).Should(Succeed()) +} + +// setJoined flips the member agent's Joined condition, which takes the cluster in and out of +// the scheduler's eligibility gate through a change the member cluster watch does not filter. +func setJoined(name string, joined metav1.ConditionStatus) { + mc := &clusterv1beta1.MemberCluster{} + Expect(k8sClient.Get(ctx, types.NamespacedName{Name: name}, mc)).Should(Succeed()) + meta.SetStatusCondition(&mc.Status.AgentStatus[0].Conditions, metav1.Condition{ + Type: string(clusterv1beta1.AgentJoined), Status: joined, Reason: "IntegrationTest", + }) + Expect(k8sClient.Status().Update(ctx, mc)).Should(Succeed()) +} + +// setRegion relabels the region of a member cluster. +func setRegion(name, region string) { + mc := &clusterv1beta1.MemberCluster{} + Expect(k8sClient.Get(ctx, types.NamespacedName{Name: name}, mc)).Should(Succeed()) + mc.Labels[testRegionLabel] = region + Expect(k8sClient.Update(ctx, mc)).Should(Succeed()) +} + +// setSelectorCount rewrites the count of a policy's only selector. +func setSelectorCount(key types.NamespacedName, count int32) { + policy := &kfplacementv1alpha1.PlacementPolicy{} + Expect(k8sClient.Get(ctx, key, policy)).Should(Succeed()) + policy.Spec.ClusterSelectors[0].Count = ptr.To(intstr.FromInt32(count)) + Expect(k8sClient.Update(ctx, policy)).Should(Succeed()) +} + +// drainMemberClusters deletes every member cluster and waits for them to be gone, since they are +// cluster-scoped and shared with the other containers of this suite. +func drainMemberClusters() { + memberClusters := &clusterv1beta1.MemberClusterList{} + Expect(k8sClient.List(ctx, memberClusters)).Should(Succeed()) + for i := range memberClusters.Items { + Expect(client.IgnoreNotFound(k8sClient.Delete(ctx, &memberClusters.Items[i]))).Should(Succeed()) + } + Eventually(func(g Gomega) { + g.Expect(k8sClient.List(ctx, memberClusters)).Should(Succeed()) + g.Expect(memberClusters.Items).Should(BeEmpty()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) +} + +var _ = Describe("placement binding lifecycle", Ordered, func() { + const ( + region = "binding-east" + policyName = "binder" + ) + policyKey := types.NamespacedName{Namespace: testNamespace, Name: policyName} + regionTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{testRegionLabel: region}}, + } + + // bindingsOf returns the policy's bindings, sorted by cluster name. + bindingsOf := func(g Gomega) []kfplacementv1alpha1.PlacementBinding { + list := &kfplacementv1alpha1.PlacementBindingList{} + g.Expect(k8sClient.List(ctx, list, client.InNamespace(testNamespace))).Should(Succeed()) + owned := make([]kfplacementv1alpha1.PlacementBinding, 0, len(list.Items)) + for _, binding := range list.Items { + if binding.Spec.PlacementPolicyName == policyName { + owned = append(owned, binding) + } + } + slices.SortFunc(owned, func(a, b kfplacementv1alpha1.PlacementBinding) int { + return cmp.Compare(a.Spec.ClusterName, b.Spec.ClusterName) + }) + return owned + } + boundClusters := func(g Gomega) []string { + names := []string{} + for _, binding := range bindingsOf(g) { + names = append(names, binding.Spec.ClusterName) + } + return names + } + wantBound := func(clusters ...string) func(Gomega) { + want := append([]string{}, clusters...) + return func(g Gomega) { + g.Expect(boundClusters(g)).Should(Equal(want)) + } + } + wantScheduled := func(status metav1.ConditionStatus, reason string, desired, scheduled int32) { + Eventually(wantScheduledState(policyKey, status, reason, desired, scheduled), eventuallyTimeout, pollInterval).Should(Succeed()) + } + + BeforeAll(func() { + // The node counts rank the clusters c1, c3, c2: the names deliberately disagree with + // the ranking, so that picking by name cannot pass. + for name, nodes := range map[string]string{"binder-c1": "1", "binder-c2": "5", "binder-c3": "3"} { + Expect(k8sClient.Create(ctx, newMemberCluster(name, map[string]string{testRegionLabel: region}))).Should(Succeed()) + markJoined(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: name}}) + setNodeCount(name, nodes) + } + Expect(k8sClient.Create(ctx, newPolicy(policyName, regionSelector(region, ptr.To(intstr.FromInt32(2)), nil)))).Should(Succeed()) + }) + + AfterAll(func() { + policy := &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: policyName, Namespace: testNamespace}} + Expect(client.IgnoreNotFound(k8sClient.Delete(ctx, policy))).Should(Succeed()) + // Policy deletion is finalizer-gated on claim cleanup; wait for both to drain. + Eventually(func(g Gomega) { + g.Expect(errors.IsNotFound(k8sClient.Get(ctx, policyKey, &kfplacementv1alpha1.PlacementPolicy{}))).Should(BeTrue()) + claims := &kfplacementv1alpha1.ClusterClaimList{} + g.Expect(k8sClient.List(ctx, claims)).Should(Succeed()) + g.Expect(claims.Items).Should(BeEmpty()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + // envtest runs no garbage collector, so the owned bindings are removed by hand. + Expect(k8sClient.DeleteAllOf(ctx, &kfplacementv1alpha1.PlacementBinding{}, client.InNamespace(testNamespace))).Should(Succeed()) + drainMemberClusters() + }) + + It("binds the best-ranked clusters up to the count, then releases the binding manager role", func() { + Eventually(wantBound("binder-c1", "binder-c3"), eventuallyTimeout, pollInterval).Should(Succeed()) + wantScheduled(metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 2, 2) + Eventually(func(g Gomega) { + policy := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, policyKey, policy)).Should(Succeed()) + g.Expect(policy.Status.BindingManager).Should(BeNil()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("fills in each binding from the policy and owns it", func() { + policy := &kfplacementv1alpha1.PlacementPolicy{} + Expect(k8sClient.Get(ctx, policyKey, policy)).Should(Succeed()) + for _, binding := range bindingsOf(Default) { + Expect(metav1.IsControlledBy(&binding, policy)).Should(BeTrue(), "binding %s should be controlled by the policy", binding.Name) + Expect(binding.Spec.ResourceSnapshotName).Should(Equal(stubSnapshotName(policy))) + Expect(binding.Spec.ClusterSelectors).Should(Equal([]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{Terms: regionTerms}})) + } + }) + + It("rolls the bindings' rollout progress up into the policy status", func() { + policy := &kfplacementv1alpha1.PlacementPolicy{} + Expect(k8sClient.Get(ctx, policyKey, policy)).Should(Succeed()) + revision := stubSnapshotName(policy) + policyStatus := func(g Gomega) *kfplacementv1alpha1.PlacementPolicyStatus { + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, policyKey, fetched)).Should(Succeed()) + return &fetched.Status + } + + By("nothing is synchronized before the work generator reports") + Eventually(func(g Gomega) { + status := policyStatus(g) + g.Expect(status.LatestResourceRevisionName).Should(HaveValue(Equal(revision))) + g.Expect(status.SynchronizedClusters).Should(HaveValue(BeZero())) + g.Expect(meta.IsStatusConditionFalse(status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized)).Should(BeTrue()) + g.Expect(meta.IsStatusConditionTrue(status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected)).Should(BeTrue()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the bindings report the revision synchronized and available") + for _, binding := range bindingsOf(Default) { + binding.Status.LastProcessedResourceSnapshotName = &revision + for _, condType := range []string{kfplacementv1alpha1.PlacementBindingCondTypeSynchronized, kfplacementv1alpha1.PlacementBindingCondTypeAvailable} { + meta.SetStatusCondition(&binding.Status.Conditions, metav1.Condition{ + Type: condType, Status: metav1.ConditionTrue, ObservedGeneration: binding.Generation, Reason: "IntegrationTest", + }) + } + Expect(k8sClient.Status().Update(ctx, &binding)).Should(Succeed()) + } + Eventually(func(g Gomega) { + status := policyStatus(g) + g.Expect(status.SynchronizedClusters).Should(HaveValue(BeEquivalentTo(2))) + g.Expect(status.ResourcesAvailableClusters).Should(HaveValue(BeEquivalentTo(2))) + g.Expect(meta.IsStatusConditionTrue(status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized)).Should(BeTrue()) + g.Expect(meta.IsStatusConditionTrue(status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeAvailable)).Should(BeTrue()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("keeps a bound cluster that goes dark and fills its place", func() { + // Seniority among bindings is their creation timestamp, which the API server records to + // the second; the replacement must land in a later second than the originals for the + // trim in the next spec to be about seniority rather than the name tie-break. + var newest time.Time + for _, binding := range bindingsOf(Default) { + if binding.CreationTimestamp.After(newest) { + newest = binding.CreationTimestamp.Time + } + } + Eventually(func() bool { return time.Now().After(newest.Add(time.Second)) }, eventuallyTimeout, pollInterval).Should(BeTrue()) + + setJoined("binder-c1", metav1.ConditionFalse) + Eventually(wantBound("binder-c1", "binder-c2", "binder-c3"), eventuallyTimeout, pollInterval).Should(Succeed()) + // The dark cluster is still bound but no longer counted. + wantScheduled(metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 2, 2) + }) + + It("trims the replacement once the cluster is back", func() { + setJoined("binder-c1", metav1.ConditionTrue) + Eventually(wantBound("binder-c1", "binder-c3"), eventuallyTimeout, pollInterval).Should(Succeed()) + wantScheduled(metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 2, 2) + }) + + It("keeps a bound cluster that stops matching the selector, still counted", func() { + setRegion("binder-c3", "elsewhere") + + // The relabel is a scheduling-relevant change, so the policy is reconciled; give that + // round time to land before asserting nothing moved. + Consistently(wantBound("binder-c1", "binder-c3"), consistentlyDuration, pollInterval).Should(Succeed()) + wantScheduled(metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 2, 2) + }) + + It("scales down by unbinding the newest binding", func() { + setSelectorCount(policyKey, 1) + Eventually(wantBound("binder-c1"), eventuallyTimeout, pollInterval).Should(Succeed()) + wantScheduled(metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 1, 1) + }) + + It("scales back up", func() { + setSelectorCount(policyKey, 2) + Eventually(wantBound("binder-c1", "binder-c2"), eventuallyTimeout, pollInterval).Should(Succeed()) + wantScheduled(metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 2, 2) + }) + + It("recreates a binding deleted out of band", func() { + policy := &kfplacementv1alpha1.PlacementPolicy{} + Expect(k8sClient.Get(ctx, policyKey, policy)).Should(Succeed()) + key := types.NamespacedName{Namespace: testNamespace, Name: bindingName(policy, "binder-c2")} + binding := &kfplacementv1alpha1.PlacementBinding{} + Expect(k8sClient.Get(ctx, key, binding)).Should(Succeed()) + deleted := binding.UID + Expect(k8sClient.Delete(ctx, binding)).Should(Succeed()) + + Eventually(func(g Gomega) { + g.Expect(k8sClient.Get(ctx, key, binding)).Should(Succeed()) + g.Expect(binding.UID).ShouldNot(Equal(deleted), "the binding should have been recreated, not resurrected") + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("unbinds a cluster that leaves the fleet", func() { + Expect(k8sClient.Delete(ctx, &clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: "binder-c1"}})).Should(Succeed()) + Eventually(wantBound("binder-c2"), eventuallyTimeout, pollInterval).Should(Succeed()) + wantScheduled(metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 2, 1) + }) + + It("re-evaluates the bound clusters from scratch when the selector terms change", func() { + By("a bound cluster that stops matching stays bound") + setRegion("binder-c2", "elsewhere") + Consistently(wantBound("binder-c2"), consistentlyDuration, pollInterval).Should(Succeed()) + + By("new terms that the cluster does not match release it") + policy := &kfplacementv1alpha1.PlacementPolicy{} + Expect(k8sClient.Get(ctx, policyKey, policy)).Should(Succeed()) + policy.Spec.ClusterSelectors[0].Terms = []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{testRegionLabel: "nowhere"}}, + } + Expect(k8sClient.Update(ctx, policy)).Should(Succeed()) + Eventually(wantBound(), eventuallyTimeout, pollInterval).Should(Succeed()) + wantScheduled(metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 2, 0) + }) +}) + +var _ = Describe("cluster placement binding lifecycle", Ordered, func() { + const ( + region = "binding-cluster-scope" + policyName = "cluster-binder" + clusterName = "cluster-binder-c1" + ) + policyKey := types.NamespacedName{Name: policyName} + + BeforeAll(func() { + Expect(k8sClient.Create(ctx, newMemberCluster(clusterName, map[string]string{testRegionLabel: region}))).Should(Succeed()) + markJoined(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: clusterName}}) + policy := &kfplacementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: policyName}, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{regionSelector(region, ptr.To(intstr.FromInt32(1)), nil)}, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "Namespace", Name: "demo"}, + }, + }, + } + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + }) + + AfterAll(func() { + policy := &kfplacementv1alpha1.ClusterPlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: policyName}} + Expect(client.IgnoreNotFound(k8sClient.Delete(ctx, policy))).Should(Succeed()) + Eventually(func() bool { + return errors.IsNotFound(k8sClient.Get(ctx, policyKey, &kfplacementv1alpha1.ClusterPlacementPolicy{})) + }, eventuallyTimeout, pollInterval).Should(BeTrue(), "the policy should be gone") + Expect(k8sClient.DeleteAllOf(ctx, &kfplacementv1alpha1.ClusterPlacementBinding{})).Should(Succeed()) + drainMemberClusters() + }) + + It("binds the cluster with a cluster-scoped binding owned by the policy", func() { + policy := &kfplacementv1alpha1.ClusterPlacementPolicy{} + Expect(k8sClient.Get(ctx, policyKey, policy)).Should(Succeed()) + binding := &kfplacementv1alpha1.ClusterPlacementBinding{} + Eventually(func(g Gomega) { + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: bindingName(policy, clusterName)}, binding)).Should(Succeed()) + g.Expect(binding.Spec.ClusterName).Should(Equal(clusterName)) + g.Expect(binding.Spec.ResourceSnapshotName).Should(Equal(stubSnapshotName(policy))) + g.Expect(metav1.IsControlledBy(binding, policy)).Should(BeTrue()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) +}) diff --git a/pkg/v1/controllers/placementpolicy/bindings_test.go b/pkg/v1/controllers/placementpolicy/bindings_test.go new file mode 100644 index 000000000..bea976549 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/bindings_test.go @@ -0,0 +1,524 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + "strings" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +// uniformClusters builds a cluster inventory in which every cluster ranks the same, so that the +// ranking falls back to name order and the tests can reason about it. +func uniformClusters(names ...string) map[string]*clusterv1beta1.MemberCluster { + clusters := make(map[string]*clusterv1beta1.MemberCluster, len(names)) + for _, name := range names { + clusters[name] = rankedCluster(name, "3", "4", "8Gi") + } + return clusters +} + +// boundSince builds the bound-cluster map chooseClusters takes, with each cluster bound the +// given number of minutes ago. +func boundSince(minutesAgo map[string]int) map[string]metav1.Time { + now := time.Now() + bound := make(map[string]metav1.Time, len(minutesAgo)) + for name, minutes := range minutesAgo { + bound[name] = metav1.NewTime(now.Add(-time.Duration(minutes) * time.Minute)) + } + return bound +} + +// TestChooseClusters pins the sticky selection: what a fresh policy picks, what a bound cluster +// keeps through ranking changes and outages, how a lowered count trims, and how one cluster +// serves several selectors. +func TestChooseClusters(t *testing.T) { + testCases := []struct { + name string + outcomes []selectorOutcome + bound map[string]metav1.Time + clusters map[string]*clusterv1beta1.MemberCluster + wantDesired map[string][]int + wantChosen [][]string + }{ + { + name: "fresh policy picks the best-ranked clusters up to the count", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"fat", "slim"}}, + }, + clusters: map[string]*clusterv1beta1.MemberCluster{ + "fat": rankedCluster("fat", "10", "4", "8Gi"), + "slim": rankedCluster("slim", "2", "4", "8Gi"), + }, + wantDesired: map[string][]int{"slim": {0}}, + wantChosen: [][]string{{"slim"}}, + }, + { + name: "a bound cluster sticks ahead of a better-ranked one", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"fat", "slim"}}, + }, + bound: boundSince(map[string]int{"fat": 5}), + clusters: map[string]*clusterv1beta1.MemberCluster{ + "fat": rankedCluster("fat", "10", "4", "8Gi"), + "slim": rankedCluster("slim", "2", "4", "8Gi"), + }, + wantDesired: map[string][]int{"fat": {0}}, + wantChosen: [][]string{{"fat"}}, + }, + { + name: "a bound cluster that went unschedulable sticks but does not count", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 2}, matched: []string{"a", "b"}, sticky: []string{"dark"}}, + }, + bound: boundSince(map[string]int{"dark": 5}), + clusters: uniformClusters("a", "b", "dark"), + wantDesired: map[string][]int{"a": {0}, "b": {0}, "dark": {0}}, + wantChosen: [][]string{{"a", "b"}}, + }, + { + name: "a bound cluster whose data cannot be evaluated sticks the same way", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"a"}, sticky: []string{"garbled"}}, + }, + bound: boundSince(map[string]int{"garbled": 5}), + clusters: uniformClusters("a", "garbled"), + wantDesired: map[string][]int{"a": {0}, "garbled": {0}}, + wantChosen: [][]string{{"a"}}, + }, + { + name: "an unbound unschedulable cluster is never picked", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 2}, matched: []string{"a"}, sticky: []string{"dark"}}, + }, + clusters: uniformClusters("a", "dark"), + wantDesired: map[string][]int{"a": {0}}, + wantChosen: [][]string{{"a"}}, + }, + { + name: "a bound cluster that matches nothing anymore is dropped", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"a"}}, + }, + bound: boundSince(map[string]int{"gone": 5, "relabeled": 5}), + clusters: uniformClusters("a", "relabeled"), + wantDesired: map[string][]int{"a": {0}}, + wantChosen: [][]string{{"a"}}, + }, + { + name: "a lowered count drops the newest bindings first", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"newest", "old", "older"}}, + }, + bound: boundSince(map[string]int{"older": 30, "old": 20, "newest": 10}), + clusters: uniformClusters("newest", "old", "older"), + wantDesired: map[string][]int{"older": {0}}, + wantChosen: [][]string{{"older"}}, + }, + { + name: "a lowered count drops a cluster that no longer matches before the newer matching ones", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 3}, matched: []string{"a", "b", "c", "d", "drifted"}, stale: []string{"drifted"}}, + }, + bound: boundSince(map[string]int{"drifted": 50, "a": 40, "b": 30, "c": 20, "d": 10}), + clusters: uniformClusters("a", "b", "c", "d", "drifted"), + wantDesired: map[string][]int{"a": {0}, "b": {0}, "c": {0}}, + wantChosen: [][]string{{"a", "b", "c"}}, + }, + { + name: "a cluster that no longer matches and went unschedulable sticks but does not count", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"a", "b"}, sticky: []string{"drifted-dark"}, stale: []string{"drifted-dark"}}, + }, + bound: boundSince(map[string]int{"drifted-dark": 50, "a": 40, "b": 30}), + clusters: uniformClusters("a", "b", "drifted-dark"), + wantDesired: map[string][]int{"a": {0}, "drifted-dark": {0}}, + wantChosen: [][]string{{"a"}}, + }, + { + name: "a cluster back from an outage keeps its binding and the replacement is trimmed", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 2}, matched: []string{"back", "kept", "replacement"}}, + }, + bound: boundSince(map[string]int{"back": 30, "kept": 30, "replacement": 5}), + clusters: uniformClusters("back", "kept", "replacement"), + wantDesired: map[string][]int{"back": {0}, "kept": {0}}, + wantChosen: [][]string{{"back", "kept"}}, + }, + { + name: "one cluster satisfies two selectors", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"a", "b"}}, + {counts: resolvedCounts{desired: 1}, matched: []string{"a", "c"}}, + }, + clusters: uniformClusters("a", "b", "c"), + wantDesired: map[string][]int{"a": {0, 1}}, + wantChosen: [][]string{{"a"}, {"a"}}, + }, + { + name: "a cluster trimmed by one selector may still serve another", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"a", "b"}}, + {counts: resolvedCounts{desired: 1}, matched: []string{"b"}}, + }, + bound: boundSince(map[string]int{"a": 30, "b": 10}), + clusters: uniformClusters("a", "b"), + wantDesired: map[string][]int{"a": {0}, "b": {1}}, + wantChosen: [][]string{{"a"}, {"b"}}, + }, + { + name: "a pick of this round is the newest when a later selector trims", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1}, matched: []string{"fresh"}}, + {counts: resolvedCounts{desired: 1}, matched: []string{"fresh", "veteran"}}, + }, + bound: boundSince(map[string]int{"veteran": 30}), + clusters: uniformClusters("fresh", "veteran"), + wantDesired: map[string][]int{"fresh": {0}, "veteran": {1}}, + wantChosen: [][]string{{"fresh"}, {"veteran"}}, + }, + { + name: "selectAll binds every schedulable match and keeps the unschedulable bound one", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{selectAll: true}, matched: []string{"a", "b"}, sticky: []string{"dark"}}, + }, + bound: boundSince(map[string]int{"dark": 5}), + clusters: uniformClusters("a", "b", "dark"), + wantDesired: map[string][]int{"a": {0}, "b": {0}, "dark": {0}}, + wantChosen: [][]string{{"a", "b"}}, + }, + { + name: "fewer matches than the count binds what there is", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 3}, matched: []string{"a"}}, + }, + clusters: uniformClusters("a"), + wantDesired: map[string][]int{"a": {0}}, + wantChosen: [][]string{{"a"}}, + }, + { + name: "no selectors", + wantDesired: map[string][]int{}, + wantChosen: [][]string{}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := chooseClusters(tc.outcomes, tc.bound, tc.clusters) + if diff := cmp.Diff(got, tc.wantDesired); diff != "" { + t.Errorf("chooseClusters() desired mismatch (-got, +want):\n%s", diff) + } + gotChosen := make([][]string, 0, len(tc.outcomes)) + for i := range tc.outcomes { + gotChosen = append(gotChosen, tc.outcomes[i].chosen) + } + if diff := cmp.Diff(gotChosen, tc.wantChosen); diff != "" { + t.Errorf("chooseClusters() chosen mismatch (-got, +want):\n%s", diff) + } + }) + } +} + +// TestBindingName guards the generated binding name against the object name limit and DNS-1123 +// rules, including a truncation point that lands on a separator, and pins that bindings of +// different clusters, policies, namespaces, and scopes never share a name. +func TestBindingName(t *testing.T) { + long := strings.Repeat("a", bindingNameSegmentMaxLength-1) + "." + strings.Repeat("a", 150) + testCases := []struct { + name string + policyName string + namespace string + cluster string + }{ + {name: "short names", policyName: "app", namespace: "tenant-a", cluster: "east-1"}, + {name: "another cluster", policyName: "app", namespace: "tenant-a", cluster: "east-2"}, + {name: "another policy", policyName: "app2", namespace: "tenant-a", cluster: "east-1"}, + {name: "another namespace", policyName: "app", namespace: "tenant-b", cluster: "east-1"}, + {name: "cluster scope", policyName: "app", cluster: "east-1"}, + {name: "long policy name with a separator at the truncation point", policyName: long, namespace: "tenant-a", cluster: "east-1"}, + {name: "long cluster name with a separator at the truncation point", policyName: "app", namespace: "tenant-a", cluster: long}, + {name: "both long", policyName: long, namespace: "tenant-a", cluster: long}, + } + + seen := sets.New[string]() + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := bindingName(policyFor(tc.policyName, tc.namespace), tc.cluster) + if errs := validation.IsDNS1123Subdomain(got); len(errs) > 0 { + t.Errorf("bindingName() = %q, want a valid DNS-1123 subdomain, got errors %v", got, errs) + } + if again := bindingName(policyFor(tc.policyName, tc.namespace), tc.cluster); again != got { + t.Errorf("bindingName() = %q on the second call, want the deterministic %q", again, got) + } + if seen.Has(got) { + t.Errorf("bindingName() = %q, want a name distinct from every other case's", got) + } + seen.Insert(got) + }) + } +} + +// withIndexes registers, on a fake client builder that already has its scheme, the binding and +// claim indexes the controller lists through, as fieldindexers.SetupWithHubAgentControllerManager +// does on the real cache. +func withIndexes(b *fake.ClientBuilder) *fake.ClientBuilder { + byPolicyName := func(obj client.Object) []string { + return []string{obj.(kfplacementv1alpha1.PlacementBindingAccessor).GetSpec().PlacementPolicyName} + } + byPolicyRef := func(obj client.Object) []string { + return []string{fieldindexers.ClusterClaimOwnedByPlacementPolicyCustomFieldVal(obj.(*kfplacementv1alpha1.ClusterClaim).Spec.PlacementPolicyRef)} + } + return b. + WithIndex(&kfplacementv1alpha1.PlacementBinding{}, fieldindexers.PlacementBindingOwnedByCustomFieldName, byPolicyName). + WithIndex(&kfplacementv1alpha1.ClusterPlacementBinding{}, fieldindexers.PlacementBindingOwnedByCustomFieldName, byPolicyName). + WithIndex(&kfplacementv1alpha1.ClusterClaim{}, fieldindexers.ClusterClaimOwnedByPlacementPolicyCustomFieldName, byPolicyRef) +} + +// bindingManagerHeldBy builds a policy whose binding manager role is held by the given +// controller, as a previous round would have left it. +func bindingManagerHeldBy(controller string) *kfplacementv1alpha1.PlacementPolicy { + policy := &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Namespace: "default", Name: "pp", UID: types.UID("pp-uid")}, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{APIVersion: "v1", Kind: "ConfigMap", Name: "demo"}}, + }, + } + policy.Status.BindingManager = &kfplacementv1alpha1.BindingManager{ + ControllerName: controller, + ObjectRefs: []kfplacementv1alpha1.ObjectReference{*policyReference(policy)}, + } + return policy +} + +// TestReconcileBindingsReleasesLeftOverRole pins that a round with nothing to do still releases a +// binding manager role a previous round left held, e.g. after a failed write whose change became +// moot; otherwise the policy would stay locked for every other binding manipulator forever. +func TestReconcileBindingsReleasesLeftOverRole(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + policy := bindingManagerHeldBy(controllerName) + c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(policy).WithStatusSubresource(policy).Build() + r := NewReconciler(c, c, snapshotStub{}) + + roleHeld, err := r.reconcileBindings(context.Background(), policy, nil, map[string][]int{}, nil, stubSnapshotName(policy)) + if err != nil || roleHeld { + t.Fatalf("reconcileBindings() = %t, %v, want false, nil", roleHeld, err) + } + fetched := &kfplacementv1alpha1.PlacementPolicy{} + if err := c.Get(context.Background(), types.NamespacedName{Namespace: "default", Name: "pp"}, fetched); err != nil { + t.Fatalf("Get() = %v, want no error", err) + } + if fetched.Status.BindingManager != nil { + t.Errorf("reconcileBindings() left the binding manager role as %v, want it released", fetched.Status.BindingManager) + } +} + +// TestReconcileBindingsBacksOffWhenRoleIsHeld pins that a role held by another process leaves +// the bindings untouched and reports the hold, rather than writing over a rollout in progress. +func TestReconcileBindingsBacksOffWhenRoleIsHeld(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + policy := bindingManagerHeldBy("rollout-controller") + c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(policy).WithStatusSubresource(policy).Build() + r := NewReconciler(c, c, snapshotStub{}) + + outcomes := []selectorOutcome{{counts: resolvedCounts{desired: 1}, matched: []string{"east-1"}, chosen: []string{"east-1"}}} + roleHeld, err := r.reconcileBindings(context.Background(), policy, outcomes, map[string][]int{"east-1": {0}}, nil, stubSnapshotName(policy)) + if err != nil || !roleHeld { + t.Fatalf("reconcileBindings() = %t, %v, want true, nil", roleHeld, err) + } + bindings := &kfplacementv1alpha1.PlacementBindingList{} + if err := c.List(context.Background(), bindings); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + if len(bindings.Items) != 0 { + t.Errorf("reconcileBindings() created %d bindings while the role was held elsewhere, want 0", len(bindings.Items)) + } +} + +// TestReconcileBindingsUpdatesSyncStrategy pins that a change to a policy's sync strategy reaches +// the bindings it already has, not only the ones it creates afterwards. +func TestReconcileBindingsUpdatesSyncStrategy(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + policy := bindingManagerHeldBy(controllerName) + c := withIndexes(fake.NewClientBuilder().WithScheme(scheme)).WithObjects(policy).WithStatusSubresource(policy).Build() + r := NewReconciler(c, c, snapshotStub{}) + outcomes := []selectorOutcome{{counts: resolvedCounts{desired: 1}, matched: []string{"east-1"}, chosen: []string{"east-1"}}} + chosen := map[string][]int{"east-1": {0}} + ctx := context.Background() + + if _, err := r.reconcileBindings(ctx, policy, outcomes, chosen, nil, stubSnapshotName(policy)); err != nil { + t.Fatalf("reconcileBindings() = %v, want no error", err) + } + if err := c.Get(ctx, types.NamespacedName{Namespace: "default", Name: "pp"}, policy); err != nil { + t.Fatalf("Get() = %v, want no error", err) + } + policy.Spec.SyncStrategy = &kfplacementv1alpha1.SyncStrategy{ApplyMethod: kfplacementv1alpha1.ApplyMethodServerSideApply} + want := policy.Spec.SyncStrategy.DeepCopy() + // The change is written, not just set in memory: claiming the binding manager role re-reads + // the policy, as a real round would. + if err := c.Update(ctx, policy); err != nil { + t.Fatalf("Update() = %v, want no error", err) + } + + // The second round sees the binding the first created, so the change has to go through an + // update of the existing binding. + existing, err := r.listBindings(ctx, policy) + if err != nil || len(existing) != 1 { + t.Fatalf("listBindings() = %d bindings, %v, want 1 binding, no error", len(existing), err) + } + if _, err := r.reconcileBindings(ctx, policy, outcomes, chosen, existing, stubSnapshotName(policy)); err != nil { + t.Fatalf("reconcileBindings() = %v, want no error", err) + } + + bindings := &kfplacementv1alpha1.PlacementBindingList{} + if err := c.List(ctx, bindings); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + if len(bindings.Items) != 1 { + t.Fatalf("reconcileBindings() left %d bindings, want 1", len(bindings.Items)) + } + if diff := cmp.Diff(want, bindings.Items[0].Spec.SyncStrategy); diff != "" { + t.Errorf("binding sync strategy mismatch (-want +got):\n%s", diff) + } +} + +// TestReconcileBindingsWithoutSnapshotLeavesRoleAlone pins that a round with no snapshot and only +// clusters to bind has nothing to do: it does not reach for the binding manager role at all, so a +// snapshot outage neither churns the role nor contends with a rollout holding it. +func TestReconcileBindingsWithoutSnapshotLeavesRoleAlone(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + policy := bindingManagerHeldBy("rollout-controller") + c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(policy).WithStatusSubresource(policy).Build() + r := NewReconciler(c, c, snapshotStub{}) + + outcomes := []selectorOutcome{{counts: resolvedCounts{desired: 1}, matched: []string{"east-1"}, chosen: []string{"east-1"}}} + roleHeld, err := r.reconcileBindings(context.Background(), policy, outcomes, map[string][]int{"east-1": {0}}, nil, "") + if err != nil || roleHeld { + t.Fatalf("reconcileBindings() = %t, %v, want false, nil: with nothing it can do, the round must not contend for the role", roleHeld, err) + } + bindings := &kfplacementv1alpha1.PlacementBindingList{} + if err := c.List(context.Background(), bindings); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + if len(bindings.Items) != 0 { + t.Errorf("reconcileBindings() created %d bindings without a snapshot, want 0", len(bindings.Items)) + } +} + +// name still controls it -- without holding up the policy's other clusters or its role. +func TestReconcileBindingsSettlesATakenName(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + testCases := []struct { + name string + owner *metav1.OwnerReference + occupantCluster string + wantErrCategory kferrors.ErrCategory + wantAdopted bool + }{ + {name: "an orphan of the policy's name", occupantCluster: "east-1", wantAdopted: true}, + {name: "an orphan of the policy's name for another cluster", occupantCluster: "west-9", wantErrCategory: kferrors.ErrCategoryUser}, + { + name: "a binding of an earlier policy of the same name", + owner: &metav1.OwnerReference{APIVersion: kfplacementv1alpha1.GroupVersion.String(), Kind: kfplacementv1alpha1.PlacementPolicyKind, Name: "pp", UID: "earlier-uid", Controller: ptr.To(true)}, + occupantCluster: "east-1", + wantErrCategory: kferrors.ErrCategoryTransient, + }, + { + name: "a binding another object controls", + owner: &metav1.OwnerReference{APIVersion: "v1", Kind: "ConfigMap", Name: "squatter", UID: "squatter-uid", Controller: ptr.To(true)}, + occupantCluster: "east-1", + wantErrCategory: kferrors.ErrCategoryUser, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + policy := bindingManagerHeldBy(controllerName) + occupant := &kfplacementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{Name: bindingName(policy, "east-1"), Namespace: "default"}, + Spec: kfplacementv1alpha1.PlacementBindingSpec{PlacementPolicyName: "pp", ClusterName: tc.occupantCluster, ResourceSnapshotName: "pp-snapshot-from-before"}, + } + if tc.owner != nil { + occupant.OwnerReferences = []metav1.OwnerReference{*tc.owner} + } + c := withIndexes(fake.NewClientBuilder().WithScheme(scheme)).WithObjects(policy, occupant).WithStatusSubresource(policy).Build() + r := NewReconciler(c, c, snapshotStub{}) + ctx := context.Background() + + outcomes := []selectorOutcome{{counts: resolvedCounts{desired: 2}, matched: []string{"east-1", "west-2"}, chosen: []string{"east-1", "west-2"}}} + _, err := r.reconcileBindings(ctx, policy, outcomes, map[string][]int{"east-1": {0}, "west-2": {0}}, nil, stubSnapshotName(policy)) + if got := kferrors.Category(err); (err != nil) != (tc.wantErrCategory != "") || (err != nil && got != tc.wantErrCategory) { + t.Fatalf("reconcileBindings() = %v (category %q), want error category %q", err, got, tc.wantErrCategory) + } + + got := &kfplacementv1alpha1.PlacementBinding{} + if err := c.Get(ctx, client.ObjectKeyFromObject(occupant), got); err != nil { + t.Fatalf("Get(occupant) = %v, want no error", err) + } + if adopted := metav1.IsControlledBy(got, policy); adopted != tc.wantAdopted { + t.Errorf("occupant controlled by the policy = %t, want %t", adopted, tc.wantAdopted) + } + if tc.wantAdopted && got.Spec.ResourceSnapshotName != stubSnapshotName(policy) { + t.Errorf("adopted binding rolls out %q, want the policy's latest snapshot %q", got.Spec.ResourceSnapshotName, stubSnapshotName(policy)) + } + if !tc.wantAdopted && got.Spec.ClusterName != tc.occupantCluster { + t.Errorf("occupant retargeted to cluster %q, want it left on %q", got.Spec.ClusterName, tc.occupantCluster) + } + // The other cluster is bound regardless, and the role is not left held. + if err := c.Get(ctx, types.NamespacedName{Namespace: "default", Name: bindingName(policy, "west-2")}, &kfplacementv1alpha1.PlacementBinding{}); err != nil { + t.Errorf("Get(west-2 binding) = %v, want the other cluster bound", err) + } + fetched := &kfplacementv1alpha1.PlacementPolicy{} + if err := c.Get(ctx, types.NamespacedName{Namespace: "default", Name: "pp"}, fetched); err != nil { + t.Fatalf("Get(policy) = %v, want no error", err) + } + if fetched.Status.BindingManager != nil { + t.Errorf("reconcileBindings() left the binding manager role as %v, want it released", fetched.Status.BindingManager) + } + }) + } +} diff --git a/pkg/v1/controllers/placementpolicy/claims.go b/pkg/v1/controllers/placementpolicy/claims.go new file mode 100644 index 000000000..ceab38476 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/claims.go @@ -0,0 +1,547 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + "fmt" + "slices" + + apiequality "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/utils/naming" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + // claimCleanupFinalizer marks policies with outstanding cluster claims; deleting such a + // policy first withdraws its claims. + claimCleanupFinalizer = "placement.kubefleet.dev/claim-cleanup" + + // maxConcurrentClaimsPerPolicy is the number of cluster claims a single policy may have in + // flight, per the FEP's default of one; the user-configurable per-fleet and per-placement + // limits are a config surface that arrives separately. + maxConcurrentClaimsPerPolicy = 1 + + // claimNameBaseMaxLength bounds the policy-name prefix inside a generated claim name so + // the full name stays well within the 253-character object name limit. + claimNameBaseMaxLength = 200 +) + +// claimOwnershipLabels returns the labels that select the claims of a policy. +func claimOwnershipLabels(policy kfplacementv1alpha1.PlacementPolicyAccessor) client.MatchingLabels { + return client.MatchingLabels{ + kfplacementv1alpha1.ClusterClaimPlacementPolicyNameLabel: naming.LabelValue(policy.GetName()), + kfplacementv1alpha1.ClusterClaimPlacementPolicyNamespaceLabel: policy.GetNamespace(), + } +} + +// claimName derives the deterministic name for the claim serving a policy's selector. The name +// embeds a hash of the policy's namespaced name because claims are cluster-scoped while +// PlacementPolicy names are only unique per namespace; two same-named policies in different +// namespaces must not collide. Determinism matters: creation is get-or-create, so a restarted +// reconciler converges on the same claim instead of issuing a duplicate. +func claimName(policy kfplacementv1alpha1.PlacementPolicyAccessor, selectorIndex int) string { + base := naming.Truncate(policy.GetName(), claimNameBaseMaxLength) + return fmt.Sprintf("%s-%d-%s", base, selectorIndex, naming.Hash(policy.GetNamespace()+"/"+policy.GetName())) +} + +// desiredClaim describes a claim the policy currently wants outstanding. +type desiredClaim struct { + name string + terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm + // outcome is the selector this claim serves. It is what lets the reconcile decide when a + // completed claim has done its job and should be rotated to provision the next cluster. + outcome *selectorOutcome +} + +// desiredClaims returns the claims the policy should have outstanding given the selector +// outcomes: one claim per unfulfilled selector that opted into AddClusterClaim, in selector +// order, capped by the per-policy concurrency limit. +func desiredClaims(policy kfplacementv1alpha1.PlacementPolicyAccessor, outcomes []selectorOutcome) []desiredClaim { + wanted := make([]desiredClaim, 0, maxConcurrentClaimsPerPolicy) + for i := range outcomes { + o := &outcomes[i] + if o.satisfiedInFull() || o.whenUnfulfilled != kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim { + continue + } + wanted = append(wanted, desiredClaim{name: claimName(policy, i), terms: o.terms, outcome: o}) + if len(wanted) >= maxConcurrentClaimsPerPolicy { + break + } + } + return wanted +} + +// claimReadyToRotate reports whether a claim has done its job -- the provisioner marked it completed +// and the cluster it provisioned is now eligible for the selector -- while the selector still needs +// more clusters. One claim yields one cluster (the claim carries no count, and its status names a +// single provisioned cluster), so a selector wanting several is filled one claim at a time: such a +// claim is withdrawn and reissued so the provisioner provisions the next cluster. +// +// The eligibility check is what makes this safe. The next claim is issued only once the current +// claim's cluster is counted toward the selector, so a provisioner is never handed a second claim +// while the first cluster is still joining -- which would ask it to provision two clusters at once +// for one selector. +func claimReadyToRotate(claim *kfplacementv1alpha1.ClusterClaim, outcome *selectorOutcome) bool { + if outcome == nil || outcome.satisfiedInFull() { + return false + } + if !meta.IsStatusConditionTrue(claim.Status.Conditions, kfplacementv1alpha1.ClusterClaimCondTypeCompleted) { + return false + } + provisioned := claim.Status.ProvisionedClusterName + return provisioned != nil && slices.Contains(outcome.matched, *provisioned) +} + +// reconcileClaims drives the policy's cluster claims toward the desired set: it withdraws +// claims whose selector is fulfilled or gone, refreshes the freshness marker on claims that +// are still wanted, and issues new claims within the concurrency budget. It returns the number +// of claims outstanding for the policy. +// +// A claim held in Terminating by a provisioner finalizer still counts toward the concurrency +// budget (its deterministic name also blocks re-creation), so a slow provisioner teardown can +// never cause double-provisioning for the same selector. +func (r *Reconciler) reconcileClaims(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor, outcomes []selectorOutcome, mostRecentClusterCreation metav1.Time) (int32, error) { + existing, err := r.listClaims(ctx, policy) + if err != nil { + return 0, err + } + if len(existing) > 0 { + // Re-assert the cleanup finalizer while any claim exists, so an out-of-band finalizer + // removal self-heals instead of leaving claims orphanable. + if err := r.ensureFinalizer(ctx, policy); err != nil { + return 0, err + } + } + + wanted := desiredClaims(policy, outcomes) + wantedByName := make(map[string]desiredClaim, len(wanted)) + for _, w := range wanted { + wantedByName[w.name] = w + } + + var outstanding int32 + // The names withdrawn this pass, whose claims are already counted as outstanding. + withdrawn := make(map[string]bool) + for i := range existing { + claim := &existing[i] + if !claim.DeletionTimestamp.IsZero() { + // Already being withdrawn: the claim still occupies its name and budget slot + // regardless of whether its terms happen to match the currently wanted set (a + // fulfillment flap can re-want identical terms mid-teardown), and an object on + // its way out receives no further status writes. + outstanding++ + delete(wantedByName, claim.Name) + continue + } + w, stillWanted := wantedByName[claim.Name] + // A claim only serves its original terms; if the policy's selector changed, the + // outstanding claim is withdrawn and a fresh one is issued on a later pass. + keep := stillWanted && apiequality.Semantic.DeepEqual(claim.Spec.ClusterSelectorTerms, w.terms) + if keep && claimReadyToRotate(claim, w.outcome) { + // The provisioner has provisioned an eligible cluster for this claim, but the selector + // still needs more. The completed claim is not kept: it falls through to the withdrawal + // below and, its entry left in wantedByName, is reissued on a later pass so the provisioner + // gets a fresh claim for the next cluster. Rotation is gated on the provisioned cluster + // being eligible, so the next claim is never issued before this one's cluster is confirmed. + keep = false + } + if keep { + delete(wantedByName, claim.Name) + outstanding++ + if err := r.reconcileClaimLabels(ctx, claim, policy); err != nil { + return outstanding, err + } + if err := r.refreshClaimFreshness(ctx, claim, mostRecentClusterCreation); err != nil { + return outstanding, err + } + continue + } + klog.V(2).InfoS("Withdrawing a cluster claim", "clusterClaim", claim.Name, "placementPolicy", klog.KObj(policy)) + // The delete is pinned to the UID the cache showed: claim names are deterministic, so a + // cache that has yet to see an earlier withdrawal can still list the predecessor after its + // successor was created under the same name, and an unpinned delete would withdraw the + // successor. A UID mismatch fails with a conflict, and the requeue retries on a fresher view. + if err := r.Delete(ctx, claim, client.Preconditions{UID: &claim.UID}); err != nil { + if errors.IsNotFound(err) { + // Already fully gone; it occupies nothing. + continue + } + return outstanding, kferrors.NewAPIServerError(err, "failed to withdraw the cluster claim", false, "clusterClaim", klog.KObj(claim)) + } + // A claim withdrawn this pass still occupies its budget slot: a provisioner finalizer + // can hold it in Terminating past this reconcile, and a differently-named claim created + // below would otherwise stand beside it, exceeding the concurrency budget. The claim + // watch re-queues the policy once the object is truly gone, and the slot frees then. + outstanding++ + withdrawn[claim.Name] = true + } + + if len(wantedByName) == 0 { + // Nothing more to create. If nothing is outstanding either, the policy is claim-free, so the + // cleanup finalizer -- whose only job is to withdraw claims before the policy is deleted -- + // has nothing left to guard and is released. Without this a policy that once had a claim would + // carry the finalizer forever, and disabling the feature and then deleting such a claim-free + // policy would hang its deletion with no controller left to clear it, outside the documented + // outstanding-claims caveat. + if outstanding == 0 { + if err := r.releaseFinalizerIfNoClaims(ctx, policy); err != nil { + return outstanding, err + } + } + return outstanding, nil + } + + // The cleanup finalizer lands on the policy before any claim is created, so a crash + // between the two writes cannot orphan a claim. + if err := r.ensureFinalizer(ctx, policy); err != nil { + return outstanding, err + } + // The budget so far was counted from the cache, which can lag a claim created moments ago: + // one issued last round for a selector that has since been satisfied would be missing, and a + // claim for the next selector, under another name, would stand beside it. Claims the cache + // has yet to show are counted from the API server whenever the cached count would still let + // one be issued. (A claim created after this read, by a replica that has just lost + // leadership, can still slip past; leader election makes that window negligible.) + // + // ponytail: lists every claim in the fleet uncached, but only on rounds that could issue + // one; the ownership labels cannot narrow it server-side, being mutable. + if outstanding < maxConcurrentClaimsPerPolicy { + unseen, err := r.countUnseenClaims(ctx, policy, existing) + if err != nil { + return outstanding, err + } + outstanding += unseen + } + for _, w := range wanted { + if _, still := wantedByName[w.name]; !still { + continue + } + claim := &kfplacementv1alpha1.ClusterClaim{ + ObjectMeta: metav1.ObjectMeta{ + Name: w.name, + // The labels select a policy's claims; spec.placementPolicyRef below carries + // the policy's authoritative identity, which a label value cannot always hold. + Labels: claimOwnershipLabels(policy), + }, + Spec: kfplacementv1alpha1.ClusterClaimSpec{ + PlacementPolicyRef: policyReference(policy), + ClusterSelectorTerms: w.terms, + }, + } + if outstanding >= maxConcurrentClaimsPerPolicy { + // Every slot is taken, by kept claims or by ones withdrawn moments ago that may + // still be terminating; the create is retried when a watch frees a slot. + break + } + klog.V(2).InfoS("Adding a cluster claim", "clusterClaim", claim.Name, "placementPolicy", klog.KObj(policy)) + if err := r.Create(ctx, claim); err != nil { + if !errors.IsAlreadyExists(err) { + return outstanding, kferrors.NewAPIServerError(err, "failed to add the cluster claim", false, "clusterClaim", klog.KObj(claim)) + } + // The deterministic name is still occupied. Most commonly it is this round's own + // withdraw of a same-named claim that a provisioner finalizer holds in Terminating; + // otherwise it may be a claim of this policy the cache has yet to show. Both are + // already counted above, and the claim watch re-queues the policy once the occupant + // is gone, so the create is retried then. A claim of another policy is reported as an + // error instead; the watch maps that claim to its own policy, so this one retries on + // the error backoff. + if withdrawn[claim.Name] { + continue + } + if _, err := r.claimNameOccupiedByPolicy(ctx, claim.Name, policy); err != nil { + return outstanding, err + } + continue + } + outstanding++ + // Stamp the freshness marker at creation, per the FEP: the claim carries the latest + // observed cluster creation timestamp from the moment provisioners can see it. + if err := r.refreshClaimFreshness(ctx, claim, mostRecentClusterCreation); err != nil { + return outstanding, err + } + } + return outstanding, nil +} + +// countUnseenClaims counts, from the API server, the claims of the policy that the given cached list +// does not show. +func (r *Reconciler) countUnseenClaims(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor, seen []kfplacementv1alpha1.ClusterClaim) (int32, error) { + claims := &kfplacementv1alpha1.ClusterClaimList{} + if err := r.uncachedReader.List(ctx, claims); err != nil { + return 0, kferrors.NewAPIServerError(err, "failed to list the cluster claims of the policy", false, "placementPolicy", klog.KObj(policy)) + } + var unseen int32 + for _, claim := range claimsForPolicy(claims.Items, policy) { + if !slices.ContainsFunc(seen, func(c kfplacementv1alpha1.ClusterClaim) bool { return c.UID == claim.UID }) { + unseen++ + } + } + return unseen, nil +} + +// claimNameOccupiedByPolicy reports whether the claim holding a name this policy failed to create +// belongs to the policy. The read is uncached, since the cache is what failed to show the occupant. +// A claim of another policy under the name is a user error, logged with the occupant's reference: +// the policy cannot issue the claim until the occupant is gone, and without the error its selector +// would stay unfulfilled with nothing saying why. The error ends the round, leaving any later +// selector's claim to the retry; the budget is a single claim per policy, so nothing is lost. +func (r *Reconciler) claimNameOccupiedByPolicy(ctx context.Context, name string, policy kfplacementv1alpha1.PlacementPolicyAccessor) (bool, error) { + occupant := &kfplacementv1alpha1.ClusterClaim{} + if err := r.uncachedReader.Get(ctx, client.ObjectKey{Name: name}, occupant); err != nil { + if errors.IsNotFound(err) { + // Gone since the create; its deletion event re-queues the policy, and that pass + // issues the claim. + return false, nil + } + return false, kferrors.NewAPIServerError(err, "failed to get the cluster claim occupying the name", false, "clusterClaim", name) + } + if !claimBelongsTo(occupant, policyReference(policy)) { + return false, kferrors.NewUserError(nil, "the cluster claim name is taken by a claim that does not belong to the policy", + "clusterClaim", name, "placementPolicy", klog.KObj(policy), "occupantPlacementPolicyRef", occupant.Spec.PlacementPolicyRef) + } + return true, nil +} + +// reconcileClaimLabels restores the ownership labels on a kept claim. The controller itself finds +// a policy's claims through the immutable spec.placementPolicyRef, so a stripped or rewritten label +// never confuses the reconcile; the labels exist for external consumers -- a provisioner watching a +// policy's claims by label -- and a provisioner or user that mutated one would hide the claim from +// them. The labels are re-asserted here for the same reason the cleanup finalizer is re-asserted +// while claims exist: out-of-band drift on an object the controller owns should self-heal. +func (r *Reconciler) reconcileClaimLabels(ctx context.Context, claim *kfplacementv1alpha1.ClusterClaim, policy kfplacementv1alpha1.PlacementPolicyAccessor) error { + want := claimOwnershipLabels(policy) + drifted := false + for k, v := range want { + if claim.Labels[k] != v { + drifted = true + break + } + } + if !drifted { + return nil + } + if claim.Labels == nil { + claim.Labels = make(map[string]string, len(want)) + } + for k, v := range want { + claim.Labels[k] = v + } + klog.V(2).InfoS("Restoring ownership labels on a cluster claim", "clusterClaim", claim.Name, "placementPolicy", klog.KObj(policy)) + // A conflict means another writer touched the claim; the claim watch re-queues the policy and + // the repair retries then. A NotFound means the claim was withdrawn out from under us, which a + // later pass reconciles. + if err := r.Update(ctx, claim); err != nil && !errors.IsNotFound(err) && !errors.IsConflict(err) { + return kferrors.NewAPIServerError(err, "failed to restore the ownership labels of the cluster claim", false, "clusterClaim", klog.KObj(claim)) + } + return nil +} + +// refreshClaimFreshness advances the claim's freshness marker when clusters joined after the +// last observation; provisioners use the marker to tell that the claim has been re-evaluated +// and is still wanted. +func (r *Reconciler) refreshClaimFreshness(ctx context.Context, claim *kfplacementv1alpha1.ClusterClaim, mostRecent metav1.Time) error { + if mostRecent.IsZero() { + return nil + } + observed := claim.Status.LastObservedMostRecentClusterCreationTimestamp + if observed != nil && !mostRecent.After(observed.Time) { + return nil + } + claim.Status.LastObservedMostRecentClusterCreationTimestamp = &mostRecent + // Conflicts are expected steady-state once a provisioner co-writes the claim status; the + // provisioner's own write re-enqueues the policy through the claim watch, so the refresh + // simply retries then. This only-advance guard above is also what terminates the + // claim-watch self-loop: a refresh fires one echo reconcile, which then no-ops here. + if err := r.Status().Update(ctx, claim); err != nil && !errors.IsNotFound(err) && !errors.IsConflict(err) { + return kferrors.NewAPIServerError(err, "failed to refresh the freshness marker of the cluster claim", false, "clusterClaim", klog.KObj(claim)) + } + return nil +} + +// cleanupClaims withdraws every claim belonging to a policy that is being deleted, releasing +// the cleanup finalizer once none remain. The claim count that gates the finalizer release is +// read from the API server directly, not the informer cache: a claim created moments before +// the policy deletion might not have reached the cache yet, and releasing the finalizer on a +// stale zero would orphan it permanently (nothing else ever looks at claims of a gone policy). +func (r *Reconciler) cleanupClaims(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor) error { + if !controllerutil.ContainsFinalizer(policy, claimCleanupFinalizer) { + return nil + } + + // The release decision cannot rest on the ownership labels: labels are mutable, and a claim + // whose labels were stripped would vanish from a label-selected list, releasing the + // finalizer over a claim that still exists -- permanently orphaned, since nothing else ever + // looks at claims of a gone policy. The claim's spec.placementPolicyRef is immutable, so + // every claim is listed and ownership is decided by the reference. + claims := &kfplacementv1alpha1.ClusterClaimList{} + if err := r.uncachedReader.List(ctx, claims); err != nil { + wrappedErr := kferrors.NewAPIServerError(err, "failed to list the cluster claims of the deleted policy", false, "placementPolicy", klog.KObj(policy)) + klog.ErrorS(wrappedErr, "Failed to list cluster claims for the deleted policy", kferrors.Args(wrappedErr)...) + return wrappedErr + } + ownRef := policyReference(policy) + remaining := 0 + for i := range claims.Items { + claim := &claims.Items[i] + if !claimBelongsTo(claim, ownRef) { + continue + } + if !claim.DeletionTimestamp.IsZero() { + remaining++ + continue + } + klog.V(2).InfoS("Withdrawing a cluster claim of a deleted policy", "clusterClaim", claim.Name, "placementPolicy", klog.KObj(policy)) + if err := r.Delete(ctx, claim, client.Preconditions{UID: &claim.UID}); err != nil { + if errors.IsNotFound(err) { + // Already fully removed between the list and the delete; nothing remains for + // this claim. + continue + } + return kferrors.NewAPIServerError(err, "failed to withdraw the cluster claim of the deleted policy", false, "clusterClaim", klog.KObj(claim)) + } + remaining++ + } + if remaining > 0 { + // Claims may be held in Terminating by provisioner finalizers; the watch on claims + // re-queues the policy as they go away. + return nil + } + + controllerutil.RemoveFinalizer(policy, claimCleanupFinalizer) + // Once the finalizer is gone the policy is deleted, and the reconcile that observes its + // absence drops the metric series. + return r.updatePolicy(ctx, policy, "failed to release the claim cleanup finalizer") +} + +// ensureFinalizer adds the claim cleanup finalizer to the policy if not present yet. +func (r *Reconciler) ensureFinalizer(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor) error { + if controllerutil.ContainsFinalizer(policy, claimCleanupFinalizer) { + return nil + } + controllerutil.AddFinalizer(policy, claimCleanupFinalizer) + return r.updatePolicy(ctx, policy, "failed to add the claim cleanup finalizer") +} + +// claimBelongsTo reports whether a claim's immutable back-reference names the given policy. +// The API enforces the reference's immutability, which is what makes it, unlike the ownership +// labels, fit to gate the cleanup finalizer's release. +// +// The comparison deliberately ignores the reference's version: an API promotion keeps the group, +// kind, namespace, and name, and a version-strict comparison would orphan every outstanding claim +// the moment the policy API moved to a new version. The group is part of the identity: a claim +// naming another group's PlacementPolicy is not this policy's. +func claimBelongsTo(claim *kfplacementv1alpha1.ClusterClaim, ref *kfplacementv1alpha1.ObjectReference) bool { + got := claim.Spec.PlacementPolicyRef + if got == nil || ref == nil { + return false + } + return got.APIGroup == ref.APIGroup && got.Kind == ref.Kind && got.Name == ref.Name && got.Namespace == ref.Namespace +} + +// listClaims returns the cluster claims belonging to a policy, matched on the immutable +// spec.placementPolicyRef rather than the ownership labels. +// +// The labels are mutable: a provisioner or a user that removes or rewrites one would make the claim +// vanish from a label-selected list while spec.placementPolicyRef still names the policy. The live +// reconcile would then stop tracking a claim it still owns -- never withdrawing it once the selector +// is fulfilled, changed, or switched to KeepSearching -- and a provisioner could keep acting on it. +// Matching on the immutable reference, as the cleanup path already does, keeps that from happening. +// +// The list is served from the informer cache through the index on that reference (registered by +// fieldindexers.SetupWithHubAgentControllerManager), so a reconcile reads only its own policy's +// claims rather than every claim in the fleet. A claim the cache has yet to observe is picked up +// on the re-queue the claim watch fires, which is why the cache is acceptable here while the +// finalizer release, whose mistake would be permanent, reads through the uncached reader instead. +func (r *Reconciler) listClaims(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor) ([]kfplacementv1alpha1.ClusterClaim, error) { + claims := &kfplacementv1alpha1.ClusterClaimList{} + if err := r.List(ctx, claims, client.MatchingFields{ + fieldindexers.ClusterClaimOwnedByPlacementPolicyCustomFieldName: fieldindexers.ClusterClaimOwnedByPlacementPolicyCustomFieldVal(policyReference(policy)), + }); err != nil { + wrappedErr := kferrors.NewAPIServerError(err, "failed to list the cluster claims of the policy", true, "placementPolicy", klog.KObj(policy)) + klog.ErrorS(wrappedErr, "Failed to list cluster claims for the policy", kferrors.Args(wrappedErr)...) + return nil, wrappedErr + } + return claims.Items, nil +} + +// claimsForPolicy returns the claims whose immutable reference names the policy. +func claimsForPolicy(all []kfplacementv1alpha1.ClusterClaim, policy kfplacementv1alpha1.PlacementPolicyAccessor) []kfplacementv1alpha1.ClusterClaim { + ref := policyReference(policy) + owned := make([]kfplacementv1alpha1.ClusterClaim, 0, len(all)) + for i := range all { + if claimBelongsTo(&all[i], ref) { + owned = append(owned, all[i]) + } + } + return owned +} + +// releaseFinalizerIfNoClaims removes the cleanup finalizer from a live policy once no claim of it +// remains, confirmed against the API server. It mirrors the release in cleanupClaims: the finalizer +// exists only to withdraw claims before the policy is deleted, so once none remain it is safe to +// drop -- and dropping it is what keeps a claim-free policy deletable even after the feature is +// turned off. The count is read uncached because releasing the finalizer over a claim a stale cache +// has yet to show would orphan it, nothing ever looking at the claims of a gone policy. +func (r *Reconciler) releaseFinalizerIfNoClaims(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor) error { + if !controllerutil.ContainsFinalizer(policy, claimCleanupFinalizer) { + return nil + } + claims := &kfplacementv1alpha1.ClusterClaimList{} + if err := r.uncachedReader.List(ctx, claims); err != nil { + wrappedErr := kferrors.NewAPIServerError(err, "failed to list the cluster claims of the policy", false, "placementPolicy", klog.KObj(policy)) + klog.ErrorS(wrappedErr, "Failed to list cluster claims to release the cleanup finalizer", kferrors.Args(wrappedErr)...) + return wrappedErr + } + if len(claimsForPolicy(claims.Items, policy)) > 0 { + return nil + } + controllerutil.RemoveFinalizer(policy, claimCleanupFinalizer) + return r.updatePolicy(ctx, policy, "failed to release the claim cleanup finalizer") +} + +// updatePolicy writes the policy object, categorizing a failure as an API server error. +func (r *Reconciler) updatePolicy(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor, desc string) error { + if err := r.Update(ctx, policy); err != nil { + return kferrors.NewAPIServerError(err, desc, false, "placementPolicy", klog.KObj(policy)) + } + return nil +} + +// policyReference builds the claim's back-reference to its policy. +func policyReference(policy kfplacementv1alpha1.PlacementPolicyAccessor) *kfplacementv1alpha1.ObjectReference { + kind := kfplacementv1alpha1.PlacementPolicyKind + if policy.GetNamespace() == "" { + kind = kfplacementv1alpha1.ClusterPlacementPolicyKind + } + return &kfplacementv1alpha1.ObjectReference{ + APIGroup: kfplacementv1alpha1.GroupVersion.Group, + APIVersion: kfplacementv1alpha1.GroupVersion.Version, + Kind: kind, + Name: policy.GetName(), + Namespace: policy.GetNamespace(), + } +} diff --git a/pkg/v1/controllers/placementpolicy/claims_integration_test.go b/pkg/v1/controllers/placementpolicy/claims_integration_test.go new file mode 100644 index 000000000..9929ce3ce --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/claims_integration_test.go @@ -0,0 +1,605 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + apiequality "k8s.io/apimachinery/pkg/api/equality" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +var _ = Describe("cluster claim lifecycle", Ordered, func() { + var counter int + + nextName := func(prefix string) string { + counter++ + return fmt.Sprintf("%s-claim-%d", prefix, counter) + } + + AfterEach(func() { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + Expect(k8sClient.List(ctx, policies, client.InNamespace(testNamespace))).Should(Succeed()) + for i := range policies.Items { + Expect(client.IgnoreNotFound(k8sClient.Delete(ctx, &policies.Items[i]))).Should(Succeed()) + } + // Policy deletion is finalizer-gated on claim cleanup; wait for both to drain. + Eventually(func(g Gomega) { + g.Expect(k8sClient.List(ctx, policies, client.InNamespace(testNamespace))).Should(Succeed()) + g.Expect(policies.Items).Should(BeEmpty()) + claims := &kfplacementv1alpha1.ClusterClaimList{} + g.Expect(k8sClient.List(ctx, claims)).Should(Succeed()) + g.Expect(claims.Items).Should(BeEmpty()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + memberClusters := &clusterv1beta1.MemberClusterList{} + Expect(k8sClient.List(ctx, memberClusters)).Should(Succeed()) + for i := range memberClusters.Items { + Expect(client.IgnoreNotFound(k8sClient.Delete(ctx, &memberClusters.Items[i]))).Should(Succeed()) + } + Eventually(func() (int, error) { + if err := k8sClient.List(ctx, memberClusters); err != nil { + return -1, err + } + return len(memberClusters.Items), nil + }, eventuallyTimeout, pollInterval).Should(Equal(0)) + }) + + It("adds a claim for an unfulfilled selector and cleans it up with the policy", func() { + policy := newPolicy(nextName("pp"), regionSelector("brazilsouth", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + By("the claim appears with the policy reference, terms, and ownership labels") + Eventually(func(g Gomega) { + claim := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim)).Should(Succeed()) + g.Expect(claim.Labels).Should(HaveKeyWithValue(kfplacementv1alpha1.ClusterClaimPlacementPolicyNameLabel, policy.Name)) + g.Expect(claim.Labels).Should(HaveKeyWithValue(kfplacementv1alpha1.ClusterClaimPlacementPolicyNamespaceLabel, testNamespace)) + g.Expect(claim.Spec.PlacementPolicyRef).NotTo(BeNil()) + g.Expect(claim.Spec.PlacementPolicyRef.Name).Should(Equal(policy.Name)) + g.Expect(claim.Spec.PlacementPolicyRef.Kind).Should(Equal(kfplacementv1alpha1.PlacementPolicyKind)) + g.Expect(apiequality.Semantic.DeepEqual(claim.Spec.ClusterSelectorTerms, policy.Spec.ClusterSelectors[0].Terms)).Should(BeTrue()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the policy carries the cleanup finalizer and counts the claim") + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), fetched)).Should(Succeed()) + g.Expect(controllerutil.ContainsFinalizer(fetched, claimCleanupFinalizer)).Should(BeTrue()) + g.Expect(fetched.Status.ActiveClusterClaims).NotTo(BeNil()) + g.Expect(*fetched.Status.ActiveClusterClaims).Should(Equal(int32(1))) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("deleting the policy withdraws the claim and releases the policy") + Expect(k8sClient.Delete(ctx, policy)).Should(Succeed()) + Eventually(func(g Gomega) { + err := k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + g.Expect(client.IgnoreNotFound(err)).Should(Succeed()) + g.Expect(err).Should(HaveOccurred(), "claim should be withdrawn") + err = k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), &kfplacementv1alpha1.PlacementPolicy{}) + g.Expect(client.IgnoreNotFound(err)).Should(Succeed()) + g.Expect(err).Should(HaveOccurred(), "policy should be gone once the finalizer is released") + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("withdraws the claim only when an eligible cluster fulfills the selector", func() { + policy := newPolicy(nextName("pp"), regionSelector("koreacentral", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("a matching but not-yet-joined cluster does not withdraw the claim (join window)") + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "koreacentral"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + Consistently(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + }, consistentlyDuration, pollInterval).Should(Succeed()) + + By("once the cluster joins, the claim is withdrawn and the count drops") + markJoined(mc) + Eventually(func(g Gomega) { + err := k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + g.Expect(client.IgnoreNotFound(err)).Should(Succeed()) + g.Expect(err).Should(HaveOccurred()) + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), fetched)).Should(Succeed()) + g.Expect(fetched.Status.ActiveClusterClaims).NotTo(BeNil()) + g.Expect(*fetched.Status.ActiveClusterClaims).Should(Equal(int32(0))) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("still withdraws a claim whose ownership labels were stripped, tracked by the immutable reference", func() { + policy := newPolicy(nextName("pp"), regionSelector("chilecentral", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("a provisioner strips the ownership labels, leaving only spec.placementPolicyRef") + Eventually(func(g Gomega) { + claim := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim)).Should(Succeed()) + claim.Labels = nil + g.Expect(k8sClient.Update(ctx, claim)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("fulfilling the selector still withdraws the claim (a label-selected list would have lost it)") + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "chilecentral"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + Eventually(func(g Gomega) { + err := k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + g.Expect(client.IgnoreNotFound(err)).Should(Succeed()) + g.Expect(err).Should(HaveOccurred(), "the label-stripped claim should still be withdrawn") + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("restores the ownership labels on a still-wanted claim after they are stripped", func() { + policy := newPolicy(nextName("pp"), regionSelector("canadacentral", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("a provisioner strips the ownership labels while the selector is still unfulfilled") + Eventually(func(g Gomega) { + claim := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim)).Should(Succeed()) + claim.Labels = nil + g.Expect(k8sClient.Update(ctx, claim)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the controller re-asserts the labels so label-watching consumers find the claim again") + Eventually(func(g Gomega) { + claim := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim)).Should(Succeed()) + g.Expect(claim.Labels).Should(HaveKeyWithValue(kfplacementv1alpha1.ClusterClaimPlacementPolicyNameLabel, policy.Name)) + g.Expect(claim.Labels).Should(HaveKeyWithValue(kfplacementv1alpha1.ClusterClaimPlacementPolicyNamespaceLabel, testNamespace)) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("releases the cleanup finalizer once the policy's last claim is withdrawn", func() { + policy := newPolicy(nextName("pp"), regionSelector("mexicocentral", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + By("the policy carries the cleanup finalizer while its claim is outstanding") + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), fetched)).Should(Succeed()) + g.Expect(controllerutil.ContainsFinalizer(fetched, claimCleanupFinalizer)).Should(BeTrue()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("fulfilling the selector withdraws the claim and releases the finalizer") + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "mexicocentral"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), fetched)).Should(Succeed()) + g.Expect(controllerutil.ContainsFinalizer(fetched, claimCleanupFinalizer)).Should(BeFalse(), "a claim-free policy should not retain the cleanup finalizer") + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the claim-free policy deletes without waiting on a finalizer") + Expect(k8sClient.Delete(ctx, policy)).Should(Succeed()) + Eventually(func(g Gomega) { + err := k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), &kfplacementv1alpha1.PlacementPolicy{}) + g.Expect(client.IgnoreNotFound(err)).Should(Succeed()) + g.Expect(err).Should(HaveOccurred()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("issues one claim at a time until a count>1 selector is fully provisioned", func() { + policy := newPolicy(nextName("pp"), regionSelector("australiaeast", ptr.To(intstr.FromInt32(3)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + claimN := claimName(policy, 0) + + // waitFreshUID returns the UID of the claim at the selector's name once it exists and has not + // been marked completed -- a claim the provisioner has not acted on yet. + waitFreshUID := func() types.UID { + claim := &kfplacementv1alpha1.ClusterClaim{} + Eventually(func(g Gomega) { + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: claimN}, claim)).Should(Succeed()) + g.Expect(meta.IsStatusConditionTrue(claim.Status.Conditions, kfplacementv1alpha1.ClusterClaimCondTypeCompleted)).Should(BeFalse()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + return claim.UID + } + // provision stands in for a provisioner acting on exactly one claim: it joins one cluster and + // marks that specific claim (identified by UID) completed for it, exactly once. Guarding on the + // UID keeps a conflict retry from marking the next, already-rotated claim. + provision := func(clusterName string, claimUID types.UID) { + mc := newMemberCluster(clusterName, map[string]string{testRegionLabel: "australiaeast"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + Eventually(func(g Gomega) { + claim := &kfplacementv1alpha1.ClusterClaim{} + err := k8sClient.Get(ctx, types.NamespacedName{Name: claimN}, claim) + if apierrors.IsNotFound(err) || (err == nil && claim.UID != claimUID) { + return // the targeted claim is gone or already rotated: our completion took effect + } + g.Expect(err).ShouldNot(HaveOccurred()) // any other read error fails loudly rather than passing as a skip + claim.Status.ProvisionedClusterName = ptr.To(clusterName) + meta.SetStatusCondition(&claim.Status.Conditions, metav1.Condition{ + Type: kfplacementv1alpha1.ClusterClaimCondTypeCompleted, Status: metav1.ConditionTrue, Reason: "Fulfilled", + }) + g.Expect(k8sClient.Status().Update(ctx, claim)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + } + // expectRotation waits for the completed claim (previous UID) to be withdrawn and a fresh one + // issued for the next cluster, and returns the fresh claim's UID. + expectRotation := func(previousUID types.UID) types.UID { + claim := &kfplacementv1alpha1.ClusterClaim{} + Eventually(func(g Gomega) { + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: claimN}, claim)).Should(Succeed()) + g.Expect(claim.UID).ShouldNot(Equal(previousUID), "the completed claim should be rotated to a fresh one") + g.Expect(meta.IsStatusConditionTrue(claim.Status.Conditions, kfplacementv1alpha1.ClusterClaimCondTypeCompleted)).Should(BeFalse()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + return claim.UID + } + + By("clusters 1 and 2 of 3: each completed claim is rotated to a fresh one for the next cluster") + uid := waitFreshUID() + provision(nextName("mc-aue"), uid) + uid = expectRotation(uid) + provision(nextName("mc-aue"), uid) + uid = expectRotation(uid) + + By("cluster 3 of 3: the selector is satisfied, so the claim is withdrawn and not reissued") + provision(nextName("mc-aue"), uid) + Eventually(func(g Gomega) { + err := k8sClient.Get(ctx, types.NamespacedName{Name: claimN}, &kfplacementv1alpha1.ClusterClaim{}) + g.Expect(client.IgnoreNotFound(err)).Should(Succeed()) + g.Expect(err).Should(HaveOccurred(), "the claim should be withdrawn once three clusters are provisioned") + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), fetched)).Should(Succeed()) + g.Expect(fetched.Status.ActiveClusterClaims).NotTo(BeNil()) + g.Expect(*fetched.Status.ActiveClusterClaims).Should(Equal(int32(0))) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("keeps at most one claim outstanding across multiple unfulfilled selectors", func() { + policy := newPolicy(nextName("pp"), + regionSelector("swedencentral", ptr.To(intstr.FromInt32(1)), nil), + regionSelector("norwayeast", ptr.To(intstr.FromInt32(1)), nil), + ) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + firstClaim := claimName(policy, 0) + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: firstClaim}, &kfplacementv1alpha1.ClusterClaim{}) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + Consistently(func(g Gomega) { + claims := &kfplacementv1alpha1.ClusterClaimList{} + g.Expect(k8sClient.List(ctx, claims, client.MatchingLabels{kfplacementv1alpha1.ClusterClaimPlacementPolicyNameLabel: policy.Name})).Should(Succeed()) + g.Expect(claims.Items).Should(HaveLen(1)) + g.Expect(claims.Items[0].Name).Should(Equal(firstClaim)) + }, consistentlyDuration, pollInterval).Should(Succeed()) + }) + + It("does not add claims for KeepSearching selectors", func() { + selector := regionSelector("qatarcentral", ptr.To(intstr.FromInt32(1)), nil) + selector.WhenUnfulfilled = kfplacementv1alpha1.WhenUnfulfilledOptionKeepSearching + policy := newPolicy(nextName("pp"), selector) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + By("the policy reports unfulfilled scheduling yet no claim appears") + Eventually(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 1, 0), + eventuallyTimeout, pollInterval).Should(Succeed()) + Consistently(func(g Gomega) { + claims := &kfplacementv1alpha1.ClusterClaimList{} + g.Expect(k8sClient.List(ctx, claims, client.MatchingLabels{kfplacementv1alpha1.ClusterClaimPlacementPolicyNameLabel: policy.Name})).Should(Succeed()) + g.Expect(claims.Items).Should(BeEmpty()) + }, consistentlyDuration, pollInterval).Should(Succeed()) + }) + + It("refreshes the claim freshness marker when a non-matching cluster joins", func() { + policy := newPolicy(nextName("pp"), regionSelector("polandcentral", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("a cluster joins that does not satisfy the selector") + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "italynorth"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + + By("the claim survives and its freshness marker is advanced") + Eventually(func(g Gomega) { + claim := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim)).Should(Succeed()) + g.Expect(claim.Status.LastObservedMostRecentClusterCreationTimestamp).NotTo(BeNil()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("leaves a Terminating claim alone when its selector becomes unfulfilled again", func() { + policy := newPolicy(nextName("pp"), regionSelector("uaenorth", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + claim := &kfplacementv1alpha1.ClusterClaim{} + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("a provisioner holds the claim with a finalizer") + claim.Finalizers = append(claim.Finalizers, "test.kubefleet.dev/provisioner") + Expect(k8sClient.Update(ctx, claim)).Should(Succeed()) + + By("a matching cluster joins and fulfills the selector, so the claim is withdrawn") + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "uaenorth"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + g.Expect(fetched.DeletionTimestamp.IsZero()).Should(BeFalse(), "claim should be Terminating") + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the cluster leaves, so the identical selector is unfulfilled again while the claim is still Terminating") + Expect(k8sClient.Delete(ctx, mc)).Should(Succeed()) + + By("the Terminating claim keeps its slot and no replacement is issued") + Consistently(func(g Gomega) { + claims := &kfplacementv1alpha1.ClusterClaimList{} + g.Expect(k8sClient.List(ctx, claims, client.MatchingLabels{kfplacementv1alpha1.ClusterClaimPlacementPolicyNameLabel: policy.Name})).Should(Succeed()) + g.Expect(claims.Items).Should(HaveLen(1)) + g.Expect(claims.Items[0].DeletionTimestamp.IsZero()).Should(BeFalse()) + }, consistentlyDuration, pollInterval).Should(Succeed()) + + By("releasing the finalizer lets a fresh claim be issued for the still-unfulfilled selector") + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + fetched.Finalizers = nil + g.Expect(k8sClient.Update(ctx, fetched)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + g.Expect(fetched.DeletionTimestamp.IsZero()).Should(BeTrue(), "a live replacement claim should exist") + }, eventuallyTimeout*2, pollInterval).Should(Succeed()) + }) + + It("waits out a provisioner finalizer when replacing a claim after a selector change", func() { + policy := newPolicy(nextName("pp"), regionSelector("israelcentral", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + claim := &kfplacementv1alpha1.ClusterClaim{} + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("a provisioner protects its in-flight work with a finalizer") + claim.Finalizers = append(claim.Finalizers, "test.kubefleet.dev/provisioner") + Expect(k8sClient.Update(ctx, claim)).Should(Succeed()) + + By("the selector is retargeted; the old claim lingers Terminating and blocks the slot") + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), fetched)).Should(Succeed()) + fetched.Spec.ClusterSelectors[0].Terms = []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{testRegionLabel: "newzealandnorth"}}, + } + g.Expect(k8sClient.Update(ctx, fetched)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + g.Expect(fetched.DeletionTimestamp.IsZero()).Should(BeFalse(), "old claim should be Terminating") + g.Expect(apiequality.Semantic.DeepEqual(fetched.Spec.ClusterSelectorTerms, claim.Spec.ClusterSelectorTerms)).Should(BeTrue(), "the Terminating claim still carries the old terms") + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("releasing the finalizer lets the replacement claim through with the new terms") + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + fetched.Finalizers = nil + g.Expect(k8sClient.Update(ctx, fetched)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + g.Expect(fetched.DeletionTimestamp.IsZero()).Should(BeTrue(), "replacement claim should be live") + g.Expect(fetched.Spec.ClusterSelectorTerms).Should(HaveLen(1)) + g.Expect(fetched.Spec.ClusterSelectorTerms[0].MatchLabels).Should(HaveKeyWithValue(testRegionLabel, "newzealandnorth")) + policy := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: fetched.Labels[kfplacementv1alpha1.ClusterClaimPlacementPolicyNameLabel], Namespace: testNamespace}, policy)).Should(Succeed()) + g.Expect(policy.Status.ActiveClusterClaims).NotTo(BeNil()) + g.Expect(*policy.Status.ActiveClusterClaims).Should(Equal(int32(1))) + }, eventuallyTimeout*2, pollInterval).Should(Succeed()) + }) + + It("holds the budget while a withdrawn claim of another selector lingers Terminating", func() { + // Two unfulfilled selectors; the budget admits only selector 0's claim. When selector 0 + // is fulfilled, its claim is withdrawn -- but a provisioner finalizer keeps it + // Terminating, and selector 1's claim, which has a different name, must wait for that + // slot rather than stand beside it. + policy := newPolicy(nextName("pp"), + regionSelector("swedencentral", ptr.To(intstr.FromInt32(1)), nil), + regionSelector("italynorth", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + claim0Name := claimName(policy, 0) + claim1Name := claimName(policy, 1) + claim0 := &kfplacementv1alpha1.ClusterClaim{} + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: claim0Name}, claim0) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("a provisioner protects the first claim with a finalizer") + claim0.Finalizers = append(claim0.Finalizers, "test.kubefleet.dev/provisioner") + Expect(k8sClient.Update(ctx, claim0)).Should(Succeed()) + + By("fulfilling selector 0 withdraws its claim into Terminating") + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "swedencentral"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: claim0Name}, fetched)).Should(Succeed()) + g.Expect(fetched.DeletionTimestamp.IsZero()).Should(BeFalse(), "the fulfilled selector's claim should be Terminating") + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the second selector's claim waits for the slot rather than exceeding the budget") + Consistently(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: claim1Name}, &kfplacementv1alpha1.ClusterClaim{}) + }, consistentlyDuration, pollInterval).ShouldNot(Succeed(), "the budget must hold while the withdrawn claim lingers") + + By("releasing the finalizer frees the slot for the second selector's claim") + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: claim0Name}, fetched)).Should(Succeed()) + fetched.Finalizers = nil + g.Expect(k8sClient.Update(ctx, fetched)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: claim1Name}, &kfplacementv1alpha1.ClusterClaim{}) + }, eventuallyTimeout*2, pollInterval).Should(Succeed()) + }) + + It("does not release the cleanup finalizer over a claim whose labels were stripped", func() { + // The ownership labels are mutable; the claim's back-reference is not. Cleanup must go + // by the reference, or a label-stripped claim would vanish from the cleanup list and be + // orphaned the moment the policy's finalizer is released. + policy := newPolicy(nextName("pp"), regionSelector("qatarcentral", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + claim := &kfplacementv1alpha1.ClusterClaim{} + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the claim's ownership labels are stripped and a provisioner finalizer holds it") + claim.Labels = nil + claim.Finalizers = append(claim.Finalizers, "test.kubefleet.dev/provisioner") + Expect(k8sClient.Update(ctx, claim)).Should(Succeed()) + + By("deleting the policy withdraws the claim by its reference and keeps the finalizer") + Expect(k8sClient.Delete(ctx, policy)).Should(Succeed()) + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + g.Expect(fetched.DeletionTimestamp.IsZero()).Should(BeFalse(), "cleanup must find the label-stripped claim through its reference") + }, eventuallyTimeout, pollInterval).Should(Succeed()) + Consistently(func() error { + return k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), &kfplacementv1alpha1.PlacementPolicy{}) + }, consistentlyDuration, pollInterval).Should(Succeed(), "the policy must outlive its lingering claim") + + By("once the claim is released, the policy goes too") + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + fetched.Finalizers = nil + g.Expect(k8sClient.Update(ctx, fetched)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + Eventually(func() error { + return k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), &kfplacementv1alpha1.PlacementPolicy{}) + }, eventuallyTimeout*2, pollInterval).ShouldNot(Succeed(), "the policy should be gone once its claim is") + }) + + It("leaves a claim of another policy alone when it holds the claim's name", func() { + policy := newPolicy(nextName("pp"), regionSelector("norwayeast", ptr.To(intstr.FromInt32(1)), nil)) + wantClaimName := claimName(policy, 0) + + By("a claim of another policy already holds the deterministic name") + squatterRef := policyReference(newPolicy("squatter")) + squatter := &kfplacementv1alpha1.ClusterClaim{ + ObjectMeta: metav1.ObjectMeta{Name: wantClaimName}, + Spec: kfplacementv1alpha1.ClusterClaimSpec{PlacementPolicyRef: squatterRef}, + } + Expect(k8sClient.Create(ctx, squatter)).Should(Succeed()) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + By("the policy neither adopts, rewrites, nor withdraws it, and counts no claim") + Consistently(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + g.Expect(fetched.DeletionTimestamp.IsZero()).Should(BeTrue(), "the other policy's claim must not be withdrawn") + g.Expect(fetched.Spec.PlacementPolicyRef).Should(Equal(squatterRef)) + g.Expect(fetched.Labels).Should(BeEmpty(), "the other policy's claim must not be relabeled") + pp := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), pp)).Should(Succeed()) + g.Expect(pp.Status.ActiveClusterClaims).Should(BeNil(), "a round that could not issue its claim publishes no count") + }, consistentlyDuration, pollInterval).Should(Succeed()) + + By("once the name frees up, the policy issues its own claim") + Expect(k8sClient.Delete(ctx, squatter)).Should(Succeed()) + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, fetched)).Should(Succeed()) + g.Expect(claimBelongsTo(fetched, policyReference(policy))).Should(BeTrue()) + }, eventuallyTimeout*2, pollInterval).Should(Succeed()) + }) + + It("replaces the outstanding claim when the selector terms change", func() { + policy := newPolicy(nextName("pp"), regionSelector("spaincentral", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + wantClaimName := claimName(policy, 0) + oldTerms := policy.Spec.ClusterSelectors[0].Terms + Eventually(func() error { + return k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, &kfplacementv1alpha1.ClusterClaim{}) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the selector is retargeted to another region") + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), fetched)).Should(Succeed()) + fetched.Spec.ClusterSelectors[0].Terms = []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{testRegionLabel: "mexicocentral"}}, + } + g.Expect(k8sClient.Update(ctx, fetched)).Should(Succeed()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + By("the claim is re-issued with the new terms under the same deterministic name") + Eventually(func(g Gomega) { + claim := &kfplacementv1alpha1.ClusterClaim{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: wantClaimName}, claim)).Should(Succeed()) + g.Expect(claim.Spec.ClusterSelectorTerms).Should(HaveLen(1)) + g.Expect(claim.Spec.ClusterSelectorTerms[0].MatchLabels).Should(HaveKeyWithValue(testRegionLabel, "mexicocentral")) + g.Expect(apiequality.Semantic.DeepEqual(claim.Spec.ClusterSelectorTerms, oldTerms)).Should(BeFalse()) + }, eventuallyTimeout*2, pollInterval).Should(Succeed()) + }) +}) diff --git a/pkg/v1/controllers/placementpolicy/claims_test.go b/pkg/v1/controllers/placementpolicy/claims_test.go new file mode 100644 index 000000000..ee25bf4a4 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/claims_test.go @@ -0,0 +1,470 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" +) + +func policyFor(name, namespace string) kfplacementv1alpha1.PlacementPolicyAccessor { + if namespace == "" { + return &kfplacementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: name}, + } + } + return &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace}, + } +} + +func TestClaimName(t *testing.T) { + policyA := policyFor("app", "tenant-a") + policyB := policyFor("app", "tenant-b") + clusterScoped := policyFor("app", "") + + nameA := claimName(policyA, 0) + nameB := claimName(policyB, 0) + nameCluster := claimName(clusterScoped, 0) + + if nameA == nameB { + t.Errorf("claimName(tenant-a/app, 0) = claimName(tenant-b/app, 0) = %s, want distinct names", nameA) + } + if nameA == nameCluster { + t.Errorf("claimName(tenant-a/app, 0) = claimName(cluster-scoped app, 0) = %s, want distinct names", nameA) + } + if got := claimName(policyA, 0); got != nameA { + t.Errorf("claimName(tenant-a/app, 0) = %s on the second call, want the deterministic %s", got, nameA) + } + if nameIdx1 := claimName(policyA, 1); nameIdx1 == nameA { + t.Errorf("claimName(tenant-a/app, 1) = claimName(tenant-a/app, 0) = %s, want distinct names per selector", nameA) + } + + longPolicy := policyFor(strings.Repeat("x", 250), "tenant-a") + if got := claimName(longPolicy, 0); len(got) > 253 { + t.Errorf("claimName(long policy, 0) has length %d, want at most 253", len(got)) + } +} + +// TestClaimNameValidity guards the generated claim name against the 253-character object name +// limit and DNS-1123 subdomain rules, including names whose truncation point lands on a +// separator. +func TestClaimNameValidity(t *testing.T) { + testCases := []struct { + name string + policyName string + }{ + { + name: "short name", + policyName: "app", + }, + { + name: "long name", + policyName: strings.Repeat("x", 250), + }, + { + name: "dot at the truncation boundary", + policyName: strings.Repeat("a", claimNameBaseMaxLength-1) + "." + strings.Repeat("a", 50), + }, + { + name: "dash at the truncation boundary", + policyName: strings.Repeat("a", claimNameBaseMaxLength-1) + "-" + strings.Repeat("a", 50), + }, + { + name: "run of separators at the truncation boundary", + policyName: strings.Repeat("a", claimNameBaseMaxLength-3) + "-.-" + strings.Repeat("a", 50), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := claimName(policyFor(tc.policyName, "tenant-a"), 0) + if errs := validation.IsDNS1123Subdomain(got); len(errs) > 0 { + t.Errorf("claimName(%q, 0) = %q, want a valid DNS-1123 subdomain, got errors %v", tc.policyName, got, errs) + } + }) + } +} + +func TestDesiredClaims(t *testing.T) { + policy := policyFor("app", "tenant-a") + regionTerms := func(region string) []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm { + return []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabel: region}}, + } + } + + testCases := []struct { + name string + outcomes []selectorOutcome + want []desiredClaim + }{ + { + name: "one claim for the first unfulfilled selector", + outcomes: []selectorOutcome{ + { + counts: resolvedCounts{desired: 1, minimum: 1}, + terms: regionTerms("eastus"), + whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + }, + want: []desiredClaim{{name: claimName(policyFor("app", "tenant-a"), 0), terms: regionTerms("eastus")}}, + }, + { + name: "budget caps at one claim across multiple unfulfilled selectors", + outcomes: []selectorOutcome{ + { + counts: resolvedCounts{desired: 1, minimum: 1}, + terms: regionTerms("eastus"), + whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + { + counts: resolvedCounts{desired: 1, minimum: 1}, + terms: regionTerms("westus"), + whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + }, + want: []desiredClaim{{name: claimName(policyFor("app", "tenant-a"), 0), terms: regionTerms("eastus")}}, + }, + { + name: "fulfilled selectors and KeepSearching selectors yield no claims", + outcomes: []selectorOutcome{ + { + counts: resolvedCounts{desired: 1, minimum: 1}, + matched: []string{"a"}, + terms: regionTerms("eastus"), + whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + { + counts: resolvedCounts{desired: 1, minimum: 1}, + terms: regionTerms("westus"), + whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionKeepSearching, + }, + }, + want: []desiredClaim{}, + }, + { + name: "a fulfilled earlier selector passes the budget to a later one", + outcomes: []selectorOutcome{ + { + counts: resolvedCounts{desired: 1, minimum: 1}, + matched: []string{"a"}, + terms: regionTerms("eastus"), + whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + { + counts: resolvedCounts{desired: 2, minimum: 2}, + matched: []string{"a"}, + terms: regionTerms("westus"), + whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + }, + want: []desiredClaim{{name: claimName(policyFor("app", "tenant-a"), 1), terms: regionTerms("westus")}}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := desiredClaims(policy, tc.outcomes) + // The outcome field is a pointer back into tc.outcomes that carries no identity of its + // own to assert here; the name/terms selection is what this test pins. Its wiring is + // exercised by TestClaimReadyToRotate and the integration tests. + if diff := cmp.Diff(got, tc.want, cmp.AllowUnexported(desiredClaim{}), cmpopts.IgnoreFields(desiredClaim{}, "outcome")); diff != "" { + t.Errorf("desiredClaims(%v) mismatch (-got, +want):\n%s", tc.outcomes, diff) + } + }) + } +} + +func TestClaimReadyToRotate(t *testing.T) { + completed := func() []metav1.Condition { + return []metav1.Condition{{Type: kfplacementv1alpha1.ClusterClaimCondTypeCompleted, Status: metav1.ConditionTrue, Reason: "Fulfilled"}} + } + claim := func(conds []metav1.Condition, provisioned *string) *kfplacementv1alpha1.ClusterClaim { + return &kfplacementv1alpha1.ClusterClaim{ + Status: kfplacementv1alpha1.ClusterClaimStatus{Conditions: conds, ProvisionedClusterName: provisioned}, + } + } + // A count-of-3 selector with one eligible cluster: a deficit remains. + deficit := &selectorOutcome{counts: resolvedCounts{desired: 3, minimum: 3}, matched: []string{"c1"}} + + testCases := []struct { + name string + claim *kfplacementv1alpha1.ClusterClaim + outcome *selectorOutcome + want bool + }{ + { + name: "completed with an eligible cluster while a deficit remains rotates", + claim: claim(completed(), ptr.To("c1")), + outcome: deficit, + want: true, + }, + { + name: "not yet completed does not rotate", + claim: claim(nil, nil), + outcome: deficit, + want: false, + }, + { + // The provisioner completed but its cluster has not joined/become eligible yet: waiting + // here is what keeps a second claim from being issued mid-join (no double-provisioning). + name: "completed but the provisioned cluster is not eligible yet does not rotate", + claim: claim(completed(), ptr.To("c2")), + outcome: deficit, + want: false, + }, + { + name: "completed with no provisioned cluster recorded does not rotate", + claim: claim(completed(), nil), + outcome: deficit, + want: false, + }, + { + // The selector is already at its desired count; the claim is withdrawn (not rotated) by + // the fulfilled path, so rotation must not fire. + name: "satisfied in full does not rotate", + claim: claim(completed(), ptr.To("c1")), + outcome: &selectorOutcome{counts: resolvedCounts{desired: 1, minimum: 1}, matched: []string{"c1"}}, + want: false, + }, + { + // count: All with a minCount floor is satisfied in full at the floor, so a completed + // claim below the floor still has a deficit and rotates, mirroring the integer case. + name: "select-all below its minCount floor rotates", + claim: claim(completed(), ptr.To("c1")), + outcome: &selectorOutcome{counts: resolvedCounts{selectAll: true, minimum: 2}, matched: []string{"c1"}}, + want: true, + }, + { + name: "a nil outcome does not rotate", + claim: claim(completed(), ptr.To("c1")), + outcome: nil, + want: false, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := claimReadyToRotate(tc.claim, tc.outcome); got != tc.want { + t.Errorf("claimReadyToRotate() = %v, want %v", got, tc.want) + } + }) + } +} + +// TestClaimBelongsTo checks that ownership survives a version change but not a group change. +func TestClaimBelongsTo(t *testing.T) { + ref := &kfplacementv1alpha1.ObjectReference{ + APIGroup: kfplacementv1alpha1.GroupVersion.Group, APIVersion: "v1alpha1", + Kind: kfplacementv1alpha1.PlacementPolicyKind, Namespace: "work", Name: "app", + } + withRef := func(mutate func(r *kfplacementv1alpha1.ObjectReference)) *kfplacementv1alpha1.ClusterClaim { + got := ref.DeepCopy() + mutate(got) + return &kfplacementv1alpha1.ClusterClaim{Spec: kfplacementv1alpha1.ClusterClaimSpec{PlacementPolicyRef: got}} + } + testCases := []struct { + name string + claim *kfplacementv1alpha1.ClusterClaim + want bool + }{ + {name: "same reference", claim: withRef(func(*kfplacementv1alpha1.ObjectReference) {}), want: true}, + {name: "another version", claim: withRef(func(r *kfplacementv1alpha1.ObjectReference) { r.APIVersion = "v1" }), want: true}, + {name: "another group", claim: withRef(func(r *kfplacementv1alpha1.ObjectReference) { r.APIGroup = "other.example.dev" }), want: false}, + {name: "another name", claim: withRef(func(r *kfplacementv1alpha1.ObjectReference) { r.Name = "other" }), want: false}, + {name: "no reference", claim: &kfplacementv1alpha1.ClusterClaim{}, want: false}, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := claimBelongsTo(tc.claim, ref); got != tc.want { + t.Errorf("claimBelongsTo() = %t, want %t", got, tc.want) + } + }) + } +} + +// TestReconcileClaimsNameOccupied pins how a create that finds its deterministic name taken is +// handled: a claim of the policy the cache has yet to show holds a budget slot, while a claim of +// another policy under the name is reported instead of leaving the selector silently unfulfilled. +func TestReconcileClaimsNameOccupied(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + terms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{MatchLabels: map[string]string{regionLabel: "eastus"}}} + outcomes := []selectorOutcome{{ + counts: resolvedCounts{desired: 1, minimum: 1}, + terms: terms, + whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }} + + testCases := []struct { + name string + occupantRef *kfplacementv1alpha1.ObjectReference + wantOutstanding int32 + wantErr bool + }{ + { + name: "a claim of the policy that the cache has yet to show", + occupantRef: policyReference(policyFor("pp", "default")), + wantOutstanding: 1, + }, + { + name: "a claim of another policy", + occupantRef: policyReference(policyFor("other", "default")), + wantErr: true, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + policy := &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: "pp", Namespace: "default"}} + occupant := &kfplacementv1alpha1.ClusterClaim{ + ObjectMeta: metav1.ObjectMeta{Name: claimName(policy, 0)}, + Spec: kfplacementv1alpha1.ClusterClaimSpec{PlacementPolicyRef: tc.occupantRef, ClusterSelectorTerms: terms}, + } + // The API server holds the occupant; the cache has yet to see it, so both its list and + // a cached read come back without it. + apiServer := fake.NewClientBuilder().WithScheme(scheme).WithObjects(policy, occupant).Build() + cache := interceptor.NewClient(apiServer, interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, isClaims := list.(*kfplacementv1alpha1.ClusterClaimList); isClaims { + return nil + } + return cl.List(ctx, list, opts...) + }, + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, isClaim := obj.(*kfplacementv1alpha1.ClusterClaim); isClaim { + return apierrors.NewNotFound(kfplacementv1alpha1.GroupVersion.WithResource("clusterclaims").GroupResource(), key.Name) + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + + r := NewReconciler(cache, apiServer, snapshotStub{}) + outstanding, err := r.reconcileClaims(context.Background(), policy, outcomes, metav1.Time{}) + if (err != nil) != tc.wantErr { + t.Fatalf("reconcileClaims() = %v, want error %t", err, tc.wantErr) + } + if tc.wantErr { + if got := kferrors.Category(err); got != kferrors.ErrCategoryUser { + t.Errorf("reconcileClaims() error category = %q, want %q", got, kferrors.ErrCategoryUser) + } + } + if outstanding != tc.wantOutstanding { + t.Errorf("reconcileClaims() = %d outstanding, want %d", outstanding, tc.wantOutstanding) + } + + got := &kfplacementv1alpha1.ClusterClaim{} + if err := apiServer.Get(context.Background(), client.ObjectKeyFromObject(occupant), got); err != nil { + t.Fatalf("Get(occupant) = %v, want the occupant left in place", err) + } + if diff := cmp.Diff(tc.occupantRef, got.Spec.PlacementPolicyRef); diff != "" { + t.Errorf("occupant placementPolicyRef mismatch (-want +got):\n%s", diff) + } + }) + } +} + +// TestReconcileClaimsCountsClaimsTheCacheHasYetToShow pins the one-claim budget under cache lag: a +// claim issued last round for selector 0, which the cache has yet to show, still holds the budget +// when selector 1 is the next one to want a claim, so no second claim is issued beside it. +func TestReconcileClaimsCountsClaimsTheCacheHasYetToShow(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + policy := &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: "pp", Namespace: "default"}} + lagging := &kfplacementv1alpha1.ClusterClaim{ + ObjectMeta: metav1.ObjectMeta{Name: claimName(policy, 0), UID: "lagging-uid"}, + Spec: kfplacementv1alpha1.ClusterClaimSpec{ + PlacementPolicyRef: policyReference(policy), + ClusterSelectorTerms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{MatchLabels: map[string]string{regionLabel: "eastus"}}}, + }, + } + apiServer := fake.NewClientBuilder().WithScheme(scheme).WithObjects(policy, lagging).Build() + cache := interceptor.NewClient(apiServer, interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, isClaims := list.(*kfplacementv1alpha1.ClusterClaimList); isClaims { + return nil + } + return cl.List(ctx, list, opts...) + }, + }) + // Selector 0 has been satisfied since; selector 1 now wants a claim. + outcomes := []selectorOutcome{ + {counts: resolvedCounts{desired: 1, minimum: 1}, matched: []string{"east-1"}, terms: lagging.Spec.ClusterSelectorTerms, whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim}, + {counts: resolvedCounts{desired: 1, minimum: 1}, terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{MatchLabels: map[string]string{regionLabel: "westus"}}}, whenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim}, + } + + r := NewReconciler(cache, apiServer, snapshotStub{}) + outstanding, err := r.reconcileClaims(context.Background(), policy, outcomes, metav1.Time{}) + if err != nil { + t.Fatalf("reconcileClaims() = %v, want no error", err) + } + if outstanding != 1 { + t.Errorf("reconcileClaims() = %d outstanding, want the 1 claim the cache has yet to show", outstanding) + } + if err := apiServer.Get(context.Background(), client.ObjectKey{Name: claimName(policy, 1)}, &kfplacementv1alpha1.ClusterClaim{}); !apierrors.IsNotFound(err) { + t.Errorf("Get(selector 1's claim) = %v, want NotFound: a second claim would exceed the budget of one", err) + } +} + +// TestCountUnseenClaims pins that claims are told apart by UID: a claim the cache shows is not +// counted again, while a successor under a name the cache still shows an old claim for is. +func TestCountUnseenClaims(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + policy := &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: "pp", Namespace: "default"}} + claim := func(name, uid string, ref *kfplacementv1alpha1.ObjectReference) *kfplacementv1alpha1.ClusterClaim { + return &kfplacementv1alpha1.ClusterClaim{ObjectMeta: metav1.ObjectMeta{Name: name, UID: types.UID(uid)}, Spec: kfplacementv1alpha1.ClusterClaimSpec{PlacementPolicyRef: ref}} + } + own := policyReference(policy) + seen := claim("pp-0", "seen", own) + unseen := claim("pp-1", "unseen", own) + foreign := claim("other-0", "foreign", policyReference(policyFor("other", "default"))) + apiServer := fake.NewClientBuilder().WithScheme(scheme).WithObjects(seen, unseen, foreign).Build() + r := NewReconciler(apiServer, apiServer, snapshotStub{}) + + // The cache shows the seen claim and a predecessor of the unseen one under its name. + cached := []kfplacementv1alpha1.ClusterClaim{*seen, *claim("pp-1", "predecessor", own)} + got, err := r.countUnseenClaims(context.Background(), policy, cached) + if err != nil { + t.Fatalf("countUnseenClaims() = %v, want no error", err) + } + if got != 1 { + t.Errorf("countUnseenClaims() = %d, want 1: only the successor the cache has yet to show", got) + } +} diff --git a/pkg/v1/controllers/placementpolicy/controller.go b/pkg/v1/controllers/placementpolicy/controller.go new file mode 100644 index 000000000..b41b9c735 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/controller.go @@ -0,0 +1,370 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package placementpolicy implements the FEP-0001 placement policy controller, which reconciles +// the PlacementPolicy and ClusterPlacementPolicy API objects (placement.kubefleet.dev API group): +// it resolves cluster selectors against the current member cluster inventory and reports +// scheduling status on the policy objects. +package placementpolicy + +import ( + "context" + "time" + + apiequality "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + runtime "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/handler" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/scheduler/clustereligibilitychecker" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/bindingmanager" + "go.goms.io/fleet/pkg/v1/utils/sourcequeue" +) + +const ( + // unfulfilledRequeueAfter is the wait before re-evaluating a policy that has not reached + // all of its desired cluster counts; it backstops cluster eligibility transitions that do + // not surface as watch events (e.g., a member agent heartbeat going stale). + unfulfilledRequeueAfter = 2 * time.Minute + // fulfilledRequeueAfter is the wait before re-evaluating a fulfilled policy, for the same + // backstop reason: a cluster that silently stops heartbeating emits no event, yet it must + // eventually stop counting toward fulfillment. Set to half the eligibility checker's + // 5-minute heartbeat timeout so the worst-case detection latency stays near the timeout + // itself (timeout + one requeue period, ~7.5 minutes) instead of doubling it. + fulfilledRequeueAfter = 150 * time.Second + // bindingManagerRetryBase and bindingManagerRetryMax bound the exponential backoff before + // retrying a binding change that found the policy's binding manager role held by another + // process (e.g., a rollout in progress). The FEP has contenders back off and retry rather + // than wait on the role; the backoff keeps a long rollout from being polled at a fixed pace. + bindingManagerRetryBase = 5 * time.Second + bindingManagerRetryMax = 5 * time.Minute +) + +// Reconciler reconciles PlacementPolicy and ClusterPlacementPolicy objects. +// +// One Reconciler instance serves both APIs: requests for cluster-scoped ClusterPlacementPolicy +// objects carry no namespace, which is how the two are told apart. +type Reconciler struct { + client.Client + + // uncachedReader reads directly from the API server; it gates the claim cleanup + // finalizer release, where a stale cache read could orphan a just-created claim. + uncachedReader client.Reader + + eligibility eligibilityChecker + + // snapshots provides the resource snapshot a new binding rolls out. + snapshots snapshotter + + // roleBackoff paces the retries of a policy whose binding manager role another process + // holds; a round that completes resets the policy's backoff. + roleBackoff workqueue.TypedRateLimiter[runtime.Request] + + // policySource and clusterPolicySource carry the re-queues EnqueuePoliciesSelecting makes when + // a selected resource changes, one per controller, since each controller has its own queue. + policySource sourcequeue.Queue[runtime.Request] + clusterPolicySource sourcequeue.Queue[runtime.Request] +} + +// NewReconciler returns a Reconciler that judges cluster fulfillment with the scheduler's +// standard cluster eligibility gate and takes resource snapshots through the given manager. +func NewReconciler(c client.Client, uncachedReader client.Reader, snapshots snapshotter) *Reconciler { + return &Reconciler{ + Client: c, + uncachedReader: uncachedReader, + eligibility: clustereligibilitychecker.New(), + snapshots: snapshots, + roleBackoff: workqueue.NewTypedItemExponentialFailureRateLimiter[runtime.Request](bindingManagerRetryBase, bindingManagerRetryMax), + } +} + +// Reconcile runs a single reconciliation round for a PlacementPolicy or ClusterPlacementPolicy object. +func (r *Reconciler) Reconcile(ctx context.Context, req runtime.Request) (runtime.Result, error) { + startTime := time.Now() + klog.V(2).InfoS("Placement policy reconciliation starts", "placementPolicy", req.NamespacedName) + defer func() { + latency := time.Since(startTime).Milliseconds() + klog.V(2).InfoS("Placement policy reconciliation ends", "placementPolicy", req.NamespacedName, "latency", latency) + }() + + policy, err := r.fetchPolicy(ctx, req) + if err != nil { + if errors.IsNotFound(err) { + klog.V(4).InfoS("Ignoring not-found placement policy", "placementPolicy", req.NamespacedName) + // The policy is gone for good; stop reporting metrics for it, and drop its role + // backoff so that a policy recreated under the name starts afresh. + forgetPolicyMetrics(req.Namespace, req.Name) + r.roleBackoff.Forget(req) + return runtime.Result{}, nil + } + klog.ErrorS(err, "Failed to get the placement policy object", append(kferrors.Args(err), "placementPolicy", req.NamespacedName)...) + return runtime.Result{}, err + } + if !policy.GetDeletionTimestamp().IsZero() { + // Cluster claims cannot be owner-referenced by the policy (cross-scope), so cleanup + // is finalizer-driven. + return runtime.Result{}, r.cleanupClaims(ctx, policy) + } + + memberClusters := &clusterv1beta1.MemberClusterList{} + if err := r.List(ctx, memberClusters); err != nil { + wrappedErr := kferrors.NewAPIServerError(err, "failed to list member clusters", true, "placementPolicy", req.NamespacedName) + klog.ErrorS(wrappedErr, "Failed to list member clusters", kferrors.Args(wrappedErr)...) + return runtime.Result{}, wrappedErr + } + var mostRecentClusterCreation metav1.Time + for i := range memberClusters.Items { + if ts := memberClusters.Items[i].CreationTimestamp; ts.After(mostRecentClusterCreation.Time) { + mostRecentClusterCreation = ts + } + } + + existingBindings, err := r.listBindings(ctx, policy) + if err != nil { + klog.ErrorS(err, "Failed to list the placement bindings", append(kferrors.Args(err), "placementPolicy", req.NamespacedName)...) + return runtime.Result{}, err + } + bound := make(map[string]metav1.Time, len(existingBindings)) + recorded := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(existingBindings)) + for _, binding := range existingBindings { + bound[binding.GetSpec().ClusterName] = binding.GetCreationTimestamp() + recorded[binding.GetSpec().ClusterName] = binding.GetSpec().ClusterSelectors + } + + outcomes, err := evaluateSelectors(policy.GetSpec(), memberClusters.Items, r.eligibility, recorded) + if err != nil { + // Selector evaluation fails only on invalid selector contents (e.g., an operator + // applied to a key it does not support); retrying cannot help until the spec changes, + // so the error surfaces on the status instead of the reconcile loop. Outstanding + // claims are deliberately left untouched: their specs reflect the last valid + // selectors. + klog.ErrorS(err, "Failed to evaluate the cluster selectors", "placementPolicy", req.NamespacedName) + // A role a failed binding write left held is kept for the retry to resume under; with the + // selectors invalid no binding change can follow until the spec is fixed, so the role is + // released rather than left blocking the policy's rollouts in the meantime. + if holdsBindingManagerRole(policy) { + if err := bindingmanager.RelinquishRoleFor(ctx, r.Client, policy, controllerName, *policyReference(policy)); err != nil { + return runtime.Result{}, err + } + } + return runtime.Result{}, r.updateStatus(ctx, policy, nil, nil, invalidSelectorsCondition(policy.GetGeneration(), err), nil, nil) + } + + clustersByName := make(map[string]*clusterv1beta1.MemberCluster, len(memberClusters.Items)) + for i := range memberClusters.Items { + clustersByName[memberClusters.Items[i].Name] = &memberClusters.Items[i] + } + desiredBindings := chooseClusters(outcomes, bound, clustersByName) + + // The latest resource snapshot is what new bindings roll out and what the status reports the + // target clusters against. Per the FEP the first one is taken when the policy is created; + // afterwards a resource change alone takes none, it only shows as the snapshot going stale. + // + // ponytail: every round re-hashes the selected resources from the informer cache and, when + // they still match, confirms with one uncached metadata read; cheap per policy, but it runs on + // every member cluster event fanned out to every policy. Cache the verdict per resource + // version if that shows up in profiles. + // + // Scheduling changes apply independently of resource changes, per the FEP, so a round that + // cannot get the snapshot still unbinds, updates, and refreshes claims and the Scheduled + // status; it only holds back new bindings. A transient failure leaves the resource status as it + // was, but one that the retries cannot fix on their own, a selected resource that is gone or a + // selection the user got wrong, reports the target clusters as not synchronized: the last + // round's verdict no longer describes the selected resources. Their availability depends only + // on the bindings, so it is still reported. + var latestRevision string + var resources *resourceStatus + snapshots, current, snapshotErr := r.snapshots.SnapshotResourcesIfNoSnapshotExists(ctx, policy) + if snapshotErr == nil && len(snapshots) == 0 { + snapshotErr = kferrors.NewUnexpectedError(nil, "the snapshot manager returned no placement resource snapshot", "placementPolicy", klog.KObj(policy)) + } + var selectionErr error + if kferrors.Category(snapshotErr) == kferrors.ErrCategoryUser || errors.IsNotFound(snapshotErr) { + selectionErr = snapshotErr + } + if snapshotErr == nil { + latestRevision = snapshots[0].GetName() + // The summary reads the bindings as listed, before reconcileBindings updates them, which is why + // it skips a binding whose sync strategy is still the old one: its conditions hold for its + // current generation but describe the old strategy. + summary := summarizeResourceStatus(latestRevision, current, policy.GetSpec().SyncStrategy, desiredBindings, existingBindings) + resources = &summary + } else if selectionErr != nil { + summary := summarizeAvailability(policy.GetSpec().SyncStrategy, desiredBindings, existingBindings) + resources = &summary + } + roleHeld, bindingErr := r.reconcileBindings(ctx, policy, outcomes, desiredBindings, existingBindings, latestRevision) + + activeClaims, claimErr := r.reconcileClaims(ctx, policy, outcomes, mostRecentClusterCreation) + // The scheduling status is written even when claim reconciliation failed partway, but the + // claim count is published only from a completed round: a failed round returns whatever it + // had counted when it stopped, which would misreport the claims that the rest of the round + // never reached. A nil count leaves the last completed round's value standing; the retry + // corrects it. + reportedClaims := &activeClaims + if claimErr != nil { + reportedClaims = nil + } + if err := r.updateStatus(ctx, policy, outcomes, reportedClaims, scheduledCondition(policy.GetGeneration(), outcomes), resources, selectionErr); err != nil { + return runtime.Result{}, err + } + if bindingErr != nil { + klog.ErrorS(bindingErr, "Failed to reconcile the placement bindings", append(kferrors.Args(bindingErr), "placementPolicy", req.NamespacedName)...) + } + if snapshotErr != nil { + klog.ErrorS(snapshotErr, "Failed to take the latest resource snapshot", append(kferrors.Args(snapshotErr), "placementPolicy", req.NamespacedName)...) + return runtime.Result{}, snapshotErr + } + if bindingErr != nil { + return runtime.Result{}, bindingErr + } + if claimErr != nil { + klog.ErrorS(claimErr, "Failed to reconcile the cluster claims", append(kferrors.Args(claimErr), "placementPolicy", req.NamespacedName)...) + return runtime.Result{}, claimErr + } + if roleHeld { + klog.V(2).InfoS("The binding manager role is held by another process; will retry the binding changes", "placementPolicy", req.NamespacedName) + return runtime.Result{RequeueAfter: r.roleBackoff.When(req)}, nil + } + r.roleBackoff.Forget(req) + + requeueAfter := fulfilledRequeueAfter + for i := range outcomes { + if !outcomes[i].satisfiedInFull() { + requeueAfter = unfulfilledRequeueAfter + break + } + } + return runtime.Result{RequeueAfter: requeueAfter}, nil +} + +// updateStatus writes the scheduling outcome onto the policy status, skipping the API call when +// nothing has changed. A nil outcome list clears the cluster counts (used when the selectors +// cannot be evaluated at all); a nil activeClaims leaves the current claim count untouched, and a +// nil resources leaves the resource revision, the synchronization and availability counts, and +// their conditions untouched. A non-nil selectionErr reports the ResourceCollected and +// Synchronized conditions false and leaves the resource revision as it was, while the counts and +// the Available condition still come from resources. +func (r *Reconciler) updateStatus(ctx context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor, outcomes []selectorOutcome, activeClaims *int32, scheduledCond metav1.Condition, resources *resourceStatus, selectionErr error) error { + status := policy.GetStatus() + observedStatus := status.DeepCopy() + + if outcomes == nil { + status.DesiredClusters = nil + status.ScheduledClusters = nil + } else { + desired, scheduled := aggregateCounts(outcomes) + status.DesiredClusters = &desired + status.ScheduledClusters = &scheduled + } + if activeClaims != nil { + status.ActiveClusterClaims = activeClaims + } + meta.SetStatusCondition(&status.Conditions, scheduledCond) + if resources != nil { + if selectionErr == nil { + status.LatestResourceRevisionName = &resources.latestRevision + meta.SetStatusCondition(&status.Conditions, resourceCollectedCondition(policy.GetGeneration(), nil)) + meta.SetStatusCondition(&status.Conditions, resources.synchronizedCondition(policy.GetGeneration())) + } + status.SynchronizedClusters = &resources.synchronized + status.ResourcesAvailableClusters = &resources.available + meta.SetStatusCondition(&status.Conditions, resources.availableCondition(policy.GetGeneration())) + } + if selectionErr != nil { + meta.SetStatusCondition(&status.Conditions, resourceCollectedCondition(policy.GetGeneration(), selectionErr)) + meta.SetStatusCondition(&status.Conditions, selectionFailedCondition(policy.GetGeneration(), selectionErr)) + } + + reportPolicyMetrics(policy, status, scheduledCond) + + if apiequality.Semantic.DeepEqual(observedStatus, status) { + return nil + } + if err := r.Status().Update(ctx, policy); err != nil { + wrappedErr := kferrors.NewAPIServerError(err, "failed to update the placement policy status", false, "placementPolicy", client.ObjectKeyFromObject(policy)) + klog.ErrorS(wrappedErr, "Failed to update the placement policy status", kferrors.Args(wrappedErr)...) + return wrappedErr + } + return nil +} + +// fetchPolicy retrieves the policy object for the given request; requests without a namespace +// concern the cluster-scoped ClusterPlacementPolicy API. +// +// The read is categorized as an API server error, which stays recognizable to errors.IsNotFound: +// it resolves through the wrap, so the caller can still tell a deleted policy from a failed read. +func (r *Reconciler) fetchPolicy(ctx context.Context, req runtime.Request) (kfplacementv1alpha1.PlacementPolicyAccessor, error) { + var policy kfplacementv1alpha1.PlacementPolicyAccessor = &kfplacementv1alpha1.PlacementPolicy{} + if req.Namespace == "" { + policy = &kfplacementv1alpha1.ClusterPlacementPolicy{} + } + if err := r.Get(ctx, req.NamespacedName, policy); err != nil { + return nil, kferrors.NewAPIServerError(err, "failed to get the placement policy", true, "placementPolicy", req.NamespacedName) + } + return policy, nil +} + +// SetupWithManagerForPlacementPolicy registers the reconciler with the manager for the +// namespaced PlacementPolicy API. The reconciler lists cluster claims through a cache field index, +// so fieldindexers.SetupWithHubAgentControllerManager must have run before the manager starts. +func (r *Reconciler) SetupWithManagerForPlacementPolicy(mgr runtime.Manager) error { + return runtime.NewControllerManagedBy(mgr). + Named(controllerName). + For(&kfplacementv1alpha1.PlacementPolicy{}). + Owns(&kfplacementv1alpha1.PlacementBinding{}, builder.WithPredicates(bindingRelevantChanges())). + Watches( + &clusterv1beta1.MemberCluster{}, + handler.EnqueueRequestsFromMapFunc(r.mapMemberClusterToPlacementPolicies), + builder.WithPredicates(memberClusterSchedulingRelevantChanges()), + ). + Watches( + &kfplacementv1alpha1.ClusterClaim{}, + handler.EnqueueRequestsFromMapFunc(r.mapClaimToPlacementPolicy), + ). + WatchesRawSource(&r.policySource). + Complete(r) +} + +// SetupWithManagerForClusterPlacementPolicy registers the reconciler with the manager for the +// cluster-scoped ClusterPlacementPolicy API. Like SetupWithManagerForPlacementPolicy, it relies on +// the cache field indexes of fieldindexers.SetupWithHubAgentControllerManager. +func (r *Reconciler) SetupWithManagerForClusterPlacementPolicy(mgr runtime.Manager) error { + return runtime.NewControllerManagedBy(mgr). + Named("cluster-placement-policy-controller"). + For(&kfplacementv1alpha1.ClusterPlacementPolicy{}). + Owns(&kfplacementv1alpha1.ClusterPlacementBinding{}, builder.WithPredicates(bindingRelevantChanges())). + Watches( + &clusterv1beta1.MemberCluster{}, + handler.EnqueueRequestsFromMapFunc(r.mapMemberClusterToClusterPlacementPolicies), + builder.WithPredicates(memberClusterSchedulingRelevantChanges()), + ). + Watches( + &kfplacementv1alpha1.ClusterClaim{}, + handler.EnqueueRequestsFromMapFunc(r.mapClaimToClusterPlacementPolicy), + ). + WatchesRawSource(&r.clusterPolicySource). + Complete(r) +} diff --git a/pkg/v1/controllers/placementpolicy/controller_integration_test.go b/pkg/v1/controllers/placementpolicy/controller_integration_test.go new file mode 100644 index 000000000..4212f4638 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/controller_integration_test.go @@ -0,0 +1,312 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "fmt" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + rbacv1 "k8s.io/api/rbac/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + eventuallyTimeout = time.Second * 15 + consistentlyDuration = time.Second * 2 + pollInterval = time.Millisecond * 250 + + testRegionLabel = "topology.kubernetes.io/region" +) + +// newMemberCluster builds a MemberCluster object; the caller decides whether to mark it joined. +func newMemberCluster(name string, labels map[string]string, taints ...clusterv1beta1.Taint) *clusterv1beta1.MemberCluster { + return &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{Name: name, Labels: labels}, + Spec: clusterv1beta1.MemberClusterSpec{ + Identity: rbacv1.Subject{ + Kind: "ServiceAccount", + Name: name, + Namespace: "fleet-system", + }, + Taints: taints, + }, + } +} + +// markJoined stamps a member agent status that passes the scheduler's eligibility gate: joined, +// healthy, and heartbeating. +func markJoined(mc *clusterv1beta1.MemberCluster) { + Expect(k8sClient.Get(ctx, types.NamespacedName{Name: mc.Name}, mc)).Should(Succeed()) + now := metav1.Now() + mc.Status.AgentStatus = []clusterv1beta1.AgentStatus{{ + Type: clusterv1beta1.MemberAgent, + Conditions: []metav1.Condition{ + {Type: string(clusterv1beta1.AgentJoined), Status: metav1.ConditionTrue, Reason: "AgentJoined", Message: "integration test: simulated join", LastTransitionTime: now}, + {Type: string(clusterv1beta1.AgentHealthy), Status: metav1.ConditionTrue, Reason: "AgentHealthy", Message: "integration test: simulated health", LastTransitionTime: now}, + }, + LastReceivedHeartbeat: now, + }} + Expect(k8sClient.Status().Update(ctx, mc)).Should(Succeed()) +} + +func newPolicy(name string, selectors ...kfplacementv1alpha1.ClusterSelector) *kfplacementv1alpha1.PlacementPolicy { + return &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: testNamespace}, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: selectors, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Name: "demo"}, + }, + }, + } +} + +func regionSelector(region string, count *intstr.IntOrString, minCount *int32) kfplacementv1alpha1.ClusterSelector { + return kfplacementv1alpha1.ClusterSelector{ + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{testRegionLabel: region}}, + }, + Count: count, + MinCount: minCount, + } +} + +// scheduledConditionOf fetches the policy and returns its Scheduled condition together with the +// desired/scheduled counts; nil pieces denote absence. +func scheduledConditionOf(key types.NamespacedName) func() (*metav1.Condition, *int32, *int32, error) { + return func() (*metav1.Condition, *int32, *int32, error) { + policy := &kfplacementv1alpha1.PlacementPolicy{} + if err := k8sClient.Get(ctx, key, policy); err != nil { + return nil, nil, nil, err + } + cond := meta.FindStatusCondition(policy.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeScheduled) + return cond, policy.Status.DesiredClusters, policy.Status.ScheduledClusters, nil + } +} + +func wantScheduledState(key types.NamespacedName, status metav1.ConditionStatus, reason string, desired, scheduled int32) func(Gomega) { + return func(g Gomega) { + cond, gotDesired, gotScheduled, err := scheduledConditionOf(key)() + g.Expect(err).Should(Succeed()) + g.Expect(cond).NotTo(BeNil(), "Scheduled condition is not set") + g.Expect(cond.Status).Should(Equal(status)) + g.Expect(cond.Reason).Should(Equal(reason)) + g.Expect(gotDesired).NotTo(BeNil(), "desiredClusters is not set") + g.Expect(*gotDesired).Should(Equal(desired)) + g.Expect(gotScheduled).NotTo(BeNil(), "scheduledClusters is not set") + g.Expect(*gotScheduled).Should(Equal(scheduled)) + } +} + +var _ = Describe("placement policy scheduling status", Ordered, func() { + var counter int + + nextName := func(prefix string) string { + counter++ + return fmt.Sprintf("%s-%d", prefix, counter) + } + + AfterEach(func() { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + Expect(k8sClient.List(ctx, policies, client.InNamespace(testNamespace))).Should(Succeed()) + for i := range policies.Items { + Expect(client.IgnoreNotFound(k8sClient.Delete(ctx, &policies.Items[i]))).Should(Succeed()) + } + clusterPolicies := &kfplacementv1alpha1.ClusterPlacementPolicyList{} + Expect(k8sClient.List(ctx, clusterPolicies)).Should(Succeed()) + for i := range clusterPolicies.Items { + Expect(client.IgnoreNotFound(k8sClient.Delete(ctx, &clusterPolicies.Items[i]))).Should(Succeed()) + } + // Policy deletion is finalizer-gated on claim cleanup (unfulfilled selectors in these + // specs spawn claims via the AddClusterClaim default); wait for policies and their + // claims to fully drain so no state leaks into other suites in this package. + Eventually(func(g Gomega) { + g.Expect(k8sClient.List(ctx, policies, client.InNamespace(testNamespace))).Should(Succeed()) + g.Expect(policies.Items).Should(BeEmpty()) + g.Expect(k8sClient.List(ctx, clusterPolicies)).Should(Succeed()) + g.Expect(clusterPolicies.Items).Should(BeEmpty()) + claims := &kfplacementv1alpha1.ClusterClaimList{} + g.Expect(k8sClient.List(ctx, claims)).Should(Succeed()) + g.Expect(claims.Items).Should(BeEmpty()) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + + memberClusters := &clusterv1beta1.MemberClusterList{} + Expect(k8sClient.List(ctx, memberClusters)).Should(Succeed()) + for i := range memberClusters.Items { + Expect(client.IgnoreNotFound(k8sClient.Delete(ctx, &memberClusters.Items[i]))).Should(Succeed()) + } + Eventually(func() (int, error) { + if err := k8sClient.List(ctx, memberClusters); err != nil { + return -1, err + } + return len(memberClusters.Items), nil + }, eventuallyTimeout, pollInterval).Should(Equal(0)) + }) + + It("reports unfulfilled scheduling when no clusters match", func() { + policy := newPolicy(nextName("pp"), regionSelector("eastus", ptr.To(intstr.FromInt32(2)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + Eventually(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 2, 0), + eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("fulfills scheduling once enough eligible clusters match", func() { + policy := newPolicy(nextName("pp"), regionSelector("eastus", ptr.To(intstr.FromInt32(2)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + for _, name := range []string{nextName("mc"), nextName("mc")} { + mc := newMemberCluster(name, map[string]string{testRegionLabel: "eastus"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + } + + Eventually(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 2, 2), + eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("does not count clusters that have not passed the eligibility gate", func() { + policy := newPolicy(nextName("pp"), regionSelector("westus", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + By("a matching cluster registers but its member agent has not reported") + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "westus"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + + By("the policy stays unfulfilled through the join window") + Eventually(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 1, 0), + eventuallyTimeout, pollInterval).Should(Succeed()) + Consistently(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 1, 0), + consistentlyDuration, pollInterval).Should(Succeed()) + + By("once the member agent reports in, the policy is fulfilled") + markJoined(mc) + Eventually(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 1, 1), + eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("distinguishes minimum fulfillment from full fulfillment", func() { + policy := newPolicy(nextName("pp"), regionSelector("centralus", ptr.To(intstr.FromInt32(3)), ptr.To(int32(1)))) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "centralus"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + + By("the floor is met but the desired count is not; the binary contract keeps Scheduled False") + Eventually(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 3, 1), + eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("excludes tainted clusters unless the policy tolerates them", func() { + taint := clusterv1beta1.Taint{Key: "workload", Value: "restricted", Effect: "NoSchedule"} + mc := newMemberCluster(nextName("mc"), map[string]string{testRegionLabel: "northeurope"}, taint) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + + policy := newPolicy(nextName("pp"), regionSelector("northeurope", ptr.To(intstr.FromInt32(1)), nil)) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + By("the tainted cluster does not count without a toleration") + Eventually(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 1, 0), + eventuallyTimeout, pollInterval).Should(Succeed()) + Consistently(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 1, 0), + consistentlyDuration, pollInterval).Should(Succeed()) + + By("adding a toleration makes the cluster count") + Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), policy)).Should(Succeed()) + policy.Spec.Tolerations = []kfplacementv1alpha1.Toleration{ + {Key: "workload", Operator: "Equal", Value: "restricted", Effect: "NoSchedule"}, + } + Expect(k8sClient.Update(ctx, policy)).Should(Succeed()) + + Eventually(wantScheduledState(client.ObjectKeyFromObject(policy), metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 1, 1), + eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + It("treats an empty selector list as matching all eligible clusters (cluster-scoped policy)", func() { + names := []string{nextName("mc"), nextName("mc")} + for _, name := range names { + mc := newMemberCluster(name, map[string]string{testRegionLabel: "eastasia"}) + Expect(k8sClient.Create(ctx, mc)).Should(Succeed()) + markJoined(mc) + } + + policy := &kfplacementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: nextName("cpp")}, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Name: "demo"}, + }, + }, + } + Expect(k8sClient.Create(ctx, policy)).Should(Succeed()) + + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.ClusterPlacementPolicy{} + g.Expect(k8sClient.Get(ctx, types.NamespacedName{Name: policy.Name}, fetched)).Should(Succeed()) + cond := meta.FindStatusCondition(fetched.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeScheduled) + g.Expect(cond).NotTo(BeNil()) + g.Expect(cond.Status).Should(Equal(metav1.ConditionTrue)) + g.Expect(cond.Reason).Should(Equal(kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters)) + g.Expect(fetched.Status.DesiredClusters).NotTo(BeNil()) + g.Expect(*fetched.Status.DesiredClusters).Should(Equal(int32(len(names)))) + g.Expect(fetched.Status.ScheduledClusters).NotTo(BeNil()) + g.Expect(*fetched.Status.ScheduledClusters).Should(Equal(int32(len(names)))) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) + + // The gap this used to pin is closed: #829 added the CEL rule bounding the integer form of + // count, so zero is now rejected at admission and never reaches the controller. + It("rejects an integer count of zero at admission", func() { + policy := newPolicy(nextName("pp"), regionSelector("eastus2", ptr.To(intstr.FromInt32(0)), nil)) + Expect(k8sClient.Create(ctx, policy)).ShouldNot(Succeed(), "count: 0 must be rejected by the CEL guard on the integer form") + }) + + // KNOWN GAP (API, pinned): numeric operators in matchLabelExpressions pass admission; the + // doc comment defers misuse to the scheduling phase. Admission could reject with a CEL rule + // on the field instead. + It("KNOWN GAP: numeric operator in matchLabelExpressions passes admission and is rejected by the controller instead", func() { + policy := newPolicy(nextName("pp"), kfplacementv1alpha1.ClusterSelector{ + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchLabelExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: "env", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}}, + }, + }}, + }) + Expect(k8sClient.Create(ctx, policy)).Should(Succeed(), "numeric label expression is currently accepted at admission — flip this once the CEL guard exists") + + Eventually(func(g Gomega) { + fetched := &kfplacementv1alpha1.PlacementPolicy{} + g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(policy), fetched)).Should(Succeed()) + cond := meta.FindStatusCondition(fetched.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeScheduled) + g.Expect(cond).NotTo(BeNil()) + g.Expect(cond.Status).Should(Equal(metav1.ConditionFalse)) + g.Expect(cond.Reason).Should(Equal(reasonInvalidClusterSelectors)) + }, eventuallyTimeout, pollInterval).Should(Succeed()) + }) +}) diff --git a/pkg/v1/controllers/placementpolicy/controller_test.go b/pkg/v1/controllers/placementpolicy/controller_test.go new file mode 100644 index 000000000..ef43e2de8 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/controller_test.go @@ -0,0 +1,325 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + "errors" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" +) + +// TestReconcilePreservesClaimCountOnFailedRound pins that a claim round failing before it could +// count anything does not publish its partial count: a zero from a failed list would falsely +// report the policy's claims gone, on the status and on the metric derived from it. +func TestReconcilePreservesClaimCountOnFailedRound(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + if err := clusterv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + + policy := &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: "pp", Namespace: "default", Generation: 1}, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{{ + Count: ptr.To(intstr.FromInt32(1)), + WhenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }}, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{APIVersion: "v1", Kind: "ConfigMap", Name: "app"}}, + }, + Status: kfplacementv1alpha1.PlacementPolicyStatus{ + // The last completed round counted one active claim. + ActiveClusterClaims: ptr.To(int32(1)), + }, + } + + listErr := errors.New("the claim list is unwell") + c := withIndexes(fake.NewClientBuilder().WithScheme(scheme)). + WithObjects(policy). + WithStatusSubresource(policy). + WithInterceptorFuncs(interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, isClaims := list.(*kfplacementv1alpha1.ClusterClaimList); isClaims { + return listErr + } + return cl.List(ctx, list, opts...) + }, + }). + Build() + + r := NewReconciler(c, c, snapshotStub{}) + _, err := r.Reconcile(context.Background(), ctrl.Request{NamespacedName: types.NamespacedName{Namespace: "default", Name: "pp"}}) + if !errors.Is(err, listErr) { + t.Fatalf("Reconcile() = %v, want the claim list error %v so the round is retried", err, listErr) + } + + fetched := &kfplacementv1alpha1.PlacementPolicy{} + if err := c.Get(context.Background(), types.NamespacedName{Namespace: "default", Name: "pp"}, fetched); err != nil { + t.Fatalf("Get() = %v, want no error", err) + } + if fetched.Status.ActiveClusterClaims == nil || *fetched.Status.ActiveClusterClaims != 1 { + t.Errorf("Reconcile() left activeClusterClaims = %v, want the last completed round's 1 preserved", fetched.Status.ActiveClusterClaims) + } +} + +// TestReconcileIgnoresDeletedPolicy pins that a policy that is already gone ends the round quietly. +// The read is wrapped as a categorized API server error, and IsNotFound has to keep resolving +// through that wrap: an error utility that flattened the cause instead would turn every deleted +// policy into an endless retry, with its metric series never forgotten. +func TestReconcileIgnoresDeletedPolicy(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + c := fake.NewClientBuilder().WithScheme(scheme).Build() + r := NewReconciler(c, c, snapshotStub{}) + + for _, name := range []types.NamespacedName{{Namespace: "default", Name: "gone"}, {Name: "gone"}} { + res, err := r.Reconcile(context.Background(), ctrl.Request{NamespacedName: name}) + if err != nil { + t.Errorf("Reconcile(%v) = %v, want no error for a policy that no longer exists", name, err) + } + if res.RequeueAfter != 0 { + t.Errorf("Reconcile(%v) requeued after %v, want no requeue for a policy that no longer exists", name, res.RequeueAfter) + } + } +} + +// TestReconcileBacksOffWhileRoleIsHeld pins that a policy whose binding manager role another +// process holds is retried with a growing delay rather than at a fixed pace, and that a round +// that completes resets the delay. +func TestReconcileBacksOffWhileRoleIsHeld(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + if err := clusterv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + policy := bindingManagerHeldBy("rollout-controller") + cluster := testCluster("east-1", nil, nil, nil) + c := withIndexes(fake.NewClientBuilder().WithScheme(scheme)). + WithObjects(policy, cluster). + WithStatusSubresource(policy). + Build() + r := NewReconciler(c, c, snapshotStub{}) + r.eligibility = allEligible{} + req := ctrl.Request{NamespacedName: types.NamespacedName{Namespace: "default", Name: "pp"}} + + var delays []time.Duration + for range 3 { + res, err := r.Reconcile(context.Background(), req) + if err != nil { + t.Fatalf("Reconcile() = %v, want no error while the role is held elsewhere", err) + } + delays = append(delays, res.RequeueAfter) + } + wantDelays := []time.Duration{bindingManagerRetryBase, 2 * bindingManagerRetryBase, 4 * bindingManagerRetryBase} + if diff := cmp.Diff(wantDelays, delays); diff != "" { + t.Errorf("Reconcile() requeue delays mismatch (-want +got):\n%s", diff) + } + + // The rollout finishes and releases the role; the round completes and the backoff resets. + fetched := &kfplacementv1alpha1.PlacementPolicy{} + if err := c.Get(context.Background(), req.NamespacedName, fetched); err != nil { + t.Fatalf("Get() = %v, want no error", err) + } + fetched.Status.BindingManager = nil + if err := c.Status().Update(context.Background(), fetched); err != nil { + t.Fatalf("Status().Update() = %v, want no error", err) + } + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Fatalf("Reconcile() = %v, want no error once the role is free", err) + } + if got := r.roleBackoff.NumRequeues(req); got != 0 { + t.Errorf("roleBackoff.NumRequeues() = %d after a completed round, want 0", got) + } +} + +// failingSnapshotter is a snapshot manager that cannot take or read any snapshot. +type failingSnapshotter struct{ err error } + +func (f failingSnapshotter) SnapshotResourcesIfNoSnapshotExists(context.Context, kfplacementv1alpha1.PlacementPolicyAccessor) ([]kfplacementv1alpha1.PlacementResourceSnapshotAccessor, bool, error) { + return nil, false, f.err +} + +// TestReconcileSchedulesWithoutSnapshot pins that scheduling does not wait on resources: a round +// whose snapshot fails still unbinds a cluster that has left and writes the Scheduled status, and +// only then reports the snapshot error for a retry. A transient failure leaves the Synchronized +// and ResourceCollected conditions and the cluster counts as they were; a selected resource that +// is gone reports both conditions false, no cluster synchronized, and the availability the +// bindings report. +func TestReconcileSchedulesWithoutSnapshot(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + if err := clusterv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + testCases := []struct { + name string + snapshotErr error + wantSyncCond metav1.ConditionStatus + wantCollectedCond metav1.ConditionStatus + wantCounts *int32 + }{ + { + name: "transient failure", + snapshotErr: errors.New("the snapshot manager is unwell"), + wantSyncCond: metav1.ConditionTrue, + wantCollectedCond: metav1.ConditionTrue, + }, + { + name: "selected resource gone", + // Wrapped as the snapshot manager wraps a resource selected by name that does not exist. + snapshotErr: kferrors.Wraps(kferrors.NewUserError(apierrors.NewNotFound(schema.GroupResource{Resource: "configmaps"}, "app"), "selected resource does not exist"), + "failed to retrieve selected resources (name-based selector)"), + wantSyncCond: metav1.ConditionFalse, + wantCollectedCond: metav1.ConditionFalse, + wantCounts: ptr.To[int32](0), + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + policy := bindingManagerHeldBy(controllerName) + policy.Status.BindingManager = nil + policy.Status.Conditions = []metav1.Condition{{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized, + ObservedGeneration: policy.Generation, + }, { + Type: kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected, + ObservedGeneration: policy.Generation, + }} + gone := &kfplacementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{Name: bindingName(policy, "gone"), Namespace: "default", OwnerReferences: []metav1.OwnerReference{ + {APIVersion: kfplacementv1alpha1.GroupVersion.String(), Kind: kfplacementv1alpha1.PlacementPolicyKind, Name: "pp", UID: policy.UID, Controller: ptr.To(true)}, + }}, + Spec: kfplacementv1alpha1.PlacementBindingSpec{PlacementPolicyName: "pp", ClusterName: "gone", ResourceSnapshotName: "pp-snapshot-0"}, + } + c := withIndexes(fake.NewClientBuilder().WithScheme(scheme)). + WithObjects(policy, gone). + WithStatusSubresource(policy). + Build() + r := NewReconciler(c, c, failingSnapshotter{err: tc.snapshotErr}) + r.eligibility = allEligible{} + req := ctrl.Request{NamespacedName: types.NamespacedName{Namespace: "default", Name: "pp"}} + + if _, err := r.Reconcile(context.Background(), req); !errors.Is(err, tc.snapshotErr) { + t.Fatalf("Reconcile() = %v, want the snapshot error %v so the round is retried", err, tc.snapshotErr) + } + bindings := &kfplacementv1alpha1.PlacementBindingList{} + if err := c.List(context.Background(), bindings); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + if len(bindings.Items) != 0 { + t.Errorf("Reconcile() left %d bindings, want the binding of the cluster that left deleted", len(bindings.Items)) + } + fetched := &kfplacementv1alpha1.PlacementPolicy{} + if err := c.Get(context.Background(), req.NamespacedName, fetched); err != nil { + t.Fatalf("Get() = %v, want no error", err) + } + if meta.FindStatusCondition(fetched.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeScheduled) == nil { + t.Errorf("Reconcile() wrote no Scheduled condition, want the scheduling status written") + } + if fetched.Status.LatestResourceRevisionName != nil { + t.Errorf("Reconcile() set latestResourceRevisionName = %q, want it left unset without a snapshot", *fetched.Status.LatestResourceRevisionName) + } + if cond := meta.FindStatusCondition(fetched.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized); cond == nil || cond.Status != tc.wantSyncCond { + t.Errorf("Reconcile() left Synchronized condition %+v, want status %s", cond, tc.wantSyncCond) + } + var gotCollected metav1.ConditionStatus + if cond := meta.FindStatusCondition(fetched.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected); cond != nil { + gotCollected = cond.Status + } + if gotCollected != tc.wantCollectedCond { + t.Errorf("Reconcile() left ResourceCollected condition status %q, want %q", gotCollected, tc.wantCollectedCond) + } + if diff := cmp.Diff(tc.wantCounts, fetched.Status.SynchronizedClusters); diff != "" { + t.Errorf("Reconcile() synchronizedClusters mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff(tc.wantCounts, fetched.Status.ResourcesAvailableClusters); diff != "" { + t.Errorf("Reconcile() resourcesAvailableClusters mismatch (-want +got):\n%s", diff) + } + if gotAvailable := meta.FindStatusCondition(fetched.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeAvailable) != nil; gotAvailable != (tc.wantCounts != nil) { + t.Errorf("Reconcile() wrote an Available condition = %t, want %t", gotAvailable, tc.wantCounts != nil) + } + }) + } +} + +// spec is fixed and the role would otherwise block the policy's rollouts. +func TestReconcileReleasesRoleOnInvalidSelectors(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + if err := clusterv1beta1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + policy := bindingManagerHeldBy(controllerName) + // Admitted by the API, but Gt cannot apply to a label. + policy.Spec.ClusterSelectors = []kfplacementv1alpha1.ClusterSelector{{ + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchLabelExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: "tier", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}}, + }, + }}, + }} + c := withIndexes(fake.NewClientBuilder().WithScheme(scheme)).WithObjects(policy).WithStatusSubresource(policy).Build() + r := NewReconciler(c, c, snapshotStub{}) + req := ctrl.Request{NamespacedName: types.NamespacedName{Namespace: "default", Name: "pp"}} + + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Fatalf("Reconcile() = %v, want no error", err) + } + fetched := &kfplacementv1alpha1.PlacementPolicy{} + if err := c.Get(context.Background(), req.NamespacedName, fetched); err != nil { + t.Fatalf("Get() = %v, want no error", err) + } + if fetched.Status.BindingManager != nil { + t.Errorf("Reconcile() left the binding manager role as %v, want it released", fetched.Status.BindingManager) + } + if cond := meta.FindStatusCondition(fetched.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeScheduled); cond == nil || cond.Reason != reasonInvalidClusterSelectors { + t.Errorf("Reconcile() Scheduled condition = %v, want reason %q", cond, reasonInvalidClusterSelectors) + } +} diff --git a/pkg/v1/controllers/placementpolicy/metrics.go b/pkg/v1/controllers/placementpolicy/metrics.go new file mode 100644 index 000000000..3e08aa78a --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/metrics.go @@ -0,0 +1,65 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "strconv" + + "github.com/prometheus/client_golang/prometheus" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + hubmetrics "go.goms.io/fleet/pkg/metrics/hub" +) + +// reportPolicyMetrics publishes the scheduling status and the outstanding claim count for a +// policy. The namespace label is empty for cluster-scoped policies, which is how the two kinds +// are told apart in the metric. +func reportPolicyMetrics(policy kfplacementv1alpha1.PlacementPolicyAccessor, status *kfplacementv1alpha1.PlacementPolicyStatus, scheduledCond metav1.Condition) { + namespace, name := policy.GetNamespace(), policy.GetName() + + hubmetrics.FleetPlacementPolicyStatusLastTimestampSeconds. + WithLabelValues( + namespace, + name, + strconv.FormatInt(policy.GetGeneration(), 10), + scheduledCond.Type, + string(scheduledCond.Status), + scheduledCond.Reason, + ).SetToCurrentTime() + + if status.ActiveClusterClaims != nil { + hubmetrics.FleetPlacementPolicyActiveClusterClaims. + WithLabelValues(namespace, name). + Set(float64(*status.ActiveClusterClaims)) + } +} + +// forgetPolicyMetrics drops the metric series of a policy that has been deleted, so that gauges +// do not keep reporting for objects that no longer exist. It is keyed by namespace and name +// rather than by object, so that it can run from the reconcile that observes the deletion, +// whether or not the policy ever held a cleanup finalizer. +func forgetPolicyMetrics(namespace, name string) { + hubmetrics.FleetPlacementPolicyStatusLastTimestampSeconds.DeletePartialMatch(prometheus.Labels{ + "namespace": namespace, + "name": name, + }) + hubmetrics.FleetPlacementPolicyActiveClusterClaims.DeletePartialMatch(prometheus.Labels{ + "namespace": namespace, + "name": name, + }) +} diff --git a/pkg/v1/controllers/placementpolicy/resourcewatch.go b/pkg/v1/controllers/placementpolicy/resourcewatch.go new file mode 100644 index 000000000..5484592ff --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/resourcewatch.go @@ -0,0 +1,92 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/utils/keys" +) + +// EnqueuePoliciesSelecting re-queues the policies that may select the hub resource the key names, +// so that a change to it shows on their status (the latest resource revision going stale) without +// waiting for the periodic requeue. The resource watcher calls it for every resource change it +// sees, deletions included, which is why only the key is needed. +// +// A namespaced policy selects only from its own namespace; a cluster-scoped one selects from the +// namespace its selector names, or from every namespace (and the cluster scope) when it names none. +// +// ponytail: label selectors are not evaluated, so a change to any resource of a selected kind in +// scope re-queues the policy; the re-queue only re-checks the snapshot's hash, so this costs a +// reconcile, never a wrong result. Match the latest snapshot's resource identifiers if the +// fan-out shows up in profiles. +func (r *Reconciler) EnqueuePoliciesSelecting(ctx context.Context, key keys.ClusterWideKey) error { + if key.Namespace != "" { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + if err := r.List(ctx, policies, client.InNamespace(key.Namespace)); err != nil { + return kferrors.NewAPIServerError(err, "failed to list the placement policies", true, "namespace", key.Namespace) + } + for i := range policies.Items { + policy := &policies.Items[i] + if maySelect(policy.Namespace, policy.Spec.ResourceSelectors, key) { + klog.V(2).InfoS("A selected resource changed; re-queuing the placement policy", "placementPolicy", klog.KObj(policy), "resource", key) + r.policySource.Add(reconcile.Request{NamespacedName: types.NamespacedName{Namespace: policy.Namespace, Name: policy.Name}}) + } + } + } + + policies := &kfplacementv1alpha1.ClusterPlacementPolicyList{} + if err := r.List(ctx, policies); err != nil { + return kferrors.NewAPIServerError(err, "failed to list the cluster placement policies", true) + } + for i := range policies.Items { + policy := &policies.Items[i] + if maySelect("", policy.Spec.ResourceSelectors, key) { + klog.V(2).InfoS("A selected resource changed; re-queuing the cluster placement policy", "clusterPlacementPolicy", klog.KObj(policy), "resource", key) + r.clusterPolicySource.Add(reconcile.Request{NamespacedName: types.NamespacedName{Name: policy.Name}}) + } + } + return nil +} + +// maySelect reports whether a policy in the given namespace ("" for a cluster-scoped policy) may +// select the resource the key names. The version is not compared: the resource watcher reports +// the preferred version, while a selector may name any served one. +func maySelect(policyNamespace string, selectors []kfplacementv1alpha1.ResourceSelector, key keys.ClusterWideKey) bool { + for i := range selectors { + selector := &selectors[i] + if selector.APIGroup != key.Group || selector.Kind != key.Kind || (selector.Name != "" && selector.Name != key.Name) { + continue + } + namespace := selector.Namespace + if policyNamespace != "" { + // A namespaced policy always selects from its own namespace. + namespace = policyNamespace + } + if namespace == "" || namespace == key.Namespace { + return true + } + } + return false +} diff --git a/pkg/v1/controllers/placementpolicy/resourcewatch_test.go b/pkg/v1/controllers/placementpolicy/resourcewatch_test.go new file mode 100644 index 000000000..a4764c992 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/resourcewatch_test.go @@ -0,0 +1,129 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/util/workqueue" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils/keys" +) + +func resourceKey(group, version, kind, namespace, name string) keys.ClusterWideKey { + return keys.ClusterWideKey{ResourceIdentifier: placementv1beta1.ResourceIdentifier{ + Group: group, Version: version, Kind: kind, Namespace: namespace, Name: name, + }} +} + +func TestMaySelect(t *testing.T) { + configMap := resourceKey("", "v1", "ConfigMap", "team-a", "app") + testCases := []struct { + name string + policyNamespace string + selector kfplacementv1alpha1.ResourceSelector + key keys.ClusterWideKey + want bool + }{ + {name: "by name", policyNamespace: "team-a", selector: kfplacementv1alpha1.ResourceSelector{Kind: "ConfigMap", Name: "app"}, key: configMap, want: true}, + {name: "another name", policyNamespace: "team-a", selector: kfplacementv1alpha1.ResourceSelector{Kind: "ConfigMap", Name: "other"}, key: configMap}, + {name: "by label selector, not evaluated", policyNamespace: "team-a", selector: kfplacementv1alpha1.ResourceSelector{ + Kind: "ConfigMap", LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{"app": "x"}}, + }, key: configMap, want: true}, + {name: "another kind", policyNamespace: "team-a", selector: kfplacementv1alpha1.ResourceSelector{Kind: "Secret", Name: "app"}, key: configMap}, + {name: "another group", policyNamespace: "team-a", selector: kfplacementv1alpha1.ResourceSelector{APIGroup: "apps", Kind: "ConfigMap", Name: "app"}, key: configMap}, + {name: "version is not compared", selector: kfplacementv1alpha1.ResourceSelector{ + APIGroup: "autoscaling", APIVersion: "v1", Kind: "HorizontalPodAutoscaler", Name: "hpa", + }, key: resourceKey("autoscaling", "v2", "HorizontalPodAutoscaler", "", "hpa"), want: true}, + {name: "namespaced policy ignores the selector namespace", policyNamespace: "team-a", selector: kfplacementv1alpha1.ResourceSelector{ + Kind: "ConfigMap", Name: "app", Namespace: "team-b", + }, key: configMap, want: true}, + {name: "namespaced policy, another namespace", policyNamespace: "team-b", selector: kfplacementv1alpha1.ResourceSelector{Kind: "ConfigMap", Name: "app"}, key: configMap}, + {name: "namespaced policy, cluster-scoped resource", policyNamespace: "team-a", selector: kfplacementv1alpha1.ResourceSelector{Kind: "Namespace", Name: "team-a"}, + key: resourceKey("", "v1", "Namespace", "", "team-a")}, + {name: "cluster policy, selector namespace", selector: kfplacementv1alpha1.ResourceSelector{Kind: "ConfigMap", Name: "app", Namespace: "team-a"}, key: configMap, want: true}, + {name: "cluster policy, another selector namespace", selector: kfplacementv1alpha1.ResourceSelector{Kind: "ConfigMap", Name: "app", Namespace: "team-b"}, key: configMap}, + {name: "cluster policy, every namespace", selector: kfplacementv1alpha1.ResourceSelector{Kind: "ConfigMap", Name: "app"}, key: configMap, want: true}, + {name: "cluster policy, cluster-scoped resource", selector: kfplacementv1alpha1.ResourceSelector{Kind: "Namespace", Name: "team-a"}, + key: resourceKey("", "v1", "Namespace", "", "team-a"), want: true}, + {name: "a namespace selector does not cover its contents", selector: kfplacementv1alpha1.ResourceSelector{Kind: "Namespace", Name: "team-a"}, key: configMap}, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := maySelect(tc.policyNamespace, []kfplacementv1alpha1.ResourceSelector{tc.selector}, tc.key); got != tc.want { + t.Errorf("maySelect(%q, %+v, %v) = %t, want %t", tc.policyNamespace, tc.selector, tc.key, got, tc.want) + } + }) + } +} + +// drain starts the source on a fresh queue and returns, sorted, the requests it hands over. +func drain(t *testing.T, start func(context.Context, workqueue.TypedRateLimitingInterface[reconcile.Request]) error) []reconcile.Request { + t.Helper() + q := workqueue.NewTypedRateLimitingQueue(workqueue.DefaultTypedControllerRateLimiter[reconcile.Request]()) + defer q.ShutDown() + if err := start(context.Background(), q); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + var reqs []reconcile.Request + for q.Len() > 0 { + req, _ := q.Get() + reqs = append(reqs, req) + q.Done(req) + } + return reqs +} + +func TestEnqueuePoliciesSelecting(t *testing.T) { + scheme := runtime.NewScheme() + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + selecting := []kfplacementv1alpha1.ResourceSelector{{Kind: "ConfigMap", Name: "app"}} + other := []kfplacementv1alpha1.ResourceSelector{{Kind: "Secret", Name: "app"}} + c := fake.NewClientBuilder().WithScheme(scheme).WithObjects( + &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Namespace: "team-a", Name: "selects"}, Spec: kfplacementv1alpha1.PlacementPolicySpec{ResourceSelectors: selecting}}, + &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Namespace: "team-a", Name: "other-kind"}, Spec: kfplacementv1alpha1.PlacementPolicySpec{ResourceSelectors: other}}, + &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Namespace: "team-b", Name: "other-namespace"}, Spec: kfplacementv1alpha1.PlacementPolicySpec{ResourceSelectors: selecting}}, + &kfplacementv1alpha1.ClusterPlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: "fleet-wide"}, Spec: kfplacementv1alpha1.PlacementPolicySpec{ResourceSelectors: selecting}}, + &kfplacementv1alpha1.ClusterPlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: "other-kind"}, Spec: kfplacementv1alpha1.PlacementPolicySpec{ResourceSelectors: other}}, + ).Build() + r := NewReconciler(c, c, snapshotStub{}) + + if err := r.EnqueuePoliciesSelecting(context.Background(), resourceKey("", "v1", "ConfigMap", "team-a", "app")); err != nil { + t.Fatalf("EnqueuePoliciesSelecting() = %v, want no error", err) + } + sortReqs := cmpopts.SortSlices(func(a, b reconcile.Request) bool { return a.String() < b.String() }) + wantPolicies := []reconcile.Request{{NamespacedName: types.NamespacedName{Namespace: "team-a", Name: "selects"}}} + if diff := cmp.Diff(wantPolicies, drain(t, r.policySource.Start), sortReqs); diff != "" { + t.Errorf("EnqueuePoliciesSelecting() re-queued placement policies mismatch (-want +got):\n%s", diff) + } + wantClusterPolicies := []reconcile.Request{{NamespacedName: types.NamespacedName{Name: "fleet-wide"}}} + if diff := cmp.Diff(wantClusterPolicies, drain(t, r.clusterPolicySource.Start), sortReqs); diff != "" { + t.Errorf("EnqueuePoliciesSelecting() re-queued cluster placement policies mismatch (-want +got):\n%s", diff) + } +} diff --git a/pkg/v1/controllers/placementpolicy/scheduling.go b/pkg/v1/controllers/placementpolicy/scheduling.go new file mode 100644 index 000000000..c60989589 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/scheduling.go @@ -0,0 +1,381 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "fmt" + "math" + "slices" + "strings" + + corev1 "k8s.io/api/core/v1" + apiequality "k8s.io/apimachinery/pkg/api/equality" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// eligibilityChecker is the subset of the scheduler's cluster eligibility gate that the +// placement policy controller depends on. Selector fulfillment is judged against this predicate +// rather than raw label matching, so that a cluster only counts once it is actually usable for +// scheduling (member agent online, heartbeating, and joined); taking it as an interface keeps +// the predicate pluggable and fakeable in tests. +type eligibilityChecker interface { + IsEligible(cluster *clusterv1beta1.MemberCluster) (eligible bool, reason string) +} + +// The Scheduled condition uses the reason constants defined on the API +// (PlacementPolicyScheduledCondReasonFoundAllClusters and +// PlacementPolicyScheduledCondReasonFailedToFindSomeClusters), which encode a binary contract: +// True only when every selector has found all of its desired clusters. One additional local +// reason distinguishes selectors that cannot be evaluated at all from selectors that are simply +// unfulfilled. +const ( + // reasonInvalidClusterSelectors denotes that the cluster selectors cannot be evaluated, + // e.g., an operator is applied to a key it does not support; unlike + // FailedToFindSomeRequiredClusters, waiting for more clusters cannot resolve it. + reasonInvalidClusterSelectors = "InvalidClusterSelectors" +) + +// selectorOutcome is the evaluation result of a single cluster selector against the current +// member cluster inventory. +type selectorOutcome struct { + counts resolvedCounts + // matched holds the names of the schedulable clusters that satisfy the selector terms, + // sorted for determinism. + matched []string + // sticky holds, sorted, the names of the clusters an existing binding may stick to but + // that are neither counted nor newly picked: they satisfy the selector terms but are not + // schedulable right now (ineligible, or carrying a taint the policy does not tolerate), or + // their reported data could not be evaluated at all. See chooseClusters. + sticky []string + // unevaluable holds, sorted, the names of the clusters whose reported data could not be + // evaluated against the selector terms; they are among the sticky ones, and the Scheduled + // condition names them, since nothing else tells the user why they are not counted. + unevaluable []string + // stale holds, sorted, the names of the clusters, matched or sticky, that the selector keeps + // only because their binding records its terms: they no longer satisfy them. A scale-down + // trims them first. See chooseClusters. + stale []string + // chosen holds the names of the clusters counted as selected for this selector, sorted; + // chooseClusters fills it in. + chosen []string + // terms references the originating selector's terms; the claim lifecycle propagates them + // onto ClusterClaim objects for unfulfilled selectors. + terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm + // whenUnfulfilled is the originating selector's unfulfillment stance; claims are issued + // only for selectors set to AddClusterClaim. + whenUnfulfilled kfplacementv1alpha1.WhenUnfulfilledOption +} + +// int32Len returns the length of a cluster-name list as an int32, saturating at the type's +// maximum; fleet sizes sit far below that bound, the saturation only exists to make the +// conversion provably safe. It is the single int-to-int32 conversion point required because +// the API status counts are int32 while Go slice lengths are int. +func int32Len(s []string) int32 { + n := len(s) + if n > math.MaxInt32 { + return math.MaxInt32 + } + return int32(n) +} + +// fulfilled reports whether the selector has reached its fulfillment floor (minCount). +func (o *selectorOutcome) fulfilled() bool { + return int32Len(o.matched) >= o.counts.minimum +} + +// satisfiedInFull reports whether the selector needs no further clusters: the desired count is +// reached, or the selector requests all matching clusters and its floor is met. +func (o *selectorOutcome) satisfiedInFull() bool { + if o.counts.selectAll { + return o.fulfilled() + } + return int32Len(o.matched) >= o.counts.desired +} + +// evaluateSelectors evaluates every cluster selector of a policy against the given member +// clusters. Only schedulable clusters count toward fulfillment: the cluster must pass the +// scheduler's eligibility gate (member agent online, heartbeating, and joined) and all of its +// taints must be tolerated by the policy. An empty selector list is interpreted as a single +// selector matching all available clusters, per the API contract. +// +// Scheduling decisions stick, as they do for ClusterResourcePlacement and ResourcePlacement: a +// cluster bound for a selector stays selected by it even once its labels or properties stop +// matching, so that a cluster whose properties hover around a selector's threshold (a node count +// moving with autoscaling) does not have its resources removed and placed again on every swing. +// recorded maps each bound cluster to the selectors its binding was made for; a cluster stays +// selected by a selector whose terms the binding still records. Changing the selector's terms is +// what re-evaluates its clusters from scratch, as a new scheduling policy does for placements; a +// cluster leaving the fleet, or its binding being deleted, releases it too. +func evaluateSelectors(spec *kfplacementv1alpha1.PlacementPolicySpec, clusters []clusterv1beta1.MemberCluster, checker eligibilityChecker, recorded map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly) ([]selectorOutcome, error) { + schedulable := make([]*clusterv1beta1.MemberCluster, 0, len(clusters)) + // unschedulable keeps the clusters a binding may stick to while they are temporarily out + // of the running. A cluster on its way out is left out, so that its binding is released. + var unschedulable []*clusterv1beta1.MemberCluster + for i := range clusters { + cluster := &clusters[i] + eligible, _ := checker.IsEligible(cluster) + switch { + case eligible && taintsTolerated(cluster.Spec.Taints, spec.Tolerations): + schedulable = append(schedulable, cluster) + case cluster.DeletionTimestamp.IsZero(): + unschedulable = append(unschedulable, cluster) + } + } + + selectors := spec.ClusterSelectors + if len(selectors) == 0 { + // The synthesized selector mirrors what CRD defaulting would have produced had the + // user written it out — including the AddClusterClaim default, so that a bare policy + // in a fleet with no schedulable clusters can still signal that a cluster is needed. + selectors = []kfplacementv1alpha1.ClusterSelector{ + { + Count: ptr.To(intstr.FromString(countAllClusters)), + WhenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + }, + } + } + + outcomes := make([]selectorOutcome, 0, len(selectors)) + for i := range selectors { + selector := &selectors[i] + counts, err := resolveCounts(selector) + if err != nil { + return nil, err + } + // Validate the terms structurally so that invalid selectors surface even when no + // schedulable cluster exists to evaluate them against. + if err := validateTerms(selector.Terms); err != nil { + return nil, err + } + + matched, unevaluable := matchingClusterNames(schedulable, selector.Terms) + alsoMatched, alsoUnevaluable := matchingClusterNames(unschedulable, selector.Terms) + staleSchedulable := selectedBefore(schedulable, selector.Terms, recorded, matched, unevaluable) + staleUnschedulable := selectedBefore(unschedulable, selector.Terms, recorded, alsoMatched, alsoUnevaluable) + matched = append(matched, staleSchedulable...) + slices.Sort(matched) + sticky := slices.Concat(alsoMatched, unevaluable, alsoUnevaluable, staleUnschedulable) + slices.Sort(sticky) + stale := slices.Concat(staleSchedulable, staleUnschedulable) + slices.Sort(stale) + unevaluableHere := slices.Concat(unevaluable, alsoUnevaluable) + slices.Sort(unevaluableHere) + outcomes = append(outcomes, selectorOutcome{ + counts: counts, + matched: matched, + sticky: sticky, + unevaluable: unevaluableHere, + stale: stale, + terms: selector.Terms, + whenUnfulfilled: selector.WhenUnfulfilled, + }) + } + return outcomes, nil +} + +// selectedBefore returns the names of the clusters, other than those already accounted for, whose +// binding records the given selector terms, i.e., that the selector picked earlier and that stay +// selected by it whether or not they still match. +func selectedBefore(clusters []*clusterv1beta1.MemberCluster, terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm, recorded map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, accountedFor ...[]string) []string { + var names []string + for _, cluster := range clusters { + if slices.ContainsFunc(accountedFor, func(s []string) bool { return slices.Contains(s, cluster.Name) }) { + continue + } + if slices.ContainsFunc(recorded[cluster.Name], func(s kfplacementv1alpha1.ClusterSelectorWithTermsOnly) bool { + return apiequality.Semantic.DeepEqual(s.Terms, terms) + }) { + names = append(names, cluster.Name) + } + } + return names +} + +// matchingClusterNames returns, sorted, the names of the clusters that satisfy the terms, and +// separately the names of those whose data could not be evaluated. validateTerms has vetted the +// selector itself, so an evaluation error stems from malformed runtime data self-reported by +// the cluster (e.g., a property value that is not a quantity); one misbehaving cluster must not +// fail the whole policy, so it is reported apart rather than failing the evaluation. +func matchingClusterNames(clusters []*clusterv1beta1.MemberCluster, terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm) (matched, unevaluable []string) { + for _, cluster := range clusters { + ok, err := matchesTerms(cluster, terms) + if err != nil { + klog.V(2).InfoS("Skipping a member cluster whose reported data cannot be evaluated", "memberCluster", cluster.Name, "error", err) + unevaluable = append(unevaluable, cluster.Name) + continue + } + if ok { + matched = append(matched, cluster.Name) + } + } + slices.Sort(matched) + return matched, unevaluable +} + +// aggregateCounts sums the desired and selected cluster counts across all selector outcomes. +// The sums are per selector: a cluster that satisfies multiple selectors of the same policy +// contributes to each of them, in symmetry with how the desired counts add up (see the FEP's +// "About overlapping cluster selectors" note). For a count: All selector the desired count +// floors at the selector's minimum, so that an unfulfilled All selector shows a gap between +// desiredClusters and scheduledClusters instead of trivially reporting desired == scheduled. +func aggregateCounts(outcomes []selectorOutcome) (desired, scheduled int32) { + for i := range outcomes { + o := &outcomes[i] + if o.counts.selectAll { + desired += max(int32Len(o.matched), o.counts.minimum) + } else { + desired += o.counts.desired + } + scheduled += int32Len(o.chosen) + } + return desired, scheduled +} + +// scheduledCondition builds the Scheduled condition for a policy from its selector outcomes, +// honoring the API's binary contract: True only when every selector has found all of its +// desired clusters. Whether the minCount floor has at least been reached everywhere is surfaced +// through the message, not a separate reason or status, as are the member clusters whose reported +// data cannot be evaluated against the selectors. +func scheduledCondition(generation int64, outcomes []selectorOutcome) metav1.Condition { + cond := scheduledConditionByCounts(generation, outcomes) + var unevaluable []string + for i := range outcomes { + unevaluable = append(unevaluable, outcomes[i].unevaluable...) + } + if len(unevaluable) > 0 { + slices.Sort(unevaluable) + cond.Message += fmt.Sprintf("; the data reported by member clusters %s cannot be evaluated against the cluster selectors, so they keep any existing binding but are not counted", + truncatedNameList(slices.Compact(unevaluable))) + } + return cond +} + +// truncatedNameList formats names for a condition message, listing at most a handful so that a +// fleet-wide problem cannot bloat the policy status. +func truncatedNameList(names []string) string { + const maxListed = 5 + if len(names) <= maxListed { + return strings.Join(names, ", ") + } + return fmt.Sprintf("%s and %d more", strings.Join(names[:maxListed], ", "), len(names)-maxListed) +} + +// scheduledConditionByCounts builds the Scheduled condition from the selectors' cluster counts. +func scheduledConditionByCounts(generation int64, outcomes []selectorOutcome) metav1.Condition { + allInFull, allFulfilled := true, true + for i := range outcomes { + if !outcomes[i].satisfiedInFull() { + allInFull = false + } + if !outcomes[i].fulfilled() { + allFulfilled = false + } + } + + switch { + case allInFull: + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, + Message: "All cluster selectors have found their desired numbers of clusters", + ObservedGeneration: generation, + } + case allFulfilled: + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, + Status: metav1.ConditionFalse, + Reason: kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, + Message: "All cluster selectors have reached their minimum cluster counts, but some are still short of their desired counts", + ObservedGeneration: generation, + } + default: + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, + Status: metav1.ConditionFalse, + Reason: kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, + Message: "One or more cluster selectors have not found their minimum numbers of clusters", + ObservedGeneration: generation, + } + } +} + +// invalidSelectorsCondition builds the Scheduled condition for a policy whose cluster selectors +// cannot be evaluated. +func invalidSelectorsCondition(generation int64, err error) metav1.Condition { + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, + Status: metav1.ConditionFalse, + Reason: reasonInvalidClusterSelectors, + Message: "The cluster selectors cannot be evaluated: " + err.Error(), + ObservedGeneration: generation, + } +} + +// taintsTolerated reports whether every given taint is tolerated by at least one of the given +// tolerations. +func taintsTolerated(taints []clusterv1beta1.Taint, tolerations []kfplacementv1alpha1.Toleration) bool { + for i := range taints { + if !taintTolerated(&taints[i], tolerations) { + return false + } + } + return true +} + +func taintTolerated(taint *clusterv1beta1.Taint, tolerations []kfplacementv1alpha1.Toleration) bool { + for i := range tolerations { + if tolerationTolerates(taint, &tolerations[i]) { + return true + } + } + return false +} + +// tolerationTolerates mirrors the core Kubernetes toleration semantics: an empty effect matches +// every effect, an empty key with the Exists operator matches every taint, and the Equal +// operator (the default) compares values. It parallels the scheduler's +// tainttoleration/filtering.go implementation, which operates on the v1beta1 Toleration type; +// semantic changes must be kept in sync between the two. +func tolerationTolerates(taint *clusterv1beta1.Taint, toleration *kfplacementv1alpha1.Toleration) bool { + if toleration.Effect != "" && toleration.Effect != taint.Effect { + return false + } + if toleration.Key == "" { + return toleration.Operator == corev1.TolerationOpExists + } + if toleration.Key != taint.Key { + return false + } + switch toleration.Operator { + case corev1.TolerationOpExists: + return true + case corev1.TolerationOpEqual, "": + return toleration.Value == taint.Value + default: + // The CRD enum admits only Equal and Exists; an unrecognized operator can only come + // from an object that bypassed admission and tolerates nothing, in consistency with + // the upstream Toleration.ToleratesTaint behavior. + return false + } +} diff --git a/pkg/v1/controllers/placementpolicy/scheduling_test.go b/pkg/v1/controllers/placementpolicy/scheduling_test.go new file mode 100644 index 000000000..35facbc64 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/scheduling_test.go @@ -0,0 +1,337 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "slices" + "testing" + + "github.com/google/go-cmp/cmp" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/propertyprovider" +) + +// allEligible is a test eligibility gate that admits every cluster. +type allEligible struct{} + +func (allEligible) IsEligible(*clusterv1beta1.MemberCluster) (bool, string) { + return true, "" +} + +func TestEvaluateSelectorsSkipsClustersWithUnevaluableData(t *testing.T) { + spec := &kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{{ + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGe, Values: []string{"2"}}, + }, + }}, + Count: ptr.To(intstr.FromInt32(1)), + }}, + } + clusters := []clusterv1beta1.MemberCluster{ + *testCluster("good-cluster", nil, map[string]string{propertyprovider.NodeCountProperty: "4"}, nil), + *testCluster("bad-data-cluster", nil, map[string]string{propertyprovider.NodeCountProperty: "not-a-number"}, nil), + } + + outcomes, err := evaluateSelectors(spec, clusters, allEligible{}, nil) + if err != nil { + t.Fatalf("evaluateSelectors(%v) returned unexpected error: %v", spec, err) + } + if len(outcomes) != 1 { + t.Fatalf("evaluateSelectors(%v) returned %d outcomes, want 1", spec, len(outcomes)) + } + wantMatched := []string{"good-cluster"} + if diff := cmp.Diff(outcomes[0].matched, wantMatched); diff != "" { + t.Errorf("evaluateSelectors(%v) matched mismatch (-got, +want):\n%s", spec, diff) + } + if diff := cmp.Diff([]string{"bad-data-cluster"}, outcomes[0].unevaluable); diff != "" { + t.Errorf("evaluateSelectors(%v) unevaluable mismatch (-want +got):\n%s", spec, diff) + } +} + +func TestAggregateCounts(t *testing.T) { + testCases := []struct { + name string + outcomes []selectorOutcome + wantDesired int32 + wantScheduled int32 + }{ + { + name: "integer counts sum per selector", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 2, minimum: 2}, matched: []string{"a", "b", "c"}, chosen: []string{"a", "b"}}, + {counts: resolvedCounts{desired: 3, minimum: 3}, matched: []string{"a"}, chosen: []string{"a"}}, + }, + wantDesired: 5, + wantScheduled: 3, + }, + { + name: "selectAll desired floors at the minimum when unfulfilled", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{selectAll: true, minimum: 2}, matched: []string{"a"}, chosen: []string{"a"}}, + }, + wantDesired: 2, + wantScheduled: 1, + }, + { + name: "selectAll desired follows the match count when fulfilled", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{selectAll: true, minimum: 1}, matched: []string{"a", "b", "c"}, chosen: []string{"a", "b", "c"}}, + }, + wantDesired: 3, + wantScheduled: 3, + }, + { + name: "empty fleet with a bare selectAll selector shows the gap", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{selectAll: true, minimum: 1}}, + }, + wantDesired: 1, + wantScheduled: 0, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + desired, scheduled := aggregateCounts(tc.outcomes) + if desired != tc.wantDesired || scheduled != tc.wantScheduled { + t.Errorf("aggregateCounts(%v) = %d, %d, want %d, %d", tc.outcomes, desired, scheduled, tc.wantDesired, tc.wantScheduled) + } + }) + } +} + +func TestScheduledCondition(t *testing.T) { + testCases := []struct { + name string + outcomes []selectorOutcome + wantStatus metav1.ConditionStatus + wantReason string + }{ + { + name: "all selectors fulfilled in full", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1, minimum: 1}, matched: []string{"a"}}, + }, + wantStatus: metav1.ConditionTrue, + wantReason: kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, + }, + { + name: "floor reached but short of desired stays False per the binary contract", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 3, minimum: 1}, matched: []string{"a"}}, + }, + wantStatus: metav1.ConditionFalse, + wantReason: kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, + }, + { + name: "below the floor", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 3, minimum: 2}, matched: []string{"a"}}, + }, + wantStatus: metav1.ConditionFalse, + wantReason: kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := scheduledCondition(7, tc.outcomes) + if got.Status != tc.wantStatus || got.Reason != tc.wantReason { + t.Errorf("scheduledCondition(7, %v) = %s/%s, want %s/%s", tc.outcomes, got.Status, got.Reason, tc.wantStatus, tc.wantReason) + } + if got.ObservedGeneration != 7 { + t.Errorf("scheduledCondition(7, %v) observedGeneration = %d, want 7", tc.outcomes, got.ObservedGeneration) + } + }) + } +} + +func TestTolerationTolerates(t *testing.T) { + noScheduleTaint := clusterv1beta1.Taint{Key: "workload", Value: "restricted", Effect: "NoSchedule"} + + testCases := []struct { + name string + toleration kfplacementv1alpha1.Toleration + want bool + }{ + { + name: "equal operator with matching value", + toleration: kfplacementv1alpha1.Toleration{Key: "workload", Operator: "Equal", Value: "restricted", Effect: "NoSchedule"}, + want: true, + }, + { + name: "equal operator with mismatched value", + toleration: kfplacementv1alpha1.Toleration{Key: "workload", Operator: "Equal", Value: "other", Effect: "NoSchedule"}, + want: false, + }, + { + name: "exists operator ignores the value", + toleration: kfplacementv1alpha1.Toleration{Key: "workload", Operator: "Exists", Effect: "NoSchedule"}, + want: true, + }, + { + name: "empty key with exists matches every taint", + toleration: kfplacementv1alpha1.Toleration{Operator: "Exists"}, + want: true, + }, + { + name: "empty effect matches every effect", + toleration: kfplacementv1alpha1.Toleration{Key: "workload", Operator: "Equal", Value: "restricted"}, + want: true, + }, + { + name: "unrecognized operator tolerates nothing", + toleration: kfplacementv1alpha1.Toleration{Key: "workload", Operator: "Fuzzy", Value: "restricted", Effect: "NoSchedule"}, + want: false, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := tolerationTolerates(&noScheduleTaint, &tc.toleration) + if got != tc.want { + t.Errorf("tolerationTolerates(%v, %v) = %t, want %t", noScheduleTaint, tc.toleration, got, tc.want) + } + }) + } +} + +// eligibleExcept is a test eligibility gate that admits every cluster but the named ones. +type eligibleExcept []string + +func (e eligibleExcept) IsEligible(cluster *clusterv1beta1.MemberCluster) (bool, string) { + return !slices.Contains(e, cluster.Name), "" +} + +// TestEvaluateSelectorsKeepsClustersSelectedBefore pins that a scheduling decision sticks: a +// cluster whose binding records a selector's terms stays selected by it, counted when it is +// schedulable and sticky when it is not, even though it no longer matches, and is marked stale; +// a binding made for other terms, or for a cluster that has left, keeps nothing. +func TestEvaluateSelectorsKeepsClustersSelectedBefore(t *testing.T) { + eastTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{MatchLabels: map[string]string{regionLabel: "east"}}} + oldTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{MatchLabels: map[string]string{regionLabel: "central"}}} + spec := &kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{{Terms: eastTerms, Count: ptr.To(intstr.FromInt32(3))}}, + } + west := map[string]string{regionLabel: "west"} + clusters := []clusterv1beta1.MemberCluster{ + *testCluster("east-1", map[string]string{regionLabel: "east"}, nil, nil), + *testCluster("relabeled", west, nil, nil), + *testCluster("relabeled-dark", west, nil, nil), + *testCluster("bound-for-old-terms", west, nil, nil), + *testCluster("never-bound", west, nil, nil), + } + recorded := map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + "east-1": {{Terms: eastTerms}}, + "relabeled": {{Terms: oldTerms}, {Terms: eastTerms}}, + "relabeled-dark": {{Terms: eastTerms}}, + "bound-for-old-terms": {{Terms: oldTerms}}, + "left-the-fleet": {{Terms: eastTerms}}, + } + + outcomes, err := evaluateSelectors(spec, clusters, eligibleExcept{"relabeled-dark"}, recorded) + if err != nil { + t.Fatalf("evaluateSelectors() = %v, want no error", err) + } + if len(outcomes) != 1 { + t.Fatalf("evaluateSelectors() = %d outcomes, want 1", len(outcomes)) + } + if diff := cmp.Diff([]string{"east-1", "relabeled"}, outcomes[0].matched); diff != "" { + t.Errorf("evaluateSelectors() matched mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff([]string{"relabeled-dark"}, outcomes[0].sticky); diff != "" { + t.Errorf("evaluateSelectors() sticky mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff([]string{"relabeled", "relabeled-dark"}, outcomes[0].stale); diff != "" { + t.Errorf("evaluateSelectors() stale mismatch (-want +got):\n%s", diff) + } +} + +// TestScheduledConditionNamesUnevaluableClusters pins that the clusters whose data cannot be +// evaluated are named on the condition, deduplicated across selectors and capped in number. +func TestScheduledConditionNamesUnevaluableClusters(t *testing.T) { + testCases := []struct { + name string + outcomes []selectorOutcome + wantMessage string + }{ + { + name: "none", + outcomes: []selectorOutcome{{counts: resolvedCounts{desired: 1, minimum: 1}, matched: []string{"a"}}}, + wantMessage: "All cluster selectors have found their desired numbers of clusters", + }, + { + name: "deduplicated across selectors", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1, minimum: 1}, matched: []string{"a"}, unevaluable: []string{"c", "b"}}, + {counts: resolvedCounts{desired: 1, minimum: 1}, matched: []string{"a"}, unevaluable: []string{"b"}}, + }, + wantMessage: "All cluster selectors have found their desired numbers of clusters; the data reported by member clusters b, c cannot be evaluated against the cluster selectors, so they keep any existing binding but are not counted", + }, + { + name: "capped", + outcomes: []selectorOutcome{ + {counts: resolvedCounts{desired: 1, minimum: 1}, matched: []string{"a"}, unevaluable: []string{"b", "c", "d", "e", "f", "g", "h"}}, + }, + wantMessage: "All cluster selectors have found their desired numbers of clusters; the data reported by member clusters b, c, d, e, f and 2 more cannot be evaluated against the cluster selectors, so they keep any existing binding but are not counted", + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := scheduledCondition(1, tc.outcomes).Message; got != tc.wantMessage { + t.Errorf("scheduledCondition().Message = %q, want %q", got, tc.wantMessage) + } + }) + } +} + +// TestEvaluateSelectorsKeepsClustersSelectedBeforeAcrossSelectors pins that a decision sticks to +// the selector it was made for by terms, not by position: a cluster recorded for one selector that +// now matches only another is selected by both, and reordering the selectors keeps the pin. +func TestEvaluateSelectorsKeepsClustersSelectedBeforeAcrossSelectors(t *testing.T) { + eastTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{MatchLabels: map[string]string{regionLabel: "east"}}} + westTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{MatchLabels: map[string]string{regionLabel: "west"}}} + spec := &kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + {Terms: westTerms, Count: ptr.To(intstr.FromInt32(1))}, + {Terms: eastTerms, Count: ptr.To(intstr.FromInt32(1))}, + }, + } + clusters := []clusterv1beta1.MemberCluster{ + *testCluster("moved-west", map[string]string{regionLabel: "west"}, nil, nil), + } + recorded := map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{"moved-west": {{Terms: eastTerms}}} + + outcomes, err := evaluateSelectors(spec, clusters, allEligible{}, recorded) + if err != nil { + t.Fatalf("evaluateSelectors() = %v, want no error", err) + } + var gotMatched [][]string + for i := range outcomes { + gotMatched = append(gotMatched, outcomes[i].matched) + } + if diff := cmp.Diff([][]string{{"moved-west"}, {"moved-west"}}, gotMatched); diff != "" { + t.Errorf("evaluateSelectors() matched per selector mismatch (-want +got):\n%s", diff) + } +} diff --git a/pkg/v1/controllers/placementpolicy/scoring.go b/pkg/v1/controllers/placementpolicy/scoring.go new file mode 100644 index 000000000..b0370a560 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/scoring.go @@ -0,0 +1,118 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "cmp" + "math" + "slices" + + corev1 "k8s.io/api/core/v1" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + "go.goms.io/fleet/pkg/propertyprovider" +) + +// criterionWeight is the weight the FEP assigns to each ranking criterion. All criteria weigh +// the same, so a cluster's total lies in [0, 100 * len(rankingCriteria)]. +const criterionWeight = 100.0 + +// rankingCriterion reads one preference metric off a member cluster. A cluster that does not +// report the metric does not take part in the criterion and scores 0 on it, which after +// normalization is the least preferred position whichever direction the criterion prefers. +type rankingCriterion struct { + preferHigher bool + value func(*clusterv1beta1.MemberCluster) (float64, bool) +} + +// rankingCriteria are the FEP's cluster preferences, in its order: spread placements as evenly +// as possible across a homogeneous fleet by preferring the smaller and emptier clusters. This is +// the same idea as the Kubernetes scheduler's LeastAllocated strategy. +var rankingCriteria = []rankingCriterion{ + {preferHigher: false, value: nodeCount}, + {preferHigher: true, value: availableResource(corev1.ResourceCPU)}, + {preferHigher: true, value: availableResource(corev1.ResourceMemory)}, +} + +func nodeCount(cluster *clusterv1beta1.MemberCluster) (float64, bool) { + q, err := numericPropertyValueFrom(cluster, propertyprovider.NodeCountProperty) + if err != nil || q == nil { + // Malformed self-reported data is handled the same way as absent data: the cluster + // simply does not take part in this criterion. Selector evaluation already logs it. + return 0, false + } + return q.AsApproximateFloat64(), true +} + +func availableResource(name corev1.ResourceName) func(*clusterv1beta1.MemberCluster) (float64, bool) { + return func(cluster *clusterv1beta1.MemberCluster) (float64, bool) { + q, reported := cluster.Status.ResourceUsage.Available[name] + if !reported { + return 0, false + } + return q.AsApproximateFloat64(), true + } +} + +// rankClusters orders the candidate clusters of one selector from most to least preferred, per +// the FEP: on every criterion a cluster scores 100 * (X - Min) / (Max - Min) against the +// candidates' own minimum and maximum (inverted when lower is better), 0 for all of them when +// they tie, and the criteria add up. Equal totals break on the cluster name, so the order is +// deterministic. Only the order leaves this function; the scores are reported nowhere. +func rankClusters(candidates []*clusterv1beta1.MemberCluster) []string { + totals := make(map[string]float64, len(candidates)) + for i := range rankingCriteria { + scoreCriterion(&rankingCriteria[i], candidates, totals) + } + + names := make([]string, 0, len(candidates)) + for _, cluster := range candidates { + names = append(names, cluster.Name) + } + slices.SortFunc(names, func(a, b string) int { + return cmp.Or(cmp.Compare(totals[b], totals[a]), cmp.Compare(a, b)) + }) + return names +} + +// scoreCriterion adds each candidate's normalized score on one criterion to its total. +func scoreCriterion(criterion *rankingCriterion, candidates []*clusterv1beta1.MemberCluster, totals map[string]float64) { + values := make(map[string]float64, len(candidates)) + lo, hi := math.Inf(1), math.Inf(-1) + for _, cluster := range candidates { + v, reported := criterion.value(cluster) + if !reported || math.IsNaN(v) || math.IsInf(v, 0) || v < 0 { + // Self-reported data: a negative or non-finite value would set the range for every + // other candidate, so it is treated as unreported instead. + continue + } + values[cluster.Name] = v + lo, hi = min(lo, v), max(hi, v) + } + if lo >= hi { + // Every reporting cluster has the same value (or none reports): the criterion cannot + // tell the candidates apart and contributes 0 to each of them. + return + } + for name, v := range values { + score := criterionWeight * (v - lo) / (hi - lo) + if !criterion.preferHigher { + score = criterionWeight - score + } + totals[name] += score + } +} diff --git a/pkg/v1/controllers/placementpolicy/scoring_test.go b/pkg/v1/controllers/placementpolicy/scoring_test.go new file mode 100644 index 000000000..08a07cd3f --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/scoring_test.go @@ -0,0 +1,148 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "testing" + + "github.com/google/go-cmp/cmp" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + "go.goms.io/fleet/pkg/propertyprovider" +) + +// rankedCluster builds a member cluster reporting the three ranking metrics; an empty string +// leaves the metric unreported. +func rankedCluster(name, nodes, cpu, memory string) *clusterv1beta1.MemberCluster { + cluster := &clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: name}} + if nodes != "" { + cluster.Status.Properties = map[clusterv1beta1.PropertyName]clusterv1beta1.PropertyValue{ + propertyprovider.NodeCountProperty: {Value: nodes}, + } + } + available := corev1.ResourceList{} + if cpu != "" { + available[corev1.ResourceCPU] = resource.MustParse(cpu) + } + if memory != "" { + available[corev1.ResourceMemory] = resource.MustParse(memory) + } + cluster.Status.ResourceUsage.Available = available + return cluster +} + +func TestRankClusters(t *testing.T) { + testCases := []struct { + name string + candidates []*clusterv1beta1.MemberCluster + want []string + }{ + { + name: "fewer nodes rank first", + candidates: []*clusterv1beta1.MemberCluster{ + rankedCluster("big", "10", "4", "8Gi"), + rankedCluster("small", "2", "4", "8Gi"), + }, + want: []string{"small", "big"}, + }, + { + name: "more available CPU ranks first", + candidates: []*clusterv1beta1.MemberCluster{ + rankedCluster("busy", "3", "1", "8Gi"), + rankedCluster("idle", "3", "7", "8Gi"), + }, + want: []string{"idle", "busy"}, + }, + { + name: "more available memory ranks first", + candidates: []*clusterv1beta1.MemberCluster{ + rankedCluster("full", "3", "4", "1Gi"), + rankedCluster("roomy", "3", "4", "32Gi"), + }, + want: []string{"roomy", "full"}, + }, + { + name: "criteria add up with equal weight", + candidates: []*clusterv1beta1.MemberCluster{ + // Best on nodes and CPU (200), worst on memory (0): 200. + rankedCluster("two-wins", "1", "8", "1Gi"), + // Worst on nodes and CPU (0), best on memory (100): 100. + rankedCluster("one-win", "9", "1", "16Gi"), + // Middle on every criterion: 150. + rankedCluster("middling", "5", "4.5", "8.5Gi"), + }, + want: []string{"two-wins", "middling", "one-win"}, + }, + { + name: "all tied falls back to name order", + candidates: []*clusterv1beta1.MemberCluster{ + rankedCluster("c", "3", "4", "8Gi"), + rankedCluster("a", "3", "4", "8Gi"), + rankedCluster("b", "3", "4", "8Gi"), + }, + want: []string{"a", "b", "c"}, + }, + { + name: "unreported metrics rank last on that criterion", + candidates: []*clusterv1beta1.MemberCluster{ + rankedCluster("silent", "", "", ""), + rankedCluster("reporting", "10", "1", "1Gi"), + }, + // The reporting cluster ties with itself (min == max) and scores 0 everywhere, just + // like the silent one; the name decides. + want: []string{"reporting", "silent"}, + }, + { + name: "a partially reporting cluster loses the criteria it skips", + candidates: []*clusterv1beta1.MemberCluster{ + // Best on nodes (100), unreported elsewhere (0): 100. + rankedCluster("nodes-only", "1", "", ""), + // Worst on nodes (0), only reporter on CPU and memory so those tie at 0: 0. + rankedCluster("full-report", "5", "4", "8Gi"), + }, + want: []string{"nodes-only", "full-report"}, + }, + { + name: "malformed node count is treated as unreported", + candidates: []*clusterv1beta1.MemberCluster{ + rankedCluster("garbage", "lots", "4", "8Gi"), + rankedCluster("sane", "3", "4", "8Gi"), + rankedCluster("big", "9", "4", "8Gi"), + }, + // The cluster with the garbage value scores 0 on node count like the biggest one, + // and the name settles that tie. + want: []string{"sane", "big", "garbage"}, + }, + { + name: "no candidates", + candidates: nil, + want: []string{}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := rankClusters(tc.candidates) + if diff := cmp.Diff(got, tc.want); diff != "" { + t.Errorf("rankClusters() mismatch (-got, +want):\n%s", diff) + } + }) + } +} diff --git a/pkg/v1/controllers/placementpolicy/selector.go b/pkg/v1/controllers/placementpolicy/selector.go new file mode 100644 index 000000000..62831c65f --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/selector.go @@ -0,0 +1,508 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "fmt" + "maps" + "slices" + "strconv" + "strings" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/selection" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/validation" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/propertyprovider" +) + +// countAllClusters is the sentinel string value of the ClusterSelector count field that selects +// every cluster matching the given terms. +const countAllClusters = "All" + +// maxClusterCount is the largest number of clusters a selector's count may request. It mirrors the +// upper bound of the count field's CRD pattern (^([1-9][0-9]{0,2}|All)$, i.e. 1..999). The pattern +// constrains only the string form of the int-or-string field, so the integer form is bounded here +// instead -- both to hold the intended limit and to keep the sum of counts across selectors from +// overflowing the int32 the desired-cluster total is accumulated in. +const maxClusterCount = 999 + +// resolvedCounts is the outcome of interpreting a ClusterSelector's count and minCount fields. +type resolvedCounts struct { + // selectAll is true when the selector requests all matching clusters (count = "All"). + selectAll bool + // desired is the desired number of clusters; meaningless when selectAll is set. + desired int32 + // minimum is the fulfillment floor: the selector counts as fulfilled once this many + // clusters match. Per the API defaults, it equals desired when count is an integer and + // 1 when count is "All", unless minCount is set explicitly. + minimum int32 +} + +// resolveCounts interprets the count and minCount fields of a cluster selector, applying the +// defaulting rules documented on the PlacementPolicy API. +func resolveCounts(selector *kfplacementv1alpha1.ClusterSelector) (resolvedCounts, error) { + rc := resolvedCounts{} + + count := selector.Count + if count == nil { + // The CRD defaults the field to 1; the object might not have passed through the API + // server defaulting chain (e.g., in unit tests), so the rule is applied here as well. + count = &intstr.IntOrString{Type: intstr.Int, IntVal: 1} + } + switch count.Type { + case intstr.String: + if count.StrVal == countAllClusters { + rc.selectAll = true + rc.minimum = 1 + break + } + // The CRD pattern admits digit strings alongside "All", and the minCount CEL rule converts + // them with int(self.count), so a manifest with count: "3" passes admission; parse it here + // rather than rejecting a value the API accepts. Anything else, or a digit string outside the + // bound (from an object that bypassed admission), is an error. + n, err := strconv.Atoi(count.StrVal) + if err != nil || n < 1 || n > maxClusterCount { + return rc, fmt.Errorf("invalid count value %q: only %q and integers in [1, %d] are supported", count.StrVal, countAllClusters, maxClusterCount) + } + desired := int32(n) //nolint:gosec // n is bounded to [1, maxClusterCount] above, so the conversion cannot overflow. + rc.desired = desired + rc.minimum = desired + case intstr.Int: + // The CRD pattern bounds only the string form, so the integer form is bounded here; the upper + // bound also keeps two maximum-valued selectors from overflowing the aggregated desired total. + if count.IntVal < 1 || count.IntVal > maxClusterCount { + return rc, fmt.Errorf("invalid count value %d: must be an integer in [1, %d]", count.IntVal, maxClusterCount) + } + rc.desired = count.IntVal + rc.minimum = count.IntVal + default: + return rc, fmt.Errorf("invalid count type %d", count.Type) + } + + if selector.MinCount != nil { + rc.minimum = *selector.MinCount + } + return rc, nil +} + +// validateTerms structurally validates selector terms without evaluating them against any +// cluster, so that invalid selectors are detected even when the fleet has no schedulable +// clusters at all: label expression operators must be label-applicable, and numeric property +// expressions must carry exactly one parseable quantity. +func validateTerms(terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm) error { + for i := range terms { + term := &terms[i] + // matchLabels is a free-form map on the CRD: admission stores whatever keys and values + // are given, and labels.SelectorFromSet performs no validation of its own. An invalid + // key or value therefore yields a selector that matches no cluster, which the policy + // would misread as an ordinarily unfulfilled selector and answer by provisioning a + // cluster no manifest can ever satisfy. Reject it structurally instead. + // + // The keys are validated in sorted order so that a term carrying more than one invalid + // entry always surfaces the same error. The message ends up on the policy's Scheduled + // condition, and a message that changed from one reconcile to the next -- as it would if + // the map were ranged in Go's randomized order -- would defeat the status no-op check and + // spin the reconcile. + for _, key := range slices.Sorted(maps.Keys(term.MatchLabels)) { + if errs := validation.IsQualifiedName(key); len(errs) != 0 { + return fmt.Errorf("invalid matchLabels key %q: %s", key, strings.Join(errs, "; ")) + } + if errs := validation.IsValidLabelValue(term.MatchLabels[key]); len(errs) != 0 { + return fmt.Errorf("invalid matchLabels value %q for key %q: %s", term.MatchLabels[key], key, strings.Join(errs, "; ")) + } + } + for j := range term.MatchLabelExpressions { + expr := &term.MatchLabelExpressions[j] + op, err := labelSelectionOperatorFor(expr.Operator) + if err != nil { + return fmt.Errorf("invalid label expression on key %s: %w", expr.Key, err) + } + if _, err := labels.NewRequirement(expr.Key, op, expr.Values); err != nil { + return fmt.Errorf("invalid label expression on key %s: %w", expr.Key, err) + } + } + for j := range term.MatchClusterPropertyExpressions { + expr := &term.MatchClusterPropertyExpressions[j] + if err := validatePropertyKey(expr.Key); err != nil { + return err + } + switch expr.Operator { + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNotIn: + // In and NotIn test set membership by comparing the property's value as a string. + // A resource property is a quantity whose canonical string form is not unique -- + // 8000m and 8 are equal -- so a membership test against it silently mismatches + // (In: ["8000m"] never matches a cluster reporting "8"), which would misreport the + // selector as unfulfilled and add a spurious claim. The API reserves these + // operators for labels and string-based properties, so a resource-prefixed key + // here is rejected rather than evaluated. + if strings.HasPrefix(expr.Key, propertyprovider.ResourcePropertyNamePrefix) { + return fmt.Errorf("operator %s is not applicable to the resource property %s: resource properties are numeric, use Gt, Lt, Ge, Le, Eq, or Ne", expr.Operator, expr.Key) + } + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist: + // Presence operators carry no numeric constraints and apply uniformly to string + // and resource properties; the CRD validation rules cover their value arity. + // Unlike In/NotIn above, these are intentionally left valid on a resource key: a + // presence check is a map-key lookup, immune to the quantity-canonicalization that + // breaks membership tests, and the evaluation path (stringPropertyValueFrom) already + // supports it. The API type doc frames the string operators more narrowly; leaving + // that wording in step with this enforced rule is a deferred, separate change -- do + // not "reconcile" the two by banning presence checks on resource keys here. + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorLt, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGe, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorLe, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorEq, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNe: + if len(expr.Values) != 1 { + return fmt.Errorf("cluster property expression on key %s must have exactly one value for operator %s", expr.Key, expr.Operator) + } + if _, err := resource.ParseQuantity(expr.Values[0]); err != nil { + return fmt.Errorf("value %s in cluster property expression on key %s is not a valid quantity: %w", expr.Values[0], expr.Key, err) + } + default: + return fmt.Errorf("invalid operator %s in cluster property expression on key %s", expr.Operator, expr.Key) + } + } + } + return nil +} + +// validatePropertyKey checks that a resource-property key names a known capacity type and a +// non-empty resource; non-resource property keys carry no structural constraints. +func validatePropertyKey(key string) error { + if !strings.HasPrefix(key, propertyprovider.ResourcePropertyNamePrefix) { + // A non-resource property key names a property a cluster reports. Admission accepts a + // present-but-malformed key (an empty string, say) as an ordinary map entry, and a + // DoesNotExist expression on such a key would then report the property absent on every + // cluster -- matching all of them and potentially withdrawing a still-needed claim. + // Reject it here rather than treat it as an always-absent property. + return validateNonResourcePropertyName(key) + } + name := strings.TrimPrefix(key, propertyprovider.ResourcePropertyNamePrefix) + capacityType, resourceName, ok := strings.Cut(name, "-") + if !ok || capacityType == "" || resourceName == "" { + return fmt.Errorf("invalid resource property name %s in cluster property expression", key) + } + switch capacityType { + case propertyprovider.TotalCapacityName, propertyprovider.AllocatableCapacityName, propertyprovider.AvailableCapacityName: + default: + return fmt.Errorf("invalid capacity type %s in cluster property expression key %s", capacityType, key) + } + // A non-empty resource-name portion is not enough: a key such as + // resources.kubernetes-fleet.io/allocatable-not a resource clears the checks above yet names a + // resource no cluster can ever report, so Exists would raise an unfulfillable claim and + // DoesNotExist would match every cluster. The resource name is what indexes into the cluster's + // reported ResourceList, so it -- not the whole key -- is validated as a qualified name: this + // rejects the malformed case while still admitting a domain-qualified extended resource such as + // nvidia.com/gpu, which a whole-key check would reject for carrying a second slash. This is a + // deliberate divergence from validateName in pkg/utils/validator, which validates the whole key + // and so rejects such extended resources; do not "align" this with that validator, as that would + // reintroduce the rejection of a resource the cluster genuinely reports. + if errs := validation.IsQualifiedName(resourceName); len(errs) != 0 { + return fmt.Errorf("invalid resource name %s in cluster property expression key %s: %s", resourceName, key, strings.Join(errs, "; ")) + } + return nil +} + +// validateNonResourcePropertyName checks a non-resource cluster property name against the grammar +// the property provider uses: one or more slash-separated segments, where an optional first segment +// is a DNS subdomain or qualified name and every remaining segment is a qualified name. This mirrors +// validateName in pkg/utils/validator so that a multi-segment provider name such as the Azure +// per-SKU key "kubernetes.azure.com/vm-sizes/Standard_D2s_v3/count" is accepted -- IsQualifiedName +// alone rejects any name carrying more than one slash and would misflag those legitimate keys. +func validateNonResourcePropertyName(name string) error { + segs := strings.Split(name, "/") + if len(segs) <= 1 { + if errs := validation.IsQualifiedName(name); len(errs) != 0 { + return fmt.Errorf("invalid cluster property key %q: %s", name, strings.Join(errs, "; ")) + } + return nil + } + prefix := segs[0] + if len(validation.IsDNS1123Subdomain(prefix)) != 0 && len(validation.IsQualifiedName(prefix)) != 0 { + return fmt.Errorf("invalid cluster property key %q: first segment %q is neither a valid DNS subdomain nor a valid qualified name", name, prefix) + } + for _, seg := range segs[1:] { + if errs := validation.IsQualifiedName(seg); len(errs) != 0 { + return fmt.Errorf("invalid cluster property key %q: segment %q is not valid: %s", name, seg, strings.Join(errs, "; ")) + } + } + return nil +} + +// matchesTerms reports whether the member cluster satisfies any of the given selector terms +// (terms are ORed). An empty term list matches every cluster, in consistency with the API +// contract for both cluster selectors and cluster claims. +func matchesTerms(cluster *clusterv1beta1.MemberCluster, terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm) (bool, error) { + if len(terms) == 0 { + return true, nil + } + // A term that cannot be evaluated does not veto the disjunction: a later term matching on + // its own proves the whole OR true regardless of what the broken term would have said. Only + // when no term matches does the error surface, since "no match" cannot be distinguished + // from "could not tell" at that point. + var firstErr error + for i := range terms { + matched, err := matchesTerm(cluster, &terms[i]) + if err != nil { + if firstErr == nil { + firstErr = fmt.Errorf("selector term %d: %w", i, err) + } + continue + } + if matched { + return true, nil + } + } + return false, firstErr +} + +// matchesTerm reports whether the member cluster satisfies a single selector term; the +// requirement groups within a term (matchLabels, matchLabelExpressions, and +// matchClusterPropertyExpressions) are ANDed, as are the requirements within each group. +func matchesTerm(cluster *clusterv1beta1.MemberCluster, term *kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm) (bool, error) { + if !labels.SelectorFromSet(term.MatchLabels).Matches(labels.Set(cluster.Labels)) { + return false, nil + } + + matched, err := labelExpressionsMatch(cluster, term.MatchLabelExpressions) + if err != nil || !matched { + return matched, err + } + + return propertyExpressionsMatch(cluster, term.MatchClusterPropertyExpressions) +} + +// labelExpressionsMatch evaluates matchLabelExpressions against the cluster's labels, using the +// upstream label selector semantics (e.g., NotIn and DoesNotExist match when the key is absent). +func labelExpressionsMatch(cluster *clusterv1beta1.MemberCluster, exprs []kfplacementv1alpha1.LabelClusterPropertyExpression) (bool, error) { + for i := range exprs { + expr := &exprs[i] + op, err := labelSelectionOperatorFor(expr.Operator) + if err != nil { + return false, fmt.Errorf("invalid label expression on key %s: %w", expr.Key, err) + } + req, err := labels.NewRequirement(expr.Key, op, expr.Values) + if err != nil { + return false, fmt.Errorf("invalid label expression on key %s: %w", expr.Key, err) + } + if !req.Matches(labels.Set(cluster.Labels)) { + return false, nil + } + } + return true, nil +} + +// labelSelectionOperatorFor maps a label-applicable expression operator to its upstream label +// selection counterpart; numeric operators are rejected, as the API reserves them for cluster +// properties. +func labelSelectionOperatorFor(op kfplacementv1alpha1.LabelClusterPropertyExpressionOperator) (selection.Operator, error) { + switch op { + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn: + return selection.In, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNotIn: + return selection.NotIn, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists: + return selection.Exists, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist: + return selection.DoesNotExist, nil + default: + return "", fmt.Errorf("operator %s is not applicable to labels", op) + } +} + +// propertyExpressionsMatch evaluates matchClusterPropertyExpressions against the cluster's +// reported properties. String-based operators (In, NotIn, Exists, DoesNotExist) compare the raw +// property value and mirror the label selector absence semantics; numeric operators (Gt, Lt, Ge, +// Le, Eq, Ne) compare quantities, and a cluster that does not report the property does not match. +func propertyExpressionsMatch(cluster *clusterv1beta1.MemberCluster, exprs []kfplacementv1alpha1.LabelClusterPropertyExpression) (bool, error) { + for i := range exprs { + matched, err := propertyExpressionMatches(cluster, &exprs[i]) + if err != nil || !matched { + return matched, err + } + } + return true, nil +} + +func propertyExpressionMatches(cluster *clusterv1beta1.MemberCluster, expr *kfplacementv1alpha1.LabelClusterPropertyExpression) (bool, error) { + switch expr.Operator { + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNotIn, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist: + return stringPropertyMatches(cluster, expr) + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorLt, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGe, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorLe, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorEq, + kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNe: + return numericPropertyMatches(cluster, expr) + default: + return false, fmt.Errorf("invalid operator %s in cluster property expression on key %s", expr.Operator, expr.Key) + } +} + +// stringPropertyMatches evaluates a string-based operator against a cluster property value, +// mirroring the absence semantics of label selectors (NotIn and DoesNotExist match when the +// property is not reported). +func stringPropertyMatches(cluster *clusterv1beta1.MemberCluster, expr *kfplacementv1alpha1.LabelClusterPropertyExpression) (bool, error) { + value, found := stringPropertyValueFrom(cluster, expr.Key) + + switch expr.Operator { + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists: + return found, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist: + return !found, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn: + if !found { + return false, nil + } + return slices.Contains(expr.Values, value), nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNotIn: + if !found { + return true, nil + } + return !slices.Contains(expr.Values, value), nil + default: + // Callers dispatch only string-based operators here. + return false, fmt.Errorf("invalid string operator %s in cluster property expression on key %s", expr.Operator, expr.Key) + } +} + +// numericPropertyMatches evaluates a numeric operator against a cluster property value; the +// values are compared as Kubernetes resource quantities. A cluster that does not report the +// property does not match, in consistency with the property selector semantics of the +// scheduler's cluster affinity plugin. +func numericPropertyMatches(cluster *clusterv1beta1.MemberCluster, expr *kfplacementv1alpha1.LabelClusterPropertyExpression) (bool, error) { + observed, err := numericPropertyValueFrom(cluster, expr.Key) + if err != nil { + return false, err + } + if observed == nil { + return false, nil + } + + if len(expr.Values) != 1 { + // The CRD validation enforces exactly one value for numeric operators; this guards + // against objects that bypassed admission. + return false, fmt.Errorf("cluster property expression on key %s must have exactly one value for operator %s", expr.Key, expr.Operator) + } + expected, err := resource.ParseQuantity(expr.Values[0]) + if err != nil { + return false, fmt.Errorf("value %s in cluster property expression on key %s is not a valid quantity: %w", expr.Values[0], expr.Key, err) + } + + cmp := observed.Cmp(expected) + switch expr.Operator { + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt: + return cmp > 0, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorLt: + return cmp < 0, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGe: + return cmp >= 0, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorLe: + return cmp <= 0, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorEq: + return cmp == 0, nil + case kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNe: + return cmp != 0, nil + default: + // Callers dispatch only numeric operators here. + return false, fmt.Errorf("invalid numeric operator %s in cluster property expression on key %s", expr.Operator, expr.Key) + } +} + +// stringPropertyValueFrom returns the raw value of a non-resource cluster property. +func stringPropertyValueFrom(cluster *clusterv1beta1.MemberCluster, name string) (string, bool) { + if strings.HasPrefix(name, propertyprovider.ResourcePropertyNamePrefix) { + // Resource properties are quantities; string-based operators still see them through + // their canonical string form so that Exists/DoesNotExist work uniformly. + q, err := resourceUsageValueFrom(cluster, strings.TrimPrefix(name, propertyprovider.ResourcePropertyNamePrefix)) + if err != nil || q == nil { + return "", false + } + return q.String(), true + } + v, found := cluster.Status.Properties[clusterv1beta1.PropertyName(name)] + if !found { + return "", false + } + return v.Value, true +} + +// numericPropertyValueFrom retrieves a property value, resource or non-resource, from a member +// cluster as a quantity. It returns nil (and no error) when the cluster does not report the +// property. +func numericPropertyValueFrom(cluster *clusterv1beta1.MemberCluster, name string) (*resource.Quantity, error) { + if strings.HasPrefix(name, propertyprovider.ResourcePropertyNamePrefix) { + q, err := resourceUsageValueFrom(cluster, strings.TrimPrefix(name, propertyprovider.ResourcePropertyNamePrefix)) + if err != nil { + return nil, fmt.Errorf("failed to retrieve resource property %s from cluster %s: %w", name, cluster.Name, err) + } + return q, nil + } + + v, found := cluster.Status.Properties[clusterv1beta1.PropertyName(name)] + if !found { + return nil, nil + } + q, err := resource.ParseQuantity(v.Value) + if err != nil { + return nil, fmt.Errorf("value %s of property %s from cluster %s is not a valid quantity: %w", v.Value, name, cluster.Name, err) + } + return &q, nil +} + +// resourceUsageValueFrom retrieves a resource usage value from a member cluster; the name is the +// property name with the resource property prefix removed, e.g. "allocatable-cpu". It returns +// nil (and no error) when the cluster does not report the resource. +func resourceUsageValueFrom(cluster *clusterv1beta1.MemberCluster, name string) (*resource.Quantity, error) { + // Resource properties follow the `[PREFIX]/[CAPACITY_TYPE]-[RESOURCE_NAME]` naming rule, + // e.g. `resources.kubernetes-fleet.io/allocatable-cpu`; the prefix has been removed by the + // caller. + capacityType, resourceName, found := strings.Cut(name, "-") + if !found || capacityType == "" || resourceName == "" { + return nil, fmt.Errorf("invalid resource property name: %s", name) + } + + var q resource.Quantity + var reported bool + switch capacityType { + case propertyprovider.TotalCapacityName: + q, reported = cluster.Status.ResourceUsage.Capacity[corev1.ResourceName(resourceName)] + case propertyprovider.AllocatableCapacityName: + q, reported = cluster.Status.ResourceUsage.Allocatable[corev1.ResourceName(resourceName)] + case propertyprovider.AvailableCapacityName: + q, reported = cluster.Status.ResourceUsage.Available[corev1.ResourceName(resourceName)] + default: + return nil, fmt.Errorf("invalid capacity type %s in resource property name %s", capacityType, name) + } + if !reported { + return nil, nil + } + return &q, nil +} diff --git a/pkg/v1/controllers/placementpolicy/selector_test.go b/pkg/v1/controllers/placementpolicy/selector_test.go new file mode 100644 index 000000000..99a7e5fb2 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/selector_test.go @@ -0,0 +1,792 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "math" + "testing" + + "github.com/google/go-cmp/cmp" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/propertyprovider" +) + +const ( + regionLabel = "topology.kubernetes.io/region" + envLabel = "environment" +) + +// testCluster builds a MemberCluster with the given labels, non-resource properties, and +// allocatable resource usage for matcher tests. +func testCluster(name string, clusterLabels map[string]string, properties map[string]string, allocatable corev1.ResourceList) *clusterv1beta1.MemberCluster { + mc := &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Labels: clusterLabels, + }, + } + if len(properties) > 0 { + mc.Status.Properties = map[clusterv1beta1.PropertyName]clusterv1beta1.PropertyValue{} + for k, v := range properties { + mc.Status.Properties[clusterv1beta1.PropertyName(k)] = clusterv1beta1.PropertyValue{Value: v} + } + } + mc.Status.ResourceUsage.Allocatable = allocatable + return mc +} + +func TestResolveCounts(t *testing.T) { + testCases := []struct { + name string + selector *kfplacementv1alpha1.ClusterSelector + want resolvedCounts + wantError bool + }{ + { + name: "count defaults to one when unset", + selector: &kfplacementv1alpha1.ClusterSelector{}, + want: resolvedCounts{desired: 1, minimum: 1}, + }, + { + name: "integer count", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromInt32(3)), + }, + want: resolvedCounts{desired: 3, minimum: 3}, + }, + { + name: "integer count with explicit minCount", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromInt32(3)), + MinCount: ptr.To(int32(2)), + }, + want: resolvedCounts{desired: 3, minimum: 2}, + }, + { + name: "count All defaults minCount to one", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromString("All")), + }, + want: resolvedCounts{selectAll: true, minimum: 1}, + }, + { + name: "count All with explicit minCount", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromString("All")), + MinCount: ptr.To(int32(5)), + }, + want: resolvedCounts{selectAll: true, minimum: 5}, + }, + { + name: "invalid string count", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromString("Some")), + }, + wantError: true, + }, + { + // The CRD pattern admits digit strings, so a manifest with count: "3" passes admission + // and must resolve the same as the integer form rather than being rejected. + name: "digit string count is parsed like an integer", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromString("3")), + }, + want: resolvedCounts{desired: 3, minimum: 3}, + }, + { + name: "digit string count with explicit minCount", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromString("3")), + MinCount: ptr.To(int32(2)), + }, + want: resolvedCounts{desired: 3, minimum: 2}, + }, + { + name: "string count at the upper bound", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromString("999")), + }, + want: resolvedCounts{desired: 999, minimum: 999}, + }, + { + // Parseable as an integer but below the floor: the string branch must reject it just as + // the integer branch rejects a non-positive count. + name: "non-positive digit string count", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromString("0")), + }, + wantError: true, + }, + { + name: "non-positive integer count", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromInt32(0)), + }, + wantError: true, + }, + { + // The CRD pattern bounds only the string form; an integer past the limit bypasses + // admission, and unchecked it would overflow the aggregated desired total. + name: "integer count above the upper bound", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromInt32(1000)), + }, + wantError: true, + }, + { + name: "maximum int32 integer count", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromInt32(math.MaxInt32)), + }, + wantError: true, + }, + { + name: "integer count at the upper bound", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromInt32(999)), + }, + want: resolvedCounts{desired: 999, minimum: 999}, + }, + { + // A digit string past the bound (only reachable by bypassing admission) is rejected. + name: "digit string count above the upper bound", + selector: &kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(intstr.FromString("1000")), + }, + wantError: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := resolveCounts(tc.selector) + if tc.wantError { + if err == nil { + t.Fatalf("resolveCounts(%v) = %v, nil, want error", tc.selector, got) + } + return + } + if err != nil { + t.Fatalf("resolveCounts(%v) returned unexpected error: %v", tc.selector, err) + } + if diff := cmp.Diff(got, tc.want, cmp.AllowUnexported(resolvedCounts{})); diff != "" { + t.Errorf("resolveCounts(%v) mismatch (-got, +want):\n%s", tc.selector, diff) + } + }) + } +} + +// TestMatchesTermsWithBrokenTerms covers the OR semantics under evaluation errors: one broken +// term must not veto a later term that matches on its own, and the error surfaces only when no +// term matches, since only then does the broken term's answer matter. +func TestMatchesTermsWithBrokenTerms(t *testing.T) { + eastusCluster := testCluster("eastus-cluster", map[string]string{regionLabel: "eastus"}, nil, nil) + // An operator the evaluator does not know fails unconditionally, unlike a numeric + // comparison, whose absence semantics can swallow a malformed value into a clean no-match. + brokenTerm := kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: "resources.kubefleet.dev/total-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperator("Bogus"), Values: []string{"1"}}, + }, + } + + testCases := []struct { + name string + terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm + want bool + wantErr bool + }{ + { + name: "a broken term does not veto a later matching term", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + brokenTerm, + {MatchLabels: map[string]string{regionLabel: "eastus"}}, + }, + want: true, + }, + { + name: "a broken term after the matching term changes nothing", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabel: "eastus"}}, + brokenTerm, + }, + want: true, + }, + { + name: "the error surfaces when no term matches", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + brokenTerm, + {MatchLabels: map[string]string{regionLabel: "westus"}}, + }, + wantErr: true, + }, + { + name: "only broken terms surface the error", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{brokenTerm}, + wantErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := matchesTerms(eastusCluster, tc.terms) + if tc.wantErr { + if err == nil { + t.Fatalf("matchesTerms() = %v, nil, want an error", got) + } + return + } + if err != nil { + t.Fatalf("matchesTerms() = %v, want no error", err) + } + if got != tc.want { + t.Errorf("matchesTerms() = %v, want %v", got, tc.want) + } + }) + } +} + +func TestMatchesTerms(t *testing.T) { + eastusCluster := testCluster("eastus-cluster", map[string]string{regionLabel: "eastus", envLabel: "prod"}, nil, nil) + + testCases := []struct { + name string + cluster *clusterv1beta1.MemberCluster + terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm + want bool + }{ + { + name: "empty terms match every cluster", + cluster: eastusCluster, + terms: nil, + want: true, + }, + { + name: "terms are ORed", + cluster: eastusCluster, + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabel: "westus"}}, + {MatchLabels: map[string]string{regionLabel: "eastus"}}, + }, + want: true, + }, + { + name: "no term matches", + cluster: eastusCluster, + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabel: "westus"}}, + {MatchLabels: map[string]string{regionLabel: "centralus"}}, + }, + want: false, + }, + { + name: "requirement groups within a term are ANDed", + cluster: eastusCluster, + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + { + MatchLabels: map[string]string{regionLabel: "eastus"}, + MatchLabelExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: envLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, Values: []string{"dev"}}, + }, + }, + }, + want: false, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := matchesTerms(tc.cluster, tc.terms) + if err != nil { + t.Fatalf("matchesTerms(%s, %v) returned unexpected error: %v", tc.cluster.Name, tc.terms, err) + } + if got != tc.want { + t.Errorf("matchesTerms(%s, %v) = %t, want %t", tc.cluster.Name, tc.terms, got, tc.want) + } + }) + } +} + +func TestMatchesTermLabelExpressions(t *testing.T) { + labeledCluster := testCluster("labeled-cluster", map[string]string{regionLabel: "eastus"}, nil, nil) + + testCases := []struct { + name string + cluster *clusterv1beta1.MemberCluster + expr kfplacementv1alpha1.LabelClusterPropertyExpression + want bool + wantError bool + }{ + { + name: "In matches present value", + cluster: labeledCluster, + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: regionLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, Values: []string{"eastus", "westus"}, + }, + want: true, + }, + { + name: "In does not match an absent key", + cluster: labeledCluster, + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: envLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, Values: []string{"prod"}, + }, + want: false, + }, + { + name: "NotIn matches an absent key", + cluster: labeledCluster, + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: envLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNotIn, Values: []string{"prod"}, + }, + want: true, + }, + { + name: "Exists matches a present key", + cluster: labeledCluster, + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: regionLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists, + }, + want: true, + }, + { + name: "DoesNotExist matches an absent key", + cluster: labeledCluster, + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: envLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist, + }, + want: true, + }, + { + name: "numeric operator on a label expression is an error", + cluster: labeledCluster, + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: regionLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}, + }, + wantError: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + term := &kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + MatchLabelExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{tc.expr}, + } + got, err := matchesTerm(tc.cluster, term) + if tc.wantError { + if err == nil { + t.Fatalf("matchesTerm(%s, %v) = %t, nil, want error", tc.cluster.Name, term, got) + } + return + } + if err != nil { + t.Fatalf("matchesTerm(%s, %v) returned unexpected error: %v", tc.cluster.Name, term, err) + } + if got != tc.want { + t.Errorf("matchesTerm(%s, %v) = %t, want %t", tc.cluster.Name, term, got, tc.want) + } + }) + } +} + +func TestMatchesTermPropertyExpressions(t *testing.T) { + propertiedCluster := testCluster( + "propertied-cluster", + nil, + map[string]string{ + propertyprovider.NodeCountProperty: "4", + "example.com/tier": "premium", + "example.com/invalid-quantity": "not-a-number", + }, + corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("8"), + corev1.ResourceEphemeralStorage: resource.MustParse("100Gi"), + }, + ) + + testCases := []struct { + name string + expr kfplacementv1alpha1.LabelClusterPropertyExpression + want bool + wantError bool + }{ + { + name: "Gt on a non-resource property", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"3"}, + }, + want: true, + }, + { + name: "Le on a non-resource property", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorLe, Values: []string{"3"}, + }, + want: false, + }, + { + name: "Eq compares quantities semantically", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorEq, Values: []string{"8000m"}, + }, + want: true, + }, + { + name: "Ne on a reported resource property", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNe, Values: []string{"8"}, + }, + want: false, + }, + { + name: "resource names with dashes resolve correctly", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-ephemeral-storage", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGe, Values: []string{"50Gi"}, + }, + want: true, + }, + { + name: "numeric operator on an unreported property does not match", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: "example.com/unreported", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}, + }, + want: false, + }, + { + name: "numeric operator on an unreported resource capacity does not match", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.ResourcePropertyNamePrefix + "available-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}, + }, + want: false, + }, + { + name: "numeric operator on a non-numeric property value is an error", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: "example.com/invalid-quantity", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}, + }, + wantError: true, + }, + { + name: "numeric operator with a non-quantity expected value is an error", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"three"}, + }, + wantError: true, + }, + { + name: "invalid capacity type in a resource property name is an error", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.ResourcePropertyNamePrefix + "reserved-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}, + }, + wantError: true, + }, + { + name: "In on a string property", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: "example.com/tier", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, Values: []string{"premium", "standard"}, + }, + want: true, + }, + { + name: "NotIn matches an unreported property", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: "example.com/unreported", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNotIn, Values: []string{"premium"}, + }, + want: true, + }, + { + name: "Exists on a reported resource property", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists, + }, + want: true, + }, + { + name: "DoesNotExist on an unreported property", + expr: kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: "example.com/unreported", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist, + }, + want: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + term := &kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{tc.expr}, + } + got, err := matchesTerm(propertiedCluster, term) + if tc.wantError { + if err == nil { + t.Fatalf("matchesTerm(%s, %v) = %t, nil, want error", propertiedCluster.Name, term, got) + } + return + } + if err != nil { + t.Fatalf("matchesTerm(%s, %v) returned unexpected error: %v", propertiedCluster.Name, term, err) + } + if got != tc.want { + t.Errorf("matchesTerm(%s, %v) = %t, want %t", propertiedCluster.Name, term, got, tc.want) + } + }) + } +} + +func TestValidateTerms(t *testing.T) { + testCases := []struct { + name string + terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm + wantError bool + }{ + { + name: "valid mixed term", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchLabels: map[string]string{regionLabel: "eastus"}, + MatchLabelExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: envLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGe, Values: []string{"2"}}, + }, + }}, + }, + { + name: "numeric operator in a label expression", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchLabelExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: envLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}}, + }, + }}, + wantError: true, + }, + { + name: "numeric property expression with two values", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1", "2"}}, + }, + }}, + wantError: true, + }, + { + name: "numeric property expression with a non-quantity value", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"three"}}, + }, + }}, + wantError: true, + }, + { + name: "resource property key with an unknown capacity type", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.ResourcePropertyNamePrefix + "reserved-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + }}, + wantError: true, + }, + { + name: "resource property key without a resource name", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}}, + }, + }}, + wantError: true, + }, + { + name: "matchLabels with an invalid key", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchLabels: map[string]string{"not a valid key": "eastus"}, + }}, + wantError: true, + }, + { + name: "matchLabels with an invalid value", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchLabels: map[string]string{regionLabel: "not a valid value"}, + }}, + wantError: true, + }, + { + name: "non-resource property key that is empty", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: "", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist}, + }, + }}, + wantError: true, + }, + { + name: "non-resource property key that is not a qualified name", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: "not a valid key", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + }}, + wantError: true, + }, + { + name: "valid non-resource property key", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + }}, + }, + { + name: "valid multi-segment non-resource property key", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + // The Azure property provider ships per-SKU capacity keys in this multi-slash form; + // a whole-string IsQualifiedName check would wrongly reject them. + {Key: "kubernetes.azure.com/vm-sizes/Standard_D2s_v3/count", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + }}, + }, + { + name: "In on a resource property is rejected", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, Values: []string{"8000m"}}, + }, + }}, + wantError: true, + }, + { + name: "NotIn on a resource property is rejected", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNotIn, Values: []string{"8"}}, + }, + }}, + wantError: true, + }, + { + name: "Exists on a resource property is allowed", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-cpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + }}, + }, + { + name: "In on a string property is allowed", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, Values: []string{"3"}}, + }, + }}, + }, + { + name: "resource property with an invalid resource name", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-not a resource", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + }}, + wantError: true, + }, + { + name: "valid resource property with a dashed resource name", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-ephemeral-storage", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + }}, + }, + { + name: "valid resource property with a domain-qualified extended resource name", + terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + // An extended resource name carries its own slash; validating the whole key would + // reject it for a second slash even though the cluster reports it and evaluation + // resolves it. Only the resource-name portion is validated, so this is accepted. + {Key: propertyprovider.ResourcePropertyNamePrefix + "allocatable-nvidia.com/gpu", Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + }, + }}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + err := validateTerms(tc.terms) + if gotError := err != nil; gotError != tc.wantError { + t.Errorf("validateTerms(%v) error = %v, want error %t", tc.terms, err, tc.wantError) + } + }) + } +} + +// TestValidateTermsDeterministicError guards against a reconcile hot-loop: the validation error +// surfaces on the policy's Scheduled condition message, so a term with more than one invalid +// matchLabels entry must always report the same one. Go randomizes map iteration, so validating in +// map order would return a different message from call to call and defeat the status no-op check. +func TestValidateTermsDeterministicError(t *testing.T) { + terms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{ + MatchLabels: map[string]string{ + "bad key one": "eastus", + "bad key two": "westus", + }, + }} + first := validateTerms(terms) + if first == nil { + t.Fatalf("validateTerms(%v) error = nil, want error", terms) + } + // Many iterations so a randomized order would almost certainly have produced a different + // message at least once. + for i := 0; i < 100; i++ { + got := validateTerms(terms) + if got == nil || got.Error() != first.Error() { + t.Fatalf("validateTerms(%v) error = %v, want stable %v", terms, got, first) + } + } +} + +func TestMatchesTermMixedGroups(t *testing.T) { + cluster := testCluster( + "mixed-cluster", + map[string]string{regionLabel: "eastus"}, + map[string]string{propertyprovider.NodeCountProperty: "4"}, + nil, + ) + + term := &kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + MatchLabels: map[string]string{regionLabel: "eastus"}, + MatchLabelExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: envLabel, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist}, + }, + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.NodeCountProperty, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGe, Values: []string{"4"}}, + }, + } + + got, err := matchesTerm(cluster, term) + if err != nil { + t.Fatalf("matchesTerm(%s, %v) returned unexpected error: %v", cluster.Name, term, err) + } + if !got { + t.Errorf("matchesTerm(%s, %v) = false, want true", cluster.Name, term) + } +} diff --git a/pkg/v1/controllers/placementpolicy/status.go b/pkg/v1/controllers/placementpolicy/status.go new file mode 100644 index 000000000..a7f817ff8 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/status.go @@ -0,0 +1,210 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "fmt" + "maps" + "slices" + + apiequality "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/condition" +) + +// resourceStatus summarizes, for the policy status, how far the policy's latest resource revision +// has reached its target clusters. +type resourceStatus struct { + // latestRevision names the policy's latest resource snapshot. + latestRevision string + // current reports whether that snapshot still matches the selected resources on the hub. When + // it does not, the resources have changed since and wait for a rollout to take a new snapshot. + current bool + // targets counts the clusters that should hold a binding, including a bound cluster kept while + // it is unschedulable: it still holds the resources, so it still counts against the status. + targets int32 + // synchronized counts the target clusters whose binding has the latest revision applied. + synchronized int32 + // outdated counts the target clusters whose binding still names an older revision; the other + // unsynchronized ones have the latest revision yet to apply, or no binding yet. + outdated int32 + // available counts the synchronized target clusters whose resources also passed the + // availability check; it never exceeds synchronized. + available int32 +} + +// summarizeResourceStatus counts, among the bindings of the clusters that should hold one, those +// that have the latest revision synchronized and, of those, the ones whose resources are +// available. A binding is synchronized only when it names the latest revision, the work generator +// has processed that revision, and its Synchronized condition is true for the binding's current +// generation; a binding still rolling out an older revision, or one that has not been processed +// yet, is not. Nor is a binding being deleted, whose conditions outlive the removal of its +// resources, or one still carrying an older sync strategy than the policy's, whose conditions +// describe the old one. Availability is judged only for the latest revision, as placements judge +// it only once their resources are synchronized. +func summarizeResourceStatus(latestRevision string, current bool, syncStrategy *kfplacementv1alpha1.SyncStrategy, targets map[string][]int, bindings []kfplacementv1alpha1.PlacementBindingAccessor) resourceStatus { + status := resourceStatus{latestRevision: latestRevision, current: current, targets: int32Len(slices.Collect(maps.Keys(targets)))} + for _, binding := range bindings { + if !countsTowardStatus(binding, syncStrategy, targets) { + continue + } + if binding.GetSpec().ResourceSnapshotName != latestRevision { + status.outdated++ + continue + } + bindingStatus := binding.GetStatus() + gen := binding.GetGeneration() + processed := bindingStatus.LastProcessedResourceSnapshotName + if processed == nil || *processed != latestRevision || + !condition.IsConditionStatusTrue(meta.FindStatusCondition(bindingStatus.Conditions, kfplacementv1alpha1.PlacementBindingCondTypeSynchronized), gen) { + continue + } + status.synchronized++ + if condition.IsConditionStatusTrue(meta.FindStatusCondition(bindingStatus.Conditions, kfplacementv1alpha1.PlacementBindingCondTypeAvailable), gen) { + status.available++ + } + } + return status +} + +// summarizeAvailability summarizes the resource status of a round whose selected resources cannot +// be collected, so there is no latest revision to judge the bindings against. No target cluster +// counts as synchronized, but availability depends only on the bindings, so it is still counted +// for whatever revision each binding holds, the way a Deployment reports its ready pods whatever +// their revision. +func summarizeAvailability(syncStrategy *kfplacementv1alpha1.SyncStrategy, targets map[string][]int, bindings []kfplacementv1alpha1.PlacementBindingAccessor) resourceStatus { + status := resourceStatus{targets: int32Len(slices.Collect(maps.Keys(targets)))} + for _, binding := range bindings { + if countsTowardStatus(binding, syncStrategy, targets) && + condition.IsConditionStatusTrue(meta.FindStatusCondition(binding.GetStatus().Conditions, kfplacementv1alpha1.PlacementBindingCondTypeAvailable), binding.GetGeneration()) { + status.available++ + } + } + return status +} + +// countsTowardStatus reports whether a binding speaks for its cluster in the policy status: the +// cluster is a target, and the binding is neither being deleted, as its conditions outlive the +// removal of its resources, nor still on an older sync strategy than the policy's, as its +// conditions describe the old one. +func countsTowardStatus(binding kfplacementv1alpha1.PlacementBindingAccessor, syncStrategy *kfplacementv1alpha1.SyncStrategy, targets map[string][]int) bool { + if _, isTarget := targets[binding.GetSpec().ClusterName]; !isTarget { + return false + } + return binding.GetDeletionTimestamp().IsZero() && apiequality.Semantic.DeepEqual(binding.GetSpec().SyncStrategy, syncStrategy) +} + +// synchronizedCondition builds the policy's Synchronized condition: true when the latest revision +// still matches the selected resources and every target cluster has it synchronized. Per the FEP, +// a resource change alone takes no snapshot; it waits for a rollout, and until then the condition +// reports ResourcesChangedPendingRollout. Otherwise a false condition reports +// SomeClustersOnOlderResourceRevision while any target cluster is still on an older revision, and +// FailedToSynchronizeSomeClusters while the rest have yet to apply the latest one. With no target +// cluster there is nothing to tell apart from the hub, so the condition is true. +func (s resourceStatus) synchronizedCondition(generation int64) metav1.Condition { + cond := metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters, + ObservedGeneration: generation, + } + switch { + case s.targets > 0 && !s.current: + cond.Reason = kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout + cond.Message = fmt.Sprintf("The selected resources have changed since resource revision %s and are waiting for a rollout to reach the target clusters", s.latestRevision) + case s.synchronized < s.targets: + if s.outdated > 0 { + cond.Reason = kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision + } + cond.Message = fmt.Sprintf("%d of %d target clusters have the latest resource revision %s synchronized (still on an older revision: %d, yet to apply the latest one: %d)", + s.synchronized, s.targets, s.latestRevision, s.outdated, s.targets-s.synchronized-s.outdated) + case s.targets == 0: + cond.Status = metav1.ConditionTrue + cond.Reason = kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized + cond.Message = fmt.Sprintf("No cluster is targeted; the latest resource revision is %s", s.latestRevision) + default: + cond.Status = metav1.ConditionTrue + cond.Reason = kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized + cond.Message = fmt.Sprintf("All %d target clusters have the latest resource revision %s synchronized", s.targets, s.latestRevision) + } + return cond +} + +// resourceCollectedCondition builds the policy's ResourceCollected condition: true when the selected +// resources were collected and the latest snapshot taken or found, whether or not it still matches +// them; false when they cannot be collected for good (until the user or the resources change). +func resourceCollectedCondition(generation int64, selectionErr error) metav1.Condition { + if selectionErr != nil { + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, + Status: metav1.ConditionFalse, + Reason: kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonFailedToCollectSomeResources, + Message: "The selected resources cannot be collected: " + selectionErr.Error(), + ObservedGeneration: generation, + } + } + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected, + Message: "All selected resources have been collected", + ObservedGeneration: generation, + } +} + +// selectionFailedCondition builds the policy's Synchronized condition for a round whose selected +// resources cannot be collected for good (until the user or the resources change): the last +// round's verdict no longer describes them. It is reported as FailedToSynchronizeSomeClusters; the +// ResourceCollected condition carries the cause. +func selectionFailedCondition(generation int64, err error) metav1.Condition { + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters, + Message: "The selected resources cannot be snapshotted: " + err.Error(), + ObservedGeneration: generation, + } +} + +// availableCondition builds the policy's Available condition: true when the resources of every +// target cluster have passed the availability check, on the latest revision or, when the selected +// resources cannot be collected, on whatever revision the cluster holds. +func (s resourceStatus) availableCondition(generation int64) metav1.Condition { + if s.available < s.targets { + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable, + Message: fmt.Sprintf("The resources are available on %d of %d target clusters", s.available, s.targets), + ObservedGeneration: generation, + } + } + message := fmt.Sprintf("The resources are available on all %d target clusters", s.targets) + if s.targets == 0 { + message = "No cluster is targeted" + } + return metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonAllClustersAvailable, + Message: message, + ObservedGeneration: generation, + } +} diff --git a/pkg/v1/controllers/placementpolicy/status_test.go b/pkg/v1/controllers/placementpolicy/status_test.go new file mode 100644 index 000000000..36ae07048 --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/status_test.go @@ -0,0 +1,261 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "errors" + "testing" + + "github.com/google/go-cmp/cmp" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// progressedBinding builds a binding of generation 2 to the cluster, rolling out the revision, with +// the given processed revision and condition statuses; an empty status leaves the condition out, and +// staleGen reports the conditions for the previous generation. +func progressedBinding(cluster, revision string, processed *string, synchronized, available metav1.ConditionStatus, staleGen bool) kfplacementv1alpha1.PlacementBindingAccessor { + binding := &kfplacementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{Name: cluster, Generation: 2}, + Spec: kfplacementv1alpha1.PlacementBindingSpec{ClusterName: cluster, ResourceSnapshotName: revision}, + Status: kfplacementv1alpha1.PlacementBindingStatus{LastProcessedResourceSnapshotName: processed}, + } + gen := int64(2) + if staleGen { + gen = 1 + } + for condType, status := range map[string]metav1.ConditionStatus{ + kfplacementv1alpha1.PlacementBindingCondTypeSynchronized: synchronized, + kfplacementv1alpha1.PlacementBindingCondTypeAvailable: available, + } { + if status != "" { + binding.Status.Conditions = append(binding.Status.Conditions, metav1.Condition{Type: condType, Status: status, ObservedGeneration: gen, Reason: "Test"}) + } + } + return binding +} + +func TestSummarizeResourceStatus(t *testing.T) { + const latest, older = "pp-snapshot-1", "pp-snapshot-0" + targets := map[string][]int{"done": {0}, "rolling": {0}, "unprocessed": {0}, "stale-gen": {0}, "failing": {0}, "no-binding-yet": {0}, "terminating": {0}, "old-strategy": {0}} + terminating := progressedBinding("terminating", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, false) + terminating.SetDeletionTimestamp(ptr.To(metav1.Now())) + oldStrategy := progressedBinding("old-strategy", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, false) + oldStrategy.GetSpec().SyncStrategy = &kfplacementv1alpha1.SyncStrategy{ApplyMethod: kfplacementv1alpha1.ApplyMethodServerSideApply} + bindings := []kfplacementv1alpha1.PlacementBindingAccessor{ + progressedBinding("done", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, false), + // Still on the older revision: available there, but not synchronized with the latest one. + progressedBinding("rolling", older, ptr.To(older), metav1.ConditionTrue, metav1.ConditionTrue, false), + // Moved to the latest revision, which the work generator has yet to process. + progressedBinding("unprocessed", latest, ptr.To(older), metav1.ConditionTrue, metav1.ConditionTrue, false), + progressedBinding("stale-gen", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, true), + progressedBinding("failing", latest, ptr.To(latest), metav1.ConditionFalse, metav1.ConditionUnknown, false), + // Not a target anymore: on its way out, and not counted at all. + progressedBinding("leaving", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, false), + // Being deleted: its conditions outlive the removal of its resources. + terminating, + // Not yet updated to the policy's sync strategy: its conditions describe the old one. + oldStrategy, + } + + got := summarizeResourceStatus(latest, true, nil, targets, bindings) + want := resourceStatus{latestRevision: latest, current: true, targets: 8, synchronized: 1, outdated: 1, available: 1} + if diff := cmp.Diff(want, got, cmp.AllowUnexported(resourceStatus{})); diff != "" { + t.Errorf("summarizeResourceStatus() mismatch (-want +got):\n%s", diff) + } +} + +// TestSummarizeAvailability pins that a round without a latest revision still counts the target +// clusters whose resources are available, whatever revision they are on, and counts none as +// synchronized. +func TestSummarizeAvailability(t *testing.T) { + const latest, older = "pp-snapshot-1", "pp-snapshot-0" + targets := map[string][]int{"done": {0}, "rolling": {0}, "stale-gen": {0}, "failing": {0}, "terminating": {0}, "old-strategy": {0}} + terminating := progressedBinding("terminating", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, false) + terminating.SetDeletionTimestamp(ptr.To(metav1.Now())) + oldStrategy := progressedBinding("old-strategy", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, false) + oldStrategy.GetSpec().SyncStrategy = &kfplacementv1alpha1.SyncStrategy{ApplyMethod: kfplacementv1alpha1.ApplyMethodServerSideApply} + bindings := []kfplacementv1alpha1.PlacementBindingAccessor{ + progressedBinding("done", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, false), + // Available on the older revision, which still counts without a latest one to judge against. + progressedBinding("rolling", older, ptr.To(older), metav1.ConditionTrue, metav1.ConditionTrue, false), + progressedBinding("stale-gen", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, true), + progressedBinding("failing", latest, ptr.To(latest), metav1.ConditionFalse, metav1.ConditionUnknown, false), + progressedBinding("leaving", latest, ptr.To(latest), metav1.ConditionTrue, metav1.ConditionTrue, false), + terminating, + oldStrategy, + } + + got := summarizeAvailability(nil, targets, bindings) + want := resourceStatus{targets: 6, available: 2} + if diff := cmp.Diff(want, got, cmp.AllowUnexported(resourceStatus{})); diff != "" { + t.Errorf("summarizeAvailability() mismatch (-want +got):\n%s", diff) + } +} + +func TestResourceStatusConditions(t *testing.T) { + testCases := []struct { + name string + status resourceStatus + wantSync metav1.Condition + wantAvailable metav1.Condition + }{ + { + name: "all target clusters synchronized and available", + status: resourceStatus{latestRevision: "r1", current: true, targets: 2, synchronized: 2, available: 2}, + wantSync: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, Status: metav1.ConditionTrue, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized, + Message: "All 2 target clusters have the latest resource revision r1 synchronized", + }, + wantAvailable: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, Status: metav1.ConditionTrue, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonAllClustersAvailable, + Message: "The resources are available on all 2 target clusters", + }, + }, + { + name: "rollout in progress", + status: resourceStatus{latestRevision: "r1", current: true, targets: 4, synchronized: 1, outdated: 2, available: 2}, + wantSync: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, Status: metav1.ConditionFalse, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision, + Message: "1 of 4 target clusters have the latest resource revision r1 synchronized (still on an older revision: 2, yet to apply the latest one: 1)", + }, + wantAvailable: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, Status: metav1.ConditionFalse, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable, + Message: "The resources are available on 2 of 4 target clusters", + }, + }, + { + name: "latest revision yet to apply on some target clusters", + status: resourceStatus{latestRevision: "r1", current: true, targets: 2, synchronized: 1, available: 1}, + wantSync: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, Status: metav1.ConditionFalse, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters, + Message: "1 of 2 target clusters have the latest resource revision r1 synchronized (still on an older revision: 0, yet to apply the latest one: 1)", + }, + wantAvailable: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, Status: metav1.ConditionFalse, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable, + Message: "The resources are available on 1 of 2 target clusters", + }, + }, + { + name: "resources changed on the hub since the latest revision", + status: resourceStatus{latestRevision: "r1", current: false, targets: 2, synchronized: 2, available: 2}, + wantSync: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, Status: metav1.ConditionFalse, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout, + Message: "The selected resources have changed since resource revision r1 and are waiting for a rollout to reach the target clusters", + }, + wantAvailable: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, Status: metav1.ConditionTrue, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonAllClustersAvailable, + Message: "The resources are available on all 2 target clusters", + }, + }, + { + name: "resources changed while some target clusters are on an older revision", + status: resourceStatus{latestRevision: "r1", current: false, targets: 2, synchronized: 1, outdated: 1, available: 1}, + wantSync: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, Status: metav1.ConditionFalse, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout, + Message: "The selected resources have changed since resource revision r1 and are waiting for a rollout to reach the target clusters", + }, + wantAvailable: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, Status: metav1.ConditionFalse, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable, + Message: "The resources are available on 1 of 2 target clusters", + }, + }, + { + name: "resources changed with no target clusters", + status: resourceStatus{latestRevision: "r0", current: false}, + wantSync: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, Status: metav1.ConditionTrue, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized, + Message: "No cluster is targeted; the latest resource revision is r0", + }, + wantAvailable: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, Status: metav1.ConditionTrue, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonAllClustersAvailable, + Message: "No cluster is targeted", + }, + }, + { + name: "no target clusters", + status: resourceStatus{latestRevision: "r0", current: true}, + wantSync: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, Status: metav1.ConditionTrue, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized, + Message: "No cluster is targeted; the latest resource revision is r0", + }, + wantAvailable: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, Status: metav1.ConditionTrue, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonAllClustersAvailable, + Message: "No cluster is targeted", + }, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if diff := cmp.Diff(tc.wantSync, tc.status.synchronizedCondition(3)); diff != "" { + t.Errorf("synchronizedCondition() mismatch (-want +got):\n%s", diff) + } + if diff := cmp.Diff(tc.wantAvailable, tc.status.availableCondition(3)); diff != "" { + t.Errorf("availableCondition() mismatch (-want +got):\n%s", diff) + } + }) + } +} + +func TestResourceCollectedCondition(t *testing.T) { + testCases := []struct { + name string + selectionErr error + want metav1.Condition + }{ + { + name: "collected", + want: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, Status: metav1.ConditionTrue, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected, + Message: "All selected resources have been collected", + }, + }, + { + name: "cannot be collected", + selectionErr: errors.New("configmaps \"app\" not found"), + want: metav1.Condition{ + Type: kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, Status: metav1.ConditionFalse, ObservedGeneration: 3, + Reason: kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonFailedToCollectSomeResources, + Message: "The selected resources cannot be collected: configmaps \"app\" not found", + }, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if diff := cmp.Diff(tc.want, resourceCollectedCondition(3, tc.selectionErr)); diff != "" { + t.Errorf("resourceCollectedCondition() mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/pkg/v1/controllers/placementpolicy/suite_test.go b/pkg/v1/controllers/placementpolicy/suite_test.go new file mode 100644 index 000000000..27d0c38bd --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/suite_test.go @@ -0,0 +1,149 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + "flag" + "path/filepath" + "testing" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/manager" + "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + testNamespace = "placement-policy-integration-test" +) + +var ( + cfg *rest.Config + mgr manager.Manager + k8sClient client.Client + testEnv *envtest.Environment + ctx context.Context + cancel context.CancelFunc +) + +// snapshotStub stands in for the placement resource snapshot manager: it hands back a snapshot +// named after the policy without creating anything, which is all the controller needs from it. +type snapshotStub struct{} + +func (snapshotStub) SnapshotResourcesIfNoSnapshotExists(_ context.Context, policy kfplacementv1alpha1.PlacementPolicyAccessor) ([]kfplacementv1alpha1.PlacementResourceSnapshotAccessor, bool, error) { + return []kfplacementv1alpha1.PlacementResourceSnapshotAccessor{ + &kfplacementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{Name: stubSnapshotName(policy)}}, + }, true, nil +} + +// stubSnapshotName is the snapshot name snapshotStub reports for a policy. +func stubSnapshotName(policy kfplacementv1alpha1.PlacementPolicyAccessor) string { + return policy.GetName() + "-snapshot-0" +} + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Placement Policy Controller Suite") +} + +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + By("bootstrapping test environment") + testEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("../../../../", "config", "crd", "bases")}, + ErrorIfCRDPathMissing: true, + } + + var err error + cfg, err = testEnv.Start() + Expect(err).Should(Succeed()) + Expect(cfg).NotTo(BeNil()) + + Expect(kfplacementv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + Expect(clusterv1beta1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + By("construct the k8s client") + k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme}) + Expect(err).Should(Succeed()) + Expect(k8sClient).NotTo(BeNil()) + + By("creating a test namespace") + ns := corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: testNamespace, + }, + } + Expect(k8sClient.Create(ctx, &ns)).Should(Succeed(), "failed to create namespace") + + By("starting the controller manager") + klog.InitFlags(flag.CommandLine) + flag.Parse() + + mgr, err = ctrl.NewManager(cfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: server.Options{ + BindAddress: "0", + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).Should(Succeed()) + + Expect(fieldindexers.SetupWithHubAgentControllerManager(ctx, mgr)).Should(Succeed()) + reconciler := NewReconciler(mgr.GetClient(), mgr.GetAPIReader(), snapshotStub{}) + Expect(reconciler.SetupWithManagerForPlacementPolicy(mgr)).Should(Succeed()) + Expect(reconciler.SetupWithManagerForClusterPlacementPolicy(mgr)).Should(Succeed()) + + go func() { + defer GinkgoRecover() + err = mgr.Start(ctx) + Expect(err).Should(Succeed(), "failed to run manager") + }() + + Eventually(func() bool { + return mgr.GetCache().WaitForCacheSync(ctx) + }, time.Second*10, time.Second*2).To(BeTrue(), "failed to wait for cache to sync") +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + + cancel() + By("tearing down the test environment") + Expect(testEnv.Stop()).Should(Succeed()) +}) diff --git a/pkg/v1/controllers/placementpolicy/watch.go b/pkg/v1/controllers/placementpolicy/watch.go new file mode 100644 index 000000000..2b28c026b --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/watch.go @@ -0,0 +1,215 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + + corev1 "k8s.io/api/core/v1" + apiequality "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/predicate" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// memberClusterSchedulingView is a projection of a MemberCluster onto exactly the attributes +// that can change a placement policy's scheduling outcome. Attributes that churn on every +// member agent heartbeat without affecting scheduling — the heartbeat timestamp itself, +// property observation times, condition transition times — are deliberately excluded, so that +// heartbeat traffic does not fan out into policy reconciliations across the fleet. +type memberClusterSchedulingView struct { + // The fields are exported so that the semantic deep-equality helper, which reflects over + // field values, can access them; the type itself stays package-private. + Labels map[string]string + Taints []clusterv1beta1.Taint + Terminating bool + + MemberAgentJoined metav1.ConditionStatus + MemberAgentHealthy metav1.ConditionStatus + + PropertyValues map[clusterv1beta1.PropertyName]string + Capacity corev1.ResourceList + Allocatable corev1.ResourceList + Available corev1.ResourceList +} + +func schedulingViewOf(cluster *clusterv1beta1.MemberCluster) memberClusterSchedulingView { + view := memberClusterSchedulingView{ + Labels: cluster.Labels, + Taints: cluster.Spec.Taints, + Terminating: !cluster.DeletionTimestamp.IsZero(), + Capacity: cluster.Status.ResourceUsage.Capacity, + Allocatable: cluster.Status.ResourceUsage.Allocatable, + Available: cluster.Status.ResourceUsage.Available, + } + + if joinedCond := cluster.GetAgentCondition(clusterv1beta1.MemberAgent, clusterv1beta1.AgentJoined); joinedCond != nil { + view.MemberAgentJoined = joinedCond.Status + } + if healthyCond := cluster.GetAgentCondition(clusterv1beta1.MemberAgent, clusterv1beta1.AgentHealthy); healthyCond != nil { + view.MemberAgentHealthy = healthyCond.Status + } + + if len(cluster.Status.Properties) > 0 { + view.PropertyValues = make(map[clusterv1beta1.PropertyName]string, len(cluster.Status.Properties)) + for name, value := range cluster.Status.Properties { + view.PropertyValues[name] = value.Value + } + } + return view +} + +// memberClusterSchedulingRelevantChanges is a predicate that admits member cluster events only +// when they can affect scheduling outcomes; in particular, updates that only advance heartbeat +// or observation timestamps are filtered out. +func memberClusterSchedulingRelevantChanges() predicate.Predicate { + return predicate.Funcs{ + CreateFunc: func(event.CreateEvent) bool { return true }, + DeleteFunc: func(event.DeleteEvent) bool { return true }, + UpdateFunc: func(e event.UpdateEvent) bool { + oldCluster, oldOK := e.ObjectOld.(*clusterv1beta1.MemberCluster) + newCluster, newOK := e.ObjectNew.(*clusterv1beta1.MemberCluster) + if !oldOK || !newOK { + // This should never happen as the predicate is registered on MemberCluster + // watches only; admit the event out of caution. + return true + } + return !apiequality.Semantic.DeepEqual(schedulingViewOf(oldCluster), schedulingViewOf(newCluster)) + }, + GenericFunc: func(event.GenericEvent) bool { return false }, + } +} + +// mapMemberClusterToPlacementPolicies enqueues every PlacementPolicy on a member cluster event. +// +// Cluster changes are not attributable to individual policies without evaluating every selector +// anyway, so all policies are re-enqueued; the scheduling-relevant-changes predicate keeps the +// event volume low. +func (r *Reconciler) mapMemberClusterToPlacementPolicies(ctx context.Context, _ client.Object) []reconcile.Request { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + if err := r.List(ctx, policies); err != nil { + klog.ErrorS(err, "Failed to list placement policies for member cluster mapping") + return nil + } + requests := make([]reconcile.Request, 0, len(policies.Items)) + for i := range policies.Items { + requests = append(requests, reconcile.Request{ + NamespacedName: client.ObjectKeyFromObject(&policies.Items[i]), + }) + } + return requests +} + +// mapClaimToPlacementPolicy resolves a cluster claim to its owning namespaced PlacementPolicy +// via its spec reference; claims of cluster-scoped policies are left to the +// ClusterPlacementPolicy controller's mapper. +func (r *Reconciler) mapClaimToPlacementPolicy(_ context.Context, obj client.Object) []reconcile.Request { + // The spec reference, not the ownership labels, is the policy's authoritative identity: a + // label value cannot hold a name longer than 63 bytes. + ref := policyRefOf(obj) + if ref == nil || ref.Name == "" || ref.Namespace == "" { + return nil + } + return []reconcile.Request{{NamespacedName: types.NamespacedName{Name: ref.Name, Namespace: ref.Namespace}}} +} + +// mapClaimToClusterPlacementPolicy resolves a cluster claim to its owning ClusterPlacementPolicy +// via its spec reference. +func (r *Reconciler) mapClaimToClusterPlacementPolicy(_ context.Context, obj client.Object) []reconcile.Request { + ref := policyRefOf(obj) + if ref == nil || ref.Name == "" || ref.Namespace != "" { + return nil + } + return []reconcile.Request{{NamespacedName: types.NamespacedName{Name: ref.Name}}} +} + +// policyRefOf returns the placement policy reference of a cluster claim, or nil for anything +// else; the claim watches this maps for are registered on ClusterClaim objects only. +func policyRefOf(obj client.Object) *kfplacementv1alpha1.ObjectReference { + claim, ok := obj.(*kfplacementv1alpha1.ClusterClaim) + if !ok { + return nil + } + return claim.Spec.PlacementPolicyRef +} + +// mapMemberClusterToClusterPlacementPolicies enqueues every ClusterPlacementPolicy on a member +// cluster event. +func (r *Reconciler) mapMemberClusterToClusterPlacementPolicies(ctx context.Context, _ client.Object) []reconcile.Request { + policies := &kfplacementv1alpha1.ClusterPlacementPolicyList{} + if err := r.List(ctx, policies); err != nil { + klog.ErrorS(err, "Failed to list cluster placement policies for member cluster mapping") + return nil + } + requests := make([]reconcile.Request, 0, len(policies.Items)) + for i := range policies.Items { + requests = append(requests, reconcile.Request{ + NamespacedName: client.ObjectKeyFromObject(&policies.Items[i]), + }) + } + return requests +} + +// bindingProgressView projects a binding onto what its policy's status reports: the revision the +// work generator last processed and the status of its Synchronized and Available conditions. +// Transition times and messages are left out, so status writes that change neither do not wake +// the policy. +type bindingProgressView struct { + processedRevision string + synchronized, available metav1.ConditionStatus + synchronizedGen, availableGen int64 +} + +func bindingProgressOf(obj client.Object) bindingProgressView { + binding, ok := obj.(kfplacementv1alpha1.PlacementBindingAccessor) + if !ok { + return bindingProgressView{} + } + status := binding.GetStatus() + var view bindingProgressView + if status.LastProcessedResourceSnapshotName != nil { + view.processedRevision = *status.LastProcessedResourceSnapshotName + } + if cond := meta.FindStatusCondition(status.Conditions, kfplacementv1alpha1.PlacementBindingCondTypeSynchronized); cond != nil { + view.synchronized, view.synchronizedGen = cond.Status, cond.ObservedGeneration + } + if cond := meta.FindStatusCondition(status.Conditions, kfplacementv1alpha1.PlacementBindingCondTypeAvailable); cond != nil { + view.available, view.availableGen = cond.Status, cond.ObservedGeneration + } + return view +} + +// bindingRelevantChanges admits the binding events the owning policy must see: any spec change, +// as before, and any change in the rollout progress the policy rolls up into its status. +func bindingRelevantChanges() predicate.Predicate { + return predicate.Or(predicate.GenerationChangedPredicate{}, predicate.Funcs{ + CreateFunc: func(event.CreateEvent) bool { return false }, + DeleteFunc: func(event.DeleteEvent) bool { return false }, + GenericFunc: func(event.GenericEvent) bool { return false }, + UpdateFunc: func(e event.UpdateEvent) bool { + return bindingProgressOf(e.ObjectOld) != bindingProgressOf(e.ObjectNew) + }, + }) +} diff --git a/pkg/v1/controllers/placementpolicy/watch_test.go b/pkg/v1/controllers/placementpolicy/watch_test.go new file mode 100644 index 000000000..dfb0400ba --- /dev/null +++ b/pkg/v1/controllers/placementpolicy/watch_test.go @@ -0,0 +1,194 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementpolicy + +import ( + "context" + "strings" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +func claimWithRef(ref *kfplacementv1alpha1.ObjectReference) *kfplacementv1alpha1.ClusterClaim { + return &kfplacementv1alpha1.ClusterClaim{ + ObjectMeta: metav1.ObjectMeta{Name: "claim"}, + Spec: kfplacementv1alpha1.ClusterClaimSpec{PlacementPolicyRef: ref}, + } +} + +// TestMapClaimToPolicies pins the claim-to-policy mapping to spec.placementPolicyRef, which is +// the only identity that survives policy names longer than a label value can hold. +func TestMapClaimToPolicies(t *testing.T) { + longName := strings.Repeat("x", 250) + + testCases := []struct { + name string + obj client.Object + wantNamespaced []reconcile.Request + wantClusterScoped []reconcile.Request + }{ + { + name: "namespaced policy reference", + obj: claimWithRef(&kfplacementv1alpha1.ObjectReference{Name: "app", Namespace: "tenant-a"}), + wantNamespaced: []reconcile.Request{{NamespacedName: types.NamespacedName{Name: "app", Namespace: "tenant-a"}}}, + }, + { + name: "cluster-scoped policy reference", + obj: claimWithRef(&kfplacementv1alpha1.ObjectReference{Name: "app"}), + wantClusterScoped: []reconcile.Request{{NamespacedName: types.NamespacedName{Name: "app"}}}, + }, + { + name: "name longer than a label value still maps", + obj: claimWithRef(&kfplacementv1alpha1.ObjectReference{Name: longName, Namespace: "tenant-a"}), + wantNamespaced: []reconcile.Request{{NamespacedName: types.NamespacedName{Name: longName, Namespace: "tenant-a"}}}, + }, + { + name: "claim without a policy reference maps to nothing", + obj: claimWithRef(nil), + }, + { + name: "reference without a name maps to nothing", + obj: claimWithRef(&kfplacementv1alpha1.ObjectReference{Namespace: "tenant-a"}), + }, + { + name: "object of another kind maps to nothing", + obj: &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: "cm", Namespace: "tenant-a"}}, + }, + } + + r := &Reconciler{} + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotNamespaced := r.mapClaimToPlacementPolicy(context.Background(), tc.obj) + if diff := cmp.Diff(gotNamespaced, tc.wantNamespaced); diff != "" { + t.Errorf("mapClaimToPlacementPolicy() mismatch (-got, +want):\n%s", diff) + } + gotClusterScoped := r.mapClaimToClusterPlacementPolicy(context.Background(), tc.obj) + if diff := cmp.Diff(gotClusterScoped, tc.wantClusterScoped); diff != "" { + t.Errorf("mapClaimToClusterPlacementPolicy() mismatch (-got, +want):\n%s", diff) + } + }) + } +} + +// TestMemberClusterSchedulingRelevantChanges pins which member cluster updates reach the policy +// reconciler: heartbeat and observation timestamps must not, anything scheduling reads must. +func TestMemberClusterSchedulingRelevantChanges(t *testing.T) { + now := metav1.Now() + base := func() *clusterv1beta1.MemberCluster { + return &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{Name: "member", Labels: map[string]string{"region": "eastus"}}, + Spec: clusterv1beta1.MemberClusterSpec{Taints: []clusterv1beta1.Taint{{Key: "k", Value: "v", Effect: corev1.TaintEffectNoSchedule}}}, + Status: clusterv1beta1.MemberClusterStatus{ + AgentStatus: []clusterv1beta1.AgentStatus{{ + Type: clusterv1beta1.MemberAgent, + Conditions: []metav1.Condition{{Type: string(clusterv1beta1.AgentJoined), Status: metav1.ConditionTrue, LastTransitionTime: now}}, + LastReceivedHeartbeat: now, + }}, + Properties: map[clusterv1beta1.PropertyName]clusterv1beta1.PropertyValue{"node-count": {Value: "3", ObservationTime: now}}, + }, + } + } + + testCases := []struct { + name string + mutate func(*clusterv1beta1.MemberCluster) + want bool + }{ + {name: "heartbeat only", mutate: func(mc *clusterv1beta1.MemberCluster) { + mc.Status.AgentStatus[0].LastReceivedHeartbeat = metav1.NewTime(now.Add(time.Minute)) + mc.Status.Properties["node-count"] = clusterv1beta1.PropertyValue{Value: "3", ObservationTime: metav1.NewTime(now.Add(time.Minute))} + }}, + {name: "label change", mutate: func(mc *clusterv1beta1.MemberCluster) { mc.Labels["region"] = "westus" }, want: true}, + {name: "taint change", mutate: func(mc *clusterv1beta1.MemberCluster) { mc.Spec.Taints = nil }, want: true}, + {name: "joined condition flips", mutate: func(mc *clusterv1beta1.MemberCluster) { + mc.Status.AgentStatus[0].Conditions[0].Status = metav1.ConditionFalse + }, want: true}, + {name: "property value change", mutate: func(mc *clusterv1beta1.MemberCluster) { + mc.Status.Properties["node-count"] = clusterv1beta1.PropertyValue{Value: "4", ObservationTime: now} + }, want: true}, + {name: "deletion starts", mutate: func(mc *clusterv1beta1.MemberCluster) { mc.DeletionTimestamp = &now }, want: true}, + } + + pred := memberClusterSchedulingRelevantChanges() + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + updated := base() + tc.mutate(updated) + if got := pred.Update(event.UpdateEvent{ObjectOld: base(), ObjectNew: updated}); got != tc.want { + t.Errorf("memberClusterSchedulingRelevantChanges().Update() = %v, want %v", got, tc.want) + } + }) + } +} + +// TestBindingRelevantChanges pins that a binding update wakes its policy when the spec or the +// rollout progress changes, and not when only a condition's message or transition time does. +func TestBindingRelevantChanges(t *testing.T) { + base := func() *kfplacementv1alpha1.PlacementBinding { + return &kfplacementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{Name: "b", Generation: 1}, + Status: kfplacementv1alpha1.PlacementBindingStatus{ + LastProcessedResourceSnapshotName: ptr.To("pp-snapshot-0"), + Conditions: []metav1.Condition{ + {Type: kfplacementv1alpha1.PlacementBindingCondTypeSynchronized, Status: metav1.ConditionFalse, ObservedGeneration: 1, Reason: "Waiting", Message: "a"}, + }, + }, + } + } + testCases := []struct { + name string + mutate func(*kfplacementv1alpha1.PlacementBinding) + want bool + }{ + {name: "spec change", mutate: func(b *kfplacementv1alpha1.PlacementBinding) { b.Generation = 2 }, want: true}, + {name: "synchronized", mutate: func(b *kfplacementv1alpha1.PlacementBinding) { b.Status.Conditions[0].Status = metav1.ConditionTrue }, want: true}, + {name: "available reported", mutate: func(b *kfplacementv1alpha1.PlacementBinding) { + b.Status.Conditions = append(b.Status.Conditions, metav1.Condition{Type: kfplacementv1alpha1.PlacementBindingCondTypeAvailable, Status: metav1.ConditionTrue, ObservedGeneration: 1}) + }, want: true}, + {name: "new revision processed", mutate: func(b *kfplacementv1alpha1.PlacementBinding) { + b.Status.LastProcessedResourceSnapshotName = ptr.To("pp-snapshot-1") + }, want: true}, + {name: "message only", mutate: func(b *kfplacementv1alpha1.PlacementBinding) { b.Status.Conditions[0].Message = "b" }, want: false}, + {name: "resource counts only", mutate: func(b *kfplacementv1alpha1.PlacementBinding) { b.Status.SelectedResources = ptr.To(int32(3)) }, want: false}, + } + // Creates and deletes still wake the owner, so that a binding deleted out of band is recreated. + if !bindingRelevantChanges().Create(event.CreateEvent{Object: base()}) || !bindingRelevantChanges().Delete(event.DeleteEvent{Object: base()}) { + t.Errorf("bindingRelevantChanges() filtered out a create or a delete, want both admitted") + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + oldObj, newObj := base(), base() + tc.mutate(newObj) + if got := bindingRelevantChanges().Update(event.UpdateEvent{ObjectOld: oldObj, ObjectNew: newObj}); got != tc.want { + t.Errorf("bindingRelevantChanges().Update() = %t, want %t", got, tc.want) + } + }) + } +} diff --git a/pkg/v1/controllers/stagedupdaterun/claim.go b/pkg/v1/controllers/stagedupdaterun/claim.go new file mode 100644 index 000000000..b7e8bc148 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/claim.go @@ -0,0 +1,73 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + + "k8s.io/klog/v2" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/bindingmanager" +) + +func stagedUpdateRunObjRef(stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor) placementv1alpha1.ObjectReference { + var kind string + if stagedUpdateRun.GetNamespace() == "" { + kind = rolloutv1alpha1.ClusterStagedUpdateRunKind + } else { + kind = rolloutv1alpha1.StagedUpdateRunKind + } + return placementv1alpha1.ObjectReference{ + APIGroup: rolloutv1alpha1.GroupVersion.Group, + APIVersion: rolloutv1alpha1.GroupVersion.Version, + Kind: kind, + Namespace: stagedUpdateRun.GetNamespace(), + Name: stagedUpdateRun.GetName(), + } +} + +func (r *Reconciler) claimAsBindingManagerFor( + ctx context.Context, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, +) (claimed bool, err error) { + stagedUpdateRunObjRef := stagedUpdateRunObjRef(stagedUpdateRun) + claimed, err = bindingmanager.ClaimRoleAs(ctx, r.hubClient, placementPolicy, controllerName, stagedUpdateRunObjRef) + if err != nil { + wrappedErr := errors.Wraps(err, "", + "stagedUpdateRun", klog.KObj(stagedUpdateRun), + "placementPolicyName", stagedUpdateRun.GetSpec().PlacementPolicyName, + "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to claim binding manager role", errors.Args(wrappedErr)...) + return false, wrappedErr + } + if !claimed { + klog.V(2).InfoS("Cannot claim the binding manager role for now; will retry later", + "stagedUpdateRun", klog.KObj(stagedUpdateRun), + "placementPolicyName", stagedUpdateRun.GetSpec().PlacementPolicyName, + "controller", controllerName) + return false, nil + } + klog.V(2).InfoS("Successfully claimed the binding manager role for the staged update run", + "stagedUpdateRun", klog.KObj(stagedUpdateRun), + "placementPolicyName", stagedUpdateRun.GetSpec().PlacementPolicyName, + "controller", controllerName) + return true, nil +} diff --git a/pkg/v1/controllers/stagedupdaterun/cleanup.go b/pkg/v1/controllers/stagedupdaterun/cleanup.go new file mode 100644 index 000000000..bc42287c3 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/cleanup.go @@ -0,0 +1,77 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/bindingmanager" +) + +const ( + stagedUpdateRunCleanupFinalizer = "rollout.kubefleet.dev/staged-update-run-cleanup" +) + +func (r *Reconciler) cleanup(ctx context.Context, stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor) error { + // Note that approval requests are created with owner references; once the staged update run is deleted, + // the approval requests will be automatically garbage collected. + + // Give up the binding manager role for the linked placement policy, if currently held. + placementPolicyAccessor, err := r.retrieveLinkedPlacementPolicy(ctx, stagedUpdateRun) + switch { + case apierrors.IsNotFound(err): + // The linked placement policy no longer exists; there is no binding manager role to relinquish, so take no action. + case err != nil: + return errors.Wraps(err, "failed to retrieve the linked placement policy") + default: + // Successfully retrieved the linked placement policy; relinquish the binding manager role if currently held. + stagedUpdateRunObjRef := stagedUpdateRunObjRef(stagedUpdateRun) + if err := bindingmanager.RelinquishRoleFor(ctx, r.hubClient, placementPolicyAccessor, controllerName, stagedUpdateRunObjRef); err != nil { + return errors.Wraps(err, "failed to relinquish the binding manager role") + } + } + + // Drop the cleanup finalizer so that the staged update run object can be deleted. + if controllerutil.ContainsFinalizer(stagedUpdateRun, stagedUpdateRunCleanupFinalizer) { + controllerutil.RemoveFinalizer(stagedUpdateRun, stagedUpdateRunCleanupFinalizer) + if err := r.hubClient.Update(ctx, stagedUpdateRun); err != nil { + return errors.NewAPIServerError(err, "failed to update staged update run object", false) + } + } + + return nil +} + +func (r *Reconciler) ensureCleanupFinalizer(ctx context.Context, stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor) error { + if controllerutil.ContainsFinalizer(stagedUpdateRun, stagedUpdateRunCleanupFinalizer) { + return nil + } + + controllerutil.AddFinalizer(stagedUpdateRun, stagedUpdateRunCleanupFinalizer) + if err := r.hubClient.Update(ctx, stagedUpdateRun); err != nil { + return errors.NewAPIServerError(err, "failed to update staged update run object", false) + } + + klog.V(2).InfoS("Added cleanup finalizer to staged update run object", "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + return nil +} diff --git a/pkg/v1/controllers/stagedupdaterun/controller.go b/pkg/v1/controllers/stagedupdaterun/controller.go new file mode 100644 index 000000000..fd1ee9bc5 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/controller.go @@ -0,0 +1,289 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + "fmt" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller" + "sigs.k8s.io/controller-runtime/pkg/handler" + "sigs.k8s.io/controller-runtime/pkg/predicate" + + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/utils/parallelizer" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot" +) + +const ( + controllerName = "staged-update-run" + + requeueImmediatelyAfterPeriod = time.Millisecond * 200 +) + +const ( + minRefreshPeriod = time.Second * 30 + + bindingManagerClaimRateLimiterBaseDelay = time.Second * 5 + bindingManagerClaimRateLimiterMaxDelay = time.Second * 180 + bindingAvailabilityCheckRateLimiterBaseDelay = time.Second * 5 + bindingAvailabilityCheckRateLimiterMaxDelay = time.Second * 180 +) + +// Reconciler helps reconcile stage update run objects. +type Reconciler struct { + hubClient client.Client + hubUncachedClient client.Reader + + placementResourceSnapshotManager *placementresourcesnapshot.Manager + + parallelizer parallelizer.Parallelizer + + bindingManagerClaimRateLimiter workqueue.TypedRateLimiter[ctrl.Request] + bindingAvailabilityCheckRateLimiter workqueue.TypedRateLimiter[types.NamespacedName] + + refreshPeriod time.Duration +} + +// New creates a Reconciler. +func New( + hubClient client.Client, + hubUncachedClient client.Reader, + placementResourceSnapshotManager *placementresourcesnapshot.Manager, + parallelizer parallelizer.Parallelizer, + bindingManagerClaimRateLimiter workqueue.TypedRateLimiter[ctrl.Request], + bindingAvailabilityCheckRateLimiter workqueue.TypedRateLimiter[types.NamespacedName], + refreshPeriod time.Duration, +) (*Reconciler, error) { + if bindingManagerClaimRateLimiter == nil { + bindingManagerClaimRateLimiter = workqueue.NewTypedItemExponentialFailureRateLimiter[ctrl.Request](bindingManagerClaimRateLimiterBaseDelay, bindingManagerClaimRateLimiterMaxDelay) + } + if bindingAvailabilityCheckRateLimiter == nil { + bindingAvailabilityCheckRateLimiter = workqueue.NewTypedItemExponentialFailureRateLimiter[types.NamespacedName](bindingAvailabilityCheckRateLimiterBaseDelay, bindingAvailabilityCheckRateLimiterMaxDelay) + } + + if refreshPeriod < minRefreshPeriod { + return nil, errors.NewUnexpectedError(nil, "refresh period is too short", "minRefreshPeriod", minRefreshPeriod, "observedRefreshPeriod", refreshPeriod) + } + + return &Reconciler{ + hubClient: hubClient, + hubUncachedClient: hubUncachedClient, + placementResourceSnapshotManager: placementResourceSnapshotManager, + parallelizer: parallelizer, + bindingManagerClaimRateLimiter: bindingManagerClaimRateLimiter, + bindingAvailabilityCheckRateLimiter: bindingAvailabilityCheckRateLimiter, + refreshPeriod: refreshPeriod, + }, nil +} + +func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + startTime := time.Now() + klog.V(2).InfoS("Reconciliation starts", "stagedUpdateRun", req.NamespacedName, "controller", controllerName) + defer func() { + latency := time.Since(startTime).Milliseconds() + klog.V(2).InfoS("Reconciliation ends", "stagedUpdateRun", req.NamespacedName, "latency", latency, "controller", controllerName) + }() + + // Retrieve the StagedUpdateRun or ClusterStagedUpdateRun resource and return it as a StagedUpdateRunAccessor interface. + stagedUpdateRun, err := r.retrieveStagedUpdateRun(ctx, req.NamespacedName) + if err != nil { + if apierrors.IsNotFound(err) { + // The staged update run cannot be found. + return ctrl.Result{}, nil + } + + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", req.NamespacedName, "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to retrieve staged update run object", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Clean things up if the staged update run object has been marked for deletion. + if !stagedUpdateRun.GetDeletionTimestamp().IsZero() { + if err := r.cleanup(ctx, stagedUpdateRun); err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to cleanup staged update run", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + return ctrl.Result{}, nil + } + + shouldSkip, err := r.shouldSkipProcessing(ctx, stagedUpdateRun) + if err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to determine if staged update run should be skipped", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + if shouldSkip { + // The staged update run is already in a terminal state (failed to initialize, complete), or has been + // suspended. Give up the binding manager role and drop the cleanup finalizer. + if err := r.cleanup(ctx, stagedUpdateRun); err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to cleanup staged update run", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + return ctrl.Result{}, nil + } + + // Add a cleanup finalizer to the staged update run object if it doesn't have one. + if err := r.ensureCleanupFinalizer(ctx, stagedUpdateRun); err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to ensure cleanup finalizer on staged update run object", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Retrieve the corresponding (cluster) placement policy resource and return it as a placement policy accessor. + placementPolicy, err := r.retrieveLinkedPlacementPolicy(ctx, stagedUpdateRun) + if err != nil { + if apierrors.IsNotFound(err) { + // The linked placement policy cannot be found. No need to process the staged update run any more. + msg := "No linked placement policy" + if err := r.markStagedUpdateRunAsInTerminalState(ctx, stagedUpdateRun, msg); err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to mark staged update run as in terminal state", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + // Requeue to handle the terminal state update. + return ctrl.Result{RequeueAfter: requeueImmediatelyAfterPeriod}, nil + } + wrappedErr := errors.Wraps(err, "failed to retrieve the linked placement policy", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to retrieve the linked placement policy", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Claim the staged update run object as the binding manager for the placement policy. + // This ensures that no other controllers (e.g., the scheduling process, the migrations, etc.) can + // interfere with the rollout process. + claimed, err := r.claimAsBindingManagerFor(ctx, placementPolicy, stagedUpdateRun) + if err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to claim binding manager role for staged update run", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + if !claimed { + backoffDuration := r.bindingManagerClaimRateLimiter.When(req) + return ctrl.Result{RequeueAfter: backoffDuration}, nil + } + // Reset the binding manager claim rate limiter for the staged update run object. + r.bindingManagerClaimRateLimiter.Forget(req) + + // Initialize the staged update run (if applicable). + if err := r.initialize(ctx, placementPolicy, stagedUpdateRun); err != nil { + if errors.Category(err) == errors.ErrCategoryUser { + // A user error has occurred during initialization; consider the initialization step failed. + msg := fmt.Sprintf("failed to initialize staged update run: %v", err) + if err := r.markStagedUpdateRunAsFailedToInitialize(ctx, stagedUpdateRun, msg); err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to mark staged update run as failed to initialize", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + return ctrl.Result{RequeueAfter: requeueImmediatelyAfterPeriod}, nil + } + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to initialize staged update run", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Before starting processing the staged update run, verify if the run has been suspended and needs to be skipped. + shouldSkip, err = r.shouldSkipProcessing(ctx, stagedUpdateRun) + if err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to determine if staged update run should be skipped", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + if shouldSkip { + // The staged update run is already in a terminal state (failed to initialize, complete), or has been + // suspended. Give up the binding manager role and drop the cleanup finalizer. + if err := r.cleanup(ctx, stagedUpdateRun); err != nil { + wrappedErr := errors.Wraps(err, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to cleanup staged update run", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + return ctrl.Result{}, nil + } + + // Process the staged update run. + allStagesProcessingRes, requeueAfter, processingErr := r.process(ctx, stagedUpdateRun) + klog.V(2).InfoS("Processed staged update run", + "stagedUpdateRun", klog.KObj(stagedUpdateRun), + "placementPolicyName", stagedUpdateRun.GetSpec().PlacementPolicyName, + "controller", controllerName, + "allStagesProcessingRes", allStagesProcessingRes, + "err", processingErr) + + // Refresh the staged update run status. + // + // Note that this is done regardless of the processing results, so as to keep the users informed of the progress + // so far. + if updateErr := r.hubClient.Status().Update(ctx, stagedUpdateRun); updateErr != nil { + wrappedErr := errors.NewAPIServerError(updateErr, "", false, + "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to update staged update run status", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Handle processing errors and requeues accordingly. + if processingErr != nil { + wrappedErr := errors.Wraps(processingErr, "", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "Failed to process staged update run", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + switch allStagesProcessingRes { + case allStagesProcessingResultSucceeded, allStagesProcessingResultFailed: + // The staged update run has been completed. Requeue it to perform the cleanup steps. + return ctrl.Result{RequeueAfter: requeueImmediatelyAfterPeriod}, nil + case allStagesProcessingResultInProgress: + // Requeue after the specified duration. + if requeueAfter != nil { + return ctrl.Result{RequeueAfter: *requeueAfter}, nil + } + return ctrl.Result{RequeueAfter: requeueImmediatelyAfterPeriod}, nil + default: + // An unexpected processing result is yielded; consider this as an unexpected error. + wrappedErr := errors.NewUnexpectedError(nil, "encountered an unexpected processing result", + "observedProcessingResult", allStagesProcessingRes, + "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + klog.ErrorS(wrappedErr, "", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } +} + +// SetupWithManager sets up the controller with the manager. +// +// The controller watches both ClusterStagedUpdateRun (cluster-scoped) and StagedUpdateRun +// (namespace-scoped) objects, funneling events for either kind into the same reconcile queue; +// Reconcile (via retrieveStagedUpdateRun) tells the two apart by whether the incoming request +// carries a namespace. +func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, maxConcurrentReconciles int) error { + return ctrl.NewControllerManagedBy(mgr). + Named(controllerName). + WithOptions(controller.Options{MaxConcurrentReconciles: maxConcurrentReconciles}). + Watches(&rolloutv1alpha1.ClusterStagedUpdateRun{}, &handler.EnqueueRequestForObject{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). + Watches(&rolloutv1alpha1.StagedUpdateRun{}, &handler.EnqueueRequestForObject{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). + Watches(&rolloutv1alpha1.ApprovalRequest{}, approvalReqEventHandlerFuncs). + Watches(&rolloutv1alpha1.ClusterApprovalRequest{}, approvalReqEventHandlerFuncs). + Complete(r) +} diff --git a/pkg/v1/controllers/stagedupdaterun/eventhandler.go b/pkg/v1/controllers/stagedupdaterun/eventhandler.go new file mode 100644 index 000000000..97f152679 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/eventhandler.go @@ -0,0 +1,167 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/handler" + + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +var ( + approvalReqEventHandlerFuncs = &handler.TypedFuncs[client.Object, ctrl.Request]{ + // Normally the staged update run controller does not need to watch for approval request create events, as the + // controller itself is the creator of the objects and needs only to wait for the created requests to be + // updated. However, a corner case still exists, where the controller might be restarted and the approval + // request gets updated during the downtime; in this situation, the update event will not be delivered and + // the controller sees only a create event for the already updated approval request, hence the handler setup + // here. + CreateFunc: func( + ctx context.Context, + e event.TypedCreateEvent[client.Object], + q workqueue.TypedRateLimitingInterface[ctrl.Request]) { + obj := e.Object + req, err := castToApprovalRequest(obj) + if err != nil { + klog.ErrorS(err, "Failed to enqueue an approval request create event", errors.Args(err)...) + return + } + + // Enqueue the request if the approval condition exists (in whatever status). + // + // If the condition is absent, the handler will be triggered again when it is added. + if meta.FindStatusCondition(req.GetStatus().Conditions, rolloutv1alpha1.ApprovalRequestCondTypeApproved) == nil { + return + } + + // Do a sanity check; verify that the approval request references a staged update run. + if req.GetSpec().StagedUpdateRunName == "" { + err := errors.NewUnexpectedError(nil, "no staged update run reference on approval request", "approvalRequest", klog.KObj(obj)) + klog.ErrorS(err, "Failed to enqueue an approval request create event", errors.Args(err)...) + return + } + + // The staged update run shares the namespace with the approval request (empty for cluster-scoped ones). + q.Add(ctrl.Request{ + NamespacedName: types.NamespacedName{ + Namespace: req.GetNamespace(), + Name: req.GetSpec().StagedUpdateRunName, + }, + }) + }, + // Handle update events for approval requests; enqueue the staged update run for processing if the approval + // status changes. + UpdateFunc: func( + ctx context.Context, + e event.TypedUpdateEvent[client.Object], + q workqueue.TypedRateLimitingInterface[ctrl.Request]) { + oldObj, newObj := e.ObjectOld, e.ObjectNew + oldReq, err := castToApprovalRequest(oldObj) + if err != nil { + klog.ErrorS(err, "Failed to cast old approval request in update event", errors.Args(err)...) + return + } + newReq, err := castToApprovalRequest(newObj) + if err != nil { + klog.ErrorS(err, "Failed to cast new approval request in update event", errors.Args(err)...) + return + } + + // Enqueue the request if the approval condition's status changed between the old and new approval request. + oldApprovalCond := meta.FindStatusCondition(oldReq.GetStatus().Conditions, rolloutv1alpha1.ApprovalRequestCondTypeApproved) + newApprovalCond := meta.FindStatusCondition(newReq.GetStatus().Conditions, rolloutv1alpha1.ApprovalRequestCondTypeApproved) + switch { + case newApprovalCond == nil: + // The approval condition is absent in the new object; no need to enqueue the request. + return + case oldApprovalCond == nil: + // The approval condition was added; enqueue the request. + case newApprovalCond.Status != oldApprovalCond.Status: + // The approval condition's status changed; enqueue the request. + default: + // The approval condition did not change in a way that requires enqueuing the request. + return + } + + // Do a sanity check; verify that the approval request references a staged update run. + if newReq.GetSpec().StagedUpdateRunName == "" { + err := errors.NewUnexpectedError(nil, "no staged update run reference on approval request", "approvalRequest", klog.KObj(newObj)) + klog.ErrorS(err, "Failed to enqueue an approval request update event", errors.Args(err)...) + return + } + + q.Add(ctrl.Request{ + NamespacedName: types.NamespacedName{ + Namespace: newReq.GetNamespace(), + Name: newReq.GetSpec().StagedUpdateRunName, + }, + }) + }, + // Handle a corner case where the approval request is deleted and needs re-creation. + DeleteFunc: func( + ctx context.Context, + e event.TypedDeleteEvent[client.Object], + q workqueue.TypedRateLimitingInterface[ctrl.Request]) { + obj := e.Object + req, err := castToApprovalRequest(obj) + if err != nil { + klog.ErrorS(err, "Failed to cast deleted approval request", errors.Args(err)...) + return + } + + // Do a sanity check; verify that the approval request references a staged update run. + if req.GetSpec().StagedUpdateRunName == "" { + err := errors.NewUnexpectedError(nil, "no staged update run reference on deleted approval request", "approvalRequest", klog.KObj(obj)) + klog.ErrorS(err, "Failed to enqueue an approval request delete event", errors.Args(err)...) + return + } + + q.Add(ctrl.Request{ + NamespacedName: types.NamespacedName{ + Namespace: req.GetNamespace(), + Name: req.GetSpec().StagedUpdateRunName, + }, + }) + }, + } +) + +func castToApprovalRequest(obj client.Object) (rolloutv1alpha1.ApprovalRequestAccessor, error) { + if obj.GetNamespace() == "" { + req, ok := obj.(*rolloutv1alpha1.ClusterApprovalRequest) + if !ok { + return nil, errors.NewUnexpectedError(nil, "failed to cast object as a cluster approval request", "gvk", obj.GetObjectKind().GroupVersionKind(), "obj", klog.KObj(obj)) + } + return req, nil + } else { + req, ok := obj.(*rolloutv1alpha1.ApprovalRequest) + if !ok { + return nil, errors.NewUnexpectedError(nil, "failed to cast object as an approval request", "gvk", obj.GetObjectKind().GroupVersionKind(), "obj", klog.KObj(obj)) + } + return req, nil + } +} diff --git a/pkg/v1/controllers/stagedupdaterun/initialize.go b/pkg/v1/controllers/stagedupdaterun/initialize.go new file mode 100644 index 000000000..4bcba77a2 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/initialize.go @@ -0,0 +1,423 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + "sort" + "strconv" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/klog/v2" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +func (r *Reconciler) initialize( + ctx context.Context, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, +) error { + initCond := meta.FindStatusCondition(stagedUpdateRun.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeInitialized) + switch { + case initCond == nil: + // The staged update run has not been initialized yet; proceed with the initialization. + case initCond.Status == metav1.ConditionTrue: + // The staged update run has already been initialized successfully; skip the initialization. + klog.V(2).InfoS("Staged update run has been initialized; skip the initialization step", + "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + // If the condition is no longer up to date (its observed generation is behind the current generation), resync it. + return r.resyncStagedUpdateRunCondIfNeeded(ctx, stagedUpdateRun, rolloutv1alpha1.StagedUpdateRunCondTypeInitialized) + case initCond.Status == metav1.ConditionFalse: + // The staged update run has already been initialized and failed; no further processing is needed. + klog.V(2).InfoS("Staged update run has failed to initialize; no further processing is needed", + "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + // If the condition is no longer up to date (its observed generation is behind the current generation), resync it. + return r.resyncStagedUpdateRunCondIfNeeded(ctx, stagedUpdateRun, rolloutv1alpha1.StagedUpdateRunCondTypeInitialized) + } + + // Retrieve the staged update strategy. + stagedUpdateStrategy, err := r.retrieveStagedUpdateStrategy(ctx, stagedUpdateRun) + switch { + case err == nil: + // The staged update strategy has been found; proceed with the initialization. + case apierrors.IsNotFound(err): + return errors.NewUserError(nil, "staged update strategy cannot be found", errors.Args(err)...) + default: + // An API server error other than NotFound error has occurred. + return errors.Wraps(err, "failed to find staged update strategy") + } + + // List all the placement bindings from the target placement policy. + // + // As the staged update run has acquired the binding manager role, it is guaranteed that no binding will be created + // or deleted when the staged update is still in progress. This is not to say that the read (list) op here is strongly + // consistent though; if the cache is severely lagging, and the placement policy just has a scheduling decision + // update, the binding list returned might be stale. Considering that the chances of this happening is low, no + // prevention measure is taken here; it is recommended that the user re-attempt a staged update if the placement policy + // still reports that some clusters are not yet synchronized. + placementBindings, err := r.listAllPlacementBindings(ctx, placementPolicy) + if err != nil { + return errors.Wraps(err, "failed to find placement bindings") + } + + // Retrieve all the clusters that are referenced by the placement bindings. + // + // Note that due to the pre-allocation, the order of the returned clusters is guaranteed to be the same as + // the order of the placement bindings. + referencedClusters, someClustersNotRetrieved, err := r.retrieveReferencedClusters(ctx, placementBindings) + if err != nil { + return errors.Wraps(err, "failed to retrieve referenced clusters") + } + + if someClustersNotRetrieved { + // Some of the referenced clusters could not be retrieved; this is not considered an error, and + // the staged update run will continue with such bindings ignored. + placementBindings, referencedClusters = filterBindingsWithReferencedClusters(placementBindings, referencedClusters) + } + if len(placementBindings) == 0 { + return errors.NewUserError(nil, "no placement bindings are found; there are no clusters to roll out changes to") + } + + // Group clusters into stages. + perStageStatuses, err := r.groupClusters(stagedUpdateStrategy, placementBindings, referencedClusters) + if err != nil { + return errors.Wraps(err, "failed to group clusters into stages") + } + + // Identify the resource snapshot to roll out. + // + // If one has been explicitly specified, check if it exists; otherwise, retrieve the latest snapshot or + // request a new one if the latest is stale. + resourceSnapshot, err := r.identifyResourceSnapshotToRollout(ctx, placementPolicy, stagedUpdateRun) + if err != nil { + return errors.Wraps(err, "failed to identify resource snapshot to roll out") + } + + // Write the per-stage status to the staged update run. + // + // Once written, all follow-up processing will be based on the per-stage status, and the staged update run + // will be considered initialized. + if err := r.syncStagedUpdateRunInitializedStatus(ctx, stagedUpdateRun, perStageStatuses, resourceSnapshot.GetName()); err != nil { + return errors.Wraps(err, "failed to mark the staged update run as initialized") + } + return nil +} + +// filterBindingsWithReferencedClusters cross-references placement bindings with their referenced clusters. +// +// Note that the returned slices preserve the original order and have matching indexes. +func filterBindingsWithReferencedClusters( + placementBindings []placementv1alpha1.PlacementBindingAccessor, + clusters []*clusterv1beta1.MemberCluster, +) ([]placementv1alpha1.PlacementBindingAccessor, []*clusterv1beta1.MemberCluster) { + filteredBindings := make([]placementv1alpha1.PlacementBindingAccessor, 0, len(placementBindings)) + filteredClusters := make([]*clusterv1beta1.MemberCluster, 0, len(clusters)) + for idx := range placementBindings { + if clusters[idx] == nil { + continue + } + filteredBindings = append(filteredBindings, placementBindings[idx]) + filteredClusters = append(filteredClusters, clusters[idx]) + } + return filteredBindings, filteredClusters +} + +func (r *Reconciler) groupClusters( + stagedUpdateStrategy rolloutv1alpha1.StagedUpdateStrategyAccessor, + placementBindings []placementv1alpha1.PlacementBindingAccessor, + clusters []*clusterv1beta1.MemberCluster, +) ( + perStageStatuses []rolloutv1alpha1.PerStageStatus, + err error, +) { + checked := sets.Set[string]{} + + perStageStatuses = make([]rolloutv1alpha1.PerStageStatus, 0, len(stagedUpdateStrategy.GetSpec().Stages)) + + stagesInStrategy := stagedUpdateStrategy.GetSpec().Stages + for idx := range stagesInStrategy { + stage := stagesInStrategy[idx] + + // Handle the special cases where the stage has nil or empty label selector. + switch { + case stage.LabelSelector == nil: + // The stage has nil label selector; it includes no clusters at all. + perStageStatus, err := buildPerStageStatus(stage, nil) + if err != nil { + return nil, errors.Wraps(err, "failed to build per stage status", "stage", stage.Name) + } + perStageStatuses = append(perStageStatuses, perStageStatus) + continue + case stage.LabelSelector != nil && (len(stage.LabelSelector.MatchLabels) == 0 && len(stage.LabelSelector.MatchExpressions) == 0): + // The stage has an empty label selector; it includes all clusters. + matched, err := matchAndSortBindings(placementBindings, clusters, nil, stage.SortingLabelKey, checked) + if err != nil { + return nil, errors.Wraps(err, "failed to match and sort bindings for empty label selector", "stage", stage.Name) + } + // Build the stage information. + perStageStatus, err := buildPerStageStatus(stage, matched) + if err != nil { + return nil, errors.Wraps(err, "failed to build per stage status", "stage", stage.Name) + } + perStageStatuses = append(perStageStatuses, perStageStatus) + continue + } + + // The stage has a non-empty label selector; group and sort the clusters accordingly. + labelSelector, err := metav1.LabelSelectorAsSelector(stage.LabelSelector) + if err != nil { + return nil, errors.NewUserError(err, "failed to convert label selector to selector", "stage", stage.Name) + } + + // Find all the clusters that match the label selector, and their corresponding placement bindings. + matched, err := matchAndSortBindings(placementBindings, clusters, labelSelector, stage.SortingLabelKey, checked) + if err != nil { + return nil, errors.Wraps(err, "failed to match and sort bindings for non-empty label selector", "stage", stage.Name) + } + // Build the stage information. + perStageStatus, err := buildPerStageStatus(stage, matched) + if err != nil { + return nil, errors.Wraps(err, "failed to build per stage status", "stage", stage.Name) + } + perStageStatuses = append(perStageStatuses, perStageStatus) + } + + return perStageStatuses, nil +} + +type matchedCluster struct { + cluster *clusterv1beta1.MemberCluster + binding placementv1alpha1.PlacementBindingAccessor + + weight int +} + +func matchAndSortBindings( + bindings []placementv1alpha1.PlacementBindingAccessor, + clusters []*clusterv1beta1.MemberCluster, + labelSelector labels.Selector, + sortingLabelKey *string, + checked sets.Set[string], +) ([]*matchedCluster, error) { + // Find all the clusters that match the label selector, and their corresponding placement bindings. + + // Pre-allocate with a reasonable capacity. + matched := make([]*matchedCluster, 0, 10) + for idx := range clusters { + cluster := clusters[idx] + binding := bindings[idx] + + if checked.Has(cluster.GetName()) { + klog.V(2).InfoS("Cluster has already been included in a previous stage; skip it", + "cluster", klog.KObj(cluster), "placementBinding", klog.KObj(binding)) + continue + } + + if labelSelector != nil { + if labelSelector.Matches(labels.Set(cluster.GetLabels())) { + matched = append(matched, &matchedCluster{cluster: cluster, binding: binding}) + checked.Insert(cluster.GetName()) + + klog.V(2).InfoS("Cluster matches the label selector; include it", + "cluster", klog.KObj(cluster), "placementBinding", klog.KObj(binding)) + } + } else { + // No label selector is specified; include all clusters. + matched = append(matched, &matchedCluster{cluster: cluster, binding: binding}) + checked.Insert(cluster.GetName()) + + klog.V(2).InfoS("The stage has empty label selector; match all clusters", + "cluster", klog.KObj(cluster), "placementBinding", klog.KObj(binding)) + } + } + + // Sort the matched clusters if a sorting label key is specified. + if sortingLabelKey != nil && len(*sortingLabelKey) > 0 { + for idx := range matched { + labelValue, ok := matched[idx].cluster.GetLabels()[*sortingLabelKey] + if !ok { + return nil, errors.NewUserError(nil, "a sorting label key is specified in staged update strategy, yet the label is missing on a cluster", + "sortingLabelKey", *sortingLabelKey, + "cluster", klog.KObj(matched[idx].cluster), "placementBinding", klog.KObj(matched[idx].binding)) + } + + sortValue, parseErr := strconv.Atoi(labelValue) + if parseErr != nil { + return nil, errors.NewUserError(parseErr, "failed to parse sorting label value as integer", + "sortingLabelKey", *sortingLabelKey, "sortingLabelValue", labelValue, + "cluster", klog.KObj(matched[idx].cluster), "placementBinding", klog.KObj(matched[idx].binding)) + } + matched[idx].weight = sortValue + } + + sort.SliceStable(matched, func(i, j int) bool { + if matched[i].weight == matched[j].weight { + return matched[i].cluster.GetName() < matched[j].cluster.GetName() + } + return matched[i].weight < matched[j].weight + }) + } else { + // Sort clusters by name in ascending order when no sorting label key is specified. + sort.SliceStable(matched, func(i, j int) bool { + return matched[i].cluster.GetName() < matched[j].cluster.GetName() + }) + } + + return matched, nil +} + +func buildPerStageStatus( + stage rolloutv1alpha1.Stage, + matched []*matchedCluster, +) (rolloutv1alpha1.PerStageStatus, error) { + // Record the clusters assigned to the stage, and their corresponding placement bindings. + perClusterStatuses := make([]rolloutv1alpha1.PerClusterStatus, 0, len(matched)) + for idx := range matched { + matchedCluster := matched[idx].cluster + matchedBinding := matched[idx].binding + + perClusterStatuses = append(perClusterStatuses, rolloutv1alpha1.PerClusterStatus{ + ClusterName: matchedCluster.GetName(), + PlacementBindingName: matchedBinding.GetName(), + }) + } + + // Record the before-stage tasks and after-stage tasks. + beforeStageTaskStatuses := make([]rolloutv1alpha1.PerStageTaskStatus, 0, len(stage.BeforeStageTasks)) + for idx := range stage.BeforeStageTasks { + beforeStageTask := stage.BeforeStageTasks[idx] + beforeStageTaskStatus := rolloutv1alpha1.PerStageTaskStatus{ + Type: beforeStageTask.Type, + } + if beforeStageTask.Type != rolloutv1alpha1.StageTaskTypeApproval { + return rolloutv1alpha1.PerStageStatus{}, errors.NewUserError(nil, + "unsupported before-stage task type (only approval task can be added as a before stage task)", + "taskType", beforeStageTask.Type, + "stage", stage.Name) + } + beforeStageTaskStatuses = append(beforeStageTaskStatuses, beforeStageTaskStatus) + } + + afterStageTaskStatuses := make([]rolloutv1alpha1.PerStageTaskStatus, 0, len(stage.AfterStageTasks)) + for idx := range stage.AfterStageTasks { + afterStageTask := stage.AfterStageTasks[idx] + afterStageTaskStatus := rolloutv1alpha1.PerStageTaskStatus{ + Type: afterStageTask.Type, + } + if afterStageTask.Type == rolloutv1alpha1.StageTaskTypeTimedWait { + afterStageTaskStatus.WaitTime = afterStageTask.WaitTime + } + afterStageTaskStatuses = append(afterStageTaskStatuses, afterStageTaskStatus) + } + + // Resolve the max concurrency value for the stage. + maxConcurrencyValue := int32(1) + if stage.MaxConcurrency != nil { + resolvedMaxConcurrency, err := intstr.GetScaledValueFromIntOrPercent(stage.MaxConcurrency, len(matched), false) + if err != nil { + return rolloutv1alpha1.PerStageStatus{}, errors.NewUserError(err, "failed to resolve max concurrency", + "stage", stage.Name, "totalClustersCnt", len(matched), "maxConcurrency", stage.MaxConcurrency) + } + if resolvedMaxConcurrency == 0 { + resolvedMaxConcurrency = 1 + } + maxConcurrencyValue = int32(resolvedMaxConcurrency) + } + + return rolloutv1alpha1.PerStageStatus{ + StageName: stage.Name, + Clusters: perClusterStatuses, + BeforeStageTasks: beforeStageTaskStatuses, + AfterStageTasks: afterStageTaskStatuses, + MaxConcurrency: &maxConcurrencyValue, + }, nil +} + +func (r *Reconciler) identifyResourceSnapshotToRollout( + ctx context.Context, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, +) (placementv1alpha1.PlacementResourceSnapshotAccessor, error) { + resourceSnapshotName := stagedUpdateRun.GetSpec().ResourceSnapshotName + if resourceSnapshotName == "" { + // If no resource snapshot is explicitly specified, retrieve the latest snapshot or request a new one if the latest is stale. + resourceSnapshots, _, err := r.placementResourceSnapshotManager.SnapshotResourcesIfStale( + ctx, placementPolicy) + if err != nil { + return nil, errors.Wraps(err, "failed to retrieve or create the latest resource snapshot") + } + return resourceSnapshots[0], nil + } + + // A resource snapshot has been explicitly specified; check if it exists. + var resourceSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + snapshotKey := types.NamespacedName{ + Namespace: placementPolicy.GetNamespace(), + Name: resourceSnapshotName, + } + if snapshotKey.Namespace == "" { + // The resource snapshot is cluster-scoped; retrieve the ClusterPlacementResourceSnapshot. + resourceSnapshot = &placementv1alpha1.ClusterPlacementResourceSnapshot{} + } else { + // The resource snapshot is namespaced; retrieve the PlacementResourceSnapshot in its namespace. + resourceSnapshot = &placementv1alpha1.PlacementResourceSnapshot{} + } + + if err := r.hubClient.Get(ctx, snapshotKey, resourceSnapshot); err != nil { + if apierrors.IsNotFound(err) { + return nil, errors.NewUserError(err, "the explicitly specified resource snapshot does not exist", + "resourceSnapshot", klog.KRef(snapshotKey.Namespace, snapshotKey.Name)) + } + return nil, errors.NewAPIServerError(err, "failed to get the explicitly specified resource snapshot", true, + "resourceSnapshot", klog.KRef(snapshotKey.Namespace, snapshotKey.Name)) + } + + // Check if the retrieved resource snapshot has been marked for deletion and if it is a primary resource snapshot. + if !resourceSnapshot.GetDeletionTimestamp().IsZero() { + // A resource snapshot marked for deletion is on its way out (most likely reclaimed by the + // revision history limit); rolling out against it would race the deletion and is never + // the user's intent, so this is reported back as a user error rather than retried. + return nil, errors.NewUserError(nil, "the resource snapshot to roll out has been marked for deletion", + "resourceSnapshot", klog.KRef(snapshotKey.Namespace, snapshotKey.Name)) + } + if subIdx := resourceSnapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey]; subIdx != "0" { + // Only the snapshot of the sub-index 0 is the primary snapshot of its index; a staged update + // run can only be rolled out against a primary snapshot, since a secondary one does not carry + // the full picture of the resources selected at that point in time. + return nil, errors.NewUserError(nil, "the resource snapshot to roll out is not a primary resource snapshot", + "resourceSnapshot", klog.KRef(snapshotKey.Namespace, snapshotKey.Name), "subIndex", subIdx) + } + + // Check if the resource snapshot is actually owned by the placement policy. + if !metav1.IsControlledBy(resourceSnapshot, placementPolicy) { + return nil, errors.NewUserError(nil, "the resource snapshot to roll out is not owned by the linked placement policy", + "resourceSnapshot", klog.KRef(snapshotKey.Namespace, snapshotKey.Name), + "placementPolicy", klog.KObj(placementPolicy)) + } + + return resourceSnapshot, nil +} diff --git a/pkg/v1/controllers/stagedupdaterun/initialize_integration_test.go b/pkg/v1/controllers/stagedupdaterun/initialize_integration_test.go new file mode 100644 index 000000000..10df1d911 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/initialize_integration_test.go @@ -0,0 +1,2976 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "fmt" + "slices" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +const ( + canaryStageAfterStageWaitTime = time.Minute * 10 + prodStageAfterStageWaitTime = time.Minute * 5 + + defaultConcurrencyStageName = "default" + fixedConcurrencyStageName = "fixed" + halfConcurrencyStageName = "half" + tenthConcurrencyStageName = "tenth" + + groupLabelValueA = "a" + groupLabelValueB = "b" + groupLabelValueC = "c" + groupLabelValueD = "d" +) + +// Substrings of the messages that the controller reports when it fails to initialize a staged update run. +const ( + missingSortingLabelMsgSubstr = "yet the label is missing on a cluster" + nonIntegerSortingLabelMsgSubstr = "failed to parse sorting label value as integer" + noPlacementBindingsMsgSubstr = "no placement bindings are found" + strategyNotFoundMsgSubstr = "staged update strategy cannot be found" + snapshotNotFoundMsgSubstr = "the explicitly specified resource snapshot does not exist" + snapshotNotPrimaryMsgSubstr = "is not a primary resource snapshot" + snapshotNotOwnedMsgSubstr = "is not owned by the linked placement policy" + snapshotDeletedMsgSubstr = "has been marked for deletion" + // The placement resource snapshot manager reports a resource selected by name that does not exist as a user error. + selectedResourceNotFoundMsgSubstr = "selected resource does not exist" +) + +var _ = Describe("initializing staged update runs", func() { + // Unless a test case verifies an initialization failure, the staged update runs in this block are created with + // the suspended flag set; the controller initializes such runs, but does not start rolling out changes, which + // leaves the runs in a stable, fully initialized state. + + Context("label-selected stages", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0 and 2 are canary clusters; member clusters 1, 3, and 4 are production clusters. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 5) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + } + + var policy placementv1alpha1.PlacementPolicyAccessor + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: canaryStageAfterStageWaitTime}, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + MaxConcurrency: ptr.To(intstr.FromInt32(2)), + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: prodStageAfterStageWaitTime}, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 2), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: canaryStageAfterStageWaitTime}, + }, + }, + // The max concurrency defaults to 1. + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1, 3, 4), + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: prodStageAfterStageWaitTime}, + }, + }, + MaxConcurrency: ptr.To(int32(2)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + It("should create a primary placement resource snapshot for the placement policy", func() { + snapshotCreatedActual := primaryPlacementResourceSnapshotCreatedActual(policy, 0) + Eventually(snapshotCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the placement resource snapshot") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("overlapping, nil, and empty label selectors", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 1 is selected by both the east stage and the prod stage; member cluster 3 is selected by + // none of the stages with a non-empty label selector. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary, regionLabelKey: regionLabelValueEast}, + {envLabelKey: envLabelValueProd, regionLabelKey: regionLabelValueEast}, + {envLabelKey: envLabelValueProd, regionLabelKey: regionLabelValueWest}, + {envLabelKey: envLabelValueCanary, regionLabelKey: regionLabelValueWest}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: eastStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{regionLabelKey: regionLabelValueEast}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + { + // A stage with a nil label selector includes no clusters at all. + Name: emptyStageName, + }, + { + // A stage with an empty label selector includes all the clusters that no previous stage has + // included. + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: eastStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + { + // Member cluster 1 has been included in the east stage, which comes first. + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: emptyStageName, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("clusters selected by no stage", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 2 has no env label, and is selected by none of the stages. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {regionLabelKey: regionLabelValueWest}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run, leaving out the clusters selected by no stage", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("clusters sorted by an integer label", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The label values are picked so that their numerical order (1, 2, 2, 10) differs from both their lexical + // order (1, 10, 2, 2) and the order of the cluster names; member clusters 1 and 3 share the same value. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueProd, orderLabelKey: "10"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "2"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "1"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "2"}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + SortingLabelKey: ptr.To(orderLabelKey), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run, with clusters sorted by the label value (and then by name)", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2, 1, 3, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("clusters missing the sorting label", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + // Member cluster 1 does not have the sorting label. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueProd, orderLabelKey: "1"}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + SortingLabelKey: ptr.To(orderLabelKey), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, missingSortingLabelMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("clusters with a non-integer sorting label", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + // The sorting label value of member cluster 1 is not an integer. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueProd, orderLabelKey: "1"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "first"}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + SortingLabelKey: ptr.To(orderLabelKey), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, nonIntegerSortingLabelMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("max concurrency resolution", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 0 is in group A; member clusters 1-2 are in group B; member clusters 3-7 are in group C; + // member clusters 8-10 are in group D. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 11) + memberClusterGroups := []string{ + groupLabelValueA, + groupLabelValueB, groupLabelValueB, + groupLabelValueC, groupLabelValueC, groupLabelValueC, groupLabelValueC, groupLabelValueC, + groupLabelValueD, groupLabelValueD, groupLabelValueD, + } + memberClusterLabels := make([]map[string]string, 0, len(memberClusterGroups)) + for _, group := range memberClusterGroups { + memberClusterLabels = append(memberClusterLabels, map[string]string{groupLabelKey: group}) + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + // No max concurrency is specified; the API server sets it to 1 by default. + Name: defaultConcurrencyStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{groupLabelKey: groupLabelValueA}, + }, + }, + { + Name: fixedConcurrencyStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{groupLabelKey: groupLabelValueB}, + }, + MaxConcurrency: ptr.To(intstr.FromInt32(3)), + }, + { + Name: halfConcurrencyStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{groupLabelKey: groupLabelValueC}, + }, + MaxConcurrency: ptr.To(intstr.FromString("50%")), + }, + { + Name: tenthConcurrencyStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{groupLabelKey: groupLabelValueD}, + }, + MaxConcurrency: ptr.To(intstr.FromString("10%")), + }, + { + Name: emptyStageName, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run, with the max concurrency of each stage resolved", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: defaultConcurrencyStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + { + // An integer value is used as it is, even if it exceeds the number of clusters in the stage. + StageName: fixedConcurrencyStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1, 2), + MaxConcurrency: ptr.To(int32(3)), + }, + { + // 50% of 5 clusters is 2.5, which is rounded down. + StageName: halfConcurrencyStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3, 4, 5, 6, 7), + MaxConcurrency: ptr.To(int32(2)), + }, + { + // 10% of 3 clusters is 0.3, which is rounded down to 0 and then raised to the minimum of 1. + StageName: tenthConcurrencyStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 8, 9, 10), + MaxConcurrency: ptr.To(int32(1)), + }, + { + // 100% of no clusters is 0, which is raised to the minimum of 1. + StageName: emptyStageName, + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings and member clusters to leave out", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + otherPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0 and 1 are bound to the placement policy as usual; the others should be left out: + // * member cluster 2 is bound to another placement policy; + // * the placement binding of member cluster 3 has been marked for deletion; + // * member cluster 4 does not exist; and + // * member cluster 5 has been marked for deletion. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 6) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + + fixtures := make([]client.Object, 0, 2*len(memberClusterNames)+2) + fixtures = append(fixtures, policy) + for idx := range memberClusterNames { + if idx != 4 { + fixtures = append(fixtures, createMemberCluster(memberClusterNames[idx], nil)) + } + bindingPolicyName := policyName + if idx == 2 { + bindingPolicyName = otherPolicyName + } + fixtures = append(fixtures, createPlacementBinding(appNamespaceName, bindingNames[idx], bindingPolicyName, memberClusterNames[idx], previousResourceSnapshotName)) + } + markObjectForDeletion(newPlacementBindingObj(appNamespaceName, bindingNames[3])) + markObjectForDeletion(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[5]}}) + + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run, leaving out the placement bindings and member clusters that do not count", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementBindingObj(appNamespaceName, bindingNames[3])) + removeTestFinalizer(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[5]}}) + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("no placement bindings that count", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + otherPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + // Member cluster 0 does not exist; member cluster 1 has been marked for deletion; member cluster 2 is bound + // to another placement policy. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 3) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + binding0 := createPlacementBinding(appNamespaceName, bindingNames[0], policyName, memberClusterNames[0], previousResourceSnapshotName) + memberCluster1 := createMemberCluster(memberClusterNames[1], nil) + binding1 := createPlacementBinding(appNamespaceName, bindingNames[1], policyName, memberClusterNames[1], previousResourceSnapshotName) + memberCluster2 := createMemberCluster(memberClusterNames[2], nil) + binding2 := createPlacementBinding(appNamespaceName, bindingNames[2], otherPolicyName, memberClusterNames[2], previousResourceSnapshotName) + markObjectForDeletion(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[1]}}) + + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(policy, binding0, memberCluster1, binding1, memberCluster2, binding2, strategy) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, noPlacementBindingsMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeTestFinalizer(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[1]}}) + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings in another namespace", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 1 is bound by a placement binding in another namespace, which carries the same placement + // policy name. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + memberCluster0 := createMemberCluster(memberClusterNames[0], nil) + binding0 := createPlacementBinding(appNamespaceName, bindingNames[0], policyName, memberClusterNames[0], previousResourceSnapshotName) + memberCluster1 := createMemberCluster(memberClusterNames[1], nil) + binding1 := createPlacementBinding(otherAppNamespaceName, bindingNames[1], policyName, memberClusterNames[1], previousResourceSnapshotName) + + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(policy, memberCluster0, binding0, memberCluster1, binding1, strategy) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run, leaving out the placement bindings in other namespaces", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + AfterAll(func() { + removeObjects( + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + newPlacementBindingObj(appNamespaceName, bindingNames[0]), + &clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[0]}}, + newPlacementBindingObj(otherAppNamespaceName, bindingNames[1]), + &clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[1]}}, + ) + }) + }) + + Context("missing staged update strategy", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + waitForObjectsToBeCached(append(fixtures, policy)...) + + // The staged update strategy has not been created yet. + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should release the binding manager role of the placement policy", func() { + // The controller claims the role before it initializes the run, and releases it once the run has failed + // to initialize. + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can create the staged update strategy", func() { + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(strategy) + }) + + It("can trigger a new reconciliation of the staged update run", func() { + // The controller does not watch staged update strategies. The trigger leaves the run as is (in particular, + // the run is not suspended): had the controller initialized the run now that the strategy exists, it would + // go on to claim the binding manager role and roll out, which the check below would catch. + triggerStagedUpdateRunReconciliation(appNamespaceName, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not initialize the staged update run again", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, strategyNotFoundMsgSubstr) + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + Consistently(func() error { + if err := failedToInitializeActual(); err != nil { + return err + } + if err := roleReleasedActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Initialized the staged update run, claimed the binding manager role, or rolled out unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, triggerApprovalRequestName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("staged update strategy in another namespace", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + // The staged update strategy is created in a namespace other than that of the staged update run. + strategy := createStagedUpdateStrategy(otherAppNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newStagedUpdateStrategyObj(otherAppNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("resource snapshot selection (no resource snapshot specified)", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + firstRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-0") + secondRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-1") + thirdRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-2") + var firstSnapshotName string + var secondSnapshotName string + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + } + + var policy placementv1alpha1.PlacementPolicyAccessor + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + firstSnapshotName = primaryPlacementResourceSnapshotName(policy, 0) + secondSnapshotName = primaryPlacementResourceSnapshotName(policy, 1) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + }) + + It("should create a resource snapshot when none exists", func() { + createStagedUpdateRun(appNamespaceName, firstRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(firstSnapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, firstRunName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + + snapshotCreatedActual := primaryPlacementResourceSnapshotCreatedActual(policy, 0) + Eventually(snapshotCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the placement resource snapshot") + }) + + It("should reuse the latest resource snapshot when the selected resources have not changed", func() { + createStagedUpdateRun(appNamespaceName, secondRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + + // Had a new resource snapshot been created, the run would roll out that one (of index 1) instead. + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(firstSnapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, secondRunName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + It("can update the selected resources", func() { + updateConfigMapData(configMapName, map[string]string{configMapDataKey: altConfigMapDataValue}) + }) + + It("should create a new resource snapshot when the selected resources have changed", func() { + createStagedUpdateRun(appNamespaceName, thirdRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(secondSnapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, thirdRunName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + + snapshotCreatedActual := primaryPlacementResourceSnapshotCreatedActual(policy, 1) + Eventually(snapshotCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the placement resource snapshot") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, firstRunName), + newStagedUpdateRunObj(appNamespaceName, secondRunName), + newStagedUpdateRunObj(appNamespaceName, thirdRunName), + newPlacementResourceSnapshotObj(appNamespaceName, firstSnapshotName), + newPlacementResourceSnapshotObj(appNamespaceName, secondSnapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("resource snapshot selection (resource snapshot specified)", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + otherPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + + // A primary snapshot of the placement policy (index 0, sub-index 0), along with its secondary snapshot + // (index 0, sub-index 1). + var primarySnapshotName string + var secondarySnapshotName string + // A primary snapshot of the placement policy (index 1) that has been marked for deletion. + var deletedSnapshotName string + // A primary snapshot of another placement policy. + var otherPolicySnapshotName string + // A snapshot that does not exist. + var missingSnapshotName string + + runNames := make([]string, 0, 5) + for idx := 0; idx < 5; idx++ { + runNames = append(runNames, fmt.Sprintf(stagedUpdateRunNameTemplate, fmt.Sprintf("%s-%d", randStr, idx))) + } + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + primarySnapshotName = primaryPlacementResourceSnapshotName(policy, 0) + secondarySnapshotName = secondaryPlacementResourceSnapshotName(policy, 0, 1) + deletedSnapshotName = primaryPlacementResourceSnapshotName(policy, 1) + missingSnapshotName = primaryPlacementResourceSnapshotName(policy, 2) + otherPolicy := createPlacementPolicy(appNamespaceName, otherPolicyName, configMapName) + otherPolicySnapshotName = primaryPlacementResourceSnapshotName(otherPolicy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + + primarySnapshot := createPlacementResourceSnapshot(primarySnapshotName, policy, 0, 0, 2) + secondarySnapshot := createPlacementResourceSnapshot(secondarySnapshotName, policy, 0, 1, 2) + deletedSnapshot := createPlacementResourceSnapshot(deletedSnapshotName, policy, 1, 0, 1) + otherPolicySnapshot := createPlacementResourceSnapshot(otherPolicySnapshotName, otherPolicy, 0, 0, 1) + markObjectForDeletion(deletedSnapshot) + + waitForObjectsToBeCached(append(fixtures, policy, otherPolicy, strategy, primarySnapshot, secondarySnapshot, deletedSnapshot, otherPolicySnapshot)...) + }) + + It("should initialize the staged update run with the specified resource snapshot", func() { + createStagedUpdateRun(appNamespaceName, runNames[0], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: primarySnapshotName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(primarySnapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runNames[0], wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + It("should fail to initialize the staged update run if the specified resource snapshot does not exist", func() { + createStagedUpdateRun(appNamespaceName, runNames[1], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: missingSnapshotName, + StagedUpdateStrategyName: strategyName, + }) + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runNames[1], snapshotNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should fail to initialize the staged update run if the specified resource snapshot is not a primary one", func() { + createStagedUpdateRun(appNamespaceName, runNames[2], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: secondarySnapshotName, + StagedUpdateStrategyName: strategyName, + }) + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runNames[2], snapshotNotPrimaryMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should fail to initialize the staged update run if the specified resource snapshot belongs to another placement policy", func() { + createStagedUpdateRun(appNamespaceName, runNames[3], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: otherPolicySnapshotName, + StagedUpdateStrategyName: strategyName, + }) + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runNames[3], snapshotNotOwnedMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should fail to initialize the staged update run if the specified resource snapshot has been marked for deletion", func() { + createStagedUpdateRun(appNamespaceName, runNames[4], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: deletedSnapshotName, + StagedUpdateStrategyName: strategyName, + }) + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runNames[4], snapshotDeletedMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementResourceSnapshotObj(appNamespaceName, deletedSnapshotName)) + + objs := make([]client.Object, 0, len(runNames)) + for _, runName := range runNames { + objs = append(objs, newStagedUpdateRunObj(appNamespaceName, runName)) + } + removeObjects(slices.Concat( + objs, + []client.Object{ + newPlacementResourceSnapshotObj(appNamespaceName, primarySnapshotName), + newPlacementResourceSnapshotObj(appNamespaceName, secondarySnapshotName), + newPlacementResourceSnapshotObj(appNamespaceName, deletedSnapshotName), + newPlacementResourceSnapshotObj(appNamespaceName, otherPolicySnapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newPlacementPolicyObj(appNamespaceName, otherPolicyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("selected resource missing", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + var snapshotName string + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + // The placement policy selects (by name) a ConfigMap that has not been created yet. + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, selectedResourceNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should release the binding manager role of the placement policy", func() { + // The controller claims the role before it initializes the run, and releases it once the run has failed + // to initialize. + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can create the selected ConfigMap", func() { + // No need to wait for the cache; the placement resource snapshot manager reads selected resources from the + // API server directly in the test environment. + createConfigMap(configMapName) + }) + + It("can trigger a new reconciliation of the staged update run", func() { + // The controller does not watch selected resources. The trigger leaves the run as is (in particular, the + // run is not suspended): had the controller initialized the run now that the ConfigMap exists, it would + // snapshot the ConfigMap, claim the binding manager role, and roll out, which the check below would catch. + triggerStagedUpdateRunReconciliation(appNamespaceName, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not initialize the staged update run again", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, selectedResourceNotFoundMsgSubstr) + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + snapshotAbsentActual := objectRemovedActual(newPlacementResourceSnapshotObj(appNamespaceName, snapshotName)) + Consistently(func() error { + if err := failedToInitializeActual(); err != nil { + return err + } + if err := roleReleasedActual(); err != nil { + return err + } + if err := bindingUntouchedActual(); err != nil { + return err + } + return snapshotAbsentActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Initialized the staged update run, snapshotted resources, claimed the binding manager role, or rolled out unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, triggerApprovalRequestName), + // Not expected to exist; removed in case a failed run of the test case created it. + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("one-shot initialization", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Initially, member cluster 0 is a canary cluster, and member clusters 1 and 2 are production clusters; + // member cluster 3 joins (with a staging label) after the staged update run has been initialized. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + } + + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1, 2), + MaxConcurrency: ptr.To(int32(1)), + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames[:3], bindingNames[:3], memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run", func() { + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + It("can change the staged update strategy, the member clusters, and the placement bindings", func() { + // Were the staged update run to be initialized again after the changes, the canary stage would include + // member cluster 3 only, and the prod stage would include member clusters 0, 1, and 2. + updateStagedUpdateStrategyStages(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueStaging}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + updateMemberClusterLabels(memberClusterNames[0], map[string]string{envLabelKey: envLabelValueProd}) + createMemberCluster(memberClusterNames[3], map[string]string{envLabelKey: envLabelValueStaging}) + createPlacementBinding(appNamespaceName, bindingNames[3], policyName, memberClusterNames[3], previousResourceSnapshotName) + + // Wait until the cache observes all the changes, so that a new initialization (if any) would see them. + Eventually(func() error { + strategy := newStagedUpdateStrategyObj(appNamespaceName, strategyName) + if err := hubClient.Get(ctx, client.ObjectKeyFromObject(strategy), strategy); err != nil { + return err + } + if len(strategy.GetSpec().Stages) != 3 { + return fmt.Errorf("the staged update strategy has %d stages, want 3", len(strategy.GetSpec().Stages)) + } + + memberCluster := &clusterv1beta1.MemberCluster{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: memberClusterNames[0]}, memberCluster); err != nil { + return err + } + if memberCluster.Labels[envLabelKey] != envLabelValueProd { + return fmt.Errorf("member cluster %s has env label %q, want %q", memberClusterNames[0], memberCluster.Labels[envLabelKey], envLabelValueProd) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the changes in the cache") + waitForObjectsToBeCached( + &clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[3]}}, + newPlacementBindingObj(appNamespaceName, bindingNames[3]), + ) + }) + + It("can resume the staged update run", func() { + updateStagedUpdateRunSuspended(appNamespaceName, runName, false) + }) + + It("should mark the staged update run as resumed", func() { + resumedActual := stagedUpdateRunConditionActual(appNamespaceName, runName, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeSuspended, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunSuspendedCondReasonResumed, + }) + Eventually(resumedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to resume the staged update run") + }) + + It("should start rolling out to the member cluster in the first stage as initialized", func() { + // Member cluster 0 belongs to the canary stage only as initialized. + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + + // The rollout is blocked on member cluster 0 (its placement binding never becomes available); in + // particular, member cluster 3 (which a new initialization would have put in the canary stage) is + // never rolled out to. + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames[1:], previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to other member clusters unexpectedly") + }) + + It("should keep the stages as initialized", func() { + // The check runs only once the rollout has started, i.e., once the controller has processed the resumed run + // (including its initialization step); the Resumed condition alone does not prove this, as the controller + // sets it before it gets to the initialization step. + stageLayoutActual := stagedUpdateRunStageLayoutActual(appNamespaceName, runName, wantStages) + Eventually(stageLayoutActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to keep the stages as initialized") + }) + + AfterAll(func() { + // Note that deleting the placement bindings also unblocks the rollout (the controller fails to find + // the binding it is waiting for), so that the staged update run can be removed in time. + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("staged update runs created as suspended", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run and mark it as suspended (without starting it)", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + It("should remove the cleanup finalizer from the staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(appNamespaceName, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("should release the binding manager role of the placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("should not roll out to any member cluster", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames, previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to member clusters unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("initializing cluster staged update runs", func() { + // Unless a test case verifies an initialization failure, the cluster staged update runs in this block are created + // with the suspended flag set; the controller initializes such runs, but does not start rolling out changes, which + // leaves the runs in a stable, fully initialized state. + + Context("label-selected stages", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0 and 2 are canary clusters; member clusters 1, 3, and 4 are production clusters. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 5) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + } + + var policy placementv1alpha1.PlacementPolicyAccessor + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: canaryStageAfterStageWaitTime}, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + MaxConcurrency: ptr.To(intstr.FromInt32(2)), + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: prodStageAfterStageWaitTime}, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 2), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: canaryStageAfterStageWaitTime}, + }, + }, + // The max concurrency defaults to 1. + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1, 3, 4), + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: prodStageAfterStageWaitTime}, + }, + }, + MaxConcurrency: ptr.To(int32(2)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + It("should create a primary cluster placement resource snapshot for the cluster placement policy", func() { + snapshotCreatedActual := primaryPlacementResourceSnapshotCreatedActual(policy, 0) + Eventually(snapshotCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster placement resource snapshot") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("overlapping, nil, and empty label selectors", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 1 is selected by both the east stage and the prod stage; member cluster 3 is selected by + // none of the stages with a non-empty label selector. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary, regionLabelKey: regionLabelValueEast}, + {envLabelKey: envLabelValueProd, regionLabelKey: regionLabelValueEast}, + {envLabelKey: envLabelValueProd, regionLabelKey: regionLabelValueWest}, + {envLabelKey: envLabelValueCanary, regionLabelKey: regionLabelValueWest}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: eastStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{regionLabelKey: regionLabelValueEast}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + { + // A stage with a nil label selector includes no clusters at all. + Name: emptyStageName, + }, + { + // A stage with an empty label selector includes all the clusters that no previous stage has + // included. + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: eastStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + { + // Member cluster 1 has been included in the east stage, which comes first. + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: emptyStageName, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("clusters selected by no stage", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 2 has no env label, and is selected by none of the stages. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {regionLabelKey: regionLabelValueWest}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run, leaving out the clusters selected by no stage", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("clusters sorted by an integer label", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The label values are picked so that their numerical order (1, 2, 2, 10) differs from both their lexical + // order (1, 10, 2, 2) and the order of the cluster names; member clusters 1 and 3 share the same value. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueProd, orderLabelKey: "10"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "2"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "1"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "2"}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + SortingLabelKey: ptr.To(orderLabelKey), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run, with clusters sorted by the label value (and then by name)", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2, 1, 3, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("clusters missing the sorting label", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + // Member cluster 1 does not have the sorting label. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueProd, orderLabelKey: "1"}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + SortingLabelKey: ptr.To(orderLabelKey), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the cluster staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, missingSortingLabelMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("clusters with a non-integer sorting label", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + // The sorting label value of member cluster 1 is not an integer. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueProd, orderLabelKey: "1"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "first"}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + SortingLabelKey: ptr.To(orderLabelKey), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the cluster staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, nonIntegerSortingLabelMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("max concurrency resolution", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 0 is in group A; member clusters 1-2 are in group B; member clusters 3-7 are in group C; + // member clusters 8-10 are in group D. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 11) + memberClusterGroups := []string{ + groupLabelValueA, + groupLabelValueB, groupLabelValueB, + groupLabelValueC, groupLabelValueC, groupLabelValueC, groupLabelValueC, groupLabelValueC, + groupLabelValueD, groupLabelValueD, groupLabelValueD, + } + memberClusterLabels := make([]map[string]string, 0, len(memberClusterGroups)) + for _, group := range memberClusterGroups { + memberClusterLabels = append(memberClusterLabels, map[string]string{groupLabelKey: group}) + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + // No max concurrency is specified; the API server sets it to 1 by default. + Name: defaultConcurrencyStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{groupLabelKey: groupLabelValueA}, + }, + }, + { + Name: fixedConcurrencyStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{groupLabelKey: groupLabelValueB}, + }, + MaxConcurrency: ptr.To(intstr.FromInt32(3)), + }, + { + Name: halfConcurrencyStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{groupLabelKey: groupLabelValueC}, + }, + MaxConcurrency: ptr.To(intstr.FromString("50%")), + }, + { + Name: tenthConcurrencyStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{groupLabelKey: groupLabelValueD}, + }, + MaxConcurrency: ptr.To(intstr.FromString("10%")), + }, + { + Name: emptyStageName, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run, with the max concurrency of each stage resolved", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: defaultConcurrencyStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + { + // An integer value is used as it is, even if it exceeds the number of clusters in the stage. + StageName: fixedConcurrencyStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1, 2), + MaxConcurrency: ptr.To(int32(3)), + }, + { + // 50% of 5 clusters is 2.5, which is rounded down. + StageName: halfConcurrencyStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3, 4, 5, 6, 7), + MaxConcurrency: ptr.To(int32(2)), + }, + { + // 10% of 3 clusters is 0.3, which is rounded down to 0 and then raised to the minimum of 1. + StageName: tenthConcurrencyStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 8, 9, 10), + MaxConcurrency: ptr.To(int32(1)), + }, + { + // 100% of no clusters is 0, which is raised to the minimum of 1. + StageName: emptyStageName, + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings and member clusters to leave out", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + otherPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0 and 1 are bound to the cluster placement policy as usual; the others should be left out: + // * member cluster 2 is bound to another cluster placement policy; + // * the placement binding of member cluster 3 has been marked for deletion; + // * member cluster 4 does not exist; and + // * member cluster 5 has been marked for deletion. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 6) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + + fixtures := make([]client.Object, 0, 2*len(memberClusterNames)+2) + fixtures = append(fixtures, policy) + for idx := range memberClusterNames { + if idx != 4 { + fixtures = append(fixtures, createMemberCluster(memberClusterNames[idx], nil)) + } + bindingPolicyName := policyName + if idx == 2 { + bindingPolicyName = otherPolicyName + } + fixtures = append(fixtures, createPlacementBinding(clusterScopedNS, bindingNames[idx], bindingPolicyName, memberClusterNames[idx], previousResourceSnapshotName)) + } + markObjectForDeletion(newPlacementBindingObj(clusterScopedNS, bindingNames[3])) + markObjectForDeletion(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[5]}}) + + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run, leaving out the placement bindings and member clusters that do not count", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementBindingObj(clusterScopedNS, bindingNames[3])) + removeTestFinalizer(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[5]}}) + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("no placement bindings that count", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + otherPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + // Member cluster 0 does not exist; member cluster 1 has been marked for deletion; member cluster 2 is bound + // to another cluster placement policy. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 3) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + binding0 := createPlacementBinding(clusterScopedNS, bindingNames[0], policyName, memberClusterNames[0], previousResourceSnapshotName) + memberCluster1 := createMemberCluster(memberClusterNames[1], nil) + binding1 := createPlacementBinding(clusterScopedNS, bindingNames[1], policyName, memberClusterNames[1], previousResourceSnapshotName) + memberCluster2 := createMemberCluster(memberClusterNames[2], nil) + binding2 := createPlacementBinding(clusterScopedNS, bindingNames[2], otherPolicyName, memberClusterNames[2], previousResourceSnapshotName) + markObjectForDeletion(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[1]}}) + + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(policy, binding0, memberCluster1, binding1, memberCluster2, binding2, strategy) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the cluster staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, noPlacementBindingsMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeTestFinalizer(&clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[1]}}) + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("missing cluster staged update strategy", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + waitForObjectsToBeCached(append(fixtures, policy)...) + + // The cluster staged update strategy has not been created yet. + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the cluster staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should release the binding manager role of the cluster placement policy", func() { + // The controller claims the role before it initializes the run, and releases it once the run has failed + // to initialize. + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can create the cluster staged update strategy", func() { + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(strategy) + }) + + It("can trigger a new reconciliation of the cluster staged update run", func() { + // The controller does not watch staged update strategies. The trigger leaves the run as is (in particular, + // the run is not suspended): had the controller initialized the run now that the strategy exists, it would + // go on to claim the binding manager role and roll out, which the check below would catch. + triggerStagedUpdateRunReconciliation(clusterScopedNS, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not initialize the cluster staged update run again", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, strategyNotFoundMsgSubstr) + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + Consistently(func() error { + if err := failedToInitializeActual(); err != nil { + return err + } + if err := roleReleasedActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Initialized the cluster staged update run, claimed the binding manager role, or rolled out unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, triggerApprovalRequestName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("resource snapshot selection (no resource snapshot specified)", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + firstRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-0") + secondRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-1") + thirdRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-2") + var firstSnapshotName string + var secondSnapshotName string + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + } + + var policy placementv1alpha1.PlacementPolicyAccessor + + BeforeAll(func() { + createConfigMap(configMapName) + policy = createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + firstSnapshotName = primaryPlacementResourceSnapshotName(policy, 0) + secondSnapshotName = primaryPlacementResourceSnapshotName(policy, 1) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + }) + + It("should create a resource snapshot when none exists", func() { + createStagedUpdateRun(clusterScopedNS, firstRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(firstSnapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, firstRunName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + + snapshotCreatedActual := primaryPlacementResourceSnapshotCreatedActual(policy, 0) + Eventually(snapshotCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster placement resource snapshot") + }) + + It("should reuse the latest resource snapshot when the selected resources have not changed", func() { + createStagedUpdateRun(clusterScopedNS, secondRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + + // Had a new resource snapshot been created, the run would roll out that one (of index 1) instead. + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(firstSnapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, secondRunName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + It("can update the selected resources", func() { + updateConfigMapData(configMapName, map[string]string{configMapDataKey: altConfigMapDataValue}) + }) + + It("should create a new resource snapshot when the selected resources have changed", func() { + createStagedUpdateRun(clusterScopedNS, thirdRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(secondSnapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, thirdRunName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + + snapshotCreatedActual := primaryPlacementResourceSnapshotCreatedActual(policy, 1) + Eventually(snapshotCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster placement resource snapshot") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, firstRunName), + newStagedUpdateRunObj(clusterScopedNS, secondRunName), + newStagedUpdateRunObj(clusterScopedNS, thirdRunName), + newPlacementResourceSnapshotObj(clusterScopedNS, firstSnapshotName), + newPlacementResourceSnapshotObj(clusterScopedNS, secondSnapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("resource snapshot selection (resource snapshot specified)", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + otherPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + + // A primary snapshot of the cluster placement policy (index 0, sub-index 0), along with its secondary snapshot + // (index 0, sub-index 1). + var primarySnapshotName string + var secondarySnapshotName string + // A primary snapshot of the cluster placement policy (index 1) that has been marked for deletion. + var deletedSnapshotName string + // A primary snapshot of another cluster placement policy. + var otherPolicySnapshotName string + // A snapshot that does not exist. + var missingSnapshotName string + + runNames := make([]string, 0, 5) + for idx := 0; idx < 5; idx++ { + runNames = append(runNames, fmt.Sprintf(stagedUpdateRunNameTemplate, fmt.Sprintf("%s-%d", randStr, idx))) + } + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + primarySnapshotName = primaryPlacementResourceSnapshotName(policy, 0) + secondarySnapshotName = secondaryPlacementResourceSnapshotName(policy, 0, 1) + deletedSnapshotName = primaryPlacementResourceSnapshotName(policy, 1) + missingSnapshotName = primaryPlacementResourceSnapshotName(policy, 2) + otherPolicy := createPlacementPolicy(clusterScopedNS, otherPolicyName, configMapName) + otherPolicySnapshotName = primaryPlacementResourceSnapshotName(otherPolicy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + + primarySnapshot := createPlacementResourceSnapshot(primarySnapshotName, policy, 0, 0, 2) + secondarySnapshot := createPlacementResourceSnapshot(secondarySnapshotName, policy, 0, 1, 2) + deletedSnapshot := createPlacementResourceSnapshot(deletedSnapshotName, policy, 1, 0, 1) + otherPolicySnapshot := createPlacementResourceSnapshot(otherPolicySnapshotName, otherPolicy, 0, 0, 1) + markObjectForDeletion(deletedSnapshot) + + waitForObjectsToBeCached(append(fixtures, policy, otherPolicy, strategy, primarySnapshot, secondarySnapshot, deletedSnapshot, otherPolicySnapshot)...) + }) + + It("should initialize the cluster staged update run with the specified resource snapshot", func() { + createStagedUpdateRun(clusterScopedNS, runNames[0], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: primarySnapshotName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(primarySnapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runNames[0], wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + It("should fail to initialize the cluster staged update run if the specified resource snapshot does not exist", func() { + createStagedUpdateRun(clusterScopedNS, runNames[1], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: missingSnapshotName, + StagedUpdateStrategyName: strategyName, + }) + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runNames[1], snapshotNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should fail to initialize the cluster staged update run if the specified resource snapshot is not a primary one", func() { + createStagedUpdateRun(clusterScopedNS, runNames[2], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: secondarySnapshotName, + StagedUpdateStrategyName: strategyName, + }) + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runNames[2], snapshotNotPrimaryMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should fail to initialize the cluster staged update run if the specified resource snapshot belongs to another cluster placement policy", func() { + createStagedUpdateRun(clusterScopedNS, runNames[3], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: otherPolicySnapshotName, + StagedUpdateStrategyName: strategyName, + }) + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runNames[3], snapshotNotOwnedMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should fail to initialize the cluster staged update run if the specified resource snapshot has been marked for deletion", func() { + createStagedUpdateRun(clusterScopedNS, runNames[4], rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + ResourceSnapshotName: deletedSnapshotName, + StagedUpdateStrategyName: strategyName, + }) + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runNames[4], snapshotDeletedMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementResourceSnapshotObj(clusterScopedNS, deletedSnapshotName)) + + objs := make([]client.Object, 0, len(runNames)) + for _, runName := range runNames { + objs = append(objs, newStagedUpdateRunObj(clusterScopedNS, runName)) + } + removeObjects(slices.Concat( + objs, + []client.Object{ + newPlacementResourceSnapshotObj(clusterScopedNS, primarySnapshotName), + newPlacementResourceSnapshotObj(clusterScopedNS, secondarySnapshotName), + newPlacementResourceSnapshotObj(clusterScopedNS, deletedSnapshotName), + newPlacementResourceSnapshotObj(clusterScopedNS, otherPolicySnapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newPlacementPolicyObj(clusterScopedNS, otherPolicyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("selected resource missing", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + var snapshotName string + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + // The cluster placement policy selects (by name) a ConfigMap that has not been created yet. + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the cluster staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, selectedResourceNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should release the binding manager role of the cluster placement policy", func() { + // The controller claims the role before it initializes the run, and releases it once the run has failed + // to initialize. + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can create the selected ConfigMap", func() { + // No need to wait for the cache; the placement resource snapshot manager reads selected resources from the + // API server directly in the test environment. + createConfigMap(configMapName) + }) + + It("can trigger a new reconciliation of the cluster staged update run", func() { + // The controller does not watch selected resources. The trigger leaves the run as is (in particular, the + // run is not suspended): had the controller initialized the run now that the ConfigMap exists, it would + // snapshot the ConfigMap, claim the binding manager role, and roll out, which the check below would catch. + triggerStagedUpdateRunReconciliation(clusterScopedNS, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not initialize the cluster staged update run again", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, selectedResourceNotFoundMsgSubstr) + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + snapshotAbsentActual := objectRemovedActual(newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName)) + Consistently(func() error { + if err := failedToInitializeActual(); err != nil { + return err + } + if err := roleReleasedActual(); err != nil { + return err + } + if err := bindingUntouchedActual(); err != nil { + return err + } + return snapshotAbsentActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Initialized the cluster staged update run, snapshotted resources, claimed the binding manager role, or rolled out unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, triggerApprovalRequestName), + // Not expected to exist; removed in case a failed run of the test case created it. + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("one-shot initialization", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Initially, member cluster 0 is a canary cluster, and member clusters 1 and 2 are production clusters; + // member cluster 3 joins (with a staging label) after the cluster staged update run has been initialized. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + } + + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1, 2), + MaxConcurrency: ptr.To(int32(1)), + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames[:3], bindingNames[:3], memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run", func() { + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + It("can change the cluster staged update strategy, the member clusters, and the placement bindings", func() { + // Were the cluster staged update run to be initialized again after the changes, the canary stage would include + // member cluster 3 only, and the prod stage would include member clusters 0, 1, and 2. + updateStagedUpdateStrategyStages(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueStaging}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + updateMemberClusterLabels(memberClusterNames[0], map[string]string{envLabelKey: envLabelValueProd}) + createMemberCluster(memberClusterNames[3], map[string]string{envLabelKey: envLabelValueStaging}) + createPlacementBinding(clusterScopedNS, bindingNames[3], policyName, memberClusterNames[3], previousResourceSnapshotName) + + // Wait until the cache observes all the changes, so that a new initialization (if any) would see them. + Eventually(func() error { + strategy := newStagedUpdateStrategyObj(clusterScopedNS, strategyName) + if err := hubClient.Get(ctx, client.ObjectKeyFromObject(strategy), strategy); err != nil { + return err + } + if len(strategy.GetSpec().Stages) != 3 { + return fmt.Errorf("the cluster staged update strategy has %d stages, want 3", len(strategy.GetSpec().Stages)) + } + + memberCluster := &clusterv1beta1.MemberCluster{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: memberClusterNames[0]}, memberCluster); err != nil { + return err + } + if memberCluster.Labels[envLabelKey] != envLabelValueProd { + return fmt.Errorf("member cluster %s has env label %q, want %q", memberClusterNames[0], memberCluster.Labels[envLabelKey], envLabelValueProd) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the changes in the cache") + waitForObjectsToBeCached( + &clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[3]}}, + newPlacementBindingObj(clusterScopedNS, bindingNames[3]), + ) + }) + + It("can resume the cluster staged update run", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, runName, false) + }) + + It("should mark the cluster staged update run as resumed", func() { + resumedActual := stagedUpdateRunConditionActual(clusterScopedNS, runName, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeSuspended, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunSuspendedCondReasonResumed, + }) + Eventually(resumedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to resume the cluster staged update run") + }) + + It("should start rolling out to the member cluster in the first stage as initialized", func() { + // Member cluster 0 belongs to the canary stage only as initialized. + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + + // The rollout is blocked on member cluster 0 (its placement binding never becomes available); in + // particular, member cluster 3 (which a new initialization would have put in the canary stage) is + // never rolled out to. + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames[1:], previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to other member clusters unexpectedly") + }) + + It("should keep the stages as initialized", func() { + // The check runs only once the rollout has started, i.e., once the controller has processed the resumed run + // (including its initialization step); the Resumed condition alone does not prove this, as the controller + // sets it before it gets to the initialization step. + stageLayoutActual := stagedUpdateRunStageLayoutActual(clusterScopedNS, runName, wantStages) + Eventually(stageLayoutActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to keep the stages as initialized") + }) + + AfterAll(func() { + // Note that deleting the placement bindings also unblocks the rollout (the controller fails to find + // the binding it is waiting for), so that the cluster staged update run can be removed in time. + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster staged update runs created as suspended", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run and mark it as suspended (without starting it)", func() { + wantStages := []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + } + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, wantStages) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + It("should remove the cleanup finalizer from the cluster staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(clusterScopedNS, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("should release the binding manager role of the cluster placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("should not roll out to any member cluster", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames, previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to member clusters unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) +}) diff --git a/pkg/v1/controllers/stagedupdaterun/lifecycle_integration_test.go b/pkg/v1/controllers/stagedupdaterun/lifecycle_integration_test.go new file mode 100644 index 000000000..1e41114a8 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/lifecycle_integration_test.go @@ -0,0 +1,2705 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "fmt" + "slices" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +// A substring of the message that the controller reports when the placement policy of a staged update run cannot +// be found. +const noLinkedPlacementPolicyMsgSubstr = "No linked placement policy" + +// The binding manager claim details of another (fictional) controller. +const ( + otherBindingManagerControllerName = "other-controller" + otherBindingManagerAPIGroup = "example.kubefleet.dev" + otherBindingManagerAPIVersion = "v1" + otherBindingManagerKind = "OtherRollout" + otherBindingManagerNameTemplate = "other-rollout-%s" +) + +// beforeStageApprovalStages returns the stages of a staged update strategy with a single (canary) stage, which has +// a before-stage approval task and includes all the member clusters. +func beforeStageApprovalStages() []rolloutv1alpha1.Stage { + return []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + } +} + +// wantWaitingForBeforeStageApprovalStatus builds the status of a staged update run (of the stages that +// beforeStageApprovalStages returns, with member cluster 0 only) that is waiting for the before-stage approval. +func wantWaitingForBeforeStageApprovalStatus( + snapshotName string, + memberClusterNames, bindingNames []string, + approvalRequestName string, +) *rolloutv1alpha1.StagedUpdateRunStatus { + return &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } +} + +var _ = Describe("managing the lifecycle of staged update runs", func() { + Context("staged update runs waiting for approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + // The (only) stage has a before-stage approval task, which is never approved; the staged update run stays + // in progress throughout the test case. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the staged update run is waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("should add the cleanup finalizer to the staged update run", func() { + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(appNamespaceName, runName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cleanup finalizer") + }) + + It("should claim the binding manager role of the placement policy", func() { + roleHeldActual := bindingManagerRoleHeldActual(appNamespaceName, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(appNamespaceName, runName), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to claim the binding manager role") + }) + + It("can trigger a reconciliation of the staged update run", func() { + // The run is not requeued while it waits for approval; trigger a reconciliation, so that the check below + // covers what the controller does when it processes the run again. + triggerStagedUpdateRunReconciliation(appNamespaceName, runName, canaryStageName, triggerApprovalRequestName) + }) + + It("should keep the cleanup finalizer and the binding manager role while waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(appNamespaceName, runName) + roleHeldActual := bindingManagerRoleHeldActual(appNamespaceName, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(appNamespaceName, runName), + }) + Consistently(func() error { + if err := rolloutStatusActual(); err != nil { + return err + } + if err := finalizerAddedActual(); err != nil { + return err + } + return roleHeldActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Failed to keep the staged update run waiting for approval, with the cleanup finalizer and the binding manager role") + }) + + AfterAll(func() { + // The staged update run is still in progress; its deletion is processed (the binding manager role is + // released and the cleanup finalizer is removed) by the controller. + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newApprovalRequestObj(appNamespaceName, triggerApprovalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("completed staged update runs", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the staged update run once completed. + wantCompletedStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantCompletedStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the staged update run", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantCompletedStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should remove the cleanup finalizer from the staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(appNamespaceName, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("should release the binding manager role of the placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can point the placement binding to another resource snapshot and mark it as unavailable", func() { + updatePlacementBindingSpec(appNamespaceName, bindingNames[0], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + setPlacementBindingConditions(appNamespaceName, bindingNames[0], conflictingResourceSnapshotName, metav1.ConditionFalse, metav1.ConditionFalse, 0) + }) + + It("can trigger a reconciliation of the completed staged update run", func() { + // The controller does not watch placement bindings. + triggerStagedUpdateRunReconciliation(appNamespaceName, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not process the completed staged update run again", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantCompletedStatus) + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], conflictingResourceSnapshotName) + Consistently(func() error { + if err := rolloutStatusActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Processed the completed staged update run again") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, triggerApprovalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("missing placement policy at creation", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, strategy)...) + + // The placement policy has not been created yet. + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, noLinkedPlacementPolicyMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should remove the cleanup finalizer from the staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(appNamespaceName, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("can create the placement policy", func() { + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + waitForObjectsToBeCached(policy) + }) + + It("can trigger a new reconciliation of the staged update run", func() { + // The controller does not watch placement policies. The trigger leaves the run as is (in particular, the + // run is not suspended): had the controller initialized the run now that the policy exists, it would go on + // to claim the binding manager role and roll out, which the check below would catch. + triggerStagedUpdateRunReconciliation(appNamespaceName, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not initialize the staged update run again", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, noLinkedPlacementPolicyMsgSubstr) + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + Consistently(func() error { + if err := failedToInitializeActual(); err != nil { + return err + } + if err := roleReleasedActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Initialized the staged update run, claimed the binding manager role, or rolled out unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, triggerApprovalRequestName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement policy deleted while waiting for approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the staged update run is waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can delete the placement policy", func() { + removeObjects(newPlacementPolicyObj(appNamespaceName, policyName)) + }) + + It("can approve the approval request", func() { + // The controller does not watch placement policies; it is the approval that triggers the processing + // of the staged update run, which then finds the placement policy gone. + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should fail the staged update run", func() { + // The staged update run fails before it processes the stage (i.e., the approval is not processed). + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the staged update run as failed") + + completedMsgActual := stagedUpdateRunConditionMessageActual(appNamespaceName, runName, + rolloutv1alpha1.StagedUpdateRunCondTypeCompleted, noLinkedPlacementPolicyMsgSubstr) + Expect(completedMsgActual()).To(Succeed(), "Failed to report why the staged update run has failed") + }) + + It("should remove the cleanup finalizer from the staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(appNamespaceName, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("should not roll out to the member cluster", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + Expect(bindingUntouchedActual()).To(Succeed(), "Rolled out to the member cluster unexpectedly") + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; the resource snapshot that the deleted + // placement policy owns must be removed explicitly. + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("binding manager role held by another controller", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The binding manager claim of another controller. + otherBindingManager := &placementv1alpha1.BindingManager{ + ControllerName: otherBindingManagerControllerName, + ObjectRefs: []placementv1alpha1.ObjectReference{ + { + APIGroup: otherBindingManagerAPIGroup, + APIVersion: otherBindingManagerAPIVersion, + Kind: otherBindingManagerKind, + Name: fmt.Sprintf(otherBindingManagerNameTemplate, randStr), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + updatePlacementPolicyBindingManager(appNamespaceName, policyName, otherBindingManager) + // Wait until the cache (which the controller shares) observes the claim of the other controller, so that + // the staged update run never sees the role as free. + otherRoleHeldActual := bindingManagerActual(appNamespaceName, policyName, otherBindingManager) + Eventually(otherRoleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the claim of the other controller") + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should add the cleanup finalizer to the staged update run", func() { + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(appNamespaceName, runName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cleanup finalizer") + }) + + It("should not initialize the staged update run while another controller holds the binding manager role", func() { + // The controller adds the cleanup finalizer right before it tries to claim the role, and keeps retrying + // (with backoff) afterwards; in the meantime, the status of the staged update run stays empty (binding + // manager conflicts are not reported in the status), the role stays with the other controller, and the + // placement binding stays untouched. + statusUnchangedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, &rolloutv1alpha1.StagedUpdateRunStatus{}) + otherRoleHeldActual := bindingManagerActual(appNamespaceName, policyName, otherBindingManager) + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + Consistently(func() error { + if err := statusUnchangedActual(); err != nil { + return err + } + if err := otherRoleHeldActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Proceeded with the staged update run unexpectedly") + }) + + It("can release the binding manager role held by the other controller", func() { + updatePlacementPolicyBindingManager(appNamespaceName, policyName, nil) + }) + + It("should initialize the staged update run", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + It("should claim the binding manager role of the placement policy", func() { + roleHeldActual := bindingManagerRoleHeldActual(appNamespaceName, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(appNamespaceName, runName), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to claim the binding manager role") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("multiple staged update runs of the same placement policy", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + var snapshotName string + + // Both staged update runs roll out to the same placement policy, and wait for approval. + runNames := []string{ + fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-0"), + fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-1"), + } + approvalRequestNames := []string{ + fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runNames[0], canaryStageName), + fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runNames[1], canaryStageName), + } + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + for _, runName := range runNames { + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + } + }) + + It("should report that both staged update runs are waiting for approval", func() { + for idx, runName := range runNames { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestNames[idx]) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of staged update run %s", runName) + } + }) + + It("should hold the binding manager role of the placement policy on behalf of both staged update runs", func() { + roleHeldActual := bindingManagerRoleHeldActual(appNamespaceName, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(appNamespaceName, runNames[0]), + wantStagedUpdateRunObjectRef(appNamespaceName, runNames[1]), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to claim the binding manager role for both staged update runs") + }) + + It("can delete the first staged update run", func() { + removeObjects(newStagedUpdateRunObj(appNamespaceName, runNames[0])) + }) + + It("should hold the binding manager role on behalf of the second staged update run only", func() { + roleHeldActual := bindingManagerRoleHeldActual(appNamespaceName, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(appNamespaceName, runNames[1]), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role for the first staged update run only") + }) + + It("can approve the approval request of the second staged update run", func() { + approveApprovalRequest(appNamespaceName, approvalRequestNames[1]) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the second staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestNames[1], perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runNames[1], wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the second staged update run") + }) + + It("should release the binding manager role of the placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runNames[0]), + newStagedUpdateRunObj(appNamespaceName, runNames[1]), + newApprovalRequestObj(appNamespaceName, approvalRequestNames[0]), + newApprovalRequestObj(appNamespaceName, approvalRequestNames[1]), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("staged update runs deleted while a member cluster is being updated", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + // The canary stage (with a before-stage approval task) updates member cluster 0 and then member cluster 1; + // the staged update run is deleted while member cluster 0 is being updated. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + var runUID types.UID + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the approval request, with the staged update run as its controller", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + + run, err := getStagedUpdateRun(appNamespaceName, runName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the staged update run") + runUID = run.GetUID() + }) + + It("can approve the approval request", func() { + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("can delete the staged update run", func() { + Expect(hubClient.Delete(ctx, newStagedUpdateRunObj(appNamespaceName, runName))).To(Succeed(), "Failed to delete the staged update run") + }) + + It("should remove the staged update run", func() { + // The controller processes the deletion once the in-flight reconciliation (which waits for member + // cluster 0 to become available) runs out of its processing time budget. + runRemovedActual := objectRemovedActual(newStagedUpdateRunObj(appNamespaceName, runName)) + Eventually(runRemovedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to remove the staged update run") + }) + + It("should release the binding manager role of the placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("should leave the placement bindings as they are", func() { + // The placement binding of member cluster 0 is not rolled back, and the one of member cluster 1 is not + // rolled out to. + Consistently(func() error { + if err := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName)(); err != nil { + return err + } + return placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], previousResourceSnapshotName)() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Changed the placement bindings unexpectedly") + }) + + It("should leave the approval request in place, still owned by the deleted staged update run", func() { + // The envtest environment does not run the garbage collector, which would otherwise remove the approval + // request as its owner is gone. + ownerRefsActual := approvalRequestOwnerRefsActual(appNamespaceName, approvalRequestName, + wantStagedUpdateRunOwnerRefs(appNamespaceName, runName, runUID)) + Expect(ownerRefsActual()).To(Succeed(), "Failed to keep the approval request as it is") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("staged update runs deleted after their placement policy", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the staged update run is waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(appNamespaceName, runName) + Expect(finalizerAddedActual()).To(Succeed(), "The staged update run does not have the cleanup finalizer") + }) + + It("can delete the placement policy", func() { + removeObjects(newPlacementPolicyObj(appNamespaceName, policyName)) + }) + + It("can delete the staged update run", func() { + Expect(hubClient.Delete(ctx, newStagedUpdateRunObj(appNamespaceName, runName))).To(Succeed(), "Failed to delete the staged update run") + }) + + It("should remove the cleanup finalizer and the staged update run, even though the placement policy is gone", func() { + runRemovedActual := objectRemovedActual(newStagedUpdateRunObj(appNamespaceName, runName)) + Eventually(runRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("staged update runs that failed to initialize", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + waitForObjectsToBeCached(append(fixtures, policy)...) + + // The staged update strategy does not exist. + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, runName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should remove the cleanup finalizer from the staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(appNamespaceName, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("can delete the staged update run", func() { + Expect(hubClient.Delete(ctx, newStagedUpdateRunObj(appNamespaceName, runName))).To(Succeed(), "Failed to delete the staged update run") + }) + + It("should remove the staged update run right away", func() { + runRemovedActual := objectRemovedActual(newStagedUpdateRunObj(appNamespaceName, runName)) + Eventually(runRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("staged update runs suspended while waiting for approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + // The staged update run is created at generation 1, suspended at generation 2, and resumed at generation 3. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the staged update run while it is suspended (waiting for approval). The controller refreshes + // the observed generation of the Initialized condition only. + wantSuspendedStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(2, stagedUpdateRunInitializedCond()), + observedAt(1, stagedUpdateRunStartedCond()), + observedAt(2, stagedUpdateRunSuspendedCond()), + }, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, observedAt(1, perStageTaskApprovalRequestCreatedCond())), + }, + Conditions: []metav1.Condition{observedAt(1, perStageStartedCond())}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantSuspendedStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the staged update run is waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can suspend the staged update run", func() { + updateStagedUpdateRunSuspended(appNamespaceName, runName, true) + }) + + It("should mark the staged update run as suspended", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, runName, wantSuspendedStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to suspend the staged update run") + }) + + It("should remove the cleanup finalizer and release the binding manager role", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(appNamespaceName, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can approve the approval request while the staged update run is suspended", func() { + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should not process the approval or roll out while the staged update run is suspended", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, runName, wantSuspendedStatus) + approvalNotAcceptedActual := approvalRequestConditionsActual(appNamespaceName, approvalRequestName, []metav1.Condition{ + approvalRequestApprovedCond(metav1.ConditionTrue), + }) + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + Consistently(func() error { + if err := rolloutStatusActual(); err != nil { + return err + } + if err := approvalNotAcceptedActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Proceeded with the suspended staged update run unexpectedly") + }) + + It("can resume the staged update run", func() { + updateStagedUpdateRunSuspended(appNamespaceName, runName, false) + }) + + It("should mark the staged update run as resumed", func() { + resumedActual := stagedUpdateRunConditionActual(appNamespaceName, runName, stagedUpdateRunResumedCond()) + Eventually(resumedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to resume the staged update run") + }) + + It("should accept the approval and start rolling out to the member cluster", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("should add the cleanup finalizer and claim the binding manager role again", func() { + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(appNamespaceName, runName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cleanup finalizer") + + roleHeldActual := bindingManagerRoleHeldActual(appNamespaceName, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(appNamespaceName, runName), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to claim the binding manager role") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the staged update run", func() { + // The controller does not refresh the observed generations of the conditions of stage tasks. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(3, stagedUpdateRunInitializedCond()), + observedAt(3, stagedUpdateRunResumedCond()), + observedAt(3, stagedUpdateRunStartedCond()), + observedAt(3, stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded)), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + observedAt(3, perClusterStartedCond()), + observedAt(3, perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded))), + }, + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, + observedAt(1, perStageTaskApprovalRequestCreatedCond()), + observedAt(3, perStageTaskApprovalRequestApprovedCond())), + }, + Conditions: []metav1.Condition{ + observedAt(3, perStageStartedCond()), + observedAt(3, perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded)), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("staged update runs suspended while a member cluster is being updated", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + // The (only) stage updates member cluster 0 and then member cluster 1. The staged update run is created at + // generation 1, suspended at generation 2 (while member cluster 0 is being updated), and resumed at + // generation 3. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + // The status of the staged update run while it is suspended. The controller refreshes the observed generation + // of the Initialized condition only. + wantSuspendedStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(2, stagedUpdateRunInitializedCond()), + observedAt(1, stagedUpdateRunStartedCond()), + observedAt(2, stagedUpdateRunSuspendedCond()), + }, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, observedAt(1, perClusterStartedCond())), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{observedAt(1, perStageStartedCond())}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantSuspendedStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("should report that member cluster 0 is being updated", func() { + // The staged update run is suspended only after the persisted status shows that the update on member + // cluster 0 has started; the reconciliation in flight at the time of the suspension fails to persist its + // status (as the run has changed in the meantime). + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can suspend the staged update run", func() { + updateStagedUpdateRunSuspended(appNamespaceName, runName, true) + }) + + It("should mark the staged update run as suspended", func() { + // The suspension takes effect once the reconciliation in flight (which waits for member cluster 0, kept + // unavailable until now) runs out of its processing time budget. + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, runName, wantSuspendedStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to suspend the staged update run") + }) + + It("should remove the cleanup finalizer and release the binding manager role", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(appNamespaceName, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can mark member cluster 0 as available while the staged update run is suspended", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("can trigger a reconciliation of the suspended staged update run", func() { + // The controller does not watch placement bindings; without a reconciliation, the check below would pass + // even if the controller ignored the suspension. + triggerStagedUpdateRunReconciliation(appNamespaceName, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not process the staged update run while it is suspended", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, runName, wantSuspendedStatus) + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], previousResourceSnapshotName) + Consistently(func() error { + if err := rolloutStatusActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Proceeded with the suspended staged update run unexpectedly") + }) + + It("can resume the staged update run", func() { + updateStagedUpdateRunSuspended(appNamespaceName, runName, false) + }) + + It("should start rolling out to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(3, stagedUpdateRunInitializedCond()), + observedAt(3, stagedUpdateRunResumedCond()), + observedAt(3, stagedUpdateRunStartedCond()), + observedAt(3, stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded)), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + observedAt(3, perClusterStartedCond()), + observedAt(3, perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded))), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, + observedAt(3, perClusterStartedCond()), + observedAt(3, perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded))), + }, + Conditions: []metav1.Condition{ + observedAt(3, perStageStartedCond()), + observedAt(3, perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded)), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should restart the update on member cluster 0 (and its wait time limit) upon resumption", func() { + run, err := getStagedUpdateRun(appNamespaceName, runName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the staged update run") + resumedCond := meta.FindStatusCondition(run.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeSuspended) + Expect(resumedCond).ToNot(BeNil(), "The staged update run has no Suspended condition") + + startedCond, err := getPerClusterCondition(appNamespaceName, runName, memberClusterNames[0], rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the Started condition of member cluster 0") + Expect(startedCond).ToNot(BeNil(), "Member cluster 0 has no Started condition") + Expect(startedCond.LastTransitionTime.Before(&resumedCond.LastTransitionTime)).To(BeFalse(), + "The update on member cluster 0 started (%v) before the staged update run was resumed (%v)", + startedCond.LastTransitionTime, resumedCond.LastTransitionTime) + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, triggerApprovalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("suspending and resuming finished staged update runs", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + missingStrategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, utils.RandStr()) + var snapshotName string + + // The first staged update run completes successfully; the second one fails to initialize (its staged update + // strategy does not exist). Both are created at generation 1, suspended at generation 2, and resumed at + // generation 3. + completedRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-0") + failedRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-1") + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // wantCompletedStatus returns the status of the completed staged update run, with its Completed condition + // observed at the given generation. The controller refreshes the observed generation of the Completed + // condition only. + wantCompletedStatus := func(completedCondObsGen int64) *rolloutv1alpha1.StagedUpdateRunStatus { + return &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(1, stagedUpdateRunInitializedCond()), + observedAt(1, stagedUpdateRunStartedCond()), + observedAt(completedCondObsGen, stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded)), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + observedAt(1, perClusterStartedCond()), + observedAt(1, perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded))), + }, + Conditions: []metav1.Condition{ + observedAt(1, perStageStartedCond()), + observedAt(1, perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded)), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, completedRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the first staged update run", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, completedRunName, wantCompletedStatus(1)) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the first staged update run") + }) + + It("can create the second staged update run", func() { + createStagedUpdateRun(appNamespaceName, failedRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: missingStrategyName, + }) + }) + + It("should fail to initialize the second staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, failedRunName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("can suspend both staged update runs", func() { + updateStagedUpdateRunSuspended(appNamespaceName, completedRunName, true) + updateStagedUpdateRunSuspended(appNamespaceName, failedRunName, true) + }) + + It("should refresh the observed generations of the terminal conditions, without marking the staged update runs as suspended", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, completedRunName, wantCompletedStatus(2)) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the Completed condition of the first staged update run") + + // The check expects the Initialized condition (the only condition) to be observed at the current + // generation of the run. + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, failedRunName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the Initialized condition of the second staged update run") + }) + + It("can resume both staged update runs", func() { + updateStagedUpdateRunSuspended(appNamespaceName, completedRunName, false) + updateStagedUpdateRunSuspended(appNamespaceName, failedRunName, false) + }) + + It("should refresh the observed generations of the terminal conditions again, without marking the staged update runs as resumed", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(appNamespaceName, completedRunName, wantCompletedStatus(3)) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the Completed condition of the first staged update run") + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(appNamespaceName, failedRunName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the Initialized condition of the second staged update run") + + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Expect(bindingUntouchedActual()).To(Succeed(), "The placement binding has been changed unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, completedRunName), + newStagedUpdateRunObj(appNamespaceName, failedRunName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("managing the lifecycle of cluster staged update runs", func() { + Context("cluster staged update runs waiting for approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + // The (only) stage has a before-stage approval task, which is never approved; the cluster staged update run stays + // in progress throughout the test case. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the cluster staged update run is waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("should add the cleanup finalizer to the cluster staged update run", func() { + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(clusterScopedNS, runName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cleanup finalizer") + }) + + It("should claim the binding manager role of the cluster placement policy", func() { + roleHeldActual := bindingManagerRoleHeldActual(clusterScopedNS, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(clusterScopedNS, runName), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to claim the binding manager role") + }) + + It("can trigger a reconciliation of the cluster staged update run", func() { + // The run is not requeued while it waits for approval; trigger a reconciliation, so that the check below + // covers what the controller does when it processes the run again. + triggerStagedUpdateRunReconciliation(clusterScopedNS, runName, canaryStageName, triggerApprovalRequestName) + }) + + It("should keep the cleanup finalizer and the binding manager role while waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(clusterScopedNS, runName) + roleHeldActual := bindingManagerRoleHeldActual(clusterScopedNS, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(clusterScopedNS, runName), + }) + Consistently(func() error { + if err := rolloutStatusActual(); err != nil { + return err + } + if err := finalizerAddedActual(); err != nil { + return err + } + return roleHeldActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Failed to keep the cluster staged update run waiting for approval, with the cleanup finalizer and the binding manager role") + }) + + AfterAll(func() { + // The cluster staged update run is still in progress; its deletion is processed (the binding manager role is + // released and the cleanup finalizer is removed) by the controller. + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newApprovalRequestObj(clusterScopedNS, triggerApprovalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("completed cluster staged update runs", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the cluster staged update run once completed. + wantCompletedStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantCompletedStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantCompletedStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + It("should remove the cleanup finalizer from the cluster staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(clusterScopedNS, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("should release the binding manager role of the cluster placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can point the placement binding to another resource snapshot and mark it as unavailable", func() { + updatePlacementBindingSpec(clusterScopedNS, bindingNames[0], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + setPlacementBindingConditions(clusterScopedNS, bindingNames[0], conflictingResourceSnapshotName, metav1.ConditionFalse, metav1.ConditionFalse, 0) + }) + + It("can trigger a reconciliation of the completed cluster staged update run", func() { + // The controller does not watch placement bindings. + triggerStagedUpdateRunReconciliation(clusterScopedNS, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not process the completed cluster staged update run again", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantCompletedStatus) + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], conflictingResourceSnapshotName) + Consistently(func() error { + if err := rolloutStatusActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Processed the completed cluster staged update run again") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, triggerApprovalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("missing cluster placement policy at creation", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, strategy)...) + + // The cluster placement policy has not been created yet. + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the cluster staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, noLinkedPlacementPolicyMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should remove the cleanup finalizer from the cluster staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(clusterScopedNS, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("can create the cluster placement policy", func() { + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + waitForObjectsToBeCached(policy) + }) + + It("can trigger a new reconciliation of the cluster staged update run", func() { + // The controller does not watch placement policies. The trigger leaves the run as is (in particular, the + // run is not suspended): had the controller initialized the run now that the policy exists, it would go on + // to claim the binding manager role and roll out, which the check below would catch. + triggerStagedUpdateRunReconciliation(clusterScopedNS, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not initialize the cluster staged update run again", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, noLinkedPlacementPolicyMsgSubstr) + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + Consistently(func() error { + if err := failedToInitializeActual(); err != nil { + return err + } + if err := roleReleasedActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Initialized the cluster staged update run, claimed the binding manager role, or rolled out unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, triggerApprovalRequestName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster placement policy deleted while waiting for approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the cluster staged update run is waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can delete the cluster placement policy", func() { + removeObjects(newPlacementPolicyObj(clusterScopedNS, policyName)) + }) + + It("can approve the cluster approval request", func() { + // The controller does not watch placement policies; it is the approval that triggers the processing + // of the cluster staged update run, which then finds the cluster placement policy gone. + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should fail the cluster staged update run", func() { + // The cluster staged update run fails before it processes the stage (i.e., the approval is not processed). + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the cluster staged update run as failed") + + completedMsgActual := stagedUpdateRunConditionMessageActual(clusterScopedNS, runName, + rolloutv1alpha1.StagedUpdateRunCondTypeCompleted, noLinkedPlacementPolicyMsgSubstr) + Expect(completedMsgActual()).To(Succeed(), "Failed to report why the cluster staged update run has failed") + }) + + It("should remove the cleanup finalizer from the cluster staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(clusterScopedNS, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("should not roll out to the member cluster", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + Expect(bindingUntouchedActual()).To(Succeed(), "Rolled out to the member cluster unexpectedly") + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; the resource snapshot that the deleted + // cluster placement policy owns must be removed explicitly. + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("binding manager role held by another controller", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The binding manager claim of another controller. + otherBindingManager := &placementv1alpha1.BindingManager{ + ControllerName: otherBindingManagerControllerName, + ObjectRefs: []placementv1alpha1.ObjectReference{ + { + APIGroup: otherBindingManagerAPIGroup, + APIVersion: otherBindingManagerAPIVersion, + Kind: otherBindingManagerKind, + Name: fmt.Sprintf(otherBindingManagerNameTemplate, randStr), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + updatePlacementPolicyBindingManager(clusterScopedNS, policyName, otherBindingManager) + // Wait until the cache (which the controller shares) observes the claim of the other controller, so that + // the cluster staged update run never sees the role as free. + otherRoleHeldActual := bindingManagerActual(clusterScopedNS, policyName, otherBindingManager) + Eventually(otherRoleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the claim of the other controller") + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should add the cleanup finalizer to the cluster staged update run", func() { + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(clusterScopedNS, runName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cleanup finalizer") + }) + + It("should not initialize the cluster staged update run while another controller holds the binding manager role", func() { + // The controller adds the cleanup finalizer right before it tries to claim the role, and keeps retrying + // (with backoff) afterwards; in the meantime, the status of the cluster staged update run stays empty + // (binding manager conflicts are not reported in the status), the role stays with the other controller, + // and the placement binding stays untouched. + statusUnchangedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, &rolloutv1alpha1.StagedUpdateRunStatus{}) + otherRoleHeldActual := bindingManagerActual(clusterScopedNS, policyName, otherBindingManager) + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + Consistently(func() error { + if err := statusUnchangedActual(); err != nil { + return err + } + if err := otherRoleHeldActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Proceeded with the cluster staged update run unexpectedly") + }) + + It("can release the binding manager role held by the other controller", func() { + updatePlacementPolicyBindingManager(clusterScopedNS, policyName, nil) + }) + + It("should initialize the cluster staged update run", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + It("should claim the binding manager role of the cluster placement policy", func() { + roleHeldActual := bindingManagerRoleHeldActual(clusterScopedNS, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(clusterScopedNS, runName), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to claim the binding manager role") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("multiple cluster staged update runs of the same cluster placement policy", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + var snapshotName string + + // Both cluster staged update runs roll out to the same cluster placement policy, and wait for approval. + runNames := []string{ + fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-0"), + fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-1"), + } + approvalRequestNames := []string{ + fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runNames[0], canaryStageName), + fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runNames[1], canaryStageName), + } + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + for _, runName := range runNames { + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + } + }) + + It("should report that both cluster staged update runs are waiting for approval", func() { + for idx, runName := range runNames { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestNames[idx]) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of cluster staged update run %s", runName) + } + }) + + It("should hold the binding manager role of the cluster placement policy on behalf of both cluster staged update runs", func() { + roleHeldActual := bindingManagerRoleHeldActual(clusterScopedNS, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(clusterScopedNS, runNames[0]), + wantStagedUpdateRunObjectRef(clusterScopedNS, runNames[1]), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to claim the binding manager role for both cluster staged update runs") + }) + + It("can delete the first cluster staged update run", func() { + removeObjects(newStagedUpdateRunObj(clusterScopedNS, runNames[0])) + }) + + It("should hold the binding manager role on behalf of the second cluster staged update run only", func() { + roleHeldActual := bindingManagerRoleHeldActual(clusterScopedNS, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(clusterScopedNS, runNames[1]), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role for the first cluster staged update run only") + }) + + It("can approve the cluster approval request of the second cluster staged update run", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestNames[1]) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the second cluster staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestNames[1], perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runNames[1], wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the second cluster staged update run") + }) + + It("should release the binding manager role of the cluster placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runNames[0]), + newStagedUpdateRunObj(clusterScopedNS, runNames[1]), + newApprovalRequestObj(clusterScopedNS, approvalRequestNames[0]), + newApprovalRequestObj(clusterScopedNS, approvalRequestNames[1]), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster staged update runs deleted while a member cluster is being updated", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + // The canary stage (with a before-stage approval task) updates member cluster 0 and then member cluster 1; + // the cluster staged update run is deleted while member cluster 0 is being updated. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + var runUID types.UID + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the cluster approval request, with the cluster staged update run as its controller", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + + run, err := getStagedUpdateRun(clusterScopedNS, runName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the cluster staged update run") + runUID = run.GetUID() + }) + + It("can approve the cluster approval request", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("can delete the cluster staged update run", func() { + Expect(hubClient.Delete(ctx, newStagedUpdateRunObj(clusterScopedNS, runName))).To(Succeed(), "Failed to delete the cluster staged update run") + }) + + It("should remove the cluster staged update run", func() { + // The controller processes the deletion once the in-flight reconciliation (which waits for member + // cluster 0 to become available) runs out of its processing time budget. + runRemovedActual := objectRemovedActual(newStagedUpdateRunObj(clusterScopedNS, runName)) + Eventually(runRemovedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster staged update run") + }) + + It("should release the binding manager role of the cluster placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("should leave the placement bindings as they are", func() { + // The placement binding of member cluster 0 is not rolled back, and the one of member cluster 1 is not + // rolled out to. + Consistently(func() error { + if err := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName)(); err != nil { + return err + } + return placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], previousResourceSnapshotName)() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Changed the placement bindings unexpectedly") + }) + + It("should leave the cluster approval request in place, still owned by the deleted cluster staged update run", func() { + // The envtest environment does not run the garbage collector, which would otherwise remove the approval + // request as its owner is gone. + ownerRefsActual := approvalRequestOwnerRefsActual(clusterScopedNS, approvalRequestName, + wantStagedUpdateRunOwnerRefs(clusterScopedNS, runName, runUID)) + Expect(ownerRefsActual()).To(Succeed(), "Failed to keep the cluster approval request as it is") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster staged update runs deleted after their cluster placement policy", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the cluster staged update run is waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(clusterScopedNS, runName) + Expect(finalizerAddedActual()).To(Succeed(), "The cluster staged update run does not have the cleanup finalizer") + }) + + It("can delete the cluster placement policy", func() { + removeObjects(newPlacementPolicyObj(clusterScopedNS, policyName)) + }) + + It("can delete the cluster staged update run", func() { + Expect(hubClient.Delete(ctx, newStagedUpdateRunObj(clusterScopedNS, runName))).To(Succeed(), "Failed to delete the cluster staged update run") + }) + + It("should remove the cleanup finalizer and the cluster staged update run, even though the cluster placement policy is gone", func() { + runRemovedActual := objectRemovedActual(newStagedUpdateRunObj(clusterScopedNS, runName)) + Eventually(runRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster staged update runs that failed to initialize", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + waitForObjectsToBeCached(append(fixtures, policy)...) + + // The cluster staged update strategy does not exist. + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should fail to initialize the cluster staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, runName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("should remove the cleanup finalizer from the cluster staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(clusterScopedNS, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("can delete the cluster staged update run", func() { + Expect(hubClient.Delete(ctx, newStagedUpdateRunObj(clusterScopedNS, runName))).To(Succeed(), "Failed to delete the cluster staged update run") + }) + + It("should remove the cluster staged update run right away", func() { + runRemovedActual := objectRemovedActual(newStagedUpdateRunObj(clusterScopedNS, runName)) + Eventually(runRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster staged update runs suspended while waiting for approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + // The cluster staged update run is created at generation 1, suspended at generation 2, and resumed at generation 3. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the cluster staged update run while it is suspended (waiting for approval). The controller refreshes + // the observed generation of the Initialized condition only. + wantSuspendedStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(2, stagedUpdateRunInitializedCond()), + observedAt(1, stagedUpdateRunStartedCond()), + observedAt(2, stagedUpdateRunSuspendedCond()), + }, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, observedAt(1, perStageTaskApprovalRequestCreatedCond())), + }, + Conditions: []metav1.Condition{observedAt(1, perStageStartedCond())}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantSuspendedStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, beforeStageApprovalStages()) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the cluster staged update run is waiting for approval", func() { + wantStatus := wantWaitingForBeforeStageApprovalStatus(snapshotName, memberClusterNames, bindingNames, approvalRequestName) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can suspend the cluster staged update run", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, runName, true) + }) + + It("should mark the cluster staged update run as suspended", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, runName, wantSuspendedStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to suspend the cluster staged update run") + }) + + It("should remove the cleanup finalizer and release the binding manager role", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(clusterScopedNS, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can approve the cluster approval request while the cluster staged update run is suspended", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should not process the approval or roll out while the cluster staged update run is suspended", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, runName, wantSuspendedStatus) + approvalNotAcceptedActual := approvalRequestConditionsActual(clusterScopedNS, approvalRequestName, []metav1.Condition{ + approvalRequestApprovedCond(metav1.ConditionTrue), + }) + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + Consistently(func() error { + if err := rolloutStatusActual(); err != nil { + return err + } + if err := approvalNotAcceptedActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Proceeded with the suspended cluster staged update run unexpectedly") + }) + + It("can resume the cluster staged update run", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, runName, false) + }) + + It("should mark the cluster staged update run as resumed", func() { + resumedActual := stagedUpdateRunConditionActual(clusterScopedNS, runName, stagedUpdateRunResumedCond()) + Eventually(resumedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to resume the cluster staged update run") + }) + + It("should accept the approval and start rolling out to the member cluster", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("should add the cleanup finalizer and claim the binding manager role again", func() { + finalizerAddedActual := stagedUpdateRunCleanupFinalizerAddedActual(clusterScopedNS, runName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cleanup finalizer") + + roleHeldActual := bindingManagerRoleHeldActual(clusterScopedNS, policyName, []placementv1alpha1.ObjectReference{ + wantStagedUpdateRunObjectRef(clusterScopedNS, runName), + }) + Eventually(roleHeldActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to claim the binding manager role") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + // The controller does not refresh the observed generations of the conditions of stage tasks. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(3, stagedUpdateRunInitializedCond()), + observedAt(3, stagedUpdateRunResumedCond()), + observedAt(3, stagedUpdateRunStartedCond()), + observedAt(3, stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded)), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + observedAt(3, perClusterStartedCond()), + observedAt(3, perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded))), + }, + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, + observedAt(1, perStageTaskApprovalRequestCreatedCond()), + observedAt(3, perStageTaskApprovalRequestApprovedCond())), + }, + Conditions: []metav1.Condition{ + observedAt(3, perStageStartedCond()), + observedAt(3, perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded)), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster staged update runs suspended while a member cluster is being updated", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + triggerApprovalRequestName := fmt.Sprintf(triggerApprovalRequestNameTemplate, randStr) + + // The (only) stage updates member cluster 0 and then member cluster 1. The cluster staged update run is created at + // generation 1, suspended at generation 2 (while member cluster 0 is being updated), and resumed at + // generation 3. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + // The status of the cluster staged update run while it is suspended. The controller refreshes the observed generation + // of the Initialized condition only. + wantSuspendedStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(2, stagedUpdateRunInitializedCond()), + observedAt(1, stagedUpdateRunStartedCond()), + observedAt(2, stagedUpdateRunSuspendedCond()), + }, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, observedAt(1, perClusterStartedCond())), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{observedAt(1, perStageStartedCond())}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantSuspendedStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("should report that member cluster 0 is being updated", func() { + // The cluster staged update run is suspended only after the persisted status shows that the update on member + // cluster 0 has started; the reconciliation in flight at the time of the suspension fails to persist its + // status (as the run has changed in the meantime). + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can suspend the cluster staged update run", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, runName, true) + }) + + It("should mark the cluster staged update run as suspended", func() { + // The suspension takes effect once the reconciliation in flight (which waits for member cluster 0, kept + // unavailable until now) runs out of its processing time budget. + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, runName, wantSuspendedStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to suspend the cluster staged update run") + }) + + It("should remove the cleanup finalizer and release the binding manager role", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(clusterScopedNS, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + It("can mark member cluster 0 as available while the cluster staged update run is suspended", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("can trigger a reconciliation of the suspended cluster staged update run", func() { + // The controller does not watch placement bindings; without a reconciliation, the check below would pass + // even if the controller ignored the suspension. + triggerStagedUpdateRunReconciliation(clusterScopedNS, runName, restStageName, triggerApprovalRequestName) + }) + + It("should not process the cluster staged update run while it is suspended", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, runName, wantSuspendedStatus) + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], previousResourceSnapshotName) + Consistently(func() error { + if err := rolloutStatusActual(); err != nil { + return err + } + return bindingUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Proceeded with the suspended cluster staged update run unexpectedly") + }) + + It("can resume the cluster staged update run", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, runName, false) + }) + + It("should start rolling out to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(3, stagedUpdateRunInitializedCond()), + observedAt(3, stagedUpdateRunResumedCond()), + observedAt(3, stagedUpdateRunStartedCond()), + observedAt(3, stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded)), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + observedAt(3, perClusterStartedCond()), + observedAt(3, perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded))), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, + observedAt(3, perClusterStartedCond()), + observedAt(3, perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded))), + }, + Conditions: []metav1.Condition{ + observedAt(3, perStageStartedCond()), + observedAt(3, perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded)), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + It("should restart the update on member cluster 0 (and its wait time limit) upon resumption", func() { + run, err := getStagedUpdateRun(clusterScopedNS, runName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the cluster staged update run") + resumedCond := meta.FindStatusCondition(run.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeSuspended) + Expect(resumedCond).ToNot(BeNil(), "The cluster staged update run has no Suspended condition") + + startedCond, err := getPerClusterCondition(clusterScopedNS, runName, memberClusterNames[0], rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the Started condition of member cluster 0") + Expect(startedCond).ToNot(BeNil(), "Member cluster 0 has no Started condition") + Expect(startedCond.LastTransitionTime.Before(&resumedCond.LastTransitionTime)).To(BeFalse(), + "The update on member cluster 0 started (%v) before the cluster staged update run was resumed (%v)", + startedCond.LastTransitionTime, resumedCond.LastTransitionTime) + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, triggerApprovalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("suspending and resuming finished cluster staged update runs", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + missingStrategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, utils.RandStr()) + var snapshotName string + + // The first cluster staged update run completes successfully; the second one fails to initialize (its staged update + // strategy does not exist). Both are created at generation 1, suspended at generation 2, and resumed at + // generation 3. + completedRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-0") + failedRunName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr+"-1") + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // wantCompletedStatus returns the status of the completed cluster staged update run, with its Completed condition + // observed at the given generation. The controller refreshes the observed generation of the Completed + // condition only. + wantCompletedStatus := func(completedCondObsGen int64) *rolloutv1alpha1.StagedUpdateRunStatus { + return &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + observedAt(1, stagedUpdateRunInitializedCond()), + observedAt(1, stagedUpdateRunStartedCond()), + observedAt(completedCondObsGen, stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded)), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + observedAt(1, perClusterStartedCond()), + observedAt(1, perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded))), + }, + Conditions: []metav1.Condition{ + observedAt(1, perStageStartedCond()), + observedAt(1, perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded)), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, completedRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the first cluster staged update run", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, completedRunName, wantCompletedStatus(1)) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the first cluster staged update run") + }) + + It("can create the second cluster staged update run", func() { + createStagedUpdateRun(clusterScopedNS, failedRunName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: missingStrategyName, + }) + }) + + It("should fail to initialize the second cluster staged update run", func() { + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, failedRunName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the initialization failure") + }) + + It("can suspend both cluster staged update runs", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, completedRunName, true) + updateStagedUpdateRunSuspended(clusterScopedNS, failedRunName, true) + }) + + It("should refresh the observed generations of the terminal conditions, without marking the cluster staged update runs as suspended", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, completedRunName, wantCompletedStatus(2)) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the Completed condition of the first cluster staged update run") + + // The check expects the Initialized condition (the only condition) to be observed at the current + // generation of the run. + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, failedRunName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the Initialized condition of the second cluster staged update run") + }) + + It("can resume both cluster staged update runs", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, completedRunName, false) + updateStagedUpdateRunSuspended(clusterScopedNS, failedRunName, false) + }) + + It("should refresh the observed generations of the terminal conditions again, without marking the cluster staged update runs as resumed", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusAsIsActual(clusterScopedNS, completedRunName, wantCompletedStatus(3)) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the Completed condition of the first cluster staged update run") + + failedToInitializeActual := stagedUpdateRunFailedToInitializeActual(clusterScopedNS, failedRunName, strategyNotFoundMsgSubstr) + Eventually(failedToInitializeActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the Initialized condition of the second cluster staged update run") + + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Expect(bindingUntouchedActual()).To(Succeed(), "The placement binding has been changed unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, completedRunName), + newStagedUpdateRunObj(clusterScopedNS, failedRunName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) +}) diff --git a/pkg/v1/controllers/stagedupdaterun/process.go b/pkg/v1/controllers/stagedupdaterun/process.go new file mode 100644 index 000000000..370a6fb69 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/process.go @@ -0,0 +1,821 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + "errors" + "fmt" + "sync" + "sync/atomic" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + utilerrors "k8s.io/apimachinery/pkg/util/errors" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/condition" + kferrors "go.goms.io/fleet/pkg/utils/errors" +) + +const ( + maxAllowedClusterProcessingConcurrency = 10 +) + +type allStagesProcessingResult string + +const ( + allStagesProcessingResultSucceeded allStagesProcessingResult = "Succeeded" + allStagesProcessingResultFailed allStagesProcessingResult = "Failed" + allStagesProcessingResultInProgress allStagesProcessingResult = "InProgress" + allStagesProcessingResultErred allStagesProcessingResult = "Erred" +) + +type stageProcessingResult string + +const ( + stageProcessingResultSucceeded stageProcessingResult = "Succeeded" + stageProcessingResultFailed stageProcessingResult = "Failed" + stageProcessingResultInProgress stageProcessingResult = "InProgress" + stageProcessingResultErred stageProcessingResult = "Erred" +) + +type clusterProcessingResult string + +const ( + clusterProcessingResultSucceeded clusterProcessingResult = "Succeeded" + clusterProcessingResultSucceededYetSuspended clusterProcessingResult = "SucceededYetSuspended" + clusterProcessingResultFailed clusterProcessingResult = "Failed" + clusterProcessingResultErred clusterProcessingResult = "Erred" +) + +func (r *Reconciler) shouldSkipProcessing(ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor) (bool, error) { + // Skip processing if the staged update run is in a terminal state or suspended after being initialized. + // + // To retry the staged update, one needs to create a new staged update run object. + + // Skip processing if the initialization has failed. + initCond := meta.FindStatusCondition(stagedUpdateRun.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeInitialized) + if initCond != nil && initCond.Status == metav1.ConditionFalse { + // Note that for this branch no observed generation check is needed. Staged update run objects have immutable spec (except + // for the suspended switch). + klog.V(2).InfoS("The staged update run has failed to initialize; skip further processing", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + return true, r.resyncStagedUpdateRunCondIfNeeded(ctx, stagedUpdateRun, rolloutv1alpha1.StagedUpdateRunCondTypeInitialized) + } + + // Skip processing if the staged update run has been completed (successfully or not). + completedCond := meta.FindStatusCondition(stagedUpdateRun.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeCompleted) + if completedCond != nil { + // Note that for this branch no observed generation check is needed. Staged update run objects have immutable spec (except + // for the suspended switch). + klog.V(2).InfoS("The staged update run has been completed; skip further processing", + "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + // For simplicity reasons, the controller will not resync the initialized condition in this case. + return true, r.resyncStagedUpdateRunCondIfNeeded(ctx, stagedUpdateRun, rolloutv1alpha1.StagedUpdateRunCondTypeCompleted) + } + + // Skip processing if the staged update run has been initialized but is suspended. + if initCond != nil && initCond.Status == metav1.ConditionTrue && stagedUpdateRun.GetSpec().Suspended { + klog.V(2).InfoS("The staged update run is suspended; skip further processing", "stagedUpdateRun", klog.KObj(stagedUpdateRun), "controller", controllerName) + if err := r.markStagedUpdateRunAsSuspended(ctx, stagedUpdateRun); err != nil { + return false, err + } + return true, r.resyncStagedUpdateRunCondIfNeeded(ctx, stagedUpdateRun, rolloutv1alpha1.StagedUpdateRunCondTypeInitialized) + } + + // The processing should continue; if the staged update run was previously suspended, mark it as resumed. + if err := r.markStagedUpdateRunAsResumedIfApplicable(ctx, stagedUpdateRun); err != nil { + return false, err + } + return false, nil +} + +func (r *Reconciler) process(ctx context.Context, stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor) ( + allStagesProcessingRes allStagesProcessingResult, requeueAfter *time.Duration, err error) { + // Verify if the staged update run has already been completed. + completedCond := meta.FindStatusCondition(stagedUpdateRun.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeCompleted) + switch { + case completedCond == nil || completedCond.Status != metav1.ConditionTrue: + // The staged update run has not yet completed; continue processing. + case completedCond.Reason == rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded: + // The staged update run has completed successfully; no further processing is needed. + return allStagesProcessingResultSucceeded, nil, nil + case completedCond.Reason == rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded: + // The staged update run has completed partially successfully; no further processing is needed. + return allStagesProcessingResultSucceeded, nil, nil + case completedCond.Reason == rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed: + // The staged update run has completed with failure; no further processing is needed. + return allStagesProcessingResultFailed, nil, nil + default: + // The staged update run has completed with an unknown reason; report an unexpected error. Normally this + // should never occur. + return allStagesProcessingResultErred, nil, kferrors.NewUnexpectedError(nil, "staged update run completed with unknown reason", "observedCompletionReason", completedCond.Reason) + } + + // Check if the staged update run has been marked as started; if not, add the Started condition. + stagedUpdateRunStatus := stagedUpdateRun.GetStatus() + startedCond := meta.FindStatusCondition(stagedUpdateRunStatus.Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeStarted) + if startedCond == nil || + startedCond.Status != metav1.ConditionTrue || + startedCond.Reason != rolloutv1alpha1.StagedUpdateRunStartedCondReasonUpdateStarted || + startedCond.ObservedGeneration != stagedUpdateRun.GetGeneration() { + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeStarted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunStartedCondReasonUpdateStarted, + Message: "The staged update run has started", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } + + // Start a child context. + // + // This context is set to time out after a specific period of time, and terminates the processing of the staged update + // run, in order to: + // + // a) avoid one staged update run monopolizing the controller's processing capacity for too long; and + // b) allow users to see timely updates on the staged update run's progress. + childCtx, childCancel := context.WithTimeout(ctx, r.refreshPeriod) + defer childCancel() + + // Prepare the failure policy. + maxAllowedClusterFailures := int32(1) + maxWaitTimePerCluster := 5 * time.Minute + + stagedUpdateRunSpec := stagedUpdateRun.GetSpec() + if stagedUpdateRunSpec.FailurePolicy != nil { + if stagedUpdateRunSpec.FailurePolicy.MaxAllowedClusterFailures != nil { + maxAllowedClusterFailures = *stagedUpdateRunSpec.FailurePolicy.MaxAllowedClusterFailures + } + if stagedUpdateRunSpec.FailurePolicy.MaxWaitTimePerClusterMinutes > 0 { + maxWaitTimePerCluster = time.Duration(stagedUpdateRunSpec.FailurePolicy.MaxWaitTimePerClusterMinutes) * time.Minute + } + } + + // Prepare the cluster update failure counter (shared across all stages). + clusterUpdateTotalFailures := atomic.Int32{} + clusterUpdateTotalFailures.Store(0) + + // Scan the stages and process them one by one. + stages := stagedUpdateRun.GetStatus().Stages + for idx := range stages { + stage := &stages[idx] + + stageProcessingRes, requeueAfter, err := r.processOneStage(childCtx, + stagedUpdateRun, stage, &clusterUpdateTotalFailures, maxAllowedClusterFailures, maxWaitTimePerCluster) + if err != nil { + if errors.Is(childCtx.Err(), context.DeadlineExceeded) { + // The per-run processing time budget has been exhausted; this is expected (not a real + // failure), so retry in the next reconciliation loop instead of reporting an error. + + // There might be another error that occurred independently; log the errors for reference. + klog.V(2).InfoS("Per-run processing time budget exhausted", + "stagedUpdateRun", klog.KObj(stagedUpdateRun), "stage", stage.StageName, + "error", err) + return allStagesProcessingResultInProgress, requeueAfter, nil + } + return allStagesProcessingResultErred, requeueAfter, err + } + + switch stageProcessingRes { + case stageProcessingResultSucceeded: + // The stage has been successfully processed; move on to the next stage. + case stageProcessingResultFailed: + // The stage has failed; stop processing further stages. + stagedUpdateRunStatus := stagedUpdateRun.GetStatus() + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed, + Message: fmt.Sprintf("The staged update run has failed (stage %q has failed)", stage.StageName), + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + return allStagesProcessingResultFailed, requeueAfter, nil + case stageProcessingResultInProgress: + // The stage is still being processed in progress; retry in the next reconciliation loop. + return allStagesProcessingResultInProgress, requeueAfter, nil + default: + // The stage failed processing for an unknown reason; report an unexpected error. Normally this + // should never occur. + return allStagesProcessingResultErred, nil, kferrors.NewUnexpectedError(nil, "stage processing yields an unexpected result", + "observedProcessingResult", stageProcessingRes) + } + } + + // All stages have completed successfully; mark the staged update run as completed (fully or partially). + if clusterUpdateTotalFailures.Load() > 0 { + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded, + Message: fmt.Sprintf("The staged update run has completed with %d failed clusters", clusterUpdateTotalFailures.Load()), + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } else { + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded, + Message: "The staged update run has completed successfully", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } + return allStagesProcessingResultSucceeded, nil, nil +} + +func (r *Reconciler) processOneStage( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, + clusterUpdateTotalFailures *atomic.Int32, + maxAllowedClusterFailures int32, + maxWaitTimePerCluster time.Duration, +) (result stageProcessingResult, requeueAfter *time.Duration, err error) { + // Skip the stage if it has already been completed. + stageCompletedCond := meta.FindStatusCondition(stage.Conditions, rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted) + switch { + case stageCompletedCond == nil || stageCompletedCond.Status != metav1.ConditionTrue: + // Move on to process this stage. + case stageCompletedCond.Reason == rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded: + // The stage has already been completed successfully, so skip further processing. + resyncPerStageCondWithNoCommit(stagedUpdateRun, stage, rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted) + return stageProcessingResultSucceeded, nil, nil + case stageCompletedCond.Reason == rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonFailed: + // The stage has already been completed and failed, so skip further processing. + resyncPerStageCondWithNoCommit(stagedUpdateRun, stage, rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted) + return stageProcessingResultFailed, nil, nil + case stageCompletedCond.Reason == rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded: + // The stage has already been completed with a partial success. Count the cluster update failures. + resyncPerStageCondWithNoCommit(stagedUpdateRun, stage, rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted) + if err := countClusterUpdateFailures(stage, clusterUpdateTotalFailures); err != nil { + return stageProcessingResultErred, nil, kferrors.Wraps(err, "failed to count cluster update failures for partially succeeded stage") + } + return stageProcessingResultSucceeded, nil, nil + } + + // Set the stage started timestamp. + if stage.StartedTimestamp == nil { + now := metav1.Now() + stage.StartedTimestamp = &now + } + + // Mark the stage as started. + startedCond := meta.FindStatusCondition(stage.Conditions, rolloutv1alpha1.StagedUpdateRunPerStageCondTypeStarted) + if startedCond == nil || startedCond.ObservedGeneration != stagedUpdateRun.GetGeneration() { + meta.SetStatusCondition(&stage.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerStageCondTypeStarted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerStageStartedCondReasonUpdateStarted, + Message: "Updating has started in the stage", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } + + // Execute the before stage tasks. + beforeStageTasksAllCleared, beforeStageTasksRequeueAfter, err := r.executeBeforeStageTasks(ctx, stagedUpdateRun, stage) + if err != nil { + return stageProcessingResultErred, beforeStageTasksRequeueAfter, err + } + if !beforeStageTasksAllCleared { + return stageProcessingResultInProgress, beforeStageTasksRequeueAfter, nil + } + + // Process updates to all clusters in the stage per given concurrency setting. + allowedClusterProcessingConcurrency := 1 + if stage.MaxConcurrency != nil && *stage.MaxConcurrency > 0 { + allowedClusterProcessingConcurrency = int(*stage.MaxConcurrency) + } + if allowedClusterProcessingConcurrency > maxAllowedClusterProcessingConcurrency { + allowedClusterProcessingConcurrency = maxAllowedClusterProcessingConcurrency + } + + // Spin up a child context. + childCtx, childCancel := context.WithCancel(ctx) + defer childCancel() + + tracker := &perStageClusterProcessingTracker{ + errs: make([]error, len(stage.Clusters)), + } + wg := &sync.WaitGroup{} + for workerIdx := 0; workerIdx < allowedClusterProcessingConcurrency; workerIdx++ { + wg.Add(1) + + go func() { + defer wg.Done() + r.doClusterProcessingWork(childCtx, childCancel, workerIdx, stagedUpdateRun, stage, tracker, + clusterUpdateTotalFailures, maxAllowedClusterFailures, maxWaitTimePerCluster) + }() + } + wg.Wait() + + // Tally the results. + klog.InfoS("Processed all clusters in the stage", + "stagedUpdateRun", klog.KObj(stagedUpdateRun), + "stage", stage.StageName, + "succeededClusterCnt", tracker.succeededClusterCnt.Load(), + "failedClusterCnt", tracker.failedClusterCnt.Load(), + "erredClusterCnt", tracker.erredClusterCnt.Load(), + ) + switch { + case clusterUpdateTotalFailures.Load() > maxAllowedClusterFailures: + // The failure threshold has been exceeded. The stage has failed. + meta.SetStatusCondition(&stage.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonFailed, + Message: fmt.Sprintf("The stage has failed (%d out of %d clusters failed to update; total failures across stages have exceeded the threshold)", + tracker.failedClusterCnt.Load(), len(stage.Clusters)), + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + // Mark all ongoing updates in the stage as canceled due to the stage-level failure. + markAllOngoingUpdatesInStageAsCanceled(stagedUpdateRun, stage) + stage.CompletedTimestamp = ptr.To(metav1.Now()) + return stageProcessingResultFailed, nil, nil + case tracker.succeededClusterCnt.Load() == int32(len(stage.Clusters)): + // All clusters in the stage have succeeded. Move on to the execution of the after stage tasks. + case tracker.succeededClusterCnt.Load()+tracker.failedClusterCnt.Load() == int32(len(stage.Clusters)): + // Some clusters have failed, but the failure threshold has not been exceeded. Move on to the execution + // of the after stage tasks. + default: + // Not all clusters have been successfully updated, but the failure threshold has not been exceeded either. + // + // Requeue and continue processing in the next reconciliation loop. + if aggregatedErrs := utilerrors.NewAggregate(tracker.errs); aggregatedErrs != nil { + return stageProcessingResultErred, nil, kferrors.NewTransientError(nil, "failed to update some clusters", "errs", aggregatedErrs) + } + return stageProcessingResultInProgress, nil, nil + } + + // Execute the after stage tasks. + afterStageTasksAllCleared, afterStageTasksRequeueAfter, err := r.executeAfterStageTasks(ctx, stagedUpdateRun, stage) + if err != nil { + return stageProcessingResultErred, afterStageTasksRequeueAfter, err + } + if !afterStageTasksAllCleared { + return stageProcessingResultInProgress, afterStageTasksRequeueAfter, nil + } + + // Mark the stage as completed (fully or partially). + if tracker.failedClusterCnt.Load() > 0 { + meta.SetStatusCondition(&stage.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded, + Message: fmt.Sprintf("The stage has completed with partial success (%d out of %d clusters failed to update)", + tracker.failedClusterCnt.Load(), len(stage.Clusters)), + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } else { + meta.SetStatusCondition(&stage.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded, + Message: "The stage has completed successfully", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } + stage.CompletedTimestamp = ptr.To(metav1.Now()) + return stageProcessingResultSucceeded, nil, nil +} + +// perStageClusterProcessingTracker is shared by the workers that process the clusters in a stage concurrently; it +// hands out the clusters to process and tallies the processing results. +type perStageClusterProcessingTracker struct { + // The index (in the stage) of the next cluster to process. + nextClusterIdx atomic.Int32 + + // The counts of clusters per processing result. + succeededClusterCnt atomic.Int32 + failedClusterCnt atomic.Int32 + erredClusterCnt atomic.Int32 + + // The errors (if any) that occurred when processing the clusters, indexed the same way as the clusters in the stage. + errs []error +} + +// doClusterProcessingWork processes the clusters in a stage one at a time, each time picking up the next cluster +// that no worker has picked up yet, until there are no clusters left, the context is cancelled, or an error occurs. +// Once the cluster update failures exceed the threshold, it cancels the context, which stops the other workers too. +func (r *Reconciler) doClusterProcessingWork( + ctx context.Context, + cancel context.CancelFunc, + workerIdx int, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, + tracker *perStageClusterProcessingTracker, + clusterUpdateTotalFailures *atomic.Int32, + maxAllowedClusterFailures int32, + maxWaitTimePerCluster time.Duration, +) { + clusterCnt := int32(len(stage.Clusters)) + for { + // Stop picking up new clusters once the processing has been cancelled or has run out of time. + if ctx.Err() != nil { + return + } + + clusterIdx := tracker.nextClusterIdx.Add(1) - 1 + if clusterIdx >= clusterCnt { + klog.V(2).InfoS("No more clusters to process in the stage", "workerIdx", workerIdx, + "stage", stage.StageName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + return + } + klog.V(2).InfoS("Processing cluster in the stage", "workerIdx", workerIdx, + "clusterIdx", clusterIdx, "stage", stage.StageName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + + cluster := &stage.Clusters[clusterIdx] + clusterProcessingRes, err := r.refreshOneCluster(ctx, cluster, stagedUpdateRun, maxWaitTimePerCluster) + if err != nil { + klog.ErrorS(err, "Failed to update a cluster in the stage", + append(kferrors.Args(err), "stagedUpdateRun", klog.KObj(stagedUpdateRun), "stage", stage.StageName, "cluster", cluster.ClusterName)...) + tracker.errs[clusterIdx] = err + } + + switch clusterProcessingRes { + case clusterProcessingResultSucceeded: + // The processing has completed successfully; the worker can move on to processing the next cluster. + tracker.succeededClusterCnt.Add(1) + case clusterProcessingResultFailed: + tracker.failedClusterCnt.Add(1) + totalFailures := clusterUpdateTotalFailures.Add(1) + if totalFailures > maxAllowedClusterFailures { + // The failure threshold has been exceeded. Cancel all ongoing cluster processing attempts. + cancel() + return + } + // The worker moves on to processing the next cluster. + case clusterProcessingResultErred: + // An error occurred while processing the cluster. It does not count as a cluster failure, however, + // the worker will stop processing clusters. + tracker.erredClusterCnt.Add(1) + return + } + } +} + +func countClusterUpdateFailures(stage *rolloutv1alpha1.PerStageStatus, clusterUpdateTotalFailures *atomic.Int32) error { + for idx := range stage.Clusters { + perClusterStatus := &stage.Clusters[idx] + + completedCond := meta.FindStatusCondition(perClusterStatus.Conditions, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted) + switch { + case completedCond == nil || completedCond.Status != metav1.ConditionTrue: + // The cluster has not completed its update, yet the stage has been marked as completed. Report + // an unexpected error. + return kferrors.NewUnexpectedError(nil, "a cluster has not completed its update, yet the stage has been marked as completed", + "clusterName", perClusterStatus.ClusterName) + case completedCond.Reason == rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed: + // Register a cluster update failure. + clusterUpdateTotalFailures.Add(1) + case completedCond.Reason == rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonCanceled: + // Register a cluster update cancellation. + clusterUpdateTotalFailures.Add(1) + default: + // The cluster has completed its update successfully. + } + } + return nil +} + +func (r *Reconciler) refreshOneCluster( + ctx context.Context, + cluster *rolloutv1alpha1.PerClusterStatus, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + maxWaitTimePerCluster time.Duration, +) (clusterProcessingResult, error) { + // Check if an update attempt has been completed before for the cluster. + completedCond := meta.FindStatusCondition(cluster.Conditions, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted) + switch { + case completedCond == nil || completedCond.Status != metav1.ConditionTrue: + // There is no update attempt before or it has not been completed yet. Continue with the update process. + case completedCond.Reason == rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded: + // The update attempt has been completed successfully before. + resyncPerClusterCondWithNoCommit(stagedUpdateRun, cluster, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted) + return clusterProcessingResultSucceeded, nil + case completedCond.Reason == rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed: + // The update attempt has failed before. + resyncPerClusterCondWithNoCommit(stagedUpdateRun, cluster, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted) + return clusterProcessingResultFailed, nil + case completedCond.Reason == rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonCanceled: + // The update attempt has been canceled before. This branch is added for completeness reasons; normally this + // will never be encountered. + resyncPerClusterCondWithNoCommit(stagedUpdateRun, cluster, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted) + return clusterProcessingResultFailed, nil + default: + // Found an unexpected completion reason. Consider this as an unexpected error. + return clusterProcessingResultErred, kferrors.NewUnexpectedError(nil, "failed to determine the update status for a cluster", "unexpectedCompletionReason", completedCond.Reason) + } + + maxWaitTime := maxWaitTimePerCluster + + // Note that at this position of the code, it is guaranteed that the staged update run has either never been + // suspended or has been resumed. + resumedCond := meta.FindStatusCondition(stagedUpdateRun.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeSuspended) + + // Check if an update attempt has started already for the cluster. + startedCond := meta.FindStatusCondition(cluster.Conditions, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + resyncPerClusterCondWithNoCommit(stagedUpdateRun, cluster, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + if startedCond != nil && !clusterUpdateStartedBeforeRunResumption(startedCond, resumedCond) { + // Check if the max wait time has elapsed. + // + // It is possible that the cluster has been updated successfully before this specific check is run (and + // it could happen still within the time limit, but no one is checking at the time); here the controller + // takes a more reserved stance and would fail a cluster as long as there is no clear signal found within + // the wait time. + // + // Note that if the cluster update started before run is suspended, we reset the per cluster wait time. + if elapsed := time.Since(startedCond.LastTransitionTime.Time); elapsed > maxWaitTime { + markClusterUpdateAsFailedDueToTimeout(stagedUpdateRun, cluster) + return clusterProcessingResultFailed, nil + } + } + + // Retrieve the placement binding. + var binding placementv1alpha1.PlacementBindingAccessor + bindingKey := types.NamespacedName{ + Namespace: stagedUpdateRun.GetNamespace(), + Name: cluster.PlacementBindingName, + } + if bindingKey.Namespace == "" { + binding = &placementv1alpha1.ClusterPlacementBinding{} + } else { + binding = &placementv1alpha1.PlacementBinding{} + } + if err := r.hubClient.Get(ctx, bindingKey, binding); err != nil { + if apierrors.IsNotFound(err) { + // The placement binding cannot be found. Consider the update on the cluster as failed. + // + // Normally this would not occur as the controller has acquired the role of binding manager for + // the placement policy. + meta.SetStatusCondition(&cluster.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed, + Message: "Failed to update the cluster as the binding cannot be found", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + wrappedErr := kferrors.NewUnexpectedError(err, "", "placementBinding", klog.KRef(bindingKey.Namespace, bindingKey.Name)) + klog.ErrorS(wrappedErr, + "Failed to retrieve a placement binding for an initialized staged update run", + append(kferrors.Args(wrappedErr), "stagedUpdateRun", klog.KObj(stagedUpdateRun))...) + return clusterProcessingResultFailed, nil + } + return clusterProcessingResultErred, kferrors.NewAPIServerError(err, "failed to get the placement binding", true, + "placementBinding", klog.KRef(bindingKey.Namespace, bindingKey.Name)) + } + + // Verify that the binding has not yet been marked for deletion and the cluster name matches, as a sanity check. + // + // Normally these branches should never run: the binding is only removed, and its target cluster only + // changed, by re-running initialization, which also refreshes this very status. + if !binding.GetDeletionTimestamp().IsZero() || binding.GetSpec().ClusterName != cluster.ClusterName { + // Mark the cluster as failed to update. + meta.SetStatusCondition(&cluster.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed, + Message: "Failed to update the cluster as the binding has been marked for deletion or its cluster name does not match the tracked cluster name", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + wrappedErr := kferrors.NewUnexpectedError(nil, + "the placement binding has been marked for deletion or its cluster name does not match the tracked cluster name", + "placementBinding", klog.KObj(binding), + "placementBindingDeletionTimestamp", binding.GetDeletionTimestamp(), + "expectedClusterName", cluster.ClusterName, "observedClusterName", binding.GetSpec().ClusterName) + klog.ErrorS(wrappedErr, "Failed to update the cluster due to placement binding issue", kferrors.Args(wrappedErr)...) + return clusterProcessingResultFailed, nil + } + + // Update the binding to use the expected resource snapshot name. + // + // Note (chenyu1): for simplicity reasons, here the control loop does not implement the rollback support + // properly (i.e., no forward only constraint), and concurrent staged update runs are implemented in a + // last writer wins manner (most likely). + wantResourceSnapshotName := stagedUpdateRun.GetStatus().ResourceSnapshotNameToRollout + if binding.GetSpec().ResourceSnapshotName != wantResourceSnapshotName { + if startedCond != nil { + // The resource snapshot name is updated after this specific staged update run starts updating the + // cluster. There might be another staged update run processing the same cluster at the same time; + // consider this as a failure. + // + // There exists a corner case where the controller here is simply reading a stale version of the binding. + // Do a quorum read first to verify if the conflict does exist. + snapshotNameChanged, latestBinding, err := r.placementBindingResourceSnapshotChanged(ctx, binding, wantResourceSnapshotName) + if err != nil { + return clusterProcessingResultErred, err + } + if snapshotNameChanged { + markClusterUpdateAsFailedDueToConflict(stagedUpdateRun, cluster) + klog.V(2).InfoS("Failed to update cluster due to resource snapshot name conflict", + "placementBinding", klog.KObj(latestBinding), "cluster", cluster.ClusterName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + return clusterProcessingResultFailed, nil + } + // The cached binding was stale; use the latest one from the API server. + binding = latestBinding + } else { + binding.GetSpec().ResourceSnapshotName = wantResourceSnapshotName + if err := r.hubClient.Update(ctx, binding); err != nil { + return clusterProcessingResultErred, kferrors.NewAPIServerError(err, + "failed to update the placement binding with the resource snapshot to roll out", false, + "placementBinding", klog.KObj(binding), "resourceSnapshotName", wantResourceSnapshotName) + } + } + } + + // Mark the cluster as started updating. Reset the condition (specifically the start time) if the cluster + // started updating before the run was suspended. + if startedCond == nil || + clusterUpdateStartedBeforeRunResumption(startedCond, resumedCond) { + meta.RemoveStatusCondition(&cluster.Conditions, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + meta.SetStatusCondition(&cluster.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterStartedCondReasonUpdateStarted, + Message: "The update on the cluster has started", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + // Read the condition back so that its (assigned) LastTransitionTime is available for the + // wait-time check in the polling loop below. + startedCond = meta.FindStatusCondition(cluster.Conditions, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + } + + // As a shortcut, if the binding has been suspended, there is no need to wait. Consider it updated right away. + if binding.GetSpec().Suspended { + klog.V(2).InfoS("Successfully updated cluster (binding suspended)", + "cluster", cluster.ClusterName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + meta.SetStatusCondition(&cluster.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + Message: "The update on the cluster has completed successfully (binding is suspended)", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + return clusterProcessingResultSucceeded, nil + } + + // Capture the generation the binding is expected to have been bumped to once the resource snapshot + // update above (if any) has been applied. This is used, rather than re-reading the generation off of + // each poll below, because the cached client used by the polling loop may for a short while still + // return the pre-update version of the binding (same generation and conditions together, both stale); + // comparing a freshly re-fetched condition's ObservedGeneration against an equally stale generation + // would otherwise spuriously look up-to-date. + wantBindingGeneration := binding.GetGeneration() + + // Wait for the placement binding to report that the rollout has been synchronized to, and is available + // on, the target cluster, polling periodically until either that happens or the context is cancelled + // (e.g., the per-run processing time budget has been exhausted, in which case the caller will pick up + // where this left off on the next reconciliation pass). + timer := time.NewTimer(r.bindingAvailabilityCheckRateLimiter.When(bindingKey)) + defer timer.Stop() + // Reset the availability check rate limiter for this binding after the refreshing completes. + defer r.bindingAvailabilityCheckRateLimiter.Forget(bindingKey) + clusterProcessingRes, err := r.waitForOneClusterToBeUpdatedAndAvailable(ctx, stagedUpdateRun, cluster, + bindingKey, binding, wantResourceSnapshotName, wantBindingGeneration, + startedCond.LastTransitionTime.Time, maxWaitTime, timer) + switch { + case err != nil || clusterProcessingRes == clusterProcessingResultErred: + return clusterProcessingResultErred, kferrors.Wraps(err, "failed to wait for cluster to be updated and available", + "placementBinding", klog.KObj(binding)) + case clusterProcessingRes == clusterProcessingResultFailed: + return clusterProcessingResultFailed, nil + case clusterProcessingRes == clusterProcessingResultSucceededYetSuspended: + meta.SetStatusCondition(&cluster.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + Message: "The update on the cluster has completed successfully (binding is suspended)", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + default: // clusterProcessingRes == clusterProcessingResultSucceeded + meta.SetStatusCondition(&cluster.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + Message: "The update on the cluster has completed successfully", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } + + klog.V(2).InfoS("Successfully updated cluster", + "cluster", cluster.ClusterName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + return clusterProcessingResultSucceeded, nil +} + +func clusterUpdateStartedBeforeRunResumption(startedCond, resumedCond *metav1.Condition) bool { + if startedCond == nil || resumedCond == nil { + return false + } + + if startedCond.LastTransitionTime.Before(&resumedCond.LastTransitionTime) { + return true + } + return false +} + +func (r *Reconciler) waitForOneClusterToBeUpdatedAndAvailable(ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + cluster *rolloutv1alpha1.PerClusterStatus, + bindingKey types.NamespacedName, + binding placementv1alpha1.PlacementBindingAccessor, + wantResourceSnapshotName string, + wantBindingGeneration int64, + updateStartedAt time.Time, + maxWaitTime time.Duration, + timer *time.Timer, +) (res clusterProcessingResult, err error) { + for { + select { + case <-ctx.Done(): + return clusterProcessingResultErred, kferrors.NewTransientError(ctx.Err(), + "the placement binding has not yet become synchronized and available", "placementBinding", klog.KObj(binding)) + case <-timer.C: + // Time to check again; continue the loop. + } + + // Check also on every loop run that the max wait time is still respected; the cluster might take + // too long to become synchronized and available even though it did not time out earlier. + // + // Similarly, it is possible that the cluster has become synchronized and available just + // before the max wait time is reached, yet the controller just wasn't able to observe it in time; + // in this case, the controller takes a more reserved stance and would fail a cluster as long as there is + // no clear signal found within the wait time. + if elapsed := time.Since(updateStartedAt); elapsed > maxWaitTime { + markClusterUpdateAsFailedDueToTimeout(stagedUpdateRun, cluster) + klog.V(2).InfoS("Failed to update cluster as the max wait time has elapsed", + "placementBinding", klog.KObj(binding), "cluster", cluster.ClusterName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + return clusterProcessingResultFailed, nil + } + + // Re-fetch the binding at the start of every iteration so that the check below always runs + // against the latest observed state; without this, the loop would keep re-checking the same + // (initial) snapshot of the binding and never notice it becoming synchronized and available. + if err := r.hubClient.Get(ctx, bindingKey, binding); err != nil { + return clusterProcessingResultErred, kferrors.NewAPIServerError(err, "failed to get the placement binding", true, + "placementBinding", klog.KRef(bindingKey.Namespace, bindingKey.Name)) + } + + if binding.GetGeneration() < wantBindingGeneration { + // The binding in cache is stale; check again later. + timer.Reset(r.bindingAvailabilityCheckRateLimiter.When(bindingKey)) + continue + } + + if binding.GetSpec().ResourceSnapshotName != wantResourceSnapshotName { + // The resource snapshot name has been modified after the staged update run started processing the cluster; + // mark the cluster as failed due to conflicts. + // + // Note that this is a best-effort check; KubeFleet at this moment does not block concurrent + // staged update runs: when multiple runs target the same cluster (binding) at the same time, + // either the last writer succeeded while the others failed, or they all failed due to timeouts (though + // unlikely). + markClusterUpdateAsFailedDueToConflict(stagedUpdateRun, cluster) + klog.V(2).InfoS("Failed to update cluster due to resource snapshot name conflict", + "placementBinding", klog.KObj(binding), "cluster", cluster.ClusterName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + return clusterProcessingResultFailed, nil + } + + if binding.GetSpec().Suspended { + // The binding is suspended; consider the update succeeded yet suspended. + klog.V(2).InfoS("The placement binding is suspended; considering the update succeeded yet suspended", + "placementBinding", klog.KObj(binding), "cluster", cluster.ClusterName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + return clusterProcessingResultSucceededYetSuspended, nil + } + + // Note again that the controller here does a best-effort check only; for performance reasons, the check + // always reads using the cached client, and as result we might observe stale data. This is alright + // as concurrent staged update runs follow a last-writer-wins approach and the binding manager lock promises + // that no other changes would be made to the binding in normal ops. + bindingStatus := binding.GetStatus() + synchronizedCond := meta.FindStatusCondition(bindingStatus.Conditions, placementv1alpha1.PlacementBindingCondTypeSynchronized) + availableCond := meta.FindStatusCondition(bindingStatus.Conditions, placementv1alpha1.PlacementBindingCondTypeAvailable) + if condition.IsConditionStatusTrue(synchronizedCond, binding.GetGeneration()) && + condition.IsConditionStatusTrue(availableCond, binding.GetGeneration()) { + // The rollout has been synchronized to, and is available on, the target cluster. + return clusterProcessingResultSucceeded, nil + } + + // Reset the timer. + timer.Reset(r.bindingAvailabilityCheckRateLimiter.When(bindingKey)) + } +} diff --git a/pkg/v1/controllers/stagedupdaterun/process_integration_test.go b/pkg/v1/controllers/stagedupdaterun/process_integration_test.go new file mode 100644 index 000000000..ab648a615 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/process_integration_test.go @@ -0,0 +1,3822 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "fmt" + "slices" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +// A substring of the message that the controller reports when it completes the update on a member cluster whose +// placement binding is suspended (whether the binding is suspended before or during the update). +const bindingSuspendedMsgSubstr = "binding is suspended" + +// Substrings of the messages that the controller reports when it fails to update a member cluster. +const ( + resourceSnapshotConflictMsgSubstr = "resource snapshot name conflict" + bindingNotFoundMsgSubstr = "the binding cannot be found" + bindingDeletingOrRetargetedMsgSubstr = "the binding has been marked for deletion or its cluster name does not match" + clusterTimeoutMsgSubstr = "has failed due to timeout" +) + +// The format of the substring of the message that the controller reports when a stage fails as the cluster failures +// have exceeded the threshold; the placeholders are the number of clusters that have failed to update in the stage +// and the number of all the clusters in the stage. +const stageFailedMsgSubstrFmt = "%d out of %d clusters failed to update; total failures across stages have exceeded the threshold" + +// The label of the test cases that take more than a minute to complete; use `--label-filter='!slow'` to skip them. +const slowTestLabel = "slow" + +const ( + emptyStageAfterStageWaitTime = time.Second * 5 + + // The resource snapshot name that a test case points a placement binding to, as if another staged update run + // has rolled out to the member cluster. + conflictingResourceSnapshotName = "conflicting-resource-snapshot" + // The member cluster name that a test case retargets a placement binding to; no member cluster of this name + // needs to exist. + retargetedClusterName = "retargeted-cluster" +) + +var _ = Describe("processing staged update runs", func() { + Context("multiple stages, one cluster at a time", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 0 is the only canary cluster; member clusters 1, 2, and 3 are production clusters, which + // the prod stage sorts by their order labels. That is, the rollout order is member cluster 0 (in the canary + // stage), and then member clusters 2, 3, and 1 (in the prod stage). + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd, orderLabelKey: "30"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "10"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "20"}, + } + + // The canary stage, as it is expected to be once completed. + wantCompletedCanaryStage := rolloutv1alpha1.PerStageStatus{ + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + SortingLabelKey: ptr.To(orderLabelKey), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the canary cluster") + }) + + It("should not roll out to the production clusters while the canary cluster is being updated", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, []string{bindingNames[1], bindingNames[2], bindingNames[3]}, previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the production clusters unexpectedly") + }) + + It("should report that the canary cluster is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2, 3, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can mark the canary cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should start rolling out to the first production cluster (by sorting order)", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the first production cluster") + }) + + It("should not roll out to other production clusters while the first one is being updated", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, []string{bindingNames[3], bindingNames[1]}, previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to other production clusters unexpectedly") + }) + + It("should report that the canary stage has completed and the first production cluster is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 3), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can mark the first production cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[2], snapshotName) + }) + + It("should start rolling out to the second production cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[3], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the second production cluster") + }) + + It("should not roll out to the last production cluster while the second one is being updated", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the last production cluster unexpectedly") + }) + + It("should report that the second production cluster is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can mark the second production cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[3], snapshotName) + }) + + It("should start rolling out to the last production cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the last production cluster") + }) + + It("should report that the last production cluster is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can mark the last production cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(appNamespaceName, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + It("should have rolled out to all the member clusters", func() { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames, snapshotName) + Expect(bindingsUpdatedActual()).To(Succeed(), "Failed to roll out to all the member clusters") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("concurrent updates within a stage", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0-2 are canary clusters, which the canary stage updates two at a time; member clusters + // 3-5 are production clusters, which the prod stage updates all at once. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 6) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + MaxConcurrency: ptr.To(intstr.FromInt32(2)), + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to two canary clusters at the same time", func() { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames[0:2], snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to two canary clusters") + }) + + It("should not roll out to more member clusters while two canary clusters are being updated", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames[2:], previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to more member clusters unexpectedly") + }) + + It("should report that two canary clusters are being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 2), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3, 4, 5), + MaxConcurrency: ptr.To(int32(3)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can mark one of the two canary clusters as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should start rolling out to the last canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the last canary cluster") + }) + + It("should not roll out to the production clusters while canary clusters are being updated", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames[3:], previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the production clusters unexpectedly") + }) + + It("can mark the other canary clusters as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[2], snapshotName) + }) + + It("should start rolling out to all the production clusters at the same time", func() { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames[3:], snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to all the production clusters") + }) + + It("should report that all the production clusters are being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 4, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 5, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(3)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can mark all the production clusters as available", func() { + for _, bindingName := range bindingNames[3:] { + markPlacementBindingAsAvailable(appNamespaceName, bindingName, snapshotName) + } + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 4, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 5, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(3)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(appNamespaceName, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement binding availability detection", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the staged update run while the (only) member cluster is being updated. + wantInProgressStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantInProgressStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("should report that the member cluster is being updated", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantInProgressStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can mark the placement binding as synchronized and available, observed at a previous generation", func() { + setPlacementBindingConditions(appNamespaceName, bindingNames[0], snapshotName, metav1.ConditionTrue, metav1.ConditionTrue, -1) + }) + + It("should not consider the member cluster as updated (stale observation)", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantInProgressStatus) + Consistently(rolloutStatusActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Considered the member cluster as updated unexpectedly") + }) + + It("can mark the placement binding as synchronized but not available, observed at the current generation", func() { + setPlacementBindingConditions(appNamespaceName, bindingNames[0], snapshotName, metav1.ConditionTrue, metav1.ConditionFalse, 0) + }) + + It("should not consider the member cluster as updated (not available)", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantInProgressStatus) + Consistently(rolloutStatusActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Considered the member cluster as updated unexpectedly") + }) + + It("can mark the placement binding as synchronized and available, observed at the current generation", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings already up to date", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The placement binding of member cluster 0 already points to the resource snapshot that the staged update + // run is going to roll out, and has been marked as available; the one of member cluster 1 points to the + // previous resource snapshot. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + memberCluster0 := createMemberCluster(memberClusterNames[0], nil) + binding0 := createPlacementBinding(appNamespaceName, bindingNames[0], policyName, memberClusterNames[0], snapshotName) + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + memberCluster1 := createMemberCluster(memberClusterNames[1], nil) + binding1 := createPlacementBinding(appNamespaceName, bindingNames[1], policyName, memberClusterNames[1], previousResourceSnapshotName) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(policy, memberCluster0, binding0, memberCluster1, binding1, strategy) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should complete the update on member cluster 0 and start rolling out to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("should report that member cluster 0 has been updated and member cluster 1 is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should not have updated the placement binding of member cluster 0", func() { + // The placement binding has stayed at its initial generation, i.e., its spec has never been updated. + bindingGenerationActual := placementBindingGenerationActual(appNamespaceName, bindingNames[0], 1) + Expect(bindingGenerationActual()).To(Succeed(), "The placement binding of member cluster 0 has been updated unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("suspended placement bindings", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The placement binding of member cluster 0 is suspended before the rollout starts; the one of member + // cluster 1 is suspended while the member cluster is being updated. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + updatePlacementBindingSuspended(appNamespaceName, bindingNames[0], true) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should roll out to member cluster 0, and move on to member cluster 1 right away", func() { + // The placement binding of member cluster 0 is suspended, so the controller does not wait for it to + // become available. + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames, snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member clusters") + }) + + It("can suspend the placement binding of member cluster 1", func() { + updatePlacementBindingSuspended(appNamespaceName, bindingNames[1], true) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should report that the updates have completed as the placement bindings are suspended", func() { + clusterCompletedMsgActual := perClusterConditionMessageActual(appNamespaceName, runName, memberClusterNames[0], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, bindingSuspendedMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report that the placement binding of member cluster 0 is suspended") + + clusterCompletedMsgActual = perClusterConditionMessageActual(appNamespaceName, runName, memberClusterNames[1], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, bindingSuspendedMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report that the placement binding of member cluster 1 is suspended") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("empty stage with after-stage tasks", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 0 is a canary cluster, and member cluster 1 is a production cluster; the empty stage, which + // comes between the canary stage and the prod stage, includes no clusters, but has an after-stage timed wait + // task. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + // The canary stage, as it is expected to be once completed. + wantCompletedCanaryStage := rolloutv1alpha1.PerStageStatus{ + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: emptyStageName, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: emptyStageAfterStageWaitTime}, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the canary cluster") + }) + + It("can mark the canary cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should start the empty stage and its after-stage timed wait task", func() { + // The controller persists the status right after it starts the timed wait task, so the status is + // observable for the whole wait time. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: emptyStageName, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: emptyStageAfterStageWaitTime}, + Conditions: []metav1.Condition{perStageTaskTimedWaitStartedCond()}, + }, + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start the empty stage") + }) + + It("should start rolling out to the production cluster once the wait time has elapsed", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to roll out to the production cluster") + }) + + It("can mark the production cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: emptyStageName, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: emptyStageAfterStageWaitTime}, + Conditions: []metav1.Condition{perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()}, + }, + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(appNamespaceName, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + + stageLastedActual := stagedUpdateRunStageLastedAtLeastActual(appNamespaceName, runName, emptyStageName, emptyStageAfterStageWaitTime) + Expect(stageLastedActual()).To(Succeed(), "Failed to wait for the after-stage timed wait task of the empty stage") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("resource snapshot conflicts", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // While member cluster 0 is being updated, its placement binding is pointed to another resource snapshot, + // as if another staged update run has rolled out to it. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + // The staged update run uses the default failure policy, which allows one cluster failure. + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("should report that member cluster 0 is being updated", func() { + // The conflict is injected only after the persisted status shows that the update has started on the + // member cluster; otherwise, a reconciliation that reads a staged update run which has yet to record the + // start might consider the placement binding as not updated yet, and point it back to the resource + // snapshot to roll out. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can point the placement binding of member cluster 0 to another resource snapshot", func() { + updatePlacementBindingSpec(appNamespaceName, bindingNames[0], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + }) + + It("should move on to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should complete the staged update run with partial success", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateFailedConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should report the conflict on member cluster 0", func() { + clusterCompletedMsgActual := perClusterConditionMessageActual(appNamespaceName, runName, memberClusterNames[0], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, resourceSnapshotConflictMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report the conflict on member cluster 0") + }) + + It("should not point the placement binding of member cluster 0 back to the resource snapshot to roll out", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], conflictingResourceSnapshotName) + Expect(bindingUntouchedActual()).To(Succeed(), "The placement binding of member cluster 0 has been updated unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings deleted after initialization", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The placement binding of member cluster 0 is deleted after the staged update run has been initialized. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + // The staged update run uses the default failure policy, which allows one cluster failure. + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the staged update run", func() { + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + }) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + It("can delete the placement binding of member cluster 0", func() { + // Note that removeObjects waits until the cache observes the deletion. + removeObjects(newPlacementBindingObj(appNamespaceName, bindingNames[0])) + }) + + It("can resume the staged update run", func() { + updateStagedUpdateRunSuspended(appNamespaceName, runName, false) + }) + + It("should move on to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should complete the staged update run with partial success", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunResumedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + // The update fails before it starts, as the placement binding cannot be found. + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed)), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should report that the placement binding of member cluster 0 cannot be found", func() { + clusterCompletedMsgActual := perClusterConditionMessageActual(appNamespaceName, runName, memberClusterNames[0], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, bindingNotFoundMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report that the placement binding of member cluster 0 cannot be found") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings marked for deletion or retargeted after initialization", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // After the staged update run has been initialized, the placement binding of member cluster 0 is marked + // for deletion, and the one of member cluster 1 is retargeted to another member cluster. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 3) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil, nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + // The staged update run allows two cluster failures. + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + FailurePolicy: &rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(int32(2)), + }, + }) + }) + + It("should initialize the staged update run", func() { + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1, 2), + MaxConcurrency: ptr.To(int32(1)), + }, + }) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(appNamespaceName, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + It("can mark the placement binding of member cluster 0 for deletion", func() { + markObjectForDeletion(newPlacementBindingObj(appNamespaceName, bindingNames[0])) + }) + + It("can retarget the placement binding of member cluster 1 to another member cluster", func() { + updatePlacementBindingSpec(appNamespaceName, bindingNames[1], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ClusterName = retargetedClusterName + }) + }) + + It("can resume the staged update run", func() { + updateStagedUpdateRunSuspended(appNamespaceName, runName, false) + }) + + It("should move on to member cluster 2", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 2") + }) + + It("can mark member cluster 2 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[2], snapshotName) + }) + + It("should complete the staged update run with partial success", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunResumedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + // The updates fail before they start, as the placement bindings no longer qualify. + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed)), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed)), + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should report why the updates on member clusters 0 and 1 have failed", func() { + for _, idx := range []int{0, 1} { + clusterCompletedMsgActual := perClusterConditionMessageActual(appNamespaceName, runName, memberClusterNames[idx], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, bindingDeletingOrRetargetedMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report why the update on member cluster %d has failed", idx) + } + }) + + It("should not roll out to member clusters 0 and 1", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames[0:2], previousResourceSnapshotName) + Expect(bindingsUntouchedActual()).To(Succeed(), "Rolled out to member clusters 0 and 1 unexpectedly") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementBindingObj(appNamespaceName, bindingNames[0])) + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster failures exceeding the threshold within a stage", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0-2 are canary clusters, which the canary stage updates two at a time; member cluster 3 is + // a production cluster. The staged update run tolerates no cluster failures: the update on member cluster 0 + // fails while member cluster 1 is still being updated (its placement binding never becomes available). + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + MaxConcurrency: ptr.To(intstr.FromInt32(2)), + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + FailurePolicy: &rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(int32(0)), + }, + }) + }) + + It("should start rolling out to member clusters 0 and 1", func() { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames[0:2], snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member clusters 0 and 1") + }) + + It("should report that member clusters 0 and 1 are being updated", func() { + // The conflict below is injected only after the persisted status shows that the updates have started. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 2), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can point the placement binding of member cluster 0 to another resource snapshot", func() { + updatePlacementBindingSpec(appNamespaceName, bindingNames[0], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + }) + + It("should fail the staged update run, canceling the in-flight update on member cluster 1", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateFailedConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, + perClusterStartedCond(), + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonCanceled)), + // The update on member cluster 2 never starts. + wantPerClusterStatus(memberClusterNames, bindingNames, 2), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonFailed), + }, + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the staged update run as failed") + }) + + It("should report that the canary stage has failed as the cluster failures have exceeded the threshold", func() { + // The message counts only the clusters that have failed to update (i.e., not the canceled ones). + stageCompletedMsgActual := perStageConditionMessageActual(appNamespaceName, runName, canaryStageName, + rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, fmt.Sprintf(stageFailedMsgSubstrFmt, 1, 3)) + Expect(stageCompletedMsgActual()).To(Succeed(), "Failed to report why the canary stage has failed") + }) + + It("should not roll out to member clusters 2 and 3", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames[2:], previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to member clusters 2 and 3 unexpectedly") + }) + + It("should remove the cleanup finalizer from the staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(appNamespaceName, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("should release the binding manager role of the placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(appNamespaceName, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster failures exceeding the threshold across stages", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0 and 1 are canary clusters; member clusters 2 and 3 are production clusters. The staged + // update run tolerates one cluster failure: the update on member cluster 0 fails in the canary stage, and + // then the one on member cluster 2 fails in the prod stage. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + } + + // The canary stage, as it is expected to be once completed. + wantCompletedCanaryStage := rolloutv1alpha1.PerStageStatus{ + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateFailedConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded), + }, + MaxConcurrency: ptr.To(int32(1)), + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + // The staged update run uses the default failure policy, which allows one cluster failure. + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("should report that member cluster 0 is being updated", func() { + // The conflict below is injected only after the persisted status shows that the update has started. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can point the placement binding of member cluster 0 to another resource snapshot", func() { + updatePlacementBindingSpec(appNamespaceName, bindingNames[0], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + }) + + It("should move on to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should start rolling out to member cluster 2", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 2") + }) + + It("should report that the canary stage has completed with partial success and member cluster 2 is being updated", func() { + // Once this status is persisted, the controller processes the prod stage in later reconciliations, where + // it counts the cluster failure in the canary stage from the persisted status; the conflict below is + // injected only after this point. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 3), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can point the placement binding of member cluster 2 to another resource snapshot", func() { + updatePlacementBindingSpec(appNamespaceName, bindingNames[2], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + }) + + It("should fail the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateFailedConds()...), + // The update on member cluster 3 never starts. + wantPerClusterStatus(memberClusterNames, bindingNames, 3), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonFailed), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the staged update run as failed") + }) + + It("should report that the prod stage has failed as the cluster failures across stages have exceeded the threshold", func() { + stageCompletedMsgActual := perStageConditionMessageActual(appNamespaceName, runName, prodStageName, + rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, fmt.Sprintf(stageFailedMsgSubstrFmt, 1, 2)) + Expect(stageCompletedMsgActual()).To(Succeed(), "Failed to report why the prod stage has failed") + }) + + It("should not roll out to member cluster 3", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[3], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to member cluster 3 unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("processing time budget expiries", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The (only) member cluster stays unavailable for multiple processing time budgets, so that the controller + // picks up the in-flight update in new reconciliations. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the staged update run while the member cluster is being updated. + wantInProgressStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + // The placement binding is created at generation 1, and is bumped to generation 2 when the controller points it + // to the resource snapshot to roll out. + wantBindingGeneration := int64(2) + + var startedCondLTT metav1.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantInProgressStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("should report that the member cluster is being updated once the processing time budget runs out", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantInProgressStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + + startedCond, err := getPerClusterCondition(appNamespaceName, runName, memberClusterNames[0], rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the Started condition of the member cluster") + Expect(startedCond).ToNot(BeNil(), "The member cluster has no Started condition") + startedCondLTT = startedCond.LastTransitionTime + }) + + It("should keep waiting for the member cluster across processing time budget expiries", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantInProgressStatus) + Consistently(rolloutStatusActual, consistentlyDurationAcrossBudgetExpiries, consistentlyInterval).Should(Succeed(), "Failed to keep waiting for the member cluster") + }) + + It("should keep the update on the member cluster as it started", func() { + // The Started condition is not reset (which would also reset the wait time limit of the cluster). + startedCond, err := getPerClusterCondition(appNamespaceName, runName, memberClusterNames[0], rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the Started condition of the member cluster") + Expect(startedCond).ToNot(BeNil(), "The member cluster has no Started condition") + Expect(startedCond.LastTransitionTime.Equal(&startedCondLTT)).To(BeTrue(), + "The Started condition of the member cluster has been reset (last transition time: %v, want %v)", startedCond.LastTransitionTime, startedCondLTT) + + // The placement binding is not updated again. + bindingGenerationActual := placementBindingGenerationActual(appNamespaceName, bindingNames[0], wantBindingGeneration) + Expect(bindingGenerationActual()).To(Succeed(), "The placement binding has been updated again unexpectedly") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should not have updated the placement binding again", func() { + bindingGenerationActual := placementBindingGenerationActual(appNamespaceName, bindingNames[0], wantBindingGeneration) + Expect(bindingGenerationActual()).To(Succeed(), "The placement binding has been updated again unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("processing cluster staged update runs", func() { + Context("multiple stages, one cluster at a time", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 0 is the only canary cluster; member clusters 1, 2, and 3 are production clusters, which + // the prod stage sorts by their order labels. That is, the rollout order is member cluster 0 (in the canary + // stage), and then member clusters 2, 3, and 1 (in the prod stage). + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd, orderLabelKey: "30"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "10"}, + {envLabelKey: envLabelValueProd, orderLabelKey: "20"}, + } + + // The canary stage, as it is expected to be once completed. + wantCompletedCanaryStage := rolloutv1alpha1.PerStageStatus{ + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + SortingLabelKey: ptr.To(orderLabelKey), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the canary cluster") + }) + + It("should not roll out to the production clusters while the canary cluster is being updated", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, []string{bindingNames[1], bindingNames[2], bindingNames[3]}, previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the production clusters unexpectedly") + }) + + It("should report that the canary cluster is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2, 3, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can mark the canary cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should start rolling out to the first production cluster (by sorting order)", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the first production cluster") + }) + + It("should not roll out to other production clusters while the first one is being updated", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, []string{bindingNames[3], bindingNames[1]}, previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to other production clusters unexpectedly") + }) + + It("should report that the canary stage has completed and the first production cluster is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 3), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can mark the first production cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[2], snapshotName) + }) + + It("should start rolling out to the second production cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[3], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the second production cluster") + }) + + It("should not roll out to the last production cluster while the second one is being updated", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the last production cluster unexpectedly") + }) + + It("should report that the second production cluster is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can mark the second production cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[3], snapshotName) + }) + + It("should start rolling out to the last production cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the last production cluster") + }) + + It("should report that the last production cluster is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can mark the last production cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(clusterScopedNS, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + It("should have rolled out to all the member clusters", func() { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames, snapshotName) + Expect(bindingsUpdatedActual()).To(Succeed(), "Failed to roll out to all the member clusters") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("concurrent updates within a stage", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0-2 are canary clusters, which the canary stage updates two at a time; member clusters + // 3-5 are production clusters, which the prod stage updates all at once. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 6) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + MaxConcurrency: ptr.To(intstr.FromInt32(2)), + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to two canary clusters at the same time", func() { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames[0:2], snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to two canary clusters") + }) + + It("should not roll out to more member clusters while two canary clusters are being updated", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames[2:], previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to more member clusters unexpectedly") + }) + + It("should report that two canary clusters are being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 2), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3, 4, 5), + MaxConcurrency: ptr.To(int32(3)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can mark one of the two canary clusters as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should start rolling out to the last canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the last canary cluster") + }) + + It("should not roll out to the production clusters while canary clusters are being updated", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames[3:], previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the production clusters unexpectedly") + }) + + It("can mark the other canary clusters as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[2], snapshotName) + }) + + It("should start rolling out to all the production clusters at the same time", func() { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames[3:], snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to all the production clusters") + }) + + It("should report that all the production clusters are being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 4, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 5, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(3)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can mark all the production clusters as available", func() { + for _, bindingName := range bindingNames[3:] { + markPlacementBindingAsAvailable(clusterScopedNS, bindingName, snapshotName) + } + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 3, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 4, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 5, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(3)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(clusterScopedNS, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement binding availability detection", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the cluster staged update run while the (only) member cluster is being updated. + wantInProgressStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantInProgressStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("should report that the member cluster is being updated", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantInProgressStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can mark the placement binding as synchronized and available, observed at a previous generation", func() { + setPlacementBindingConditions(clusterScopedNS, bindingNames[0], snapshotName, metav1.ConditionTrue, metav1.ConditionTrue, -1) + }) + + It("should not consider the member cluster as updated (stale observation)", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantInProgressStatus) + Consistently(rolloutStatusActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Considered the member cluster as updated unexpectedly") + }) + + It("can mark the placement binding as synchronized but not available, observed at the current generation", func() { + setPlacementBindingConditions(clusterScopedNS, bindingNames[0], snapshotName, metav1.ConditionTrue, metav1.ConditionFalse, 0) + }) + + It("should not consider the member cluster as updated (not available)", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantInProgressStatus) + Consistently(rolloutStatusActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Considered the member cluster as updated unexpectedly") + }) + + It("can mark the placement binding as synchronized and available, observed at the current generation", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings already up to date", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The placement binding of member cluster 0 already points to the resource snapshot that the staged update + // run is going to roll out, and has been marked as available; the one of member cluster 1 points to the + // previous resource snapshot. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + memberCluster0 := createMemberCluster(memberClusterNames[0], nil) + binding0 := createPlacementBinding(clusterScopedNS, bindingNames[0], policyName, memberClusterNames[0], snapshotName) + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + memberCluster1 := createMemberCluster(memberClusterNames[1], nil) + binding1 := createPlacementBinding(clusterScopedNS, bindingNames[1], policyName, memberClusterNames[1], previousResourceSnapshotName) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(policy, memberCluster0, binding0, memberCluster1, binding1, strategy) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should complete the update on member cluster 0 and start rolling out to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("should report that member cluster 0 has been updated and member cluster 1 is being updated", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + It("should not have updated the placement binding of member cluster 0", func() { + // The placement binding has stayed at its initial generation, i.e., its spec has never been updated. + bindingGenerationActual := placementBindingGenerationActual(clusterScopedNS, bindingNames[0], 1) + Expect(bindingGenerationActual()).To(Succeed(), "The placement binding of member cluster 0 has been updated unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("suspended placement bindings", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The placement binding of member cluster 0 is suspended before the rollout starts; the one of member + // cluster 1 is suspended while the member cluster is being updated. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + updatePlacementBindingSuspended(clusterScopedNS, bindingNames[0], true) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should roll out to member cluster 0, and move on to member cluster 1 right away", func() { + // The placement binding of member cluster 0 is suspended, so the controller does not wait for it to + // become available. + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames, snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member clusters") + }) + + It("can suspend the placement binding of member cluster 1", func() { + updatePlacementBindingSuspended(clusterScopedNS, bindingNames[1], true) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + It("should report that the updates have completed as the placement bindings are suspended", func() { + clusterCompletedMsgActual := perClusterConditionMessageActual(clusterScopedNS, runName, memberClusterNames[0], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, bindingSuspendedMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report that the placement binding of member cluster 0 is suspended") + + clusterCompletedMsgActual = perClusterConditionMessageActual(clusterScopedNS, runName, memberClusterNames[1], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, bindingSuspendedMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report that the placement binding of member cluster 1 is suspended") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("empty stage with after-stage tasks", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 0 is a canary cluster, and member cluster 1 is a production cluster; the empty stage, which + // comes between the canary stage and the prod stage, includes no clusters, but has an after-stage timed wait + // task. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + // The canary stage, as it is expected to be once completed. + wantCompletedCanaryStage := rolloutv1alpha1.PerStageStatus{ + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: emptyStageName, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: emptyStageAfterStageWaitTime}, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the canary cluster") + }) + + It("can mark the canary cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should start the empty stage and its after-stage timed wait task", func() { + // The controller persists the status right after it starts the timed wait task, so the status is + // observable for the whole wait time. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: emptyStageName, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: emptyStageAfterStageWaitTime}, + Conditions: []metav1.Condition{perStageTaskTimedWaitStartedCond()}, + }, + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start the empty stage") + }) + + It("should start rolling out to the production cluster once the wait time has elapsed", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to roll out to the production cluster") + }) + + It("can mark the production cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: emptyStageName, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: emptyStageAfterStageWaitTime}, + Conditions: []metav1.Condition{perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()}, + }, + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(clusterScopedNS, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + + stageLastedActual := stagedUpdateRunStageLastedAtLeastActual(clusterScopedNS, runName, emptyStageName, emptyStageAfterStageWaitTime) + Expect(stageLastedActual()).To(Succeed(), "Failed to wait for the after-stage timed wait task of the empty stage") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("resource snapshot conflicts", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // While member cluster 0 is being updated, its placement binding is pointed to another resource snapshot, + // as if another cluster staged update run has rolled out to it. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + // The cluster staged update run uses the default failure policy, which allows one cluster failure. + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("should report that member cluster 0 is being updated", func() { + // The conflict is injected only after the persisted status shows that the update has started on the + // member cluster; otherwise, a reconciliation that reads a cluster staged update run which has yet to record the + // start might consider the placement binding as not updated yet, and point it back to the resource + // snapshot to roll out. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can point the placement binding of member cluster 0 to another resource snapshot", func() { + updatePlacementBindingSpec(clusterScopedNS, bindingNames[0], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + }) + + It("should move on to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should complete the cluster staged update run with partial success", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateFailedConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + It("should report the conflict on member cluster 0", func() { + clusterCompletedMsgActual := perClusterConditionMessageActual(clusterScopedNS, runName, memberClusterNames[0], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, resourceSnapshotConflictMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report the conflict on member cluster 0") + }) + + It("should not point the placement binding of member cluster 0 back to the resource snapshot to roll out", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], conflictingResourceSnapshotName) + Expect(bindingUntouchedActual()).To(Succeed(), "The placement binding of member cluster 0 has been updated unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings deleted after initialization", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The placement binding of member cluster 0 is deleted after the cluster staged update run has been initialized. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + // The cluster staged update run uses the default failure policy, which allows one cluster failure. + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + }) + }) + + It("should initialize the cluster staged update run", func() { + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + }) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + It("can delete the placement binding of member cluster 0", func() { + // Note that removeObjects waits until the cache observes the deletion. + removeObjects(newPlacementBindingObj(clusterScopedNS, bindingNames[0])) + }) + + It("can resume the cluster staged update run", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, runName, false) + }) + + It("should move on to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should complete the cluster staged update run with partial success", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunResumedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + // The update fails before it starts, as the placement binding cannot be found. + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed)), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + It("should report that the placement binding of member cluster 0 cannot be found", func() { + clusterCompletedMsgActual := perClusterConditionMessageActual(clusterScopedNS, runName, memberClusterNames[0], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, bindingNotFoundMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report that the placement binding of member cluster 0 cannot be found") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("placement bindings marked for deletion or retargeted after initialization", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // After the cluster staged update run has been initialized, the placement binding of member cluster 0 is marked + // for deletion, and the one of member cluster 1 is retargeted to another member cluster. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 3) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil, nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + // The cluster staged update run allows two cluster failures. + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + Suspended: true, + FailurePolicy: &rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(int32(2)), + }, + }) + }) + + It("should initialize the cluster staged update run", func() { + wantStatus := wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName, []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1, 2), + MaxConcurrency: ptr.To(int32(1)), + }, + }) + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(clusterScopedNS, runName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the cluster staged update run") + }) + + It("can mark the placement binding of member cluster 0 for deletion", func() { + markObjectForDeletion(newPlacementBindingObj(clusterScopedNS, bindingNames[0])) + }) + + It("can retarget the placement binding of member cluster 1 to another member cluster", func() { + updatePlacementBindingSpec(clusterScopedNS, bindingNames[1], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ClusterName = retargetedClusterName + }) + }) + + It("can resume the cluster staged update run", func() { + updateStagedUpdateRunSuspended(clusterScopedNS, runName, false) + }) + + It("should move on to member cluster 2", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 2") + }) + + It("can mark member cluster 2 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[2], snapshotName) + }) + + It("should complete the cluster staged update run with partial success", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunResumedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + // The updates fail before they start, as the placement bindings no longer qualify. + wantPerClusterStatus(memberClusterNames, bindingNames, 0, + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed)), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed)), + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + It("should report why the updates on member clusters 0 and 1 have failed", func() { + for _, idx := range []int{0, 1} { + clusterCompletedMsgActual := perClusterConditionMessageActual(clusterScopedNS, runName, memberClusterNames[idx], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, bindingDeletingOrRetargetedMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report why the update on member cluster %d has failed", idx) + } + }) + + It("should not roll out to member clusters 0 and 1", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames[0:2], previousResourceSnapshotName) + Expect(bindingsUntouchedActual()).To(Succeed(), "Rolled out to member clusters 0 and 1 unexpectedly") + }) + + AfterAll(func() { + removeTestFinalizer(newPlacementBindingObj(clusterScopedNS, bindingNames[0])) + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster failures exceeding the threshold within a stage", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0-2 are canary clusters, which the canary stage updates two at a time; member cluster 3 is + // a production cluster. The cluster staged update run tolerates no cluster failures: the update on member cluster 0 + // fails while member cluster 1 is still being updated (its placement binding never becomes available). + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + MaxConcurrency: ptr.To(intstr.FromInt32(2)), + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + FailurePolicy: &rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(int32(0)), + }, + }) + }) + + It("should start rolling out to member clusters 0 and 1", func() { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames[0:2], snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member clusters 0 and 1") + }) + + It("should report that member clusters 0 and 1 are being updated", func() { + // The conflict below is injected only after the persisted status shows that the updates have started. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 2), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can point the placement binding of member cluster 0 to another resource snapshot", func() { + updatePlacementBindingSpec(clusterScopedNS, bindingNames[0], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + }) + + It("should fail the cluster staged update run, canceling the in-flight update on member cluster 1", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateFailedConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, + perClusterStartedCond(), + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonCanceled)), + // The update on member cluster 2 never starts. + wantPerClusterStatus(memberClusterNames, bindingNames, 2), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonFailed), + }, + MaxConcurrency: ptr.To(int32(2)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the cluster staged update run as failed") + }) + + It("should report that the canary stage has failed as the cluster failures have exceeded the threshold", func() { + // The message counts only the clusters that have failed to update (i.e., not the canceled ones). + stageCompletedMsgActual := perStageConditionMessageActual(clusterScopedNS, runName, canaryStageName, + rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, fmt.Sprintf(stageFailedMsgSubstrFmt, 1, 3)) + Expect(stageCompletedMsgActual()).To(Succeed(), "Failed to report why the canary stage has failed") + }) + + It("should not roll out to member clusters 2 and 3", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames[2:], previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to member clusters 2 and 3 unexpectedly") + }) + + It("should remove the cleanup finalizer from the cluster staged update run", func() { + finalizerRemovedActual := stagedUpdateRunCleanupFinalizerRemovedActual(clusterScopedNS, runName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("should release the binding manager role of the cluster placement policy", func() { + roleReleasedActual := bindingManagerRoleReleasedActual(clusterScopedNS, policyName) + Eventually(roleReleasedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to release the binding manager role") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster failures exceeding the threshold across stages", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member clusters 0 and 1 are canary clusters; member clusters 2 and 3 are production clusters. The staged + // update run tolerates one cluster failure: the update on member cluster 0 fails in the canary stage, and + // then the one on member cluster 2 fails in the prod stage. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 4) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + {envLabelKey: envLabelValueProd}, + } + + // The canary stage, as it is expected to be once completed. + wantCompletedCanaryStage := rolloutv1alpha1.PerStageStatus{ + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateFailedConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded), + }, + MaxConcurrency: ptr.To(int32(1)), + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + // The cluster staged update run uses the default failure policy, which allows one cluster failure. + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("should report that member cluster 0 is being updated", func() { + // The conflict below is injected only after the persisted status shows that the update has started. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 1), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2, 3), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can point the placement binding of member cluster 0 to another resource snapshot", func() { + updatePlacementBindingSpec(clusterScopedNS, bindingNames[0], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + }) + + It("should move on to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should start rolling out to member cluster 2", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 2") + }) + + It("should report that the canary stage has completed with partial success and member cluster 2 is being updated", func() { + // Once this status is persisted, the controller processes the prod stage in later reconciliations, where + // it counts the cluster failure in the canary stage from the persisted status; the conflict below is + // injected only after this point. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterStartedCond()), + wantPerClusterStatus(memberClusterNames, bindingNames, 3), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can point the placement binding of member cluster 2 to another resource snapshot", func() { + updatePlacementBindingSpec(clusterScopedNS, bindingNames[2], func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.ResourceSnapshotName = conflictingResourceSnapshotName + }) + }) + + It("should fail the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateFailedConds()...), + // The update on member cluster 3 never starts. + wantPerClusterStatus(memberClusterNames, bindingNames, 3), + }, + Conditions: []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonFailed), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the cluster staged update run as failed") + }) + + It("should report that the prod stage has failed as the cluster failures across stages have exceeded the threshold", func() { + stageCompletedMsgActual := perStageConditionMessageActual(clusterScopedNS, runName, prodStageName, + rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, fmt.Sprintf(stageFailedMsgSubstrFmt, 1, 2)) + Expect(stageCompletedMsgActual()).To(Succeed(), "Failed to report why the prod stage has failed") + }) + + It("should not roll out to member cluster 3", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[3], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to member cluster 3 unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("processing time budget expiries", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // The (only) member cluster stays unavailable for multiple processing time budgets, so that the controller + // picks up the in-flight update in new reconciliations. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the cluster staged update run while the member cluster is being updated. + wantInProgressStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + // The placement binding is created at generation 1, and is bumped to generation 2 when the controller points it + // to the resource snapshot to roll out. + wantBindingGeneration := int64(2) + + var startedCondLTT metav1.Time + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantInProgressStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("should report that the member cluster is being updated once the processing time budget runs out", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantInProgressStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + + startedCond, err := getPerClusterCondition(clusterScopedNS, runName, memberClusterNames[0], rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the Started condition of the member cluster") + Expect(startedCond).ToNot(BeNil(), "The member cluster has no Started condition") + startedCondLTT = startedCond.LastTransitionTime + }) + + It("should keep waiting for the member cluster across processing time budget expiries", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantInProgressStatus) + Consistently(rolloutStatusActual, consistentlyDurationAcrossBudgetExpiries, consistentlyInterval).Should(Succeed(), "Failed to keep waiting for the member cluster") + }) + + It("should keep the update on the member cluster as it started", func() { + // The Started condition is not reset (which would also reset the wait time limit of the cluster). + startedCond, err := getPerClusterCondition(clusterScopedNS, runName, memberClusterNames[0], rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the Started condition of the member cluster") + Expect(startedCond).ToNot(BeNil(), "The member cluster has no Started condition") + Expect(startedCond.LastTransitionTime.Equal(&startedCondLTT)).To(BeTrue(), + "The Started condition of the member cluster has been reset (last transition time: %v, want %v)", startedCond.LastTransitionTime, startedCondLTT) + + // The placement binding is not updated again. + bindingGenerationActual := placementBindingGenerationActual(clusterScopedNS, bindingNames[0], wantBindingGeneration) + Expect(bindingGenerationActual()).To(Succeed(), "The placement binding has been updated again unexpectedly") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + It("should not have updated the placement binding again", func() { + bindingGenerationActual := placementBindingGenerationActual(clusterScopedNS, bindingNames[0], wantBindingGeneration) + Expect(bindingGenerationActual()).To(Succeed(), "The placement binding has been updated again unexpectedly") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("processing staged update runs and cluster staged update runs (per-cluster wait time limits)", Label(slowTestLabel), func() { + // The API requires the per-cluster wait time limit to be at least one minute. To pay for the wait only once, the + // context below creates all of its staged update runs, of both scopes, at the same time, and verifies them + // together, rather than following the one-Describe-per-scope layout of the other test cases. + Context("member clusters that time out", Ordered, func() { + type testCase struct { + ns string + maxAllowedClusterFailures int32 + + configMapName string + policyName string + strategyName string + runName string + snapshotName string + memberClusterNames []string + bindingNames []string + } + + // newTestCase prepares a staged update run (cluster-scoped if the namespace is empty) whose member cluster 0 + // never becomes available; if the run tolerates a cluster failure, its stage also has member cluster 1, which + // becomes available. + newTestCase := func(ns string, maxAllowedClusterFailures int32) *testCase { + randStr := utils.RandStr() + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + clusterCount := 1 + if maxAllowedClusterFailures > 0 { + clusterCount = 2 + } + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, clusterCount) + return &testCase{ + ns: ns, + maxAllowedClusterFailures: maxAllowedClusterFailures, + configMapName: fmt.Sprintf(configMapNameTemplate, randStr), + policyName: policyName, + strategyName: fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr), + runName: fmt.Sprintf(stagedUpdateRunNameTemplate, randStr), + memberClusterNames: memberClusterNames, + bindingNames: bindingNames, + } + } + + // For each scope, one staged update run tolerates one cluster failure, and the other tolerates none. + testCases := []*testCase{ + newTestCase(appNamespaceName, 1), + newTestCase(appNamespaceName, 0), + newTestCase(clusterScopedNS, 1), + newTestCase(clusterScopedNS, 0), + } + + BeforeAll(func() { + for _, tc := range testCases { + createConfigMap(tc.configMapName) + policy := createPlacementPolicy(tc.ns, tc.policyName, tc.configMapName) + // The snapshot name factors in the UID of the placement policy, which is only known once it is created. + tc.snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(tc.ns, tc.policyName, tc.memberClusterNames, tc.bindingNames, + make([]map[string]string, len(tc.memberClusterNames))) + // All the member clusters in the stage are updated at the same time. + strategy := createStagedUpdateStrategy(tc.ns, tc.strategyName, []rolloutv1alpha1.Stage{ + { + Name: restStageName, + LabelSelector: &metav1.LabelSelector{}, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(tc.ns, tc.runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: tc.policyName, + StagedUpdateStrategyName: tc.strategyName, + FailurePolicy: &rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(tc.maxAllowedClusterFailures), + MaxWaitTimePerClusterMinutes: 1, + }, + }) + } + }) + + It("should start rolling out to all the member clusters", func() { + for _, tc := range testCases { + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(tc.ns, tc.bindingNames, tc.snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member clusters of run %s", tc.runName) + } + }) + + It("can mark the member clusters that do not time out as available", func() { + for _, tc := range testCases { + if len(tc.bindingNames) > 1 { + markPlacementBindingAsAvailable(tc.ns, tc.bindingNames[1], tc.snapshotName) + } + } + }) + + It("should fail the updates on the member clusters that time out, and complete the runs accordingly", func() { + for _, tc := range testCases { + // The update on member cluster 0 fails once the wait time limit is reached. + clusterStatuses := []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(tc.memberClusterNames, tc.bindingNames, 0, perClusterUpdateFailedConds()...), + } + runCompletedReason := rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed + stageCompletedReason := rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonFailed + if tc.maxAllowedClusterFailures > 0 { + // The failure is tolerated; the update on member cluster 1 succeeds. + clusterStatuses = append(clusterStatuses, + wantPerClusterStatus(tc.memberClusterNames, tc.bindingNames, 1, perClusterUpdateSucceededConds()...)) + runCompletedReason = rolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded + stageCompletedReason = rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonPartiallySucceeded + } + + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(runCompletedReason), + }, + ResourceSnapshotNameToRollout: tc.snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: restStageName, + Clusters: clusterStatuses, + Conditions: []metav1.Condition{perStageStartedCond(), perStageCompletedCond(stageCompletedReason)}, + MaxConcurrency: ptr.To(int32(len(clusterStatuses))), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(tc.ns, tc.runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForClusterTimeout, eventuallyInterval).Should(Succeed(), "Failed to complete run %s", tc.runName) + } + }) + + It("should report the timeouts", func() { + for _, tc := range testCases { + clusterCompletedMsgActual := perClusterConditionMessageActual(tc.ns, tc.runName, tc.memberClusterNames[0], + rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, clusterTimeoutMsgSubstr) + Expect(clusterCompletedMsgActual()).To(Succeed(), "Failed to report the timeout in run %s", tc.runName) + + // The stage lasts at least as long as the wait time limit. + stageLastedActual := stagedUpdateRunStageLastedAtLeastActual(tc.ns, tc.runName, restStageName, time.Minute) + Expect(stageLastedActual()).To(Succeed(), "Failed to wait for the wait time limit in run %s", tc.runName) + } + }) + + AfterAll(func() { + for _, tc := range testCases { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(tc.ns, tc.runName), + newPlacementResourceSnapshotObj(tc.ns, tc.snapshotName), + newStagedUpdateStrategyObj(tc.ns, tc.strategyName), + newPlacementPolicyObj(tc.ns, tc.policyName), + newConfigMapObj(tc.configMapName), + }, + memberClusterAndBindingObjs(tc.ns, tc.memberClusterNames, tc.bindingNames), + )...) + } + }) + }) +}) diff --git a/pkg/v1/controllers/stagedupdaterun/quorumread.go b/pkg/v1/controllers/stagedupdaterun/quorumread.go new file mode 100644 index 000000000..9a557de53 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/quorumread.go @@ -0,0 +1,53 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +// placementBindingResourceSnapshotChanged reads the placement binding straight from the API server, and reports +// whether the resource snapshot name in its spec differs from a given desired resource snapshot name. +// +// A binding that no longer exists is reported as an API server error; the returned error wraps the original one, +// so callers can check it with apierrors.IsNotFound. +func (r *Reconciler) placementBindingResourceSnapshotChanged( + ctx context.Context, + placementBinding placementv1alpha1.PlacementBindingAccessor, + wantResourceSnapshotName string, +) (bool, placementv1alpha1.PlacementBindingAccessor, error) { + var latestBinding placementv1alpha1.PlacementBindingAccessor + if placementBinding.GetNamespace() == "" { + latestBinding = &placementv1alpha1.ClusterPlacementBinding{} + } else { + latestBinding = &placementv1alpha1.PlacementBinding{} + } + + // Read from the API server directly, bypassing the (possibly stale) cache. + key := types.NamespacedName{Namespace: placementBinding.GetNamespace(), Name: placementBinding.GetName()} + if err := r.hubUncachedClient.Get(ctx, key, latestBinding); err != nil { + return false, nil, errors.NewAPIServerError(err, "failed to get the placement binding", false, + "placementBinding", klog.KObj(placementBinding)) + } + return latestBinding.GetSpec().ResourceSnapshotName != wantResourceSnapshotName, latestBinding, nil +} diff --git a/pkg/v1/controllers/stagedupdaterun/retrieval.go b/pkg/v1/controllers/stagedupdaterun/retrieval.go new file mode 100644 index 000000000..1fa20e23a --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/retrieval.go @@ -0,0 +1,199 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + "sync/atomic" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + utilerrors "k8s.io/apimachinery/pkg/util/errors" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +func (r *Reconciler) retrieveStagedUpdateRun(ctx context.Context, key types.NamespacedName) (rolloutv1alpha1.StagedUpdateRunAccessor, error) { + var stagedUpdateRunAccessor rolloutv1alpha1.StagedUpdateRunAccessor + var gvk schema.GroupVersionKind + if key.Namespace == "" { + // The object to retrieve is a ClusterStagedUpdateRun. + stagedUpdateRunAccessor = &rolloutv1alpha1.ClusterStagedUpdateRun{} + gvk = rolloutv1alpha1.GroupVersion.WithKind(rolloutv1alpha1.ClusterStagedUpdateRunKind) + } else { + // The object to retrieve is a StagedUpdateRun. + stagedUpdateRunAccessor = &rolloutv1alpha1.StagedUpdateRun{} + gvk = rolloutv1alpha1.GroupVersion.WithKind(rolloutv1alpha1.StagedUpdateRunKind) + } + + if err := r.hubClient.Get(ctx, key, stagedUpdateRunAccessor); err != nil { + return nil, errors.NewAPIServerError(err, "failed to get staged update run object", true) + } + // The controller-runtime client (and the informer cache backing it) does not populate TypeMeta + // (apiVersion/kind) on typed Get calls, so the controller sets the GVK explicitly here. This ensures + // that any code relying on stagedUpdateRunAccessor.GetObjectKind() afterwards (e.g., when building an + // ObjectReference for the binding manager) sees a correctly populated Kind/APIVersion. + stagedUpdateRunAccessor.GetObjectKind().SetGroupVersionKind(gvk) + return stagedUpdateRunAccessor, nil +} + +func (r *Reconciler) retrieveLinkedPlacementPolicy( + ctx context.Context, stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor) (placementv1alpha1.PlacementPolicyAccessor, error) { + placementPolicyName := stagedUpdateRun.GetSpec().PlacementPolicyName + placementPolicyKey := types.NamespacedName{ + Namespace: stagedUpdateRun.GetNamespace(), + Name: placementPolicyName, + } + + var placementPolicyAccessor placementv1alpha1.PlacementPolicyAccessor + if placementPolicyKey.Namespace == "" { + placementPolicyAccessor = &placementv1alpha1.ClusterPlacementPolicy{} + } else { + placementPolicyAccessor = &placementv1alpha1.PlacementPolicy{} + } + + if err := r.hubClient.Get(ctx, placementPolicyKey, placementPolicyAccessor); err != nil { + return nil, errors.NewAPIServerError(err, "failed to get placement policy object", true) + } + return placementPolicyAccessor, nil +} + +func (r *Reconciler) retrieveStagedUpdateStrategy( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, +) (rolloutv1alpha1.StagedUpdateStrategyAccessor, error) { + strategyKey := types.NamespacedName{ + Namespace: stagedUpdateRun.GetNamespace(), + Name: stagedUpdateRun.GetSpec().StagedUpdateStrategyName, + } + + var stagedUpdateStrategy rolloutv1alpha1.StagedUpdateStrategyAccessor + if strategyKey.Namespace == "" { + // The staged update run is cluster-scoped; retrieve the ClusterStagedUpdateStrategy. + stagedUpdateStrategy = &rolloutv1alpha1.ClusterStagedUpdateStrategy{} + } else { + // The staged update run is namespaced; retrieve the StagedUpdateStrategy in its namespace. + stagedUpdateStrategy = &rolloutv1alpha1.StagedUpdateStrategy{} + } + + if err := r.hubClient.Get(ctx, strategyKey, stagedUpdateStrategy); err != nil { + return nil, errors.NewAPIServerError(err, "failed to get staged update strategy object", true, + "stagedUpdateStrategy", klog.KRef(strategyKey.Namespace, strategyKey.Name)) + } + return stagedUpdateStrategy, nil +} + +func (r *Reconciler) listAllPlacementBindings( + ctx context.Context, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, +) ([]placementv1alpha1.PlacementBindingAccessor, error) { + // Bindings are matched by the placement policy name in their specs, as the owner label value might be truncated. + ownerFieldSelector := client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByCustomFieldName: placementPolicy.GetName(), + } + + var placementBindings []placementv1alpha1.PlacementBindingAccessor + if placementPolicy.GetNamespace() == "" { + // The placement policy is cluster-scoped; list all cluster placement bindings owned by it. + cpbs := &placementv1alpha1.ClusterPlacementBindingList{} + if err := r.hubClient.List(ctx, cpbs, ownerFieldSelector); err != nil { + return nil, errors.NewAPIServerError(err, "failed to list cluster placement bindings", true, + "placementPolicy", klog.KRef(placementPolicy.GetNamespace(), placementPolicy.GetName())) + } + for i := range cpbs.Items { + cpb := &cpbs.Items[i] + if cpb.DeletionTimestamp.IsZero() { + // Count only placement bindings that have not been marked for deletion. + placementBindings = append(placementBindings, cpb) + } + } + } else { + // The placement policy is namespaced; list all placement bindings owned by it in its namespace. + namespaceSelector := client.InNamespace(placementPolicy.GetNamespace()) + pbs := &placementv1alpha1.PlacementBindingList{} + if err := r.hubClient.List(ctx, pbs, namespaceSelector, ownerFieldSelector); err != nil { + return nil, errors.NewAPIServerError(err, "failed to list placement bindings", true, + "placementPolicy", klog.KRef(placementPolicy.GetNamespace(), placementPolicy.GetName())) + } + for i := range pbs.Items { + pb := &pbs.Items[i] + if pb.DeletionTimestamp.IsZero() { + // Count only placement bindings that have not been marked for deletion. + placementBindings = append(placementBindings, pb) + } + } + } + return placementBindings, nil +} + +func (r *Reconciler) retrieveReferencedClusters( + ctx context.Context, + placementBindings []placementv1alpha1.PlacementBindingAccessor, +) ([]*clusterv1beta1.MemberCluster, bool, error) { + // Prepare a child context. + childCtx, childCancel := context.WithCancel(ctx) + defer childCancel() + + // Prepare a flag that signals whether some of the member clusters could not be retrieved. + someClustersNotRetrieved := atomic.Bool{} + + // Pre-allocate the slice to avoid contention. + memberClusters := make([]*clusterv1beta1.MemberCluster, len(placementBindings)) + errs := make([]error, len(placementBindings)) + // Retrieve member cluster objects in parallel. + doWork := func(pieces int) { + clusterName := placementBindings[pieces].GetSpec().ClusterName + memberCluster := &clusterv1beta1.MemberCluster{} + if err := r.hubClient.Get(childCtx, types.NamespacedName{Name: clusterName}, memberCluster); err != nil { + if apierrors.IsNotFound(err) { + // The member cluster does not exist; skip it. + klog.V(2).InfoS("Cannot find the member cluster referenced by a placement binding", + "memberClusterName", clusterName, "placementBinding", klog.KObj(placementBindings[pieces])) + someClustersNotRetrieved.Store(true) + return + } + errs[pieces] = errors.NewAPIServerError(err, "failed to get member cluster object", true, + "memberCluster", klog.KRef("", clusterName), "placementBinding", klog.KObj(placementBindings[pieces])) + // Fail fast. + childCancel() + return + } + if !memberCluster.DeletionTimestamp.IsZero() { + // The member cluster has been marked for deletion; skip it. + klog.V(2).InfoS("The member cluster referenced by a placement binding has been marked for deletion; skip it", + "memberCluster", klog.KObj(memberCluster), "placementBinding", klog.KObj(placementBindings[pieces])) + someClustersNotRetrieved.Store(true) + return + } + memberClusters[pieces] = memberCluster + } + r.parallelizer.ParallelizeUntil(childCtx, len(placementBindings), doWork, "retrieveReferencedClusters") + + if err := utilerrors.NewAggregate(errs); err != nil { + return nil, someClustersNotRetrieved.Load(), errors.Wraps(err, "aggregated errors occurred when retrieving referenced clusters") + } + // Return the context error (nil if the context has not been cancelled) here to avoid nil cluster entries + // being returned without errors. + return memberClusters, someClustersNotRetrieved.Load(), ctx.Err() +} diff --git a/pkg/v1/controllers/stagedupdaterun/status.go b/pkg/v1/controllers/stagedupdaterun/status.go new file mode 100644 index 000000000..19c2aa424 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/status.go @@ -0,0 +1,273 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + "fmt" + + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/condition" + "go.goms.io/fleet/pkg/utils/errors" +) + +func (r *Reconciler) markStagedUpdateRunAsInTerminalState( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + message string, +) error { + initCond := meta.FindStatusCondition(stagedUpdateRun.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeInitialized) + completedCond := meta.FindStatusCondition(stagedUpdateRun.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeCompleted) + stagedUpdateRunStatus := stagedUpdateRun.GetStatus() + + switch { + case initCond == nil: + // Add the failed to initialize condition if the staged update run has not been initialized yet; this alone + // puts the staged update run in a terminal state, so the completed condition is not needed. + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeInitialized, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunInitializedCondReasonFailed, + Message: message, + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + case initCond.Status == metav1.ConditionFalse: + // The staged update run has already failed to initialize; there is nothing to add. + case completedCond == nil: + // The staged update run has been initialized but has not completed yet; add the failed to complete condition. + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed, + Message: message, + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + default: + // The staged update run has already failed to complete; there is nothing to add. + } + + if err := r.hubClient.Status().Update(ctx, stagedUpdateRun); err != nil { + return errors.NewAPIServerError(err, "failed to update the staged update run status", false) + } + return nil +} + +func (r *Reconciler) markStagedUpdateRunAsFailedToInitialize( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + message string, +) error { + stagedUpdateRunStatus := stagedUpdateRun.GetStatus() + + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeInitialized, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunInitializedCondReasonFailed, + Message: message, + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + + if err := r.hubClient.Status().Update(ctx, stagedUpdateRun); err != nil { + return errors.NewAPIServerError(err, "failed to update the staged update run status", false) + } + return nil +} + +func (r *Reconciler) syncStagedUpdateRunInitializedStatus( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + perStageStatuses []rolloutv1alpha1.PerStageStatus, + resourceSnapshotName string, +) error { + stagedUpdateRunStatus := stagedUpdateRun.GetStatus() + + // Set the resource snapshot name to roll out, the prepped stages, and the initialized condition. + stagedUpdateRunStatus.ResourceSnapshotNameToRollout = resourceSnapshotName + stagedUpdateRunStatus.Stages = perStageStatuses + + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeInitialized, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunInitializedCondReasonPreppedResourceSnapshotAndAllStages, + Message: fmt.Sprintf("the staged update run has been initialized (resource snapshot to roll out: %s; prepped %d stages)", + resourceSnapshotName, len(perStageStatuses)), + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + + if err := r.hubClient.Status().Update(ctx, stagedUpdateRun); err != nil { + return errors.NewAPIServerError(err, "failed to update the staged update run status", false) + } + return nil +} + +func (r *Reconciler) markStagedUpdateRunAsSuspended( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, +) error { + stagedUpdateRunStatus := stagedUpdateRun.GetStatus() + suspendedCond := meta.FindStatusCondition(stagedUpdateRunStatus.Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeSuspended) + if condition.IsConditionStatusTrue(suspendedCond, stagedUpdateRun.GetGeneration()) { + // The staged update run is already marked as suspended; no further action is needed. + return nil + } + + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeSuspended, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunSuspendedCondReasonSuspended, + Message: "The staged update run has been suspended", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + + if err := r.hubClient.Status().Update(ctx, stagedUpdateRun); err != nil { + return errors.NewAPIServerError(err, "failed to update the staged update run status", false) + } + return nil +} + +func (r *Reconciler) markStagedUpdateRunAsResumedIfApplicable( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, +) error { + stagedUpdateRunStatus := stagedUpdateRun.GetStatus() + suspendedCond := meta.FindStatusCondition(stagedUpdateRunStatus.Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeSuspended) + if suspendedCond == nil { + // No need to add this condition if the staged update run has never been suspended. + return nil + } + if condition.IsConditionStatusFalse(suspendedCond, stagedUpdateRun.GetGeneration()) && + suspendedCond.Reason == rolloutv1alpha1.StagedUpdateRunSuspendedCondReasonResumed { + // The staged update run is already marked as resumed; no further action is needed. + return nil + } + + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeSuspended, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunSuspendedCondReasonResumed, + Message: "The staged update run has been resumed", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + + if err := r.hubClient.Status().Update(ctx, stagedUpdateRun); err != nil { + return errors.NewAPIServerError(err, "failed to update the staged update run status", false) + } + return nil +} + +// resyncStagedUpdateRunCondIfNeeded refreshes the observed generation of the given condition (e.g., Initialized or +// Completed) if it has fallen behind the generation of the staged update run (e.g., after the run is suspended or resumed). +func (r *Reconciler) resyncStagedUpdateRunCondIfNeeded( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + condType string, +) error { + stagedUpdateRunStatus := stagedUpdateRun.GetStatus() + cond := meta.FindStatusCondition(stagedUpdateRunStatus.Conditions, condType) + if cond == nil || cond.ObservedGeneration == stagedUpdateRun.GetGeneration() { + // The condition has not been set yet, or it is up to date. + return nil + } + + // Re-set the condition with only the observed generation changed; as the status stays the same, the last + // transition time is preserved. + updatedCond := *cond + updatedCond.ObservedGeneration = stagedUpdateRun.GetGeneration() + meta.SetStatusCondition(&stagedUpdateRunStatus.Conditions, updatedCond) + + if err := r.hubClient.Status().Update(ctx, stagedUpdateRun); err != nil { + return errors.NewAPIServerError(err, "failed to resync a condition of the staged update run", false, + "conditionType", condType) + } + return nil +} + +func resyncPerStageCondWithNoCommit( + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, + condType string, +) { + cond := meta.FindStatusCondition(stage.Conditions, condType) + if cond == nil || cond.ObservedGeneration == stagedUpdateRun.GetGeneration() { + return + } + updatedCond := *cond + updatedCond.ObservedGeneration = stagedUpdateRun.GetGeneration() + meta.SetStatusCondition(&stage.Conditions, updatedCond) +} + +func markAllOngoingUpdatesInStageAsCanceled( + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, +) { + for i := range stage.Clusters { + startedCond := meta.FindStatusCondition(stage.Clusters[i].Conditions, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted) + completedCond := meta.FindStatusCondition(stage.Clusters[i].Conditions, rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted) + if startedCond != nil && completedCond == nil { + meta.SetStatusCondition(&stage.Clusters[i].Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonCanceled, + Message: "The update on the cluster has been canceled due to stage-level failures", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } + } +} + +func resyncPerClusterCondWithNoCommit( + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + cluster *rolloutv1alpha1.PerClusterStatus, + condType string, +) { + cond := meta.FindStatusCondition(cluster.Conditions, condType) + if cond == nil || cond.ObservedGeneration == stagedUpdateRun.GetGeneration() { + return + } + updatedCond := *cond + updatedCond.ObservedGeneration = stagedUpdateRun.GetGeneration() + meta.SetStatusCondition(&cluster.Conditions, updatedCond) +} + +func markClusterUpdateAsFailedDueToTimeout( + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + cluster *rolloutv1alpha1.PerClusterStatus, +) { + meta.SetStatusCondition(&cluster.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed, + Message: "The update on the cluster has failed due to timeout", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) +} + +func markClusterUpdateAsFailedDueToConflict( + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + cluster *rolloutv1alpha1.PerClusterStatus, +) { + meta.SetStatusCondition(&cluster.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed, + Message: "The update on the cluster has failed due to resource snapshot name conflict", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) +} diff --git a/pkg/v1/controllers/stagedupdaterun/suite_test.go b/pkg/v1/controllers/stagedupdaterun/suite_test.go new file mode 100644 index 000000000..841474aa1 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/suite_test.go @@ -0,0 +1,244 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + "flag" + "log" + "os" + "path/filepath" + "sync" + "testing" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + "sigs.k8s.io/controller-runtime/pkg/manager" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/parallelizer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + // The namespace in which namespace-scoped objects (e.g., PlacementPolicy, StagedUpdateStrategy, + // and StagedUpdateRun objects) are created. + appNamespaceName = "app" + // Another namespace, for verifying that namespace-scoped objects are not looked up across namespaces. + // + // Note that the namespaces are created once per (Ginkgo) process, as the envtest environment does not + // support namespace deletion. + otherAppNamespaceName = "other-app" + + // The number of max. concurrent reconciliations for the staged update run controller. + maxConcurrentReconciles = 4 + // The number of workers in the parallelizer that the staged update run controller uses (e.g., to retrieve the + // member clusters referenced by placement bindings in parallel). + workerCount = 4 + + // The number of mutex slots used by the placement resource snapshot manager. + placementResourceSnapshotMuSlotCnt = 256 + + // The base and max. delays of the rate limiters (for binding manager claims and binding availability + // checks) used by the staged update run controller. + // + // The defaults (5 seconds and 180 seconds respectively) are too long for the integration tests; + // shorter delays are used here to keep the tests fast. + rateLimiterBaseDelay = time.Millisecond * 250 + rateLimiterMaxDelay = time.Second * 2 + + // The refresh period (i.e., the processing time budget per reconciliation) of the staged update run controller. + // + // The controller only persists the status of a staged update run at the end of a reconciliation; with the + // minimum refresh period New() accepts (30 seconds), an in-progress rollout would not report its progress for + // that long. The suite overrides the refresh period with a shorter value, so that the tests can observe the + // progress in a timely manner; note that this deviates from what production deployments may use. + refreshPeriodForTesting = time.Second * 5 +) + +var ( + hubCfg *rest.Config + hubEnv *envtest.Environment + hubMgr manager.Manager + // hubClient is the client from the controller manager; reads are served from its cache, which + // matches how the staged update run controller reads objects. + hubClient client.Client + + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup +) + +const ( + // The environment variable that the CI pipeline sets to tell which runner invokes the tests. + testRunnerNameEnvVarName = "KUBEFLEET_CI_TEST_RUNNER_NAME" + // The runner (`go test`, via `make test`) with which the tests are skipped in the CI pipeline; the pipeline runs + // this package in a separate step with the Ginkgo CLI (in parallel) instead. + runnerNameToSkipTestsInCI = "default" +) + +func TestMain(m *testing.M) { + // Skip the tests if in the CI environment the tests are invoked with `go test` instead of the Ginkgo CLI. + // This has no effect outside the CI environment. + if v := os.Getenv(testRunnerNameEnvVarName); v == runnerNameToSkipTestsInCI { + log.Println("Skipping the tests in CI as they are not run with the expected runner") + os.Exit(0) + } + + os.Exit(m.Run()) +} + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Staged Update Run Controller Integration Test Suite") +} + +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + logger := zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)) + klog.SetLogger(logger) + ctrl.SetLogger(logger) + + By("Bootstrapping the test environment") + hubEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("../../../../", "config", "crd", "bases")}, + ErrorIfCRDPathMissing: true, + } + + var err error + hubCfg, err = hubEnv.Start() + Expect(err).Should(Succeed()) + Expect(hubCfg).NotTo(BeNil()) + + By("Setting up the scheme") + Expect(clusterv1beta1.AddToScheme(scheme.Scheme)).Should(Succeed()) + Expect(placementv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + Expect(rolloutv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + By("Setting up the controller manager") + hubMgr, err = ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{ + BindAddress: "0", + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).Should(Succeed()) + + // Use the same (cache-backed) client as the controller, so that test cases read objects the same way the + // controller does; a test case that has waited for the cache to observe a change can be sure that the controller + // observes it too. + By("Setting the hub client to the one used by the controller manager") + hubClient = hubMgr.GetClient() + + // The staged update run controller (and the placement resource snapshot manager) list objects via + // custom field indexes; the indexes must be set up before the manager starts. + By("Setting up the field indexes") + Expect(fieldindexers.SetupWithHubAgentControllerManager(ctx, hubMgr)).Should(Succeed()) + + By("Setting up the placement resource snapshot manager") + hubDynamicClient, err := dynamic.NewForConfig(hubCfg) + Expect(err).Should(Succeed()) + // No resource change detector runs in the test environment, so no informers are registered with the + // dynamic informer manager; the placement resource snapshot manager reads selected resources directly + // from the API server in this case. + hubDynamicInformerManager := informer.NewInformerManager(hubDynamicClient, 0, ctx.Done()) + placementResourceSnapshotManager, err := placementresourcesnapshot.New( + hubMgr, + hubDynamicClient, + hubDynamicInformerManager, + resourceeligibility.Baseline(hubMgr.GetRESTMapper()), + hubMgr.GetRESTMapper(), + placementResourceSnapshotMuSlotCnt, + placementresourcesnapshot.DefaultMaxPerSnapshotResourceDataSizeBytes, + placementresourcesnapshot.DefaultMaxPerSnapshotResourceCnt, + ) + Expect(err).Should(Succeed()) + // Add the placement resource snapshot manager to the controller manager so that its snapshot GC runs. + Expect(hubMgr.Add(placementResourceSnapshotManager)).Should(Succeed()) + + By("Setting up the staged update run controller") + reconciler, err := New( + hubMgr.GetClient(), + hubMgr.GetAPIReader(), + placementResourceSnapshotManager, + parallelizer.NewParallelizer(workerCount), + workqueue.NewTypedItemExponentialFailureRateLimiter[ctrl.Request](rateLimiterBaseDelay, rateLimiterMaxDelay), + workqueue.NewTypedItemExponentialFailureRateLimiter[types.NamespacedName](rateLimiterBaseDelay, rateLimiterMaxDelay), + minRefreshPeriod, + ) + Expect(err).Should(Succeed()) + // New() does not accept refresh periods shorter than the minimum; set the refresh period directly. + reconciler.refreshPeriod = refreshPeriodForTesting + Expect(reconciler.SetupWithManager(hubMgr, maxConcurrentReconciles)).Should(Succeed()) + + By("Starting the controller manager") + wg = sync.WaitGroup{} + wg.Add(1) + go func() { + defer GinkgoRecover() + defer wg.Done() + Expect(hubMgr.Start(ctx)).Should(Succeed(), "Failed to run the controller manager") + }() + Expect(hubMgr.GetCache().WaitForCacheSync(ctx)).Should(BeTrue(), "Failed to sync the controller manager cache") + + By("Creating the application namespaces") + for _, nsName := range []string{appNamespaceName, otherAppNamespaceName} { + Expect(hubClient.Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: nsName}, + })).Should(Succeed(), "Failed to create the application namespace %s", nsName) + } +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + + cancel() + // Wait for the controller manager to stop first; otherwise the API server might not shut down in time + // as the manager's watches are still open. + wg.Wait() + + By("Tearing down the test environment") + Expect(hubEnv.Stop()).Should(Succeed()) +}) diff --git a/pkg/v1/controllers/stagedupdaterun/tasks.go b/pkg/v1/controllers/stagedupdaterun/tasks.go new file mode 100644 index 000000000..f9d7145ad --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/tasks.go @@ -0,0 +1,342 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "context" + "fmt" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/ownerreferences" +) + +// executeBeforeStageTasks executes all the before stage tasks of a stage. It reports whether all the tasks have +// been cleared; if not, the returned requeue period is the longest one requested by the tasks. +func (r *Reconciler) executeBeforeStageTasks( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, +) (allCleared bool, requeueAfter *time.Duration, err error) { + allCleared = true + longestRequeueAfter := time.Duration(0) + for idx := range stage.BeforeStageTasks { + task := &stage.BeforeStageTasks[idx] + shouldContinue, taskRequeueAfter, err := r.executeOneStageTask(ctx, stagedUpdateRun, stage, task, true) + if err != nil { + return false, taskRequeueAfter, kferrors.Wraps(err, "failed to complete before stage task", "task", task.Type) + } + if !shouldContinue { + allCleared = false + if taskRequeueAfter != nil && *taskRequeueAfter > longestRequeueAfter { + longestRequeueAfter = *taskRequeueAfter + } + } + } + return allCleared, &longestRequeueAfter, nil +} + +// executeAfterStageTasks executes all the after stage tasks of a stage. It reports whether all the tasks have +// been cleared; if not, the returned requeue period is the longest one requested by the tasks. +func (r *Reconciler) executeAfterStageTasks( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, +) (allCleared bool, requeueAfter *time.Duration, err error) { + allCleared = true + longestRequeueAfter := time.Duration(0) + for idx := range stage.AfterStageTasks { + task := &stage.AfterStageTasks[idx] + shouldContinue, taskRequeueAfter, err := r.executeOneStageTask(ctx, stagedUpdateRun, stage, task, false) + if err != nil { + return false, taskRequeueAfter, kferrors.Wraps(err, "failed to complete after stage task", "task", task.Type) + } + if !shouldContinue { + allCleared = false + if taskRequeueAfter != nil && *taskRequeueAfter > longestRequeueAfter { + longestRequeueAfter = *taskRequeueAfter + } + } + } + return allCleared, &longestRequeueAfter, nil +} + +func (r *Reconciler) executeOneStageTask( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, + task *rolloutv1alpha1.PerStageTaskStatus, + isBeforeStageTask bool, +) (shouldContinue bool, requeueAfter *time.Duration, err error) { + switch task.Type { + case rolloutv1alpha1.StageTaskTypeTimedWait: + return r.executeTimedWaitStageTask(stagedUpdateRun, task) + case rolloutv1alpha1.StageTaskTypeApproval: + return r.executeApprovalStageTask(ctx, stagedUpdateRun, stage, task, isBeforeStageTask) + default: + // Normally this will never happen. + return false, nil, kferrors.NewUnexpectedError(nil, "an unknown stage task type was found", "taskType", task.Type) + } +} + +func (r *Reconciler) executeApprovalStageTask( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, + task *rolloutv1alpha1.PerStageTaskStatus, + isBeforeStageTask bool, +) (shouldContinue bool, requeueAfter *time.Duration, err error) { + // Skip the task if it has already been completed (the approval has been received). + approvalAcceptedCond := meta.FindStatusCondition(task.Conditions, rolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestApproved) + if approvalAcceptedCond != nil && approvalAcceptedCond.Status == metav1.ConditionTrue { + return true, nil, nil + } + + // Check if an approval request has been created. If not, create one, and add the ApprovalRequestCreated condition. + requestCreatedCond := meta.FindStatusCondition(task.Conditions, rolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated) + if requestCreatedCond == nil { + // Create an approval request. + if err := r.createApprovalRequest(ctx, stagedUpdateRun, stage, task, isBeforeStageTask); err != nil { + return false, nil, err + } + // No need to wait for the approval to be granted; the reconciliation loop will be triggered again + // when the approval request is approved. + return false, nil, nil + } + + // The approval request has been created. Check if it has been approved. If not, wait for the next reconciliation attempt. + approvalRequestName := task.ApprovalRequestName + if approvalRequestName == "" { + // Do a sanity check; normally this branch will never run. + return false, nil, kferrors.NewUnexpectedError(nil, "approval request name is empty in the task status") + } + + var approvalRequest rolloutv1alpha1.ApprovalRequestAccessor + var approvalRequestNamespacedName types.NamespacedName + if stagedUpdateRun.GetNamespace() == "" { + approvalRequest = &rolloutv1alpha1.ClusterApprovalRequest{} + approvalRequestNamespacedName = types.NamespacedName{Name: approvalRequestName} + } else { + approvalRequest = &rolloutv1alpha1.ApprovalRequest{} + approvalRequestNamespacedName = types.NamespacedName{Namespace: stagedUpdateRun.GetNamespace(), Name: approvalRequestName} + } + + if err := r.hubClient.Get(ctx, approvalRequestNamespacedName, approvalRequest); err != nil { + if apierrors.IsNotFound(err) { + // The approval request cannot be found. It might have been deleted by the user; re-create it. + if err := r.createApprovalRequest(ctx, stagedUpdateRun, stage, task, isBeforeStageTask); err != nil { + return false, nil, err + } + // No need to wait for the approval to be granted; the reconciliation loop will be triggered again + // when the approval request is approved. + return false, nil, nil + } + return false, nil, kferrors.NewAPIServerError(err, "failed to get approval request", true, + "approvalRequest", approvalRequestName) + } + + // Check if the approval request is owned by the staged update run. If not, delete it and return an error. + // + // This is primarily added as a safeguard against situations where staged update runs are re-created yet + // the GC process hasn't caught up and old approval requests from same-named staged update runs might still exist. + // + // Note that approval requests are typically considered to be objects always created by the staged update run + // controller; users are not expected to create or manage them directly. + if !metav1.IsControlledBy(approvalRequest, stagedUpdateRun) { + isOwnedBySameNamed, err := ownerreferences.IsOwnedByWithNoUIDCheck(r.hubClient, approvalRequest, stagedUpdateRun) + if err != nil { + return false, nil, kferrors.Wraps(err, "failed to check ownership of approval request", + "approvalRequest", klog.KObj(approvalRequest)) + } + if isOwnedBySameNamed { + // The approval request is owned by a same-named staged update run but not the current one; it should be + // deleted so that it can be re-created. + klog.V(2).InfoS("Approval request is not owned by the staged update run; deleting it", + "approvalRequest", klog.KObj(approvalRequest), "stage", stage.StageName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + if err := r.hubClient.Delete(ctx, approvalRequest); err != nil && !apierrors.IsNotFound(err) { + return false, nil, kferrors.NewAPIServerError(err, "failed to delete approval request not owned by the staged update run", false, + "approvalRequest", klog.KObj(approvalRequest)) + } + // The approval request will be re-created in the next reconciliation attempt. + return false, nil, kferrors.NewTransientError(nil, "the approval request is not owned by the staged update run; deleted it", + "approvalRequest", klog.KObj(approvalRequest)) + } + // The approval request is owned by a differently named staged update run or no staged update run at all. + klog.V(2).InfoS("Approval request is owned by a differently named staged update run or no staged update run at all", + "approvalRequest", klog.KObj(approvalRequest), "stage", stage.StageName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + + // Flip the ApprovalRequestCreated condition to False so that users are informed of the ownership conflict. + meta.SetStatusCondition(&task.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunTaskApprovalRequestCreatedCondReasonOwnershipMismatch, + Message: fmt.Sprintf("An approval request %q is found but it is not owned by the staged update run; "+ + "if the request is manually created, remove it to proceed", approvalRequestName), + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + return false, nil, kferrors.NewUserError(nil, "an approval request is found but its ownership does not align; if the request is manually created, remove it to proceed", + "approvalRequest", klog.KObj(approvalRequest)) + } + + // Check if the approval request has been approved. + requestApprovedCond := meta.FindStatusCondition(approvalRequest.GetStatus().Conditions, rolloutv1alpha1.ApprovalRequestCondTypeApproved) + if requestApprovedCond == nil || requestApprovedCond.Status != metav1.ConditionTrue { + // The approval request has not been approved yet. + return false, nil, nil + } + + // Add an approval accepted condition to the approval request. + if !meta.IsStatusConditionTrue(approvalRequest.GetStatus().Conditions, rolloutv1alpha1.ApprovalRequestCondTypeApprovalAccepted) { + meta.SetStatusCondition(&approvalRequest.GetStatus().Conditions, metav1.Condition{ + Type: rolloutv1alpha1.ApprovalRequestCondTypeApprovalAccepted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.ApprovalRequestApprovalAcceptedCondReasonAccepted, + Message: "The approval has been accepted by the staged update run", + ObservedGeneration: approvalRequest.GetGeneration(), + }) + // Use an update (rather than a patch) so that the write fails with a conflict if the approval request has + // changed (e.g., the approval has been revoked) since it was read. + if err := r.hubClient.Status().Update(ctx, approvalRequest); err != nil { + return false, nil, kferrors.NewAPIServerError(err, "failed to add the approval accepted condition to the approval request", false, + "approvalRequest", klog.KObj(approvalRequest)) + } + klog.V(2).InfoS("Accepted the approval of the approval request", + "approvalRequest", klog.KObj(approvalRequest), "stage", stage.StageName, "stagedUpdateRun", klog.KObj(stagedUpdateRun)) + } + + // Mark the approval task as completed in the staged update run status. + meta.SetStatusCondition(&task.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestApproved, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunTaskApprovalRequestApprovedCondReasonApproved, + Message: "The approval request has been approved", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + + // The approval request has been approved. + return true, nil, nil +} + +// createApprovalRequest creates an approval request for the given stage of the staged update run. The +// request is owned by the staged update run; it is not an error if the request already exists. +func (r *Reconciler) createApprovalRequest( + ctx context.Context, + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + stage *rolloutv1alpha1.PerStageStatus, + task *rolloutv1alpha1.PerStageTaskStatus, + isBeforeStageTask bool, +) error { + var approvalRequestName string + if isBeforeStageTask { + approvalRequestName = formatBeforeStageApprovalRequestName(stagedUpdateRun.GetName(), stage.StageName) + } else { + approvalRequestName = formatAfterStageApprovalRequestName(stagedUpdateRun.GetName(), stage.StageName) + } + // Set the approval request name in the task status. + task.ApprovalRequestName = approvalRequestName + + var approvalRequest rolloutv1alpha1.ApprovalRequestAccessor + var ownerGVK schema.GroupVersionKind + if stagedUpdateRun.GetNamespace() == "" { + ownerGVK = rolloutv1alpha1.GroupVersion.WithKind(rolloutv1alpha1.ClusterStagedUpdateRunKind) + approvalRequest = &rolloutv1alpha1.ClusterApprovalRequest{} + } else { + ownerGVK = rolloutv1alpha1.GroupVersion.WithKind(rolloutv1alpha1.StagedUpdateRunKind) + approvalRequest = &rolloutv1alpha1.ApprovalRequest{} + } + + approvalRequest.SetName(approvalRequestName) + approvalRequest.SetNamespace(stagedUpdateRun.GetNamespace()) + approvalRequest.SetOwnerReferences([]metav1.OwnerReference{ + *metav1.NewControllerRef(stagedUpdateRun, ownerGVK), + }) + approvalRequest.SetSpec(rolloutv1alpha1.ApprovalRequestSpec{ + StagedUpdateRunName: stagedUpdateRun.GetName(), + StageName: stage.StageName, + }) + + err := r.hubClient.Create(ctx, approvalRequest) + if err == nil || apierrors.IsAlreadyExists(err) { + // Set the request created condition. + meta.SetStatusCondition(&task.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunTaskApprovalRequestCreatedCondReasonCreated, + Message: "The approval request has been created", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + } + // The controller must return the AlreadyExists error instead of ignoring it, as the approval request found right now + // might have already been processed. + if err != nil { + return kferrors.NewAPIServerError(err, "failed to create approval request", false, "approvalRequest", klog.KObj(approvalRequest)) + } + return nil +} + +func (r *Reconciler) executeTimedWaitStageTask( + stagedUpdateRun rolloutv1alpha1.StagedUpdateRunAccessor, + task *rolloutv1alpha1.PerStageTaskStatus, +) (shouldContinue bool, requeueAfter *time.Duration, err error) { + // Skip the task if it has already been completed (the wait time has elapsed). + waitTimeElapsedCond := meta.FindStatusCondition(task.Conditions, rolloutv1alpha1.StagedUpdateRunTaskCondTypeWaitTimeElapsed) + if waitTimeElapsedCond != nil && waitTimeElapsedCond.Status == metav1.ConditionTrue { + return true, nil, nil + } + + // Execute a TimedWait task. + timedWaitStartedCond := meta.FindStatusCondition(task.Conditions, rolloutv1alpha1.StagedUpdateRunTaskCondTypeTimedWaitStarted) + if timedWaitStartedCond == nil { + // The task is being run for the first time. Add the TimedWaitStarted condition to track the start time. + timedWaitStartedCond = &metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeTimedWaitStarted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunTaskTimedWaitStartedCondReasonTimerStarted, + Message: "The TimedWait task has started", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + } + meta.SetStatusCondition(&task.Conditions, *timedWaitStartedCond) + // Read the TimedWaitStarted condition again to ensure that the last transition timestamp value is populated. + timedWaitStartedCond = meta.FindStatusCondition(task.Conditions, rolloutv1alpha1.StagedUpdateRunTaskCondTypeTimedWaitStarted) + } + + // Calculate how long the task has been waiting based on the last transition time of the TimedWaitStarted condition. + elapsed := time.Since(timedWaitStartedCond.LastTransitionTime.Time) + remaining := task.WaitTime.Duration - elapsed + if remaining > 0 { + // The wait time has not yet elapsed. Return the remaining wait time. + return false, &remaining, nil + } + + // The wait time has elapsed. Mark the task as completed. + meta.SetStatusCondition(&task.Conditions, metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeWaitTimeElapsed, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunTaskWaitTimeElapsedCondReasonTimerElapsed, + Message: "The TimedWait task is completed (the wait time has elapsed)", + ObservedGeneration: stagedUpdateRun.GetGeneration(), + }) + return true, nil, nil +} diff --git a/pkg/v1/controllers/stagedupdaterun/tasks_integration_test.go b/pkg/v1/controllers/stagedupdaterun/tasks_integration_test.go new file mode 100644 index 000000000..4bfb99272 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/tasks_integration_test.go @@ -0,0 +1,2508 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "fmt" + "slices" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +// The name templates of the approval requests that the controller creates for before-stage and after-stage approval +// tasks; the placeholders are the name of the staged update run and the name of the stage. +const ( + beforeStageApprovalRequestNameTemplate = "%s-before-stage-%s" + afterStageApprovalRequestNameTemplate = "%s-after-stage-%s" +) + +const ( + afterStageWaitTime = time.Second * 5 + longAfterStageWaitTime = time.Second * 10 +) + +// wantSingleClusterCompletedStatus builds the status of a completed staged update run whose only (canary) stage has +// member cluster 0 only, along with the given before-stage tasks. +func wantSingleClusterCompletedStatus( + snapshotName string, + memberClusterNames, bindingNames []string, + beforeStageTasks []rolloutv1alpha1.PerStageTaskStatus, +) *rolloutv1alpha1.StagedUpdateRunStatus { + return &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + BeforeStageTasks: beforeStageTasks, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } +} + +var _ = Describe("processing stage tasks of staged update runs", func() { + Context("before-stage approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + // The canary stage, which has a before-stage approval task, updates both member clusters at the same time. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the approval request for the canary stage", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + }) + + It("should report that the canary stage has started and is waiting for approval", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{perStageTaskApprovalRequestCreatedCond()}, + }, + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(2)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("should not roll out to any member cluster before the approval", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames, previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to member clusters unexpectedly") + }) + + It("can approve the approval request", func() { + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member clusters", func() { + // The controller does not requeue a staged update run that is waiting for approval; it is the watch on + // approval requests that triggers the processing of the run once the request is approved. + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(appNamespaceName, bindingNames, snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member clusters") + }) + + It("can mark the member clusters as available", func() { + for _, bindingName := range bindingNames { + markPlacementBindingAsAvailable(appNamespaceName, bindingName, snapshotName) + } + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{ + perStageTaskApprovalRequestCreatedCond(), + perStageTaskApprovalRequestApprovedCond(), + }, + }, + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(2)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; the approval request, which the staged + // update run owns, must be removed explicitly. + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("after-stage approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(afterStageApprovalRequestNameTemplate, runName, canaryStageName) + + // Member clusters 0 and 1 are canary clusters, which the canary stage (with an after-stage approval task) + // updates one at a time; member cluster 2 is a production cluster. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 3) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("should not create the approval request while member cluster 0 is being updated", func() { + approvalRequestAbsentActual := approvalRequestAbsentActual(appNamespaceName, approvalRequestName) + Consistently(approvalRequestAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Created the approval request unexpectedly") + }) + + It("can mark member cluster 0 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should start rolling out to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("should not create the approval request while member cluster 1 is being updated", func() { + approvalRequestAbsentActual := approvalRequestAbsentActual(appNamespaceName, approvalRequestName) + Consistently(approvalRequestAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Created the approval request unexpectedly") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should create the approval request once all the member clusters in the canary stage have been updated", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + }) + + It("should report that the canary stage is waiting for approval", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + + // The canary stage does not complete, and the prod stage does not start, before the approval. + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[2], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the prod stage unexpectedly") + Expect(rolloutStatusActual()).To(Succeed(), "The status of the staged update run has changed unexpectedly") + }) + + It("can approve the approval request", func() { + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the prod stage", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the prod stage") + }) + + It("can mark member cluster 2 as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[2], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(appNamespaceName, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("rejected approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the staged update run while it waits for approval. + wantWaitingForApprovalStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantWaitingForApprovalStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the approval request", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + }) + + It("should report that the canary stage is waiting for approval", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantWaitingForApprovalStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the staged update run") + }) + + It("can reject the approval request", func() { + rejectApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should not accept the rejection or roll out to the member cluster", func() { + Consistently(func() error { + if err := approvalRequestConditionsActual(appNamespaceName, approvalRequestName, []metav1.Condition{ + approvalRequestApprovedCond(metav1.ConditionFalse), + })(); err != nil { + return err + } + return placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName)() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Accepted the rejection or rolled out to the member cluster unexpectedly") + }) + + It("should keep the canary stage waiting for approval", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantWaitingForApprovalStatus) + Expect(rolloutStatusActual()).To(Succeed(), "The status of the staged update run has changed unexpectedly") + }) + + It("can approve the approval request", func() { + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("after-stage timed wait", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 0 is a canary cluster, and member cluster 1 is a production cluster; the canary stage has an + // after-stage timed wait task. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: afterStageWaitTime}, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the canary cluster") + }) + + It("can mark the canary cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should start the after-stage timed wait task of the canary stage", func() { + // The controller persists the status right after it starts the timed wait task, so the status is + // observable for the whole wait time. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantTimedWaitTaskStatus(afterStageWaitTime, perStageTaskTimedWaitStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start the timed wait task") + }) + + It("should start rolling out to the production cluster once the wait time has elapsed", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to roll out to the production cluster") + }) + + It("can mark the production cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantTimedWaitTaskStatus(afterStageWaitTime, perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + + // The wait time elapses at least the wait time after the task starts, and the canary stage completes (and + // the prod stage starts) only afterwards. + timedWaitLastedActual := perStageTaskTimedWaitLastedAtLeastActual(appNamespaceName, runName, canaryStageName, afterStageWaitTime) + Expect(timedWaitLastedActual()).To(Succeed(), "Failed to wait for the wait time") + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(appNamespaceName, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("after-stage approval and timed wait", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + canaryApprovalRequestName := fmt.Sprintf(afterStageApprovalRequestNameTemplate, runName, canaryStageName) + prodApprovalRequestName := fmt.Sprintf(afterStageApprovalRequestNameTemplate, runName, prodStageName) + + // Member cluster 0 is a canary cluster, and member cluster 1 is a production cluster. Both stages have an + // after-stage approval task and an after-stage timed wait task: in the canary stage, the approval comes + // before the wait time elapses; in the prod stage, the wait time elapses before the approval. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + // The canary stage, as it is expected to be once completed. + wantCompletedCanaryStage := rolloutv1alpha1.PerStageStatus{ + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(canaryApprovalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + wantTimedWaitTaskStatus(longAfterStageWaitTime, perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: longAfterStageWaitTime}, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: afterStageWaitTime}, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the canary cluster") + }) + + It("can mark the canary cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should create the approval request for the canary stage", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, canaryApprovalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + }) + + It("can approve the approval request for the canary stage before the wait time elapses", func() { + approveApprovalRequest(appNamespaceName, canaryApprovalRequestName) + }) + + It("should accept the approval, but keep waiting for the wait time to elapse", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, canaryApprovalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(canaryApprovalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + wantTimedWaitTaskStatus(longAfterStageWaitTime, perStageTaskTimedWaitStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1), + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(""), + wantTimedWaitTaskStatus(afterStageWaitTime), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the production cluster once the wait time has elapsed", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to roll out to the production cluster") + + // The completion of the canary stage is persisted once the processing time budget runs out (as the + // controller waits for the production cluster to become available in the meantime). + timedWaitLastedActual := perStageTaskTimedWaitLastedAtLeastActual(appNamespaceName, runName, canaryStageName, longAfterStageWaitTime) + Eventually(timedWaitLastedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to wait for the wait time of the canary stage") + }) + + It("can mark the production cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[1], snapshotName) + }) + + It("should create the approval request for the prod stage", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, prodApprovalRequestName, prodStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + }) + + It("should keep waiting for approval after the wait time of the prod stage elapses", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(prodApprovalRequestName, perStageTaskApprovalRequestCreatedCond()), + wantTimedWaitTaskStatus(afterStageWaitTime, perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to elapse the wait time of the prod stage") + Consistently(rolloutStatusActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Failed to keep waiting for approval") + }) + + It("can approve the approval request for the prod stage", func() { + approveApprovalRequest(appNamespaceName, prodApprovalRequestName) + }) + + It("should accept the approval for the prod stage", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, prodApprovalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should complete the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(prodApprovalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + wantTimedWaitTaskStatus(afterStageWaitTime, perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(appNamespaceName, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, canaryApprovalRequestName), + newApprovalRequestObj(appNamespaceName, prodApprovalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("deleted approval request", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + var oldApprovalRequestUID types.UID + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the approval request", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + + request, err := getApprovalRequest(appNamespaceName, approvalRequestName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the approval request") + oldApprovalRequestUID = request.GetUID() + }) + + It("can delete the approval request", func() { + // The controller re-creates the approval request right away; do not wait for the name to be gone. + Expect(hubClient.Delete(ctx, newApprovalRequestObj(appNamespaceName, approvalRequestName))).To(Succeed(), "Failed to delete the approval request") + }) + + It("should re-create the approval request", func() { + approvalRequestReplacedActual := approvalRequestReplacedActual(appNamespaceName, approvalRequestName, oldApprovalRequestUID) + Eventually(approvalRequestReplacedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the approval request") + + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Expect(approvalRequestCreatedActual()).To(Succeed(), "Failed to re-create the approval request as expected") + }) + + It("should not roll out to the member cluster before the approval", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the member cluster unexpectedly") + }) + + It("can approve the re-created approval request", func() { + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("stale approval request from a deleted staged update run of the same name", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + var oldApprovalRequestUID types.UID + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the approval request", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + + request, err := getApprovalRequest(appNamespaceName, approvalRequestName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the approval request") + oldApprovalRequestUID = request.GetUID() + }) + + It("can delete the staged update run", func() { + removeObjects(newStagedUpdateRunObj(appNamespaceName, runName)) + + // The envtest environment does not run the garbage collector; the approval request stays around. + _, err := getApprovalRequest(appNamespaceName, approvalRequestName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the approval request of the deleted staged update run") + }) + + It("can approve the approval request of the deleted staged update run", func() { + // The approval of the stale approval request should not carry over to the re-created staged update run. + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("can re-create the staged update run", func() { + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should replace the stale approval request with a new one owned by the re-created staged update run", func() { + approvalRequestReplacedActual := approvalRequestReplacedActual(appNamespaceName, approvalRequestName, oldApprovalRequestUID) + Eventually(approvalRequestReplacedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to replace the stale approval request") + + // The check below verifies that the new approval request is owned by the re-created staged update run + // (with the UID of the re-created run). + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the new approval request as expected") + + approvalRequestNotApprovedActual := approvalRequestConditionsActual(appNamespaceName, approvalRequestName, nil) + Expect(approvalRequestNotApprovedActual()).To(Succeed(), "The new approval request has been approved unexpectedly") + }) + + It("should not roll out to the member cluster before the new approval request is approved", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the member cluster unexpectedly") + }) + + It("can approve the new approval request", func() { + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("approval request not owned by the staged update run", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(appNamespaceName, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(appNamespaceName, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(appNamespaceName, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + + // Create (and approve) an approval request of the expected name manually, before the staged update run is + // created. Note that the request references the staged update run in its spec, so that its deletion + // triggers the processing of the run. + createManualApprovalRequest(appNamespaceName, approvalRequestName, runName, canaryStageName) + approveApprovalRequest(appNamespaceName, approvalRequestName) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(appNamespaceName, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the approval request is not owned by the staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestOwnershipMismatchCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the ownership mismatch") + }) + + It("should not roll out to the member cluster, even though the approval request has been approved", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the member cluster unexpectedly") + }) + + It("can delete the manually created approval request", func() { + // The controller re-creates the approval request right away; do not wait for the name to be gone. + Expect(hubClient.Delete(ctx, newApprovalRequestObj(appNamespaceName, approvalRequestName))).To(Succeed(), "Failed to delete the approval request") + }) + + It("should create its own approval request", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(appNamespaceName, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request") + }) + + It("should report that the approval request has been created", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the creation of the approval request") + }) + + It("can approve the approval request", func() { + approveApprovalRequest(appNamespaceName, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(appNamespaceName, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(appNamespaceName, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(appNamespaceName, bindingNames[0], snapshotName) + }) + + It("should complete the staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(appNamespaceName, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(appNamespaceName, runName), + newApprovalRequestObj(appNamespaceName, approvalRequestName), + newPlacementResourceSnapshotObj(appNamespaceName, snapshotName), + newStagedUpdateStrategyObj(appNamespaceName, strategyName), + newPlacementPolicyObj(appNamespaceName, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(appNamespaceName, memberClusterNames, bindingNames), + )...) + }) + }) +}) + +var _ = Describe("processing stage tasks of cluster staged update runs", func() { + Context("before-stage approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + // The canary stage, which has a before-stage approval task, updates both member clusters at the same time. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil, nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the cluster approval request for the canary stage", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + }) + + It("should report that the canary stage has started and is waiting for approval", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0, 1), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{perStageTaskApprovalRequestCreatedCond()}, + }, + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(2)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("should not roll out to any member cluster before the approval", func() { + bindingsUntouchedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames, previousResourceSnapshotName) + Consistently(bindingsUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to member clusters unexpectedly") + }) + + It("can approve the cluster approval request", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member clusters", func() { + // The controller does not requeue a cluster staged update run that is waiting for approval; it is the watch on + // cluster approval requests that triggers the processing of the run once the request is approved. + bindingsUpdatedActual := placementBindingsResourceSnapshotActual(clusterScopedNS, bindingNames, snapshotName) + Eventually(bindingsUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member clusters") + }) + + It("can mark the member clusters as available", func() { + for _, bindingName := range bindingNames { + markPlacementBindingAsAvailable(clusterScopedNS, bindingName, snapshotName) + } + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: []metav1.Condition{ + perStageTaskApprovalRequestCreatedCond(), + perStageTaskApprovalRequestApprovedCond(), + }, + }, + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(2)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; the cluster approval request, which the staged + // update run owns, must be removed explicitly. + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("after-stage approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(afterStageApprovalRequestNameTemplate, runName, canaryStageName) + + // Member clusters 0 and 1 are canary clusters, which the canary stage (with an after-stage approval task) + // updates one at a time; member cluster 2 is a production cluster. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 3) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to member cluster 0", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 0") + }) + + It("should not create the cluster approval request while member cluster 0 is being updated", func() { + approvalRequestAbsentActual := approvalRequestAbsentActual(clusterScopedNS, approvalRequestName) + Consistently(approvalRequestAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Created the cluster approval request unexpectedly") + }) + + It("can mark member cluster 0 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should start rolling out to member cluster 1", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to member cluster 1") + }) + + It("should not create the cluster approval request while member cluster 1 is being updated", func() { + approvalRequestAbsentActual := approvalRequestAbsentActual(clusterScopedNS, approvalRequestName) + Consistently(approvalRequestAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Created the cluster approval request unexpectedly") + }) + + It("can mark member cluster 1 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should create the cluster approval request once all the member clusters in the canary stage have been updated", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + }) + + It("should report that the canary stage is waiting for approval", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 2), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + + // The canary stage does not complete, and the prod stage does not start, before the approval. + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[2], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the prod stage unexpectedly") + Expect(rolloutStatusActual()).To(Succeed(), "The status of the cluster staged update run has changed unexpectedly") + }) + + It("can approve the cluster approval request", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the prod stage", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[2], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the prod stage") + }) + + It("can mark member cluster 2 as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[2], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 2, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(clusterScopedNS, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("rejected approval", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + // The status of the cluster staged update run while it waits for approval. + wantWaitingForApprovalStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + wantWaitingForApprovalStatus.ResourceSnapshotNameToRollout = snapshotName + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the cluster approval request", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + }) + + It("should report that the canary stage is waiting for approval", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantWaitingForApprovalStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the progress of the cluster staged update run") + }) + + It("can reject the cluster approval request", func() { + rejectApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should not accept the rejection or roll out to the member cluster", func() { + Consistently(func() error { + if err := approvalRequestConditionsActual(clusterScopedNS, approvalRequestName, []metav1.Condition{ + approvalRequestApprovedCond(metav1.ConditionFalse), + })(); err != nil { + return err + } + return placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName)() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Accepted the rejection or rolled out to the member cluster unexpectedly") + }) + + It("should keep the canary stage waiting for approval", func() { + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantWaitingForApprovalStatus) + Expect(rolloutStatusActual()).To(Succeed(), "The status of the cluster staged update run has changed unexpectedly") + }) + + It("can approve the cluster approval request", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("after-stage timed wait", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + + // Member cluster 0 is a canary cluster, and member cluster 1 is a production cluster; the canary stage has an + // after-stage timed wait task. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: afterStageWaitTime}, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the canary cluster") + }) + + It("can mark the canary cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should start the after-stage timed wait task of the canary stage", func() { + // The controller persists the status right after it starts the timed wait task, so the status is + // observable for the whole wait time. + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantTimedWaitTaskStatus(afterStageWaitTime, perStageTaskTimedWaitStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start the timed wait task") + }) + + It("should start rolling out to the production cluster once the wait time has elapsed", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to roll out to the production cluster") + }) + + It("can mark the production cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantTimedWaitTaskStatus(afterStageWaitTime, perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + + // The wait time elapses at least the wait time after the task starts, and the canary stage completes (and + // the prod stage starts) only afterwards. + timedWaitLastedActual := perStageTaskTimedWaitLastedAtLeastActual(clusterScopedNS, runName, canaryStageName, afterStageWaitTime) + Expect(timedWaitLastedActual()).To(Succeed(), "Failed to wait for the wait time") + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(clusterScopedNS, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("after-stage approval and timed wait", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + canaryApprovalRequestName := fmt.Sprintf(afterStageApprovalRequestNameTemplate, runName, canaryStageName) + prodApprovalRequestName := fmt.Sprintf(afterStageApprovalRequestNameTemplate, runName, prodStageName) + + // Member cluster 0 is a canary cluster, and member cluster 1 is a production cluster. Both stages have an + // after-stage approval task and an after-stage timed wait task: in the canary stage, the approval comes + // before the wait time elapses; in the prod stage, the wait time elapses before the approval. + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 2) + memberClusterLabels := []map[string]string{ + {envLabelKey: envLabelValueCanary}, + {envLabelKey: envLabelValueProd}, + } + + // The canary stage, as it is expected to be once completed. + wantCompletedCanaryStage := rolloutv1alpha1.PerStageStatus{ + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(canaryApprovalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + wantTimedWaitTaskStatus(longAfterStageWaitTime, perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + } + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, memberClusterLabels) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueCanary}, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: longAfterStageWaitTime}, + }, + }, + }, + { + Name: prodStageName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{envLabelKey: envLabelValueProd}, + }, + AfterStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: afterStageWaitTime}, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should start rolling out to the canary cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the canary cluster") + }) + + It("can mark the canary cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should create the cluster approval request for the canary stage", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, canaryApprovalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + }) + + It("can approve the cluster approval request for the canary stage before the wait time elapses", func() { + approveApprovalRequest(clusterScopedNS, canaryApprovalRequestName) + }) + + It("should accept the approval, but keep waiting for the wait time to elapse", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, canaryApprovalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 0, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(canaryApprovalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + wantTimedWaitTaskStatus(longAfterStageWaitTime, perStageTaskTimedWaitStartedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: prodStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 1), + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(""), + wantTimedWaitTaskStatus(afterStageWaitTime), + }, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the production cluster once the wait time has elapsed", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[1], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to roll out to the production cluster") + + // The completion of the canary stage is persisted once the processing time budget runs out (as the + // controller waits for the production cluster to become available in the meantime). + timedWaitLastedActual := perStageTaskTimedWaitLastedAtLeastActual(clusterScopedNS, runName, canaryStageName, longAfterStageWaitTime) + Eventually(timedWaitLastedActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to wait for the wait time of the canary stage") + }) + + It("can mark the production cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[1], snapshotName) + }) + + It("should create the cluster approval request for the prod stage", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, prodApprovalRequestName, prodStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + }) + + It("should keep waiting for approval after the wait time of the prod stage elapses", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(prodApprovalRequestName, perStageTaskApprovalRequestCreatedCond()), + wantTimedWaitTaskStatus(afterStageWaitTime, perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDurationForBudgetExpiry, eventuallyInterval).Should(Succeed(), "Failed to elapse the wait time of the prod stage") + Consistently(rolloutStatusActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Failed to keep waiting for approval") + }) + + It("can approve the cluster approval request for the prod stage", func() { + approveApprovalRequest(clusterScopedNS, prodApprovalRequestName) + }) + + It("should accept the approval for the prod stage", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, prodApprovalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should complete the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + stagedUpdateRunInitializedCond(), + stagedUpdateRunStartedCond(), + stagedUpdateRunCompletedCond(rolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + }, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + wantCompletedCanaryStage, + { + StageName: prodStageName, + Clusters: []rolloutv1alpha1.PerClusterStatus{ + wantPerClusterStatus(memberClusterNames, bindingNames, 1, perClusterUpdateSucceededConds()...), + }, + AfterStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(prodApprovalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + wantTimedWaitTaskStatus(afterStageWaitTime, perStageTaskTimedWaitStartedCond(), perStageTaskWaitTimeElapsedCond()), + }, + Conditions: perStageSucceededConds(), + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + + stagesCompletedInOrderActual := stagedUpdateRunStagesCompletedInOrderActual(clusterScopedNS, runName) + Expect(stagesCompletedInOrderActual()).To(Succeed(), "Failed to complete the stages in order") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, canaryApprovalRequestName), + newApprovalRequestObj(clusterScopedNS, prodApprovalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("deleted cluster approval request", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + var oldApprovalRequestUID types.UID + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the cluster approval request", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + + request, err := getApprovalRequest(clusterScopedNS, approvalRequestName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the cluster approval request") + oldApprovalRequestUID = request.GetUID() + }) + + It("can delete the cluster approval request", func() { + // The controller re-creates the cluster approval request right away; do not wait for the name to be gone. + Expect(hubClient.Delete(ctx, newApprovalRequestObj(clusterScopedNS, approvalRequestName))).To(Succeed(), "Failed to delete the cluster approval request") + }) + + It("should re-create the cluster approval request", func() { + approvalRequestReplacedActual := approvalRequestReplacedActual(clusterScopedNS, approvalRequestName, oldApprovalRequestUID) + Eventually(approvalRequestReplacedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the cluster approval request") + + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Expect(approvalRequestCreatedActual()).To(Succeed(), "Failed to re-create the cluster approval request as expected") + }) + + It("should not roll out to the member cluster before the approval", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the member cluster unexpectedly") + }) + + It("can approve the re-created cluster approval request", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("stale cluster approval request from a deleted cluster staged update run of the same name", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + var oldApprovalRequestUID types.UID + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should create the cluster approval request", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + + request, err := getApprovalRequest(clusterScopedNS, approvalRequestName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the cluster approval request") + oldApprovalRequestUID = request.GetUID() + }) + + It("can delete the cluster staged update run", func() { + removeObjects(newStagedUpdateRunObj(clusterScopedNS, runName)) + + // The envtest environment does not run the garbage collector; the cluster approval request stays around. + _, err := getApprovalRequest(clusterScopedNS, approvalRequestName) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve the cluster approval request of the deleted cluster staged update run") + }) + + It("can approve the cluster approval request of the deleted cluster staged update run", func() { + // The approval of the stale cluster approval request should not carry over to the re-created cluster staged update run. + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("can re-create the cluster staged update run", func() { + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should replace the stale cluster approval request with a new one owned by the re-created cluster staged update run", func() { + approvalRequestReplacedActual := approvalRequestReplacedActual(clusterScopedNS, approvalRequestName, oldApprovalRequestUID) + Eventually(approvalRequestReplacedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to replace the stale cluster approval request") + + // The check below verifies that the new cluster approval request is owned by the re-created cluster staged update run + // (with the UID of the re-created run). + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the new cluster approval request as expected") + + approvalRequestNotApprovedActual := approvalRequestConditionsActual(clusterScopedNS, approvalRequestName, nil) + Expect(approvalRequestNotApprovedActual()).To(Succeed(), "The new cluster approval request has been approved unexpectedly") + }) + + It("should not roll out to the member cluster before the new cluster approval request is approved", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the member cluster unexpectedly") + }) + + It("can approve the new cluster approval request", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) + + Context("cluster approval request not owned by the cluster staged update run", Ordered, func() { + randStr := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, randStr) + policyName := fmt.Sprintf(placementPolicyNameTemplate, randStr) + strategyName := fmt.Sprintf(stagedUpdateStrategyNameTemplate, randStr) + runName := fmt.Sprintf(stagedUpdateRunNameTemplate, randStr) + var snapshotName string + approvalRequestName := fmt.Sprintf(beforeStageApprovalRequestNameTemplate, runName, canaryStageName) + + memberClusterNames, bindingNames := memberClusterAndBindingNames(randStr, 1) + + BeforeAll(func() { + createConfigMap(configMapName) + policy := createPlacementPolicy(clusterScopedNS, policyName, configMapName) + // The snapshot names factor in the UID of the placement policy, which is only known once it is created. + snapshotName = primaryPlacementResourceSnapshotName(policy, 0) + fixtures := createMemberClustersAndBindings(clusterScopedNS, policyName, memberClusterNames, bindingNames, []map[string]string{nil}) + strategy := createStagedUpdateStrategy(clusterScopedNS, strategyName, []rolloutv1alpha1.Stage{ + { + Name: canaryStageName, + LabelSelector: &metav1.LabelSelector{}, + BeforeStageTasks: []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeApproval, + }, + }, + }, + }) + + // Create (and approve) a cluster approval request of the expected name manually, before the cluster staged update run is + // created. Note that the request references the cluster staged update run in its spec, so that its deletion + // triggers the processing of the run. + createManualApprovalRequest(clusterScopedNS, approvalRequestName, runName, canaryStageName) + approveApprovalRequest(clusterScopedNS, approvalRequestName) + waitForObjectsToBeCached(append(fixtures, policy, strategy)...) + + createStagedUpdateRun(clusterScopedNS, runName, rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyName, + StagedUpdateStrategyName: strategyName, + }) + }) + + It("should report that the cluster approval request is not owned by the cluster staged update run", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestOwnershipMismatchCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the ownership mismatch") + }) + + It("should not roll out to the member cluster, even though the cluster approval request has been approved", func() { + bindingUntouchedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], previousResourceSnapshotName) + Consistently(bindingUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Rolled out to the member cluster unexpectedly") + }) + + It("can delete the manually created cluster approval request", func() { + // The controller re-creates the cluster approval request right away; do not wait for the name to be gone. + Expect(hubClient.Delete(ctx, newApprovalRequestObj(clusterScopedNS, approvalRequestName))).To(Succeed(), "Failed to delete the cluster approval request") + }) + + It("should create its own cluster approval request", func() { + approvalRequestCreatedActual := approvalRequestCreatedActual(clusterScopedNS, runName, approvalRequestName, canaryStageName) + Eventually(approvalRequestCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the cluster approval request") + }) + + It("should report that the cluster approval request has been created", func() { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunStartedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: []rolloutv1alpha1.PerStageStatus{ + { + StageName: canaryStageName, + Clusters: wantPerClusterStatuses(memberClusterNames, bindingNames, 0), + BeforeStageTasks: []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond()), + }, + Conditions: []metav1.Condition{perStageStartedCond()}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the creation of the cluster approval request") + }) + + It("can approve the cluster approval request", func() { + approveApprovalRequest(clusterScopedNS, approvalRequestName) + }) + + It("should accept the approval", func() { + approvalAcceptedActual := approvalRequestApprovalAcceptedActual(clusterScopedNS, approvalRequestName) + Eventually(approvalAcceptedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to accept the approval") + }) + + It("should start rolling out to the member cluster", func() { + bindingUpdatedActual := placementBindingResourceSnapshotActual(clusterScopedNS, bindingNames[0], snapshotName) + Eventually(bindingUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll out to the member cluster") + }) + + It("can mark the member cluster as available", func() { + markPlacementBindingAsAvailable(clusterScopedNS, bindingNames[0], snapshotName) + }) + + It("should complete the cluster staged update run", func() { + wantStatus := wantSingleClusterCompletedStatus(snapshotName, memberClusterNames, bindingNames, + []rolloutv1alpha1.PerStageTaskStatus{ + wantApprovalTaskStatus(approvalRequestName, perStageTaskApprovalRequestCreatedCond(), perStageTaskApprovalRequestApprovedCond()), + }) + rolloutStatusActual := stagedUpdateRunRolloutStatusActual(clusterScopedNS, runName, wantStatus) + Eventually(rolloutStatusActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to complete the cluster staged update run") + }) + + AfterAll(func() { + removeObjects(slices.Concat( + []client.Object{ + newStagedUpdateRunObj(clusterScopedNS, runName), + newApprovalRequestObj(clusterScopedNS, approvalRequestName), + newPlacementResourceSnapshotObj(clusterScopedNS, snapshotName), + newStagedUpdateStrategyObj(clusterScopedNS, strategyName), + newPlacementPolicyObj(clusterScopedNS, policyName), + newConfigMapObj(configMapName), + }, + memberClusterAndBindingObjs(clusterScopedNS, memberClusterNames, bindingNames), + )...) + }) + }) +}) diff --git a/pkg/v1/controllers/stagedupdaterun/uniquename.go b/pkg/v1/controllers/stagedupdaterun/uniquename.go new file mode 100644 index 000000000..678838d27 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/uniquename.go @@ -0,0 +1,84 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "crypto/sha256" + "encoding/hex" + "fmt" + "strings" +) + +const ( + beforeStageApprovalRequestNameFmt = "%s-before-stage-%s" + beforeStageApprovalRequestWithHashNameFmt = "%s-before-stage-%s-%s" + afterStageApprovalRequestNameFmt = "%s-after-stage-%s" + afterStageApprovalRequestWithHashNameFmt = "%s-after-stage-%s-%s" + + // Kubernetes custom resources have a name limit of 253 characters. Here the limit is set to 200 to allow + // a small safety margin. + approvalRequestNameMaxLength = 200 + approvalRequestNameHashLength = 8 +) + +// formatBeforeStageApprovalRequestName formats the approval request name as +// [STAGED-UPDATE-RUN-NAME]-before-stage-[STAGE-NAME]. +// +// If the full name is longer than 200 characters, it truncates the two dynamic segments and +// appends an 8-char hash of the full (untruncated) name to avoid collisions. +func formatBeforeStageApprovalRequestName(stagedUpdateRunName, stageName string) string { + // Note: 14 is the length of the static part of the name: "-before-stage-". + return formatApprovalRequestName(beforeStageApprovalRequestNameFmt, + beforeStageApprovalRequestWithHashNameFmt, 14, stagedUpdateRunName, stageName) +} + +// formatAfterStageApprovalRequestName formats the approval request name as +// [STAGED-UPDATE-RUN-NAME]-after-stage-[STAGE-NAME]. +// +// If the full name is longer than 200 characters, it truncates the two dynamic +// segments and appends an 8-char hash of the full (untruncated) name to avoid +// collisions. +func formatAfterStageApprovalRequestName(stagedUpdateRunName, stageName string) string { + // Note: 13 is the length of the static part of the name: "-after-stage-". + return formatApprovalRequestName(afterStageApprovalRequestNameFmt, + afterStageApprovalRequestWithHashNameFmt, 13, stagedUpdateRunName, stageName) +} + +func formatApprovalRequestName(noHashTemplate, withHashTemplate string, staticCharLen int, stagedUpdateRunName, stageName string) string { + fullName := fmt.Sprintf(noHashTemplate, stagedUpdateRunName, stageName) + if len(fullName) <= approvalRequestNameMaxLength { + return fullName + } + + // The additional one is the extra hyphen before the hash. + availableSlots := approvalRequestNameMaxLength - staticCharLen - approvalRequestNameHashLength - 1 + stagedUpdateRunSlots := availableSlots / 2 + stageSlots := stagedUpdateRunSlots + truncatedRunName := stagedUpdateRunName + if len(truncatedRunName) > stagedUpdateRunSlots { + truncatedRunName = truncatedRunName[:stagedUpdateRunSlots] + truncatedRunName = strings.TrimRight(truncatedRunName, ".") + } + truncatedStageName := stageName + if len(truncatedStageName) > stageSlots { + truncatedStageName = truncatedStageName[:stageSlots] + } + + sum := sha256.Sum256([]byte(fullName)) + hash := hex.EncodeToString(sum[:])[:approvalRequestNameHashLength] + return fmt.Sprintf(withHashTemplate, truncatedRunName, truncatedStageName, hash) +} diff --git a/pkg/v1/controllers/stagedupdaterun/uniquename_test.go b/pkg/v1/controllers/stagedupdaterun/uniquename_test.go new file mode 100644 index 000000000..41f867bf5 --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/uniquename_test.go @@ -0,0 +1,263 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "fmt" + "regexp" + "strings" + "testing" + + "k8s.io/apimachinery/pkg/util/validation" +) + +const ( + // The maximum length of approval request names, as documented. + wantMaxApprovalRequestNameLength = 200 + // The maximum length of staged update run names (i.e., Kubernetes object names). + maxStagedUpdateRunNameLength = 253 + // The maximum length of stage names, as the staged update strategy API enforces. + maxStageNameLength = 63 +) + +// Truncated approval request names end with a hyphen and an 8-char (hex) hash. +var hashSuffixRegexp = regexp.MustCompile(`-[0-9a-f]{8}$`) + +// dottedName returns a valid DNS subdomain name of the given length, with a dot as every 10th character, starting +// from the given offset (1-9), except for the last character, e.g., dottedName(23, 9) = "abcdefghi.klmnopqrs.uvw". +func dottedName(length, dotOffset int) string { + var sb strings.Builder + for idx := 0; idx < length; idx++ { + if idx%10 == dotOffset && idx != length-1 { + sb.WriteByte('.') + continue + } + sb.WriteByte(byte('a' + idx%26)) + } + return sb.String() +} + +// TestFormatBeforeStageApprovalRequestName tests the formatBeforeStageApprovalRequestName function. +func TestFormatBeforeStageApprovalRequestName(t *testing.T) { + testCases := []struct { + name string + stagedUpdateRunName string + stageName string + wantName string + }{ + { + name: "short names", + stagedUpdateRunName: "run-1", + stageName: canaryStageName, + wantName: "run-1-before-stage-canary", + }, + { + // 123 + 14 + 63 = 200 characters. + name: "full name at the length limit (not truncated)", + stagedUpdateRunName: strings.Repeat("a", 123), + stageName: strings.Repeat("s", 63), + wantName: strings.Repeat("a", 123) + "-before-stage-" + strings.Repeat("s", 63), + }, + { + // 124 + 14 + 63 = 201 characters; the run name is truncated to 88 characters ((200 - 14 - 9) / 2), and the + // stage name fits as is. + name: "full name over the length limit (truncated, with hash)", + stagedUpdateRunName: strings.Repeat("a", 124), + stageName: strings.Repeat("s", 63), + wantName: strings.Repeat("a", 88) + "-before-stage-" + strings.Repeat("s", 63) + "-bac2165e", + }, + { + // The run name is cut right after a dot (the 88th character), which is trimmed. + name: "truncated run name ending with a dot", + stagedUpdateRunName: strings.Repeat("a", 87) + "." + strings.Repeat("b", 165), + stageName: canaryStageName, + wantName: strings.Repeat("a", 87) + "-before-stage-canary-0ff3d5af", + }, + { + // The staged update strategy API limits stage names to 63 characters, which never need truncation; + // the function still truncates longer ones. + name: "both names truncated", + stagedUpdateRunName: strings.Repeat("a", 150), + stageName: strings.Repeat("s", 100), + wantName: strings.Repeat("a", 88) + "-before-stage-" + strings.Repeat("s", 88) + "-bf938dd2", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotName := formatBeforeStageApprovalRequestName(tc.stagedUpdateRunName, tc.stageName) + if gotName != tc.wantName { + t.Errorf("formatBeforeStageApprovalRequestName(%q, %q) = %q, want %q", tc.stagedUpdateRunName, tc.stageName, gotName, tc.wantName) + } + }) + } +} + +// TestFormatAfterStageApprovalRequestName tests the formatAfterStageApprovalRequestName function. +func TestFormatAfterStageApprovalRequestName(t *testing.T) { + testCases := []struct { + name string + stagedUpdateRunName string + stageName string + wantName string + }{ + { + name: "short names", + stagedUpdateRunName: "run-1", + stageName: canaryStageName, + wantName: "run-1-after-stage-canary", + }, + { + // 124 + 13 + 63 = 200 characters. + name: "full name at the length limit (not truncated)", + stagedUpdateRunName: strings.Repeat("a", 124), + stageName: strings.Repeat("s", 63), + wantName: strings.Repeat("a", 124) + "-after-stage-" + strings.Repeat("s", 63), + }, + { + // 125 + 13 + 63 = 201 characters; the run name is truncated to 89 characters ((200 - 13 - 9) / 2), and the + // stage name fits as is. + name: "full name over the length limit (truncated, with hash)", + stagedUpdateRunName: strings.Repeat("a", 125), + stageName: strings.Repeat("s", 63), + wantName: strings.Repeat("a", 89) + "-after-stage-" + strings.Repeat("s", 63) + "-60f7932f", + }, + { + // The run name is cut right after a dot (the 89th character), which is trimmed. + name: "truncated run name ending with a dot", + stagedUpdateRunName: strings.Repeat("a", 88) + "." + strings.Repeat("b", 164), + stageName: canaryStageName, + wantName: strings.Repeat("a", 88) + "-after-stage-canary-cdbf87d2", + }, + { + // The staged update strategy API limits stage names to 63 characters, which never need truncation; + // the function still truncates longer ones. + name: "both names truncated", + stagedUpdateRunName: strings.Repeat("a", 150), + stageName: strings.Repeat("s", 100), + wantName: strings.Repeat("a", 89) + "-after-stage-" + strings.Repeat("s", 89) + "-551fb0a0", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotName := formatAfterStageApprovalRequestName(tc.stagedUpdateRunName, tc.stageName) + if gotName != tc.wantName { + t.Errorf("formatAfterStageApprovalRequestName(%q, %q) = %q, want %q", tc.stagedUpdateRunName, tc.stageName, gotName, tc.wantName) + } + }) + } +} + +// TestFormatApprovalRequestNameUniqueness verifies that the hash suffix keeps truncated approval request names apart. +func TestFormatApprovalRequestNameUniqueness(t *testing.T) { + // The two run names share their first 100 characters, i.e., their truncated forms are the same. + runName1 := strings.Repeat("a", 100) + "x" + strings.Repeat("c", 99) + runName2 := strings.Repeat("a", 100) + "y" + strings.Repeat("c", 99) + stageName := canaryStageName + + formatFuncs := map[string]func(string, string) string{ + "formatBeforeStageApprovalRequestName": formatBeforeStageApprovalRequestName, + "formatAfterStageApprovalRequestName": formatAfterStageApprovalRequestName, + } + for funcName, formatFunc := range formatFuncs { + t.Run(funcName, func(t *testing.T) { + name1 := formatFunc(runName1, stageName) + name2 := formatFunc(runName2, stageName) + if name1 == name2 { + t.Errorf("%s(%q, %q) = %s(%q, %q) = %q, want different names", funcName, runName1, stageName, funcName, runName2, stageName, name1) + } + + // The names differ only in their hash suffixes. + if !hashSuffixRegexp.MatchString(name1) || !hashSuffixRegexp.MatchString(name2) { + t.Fatalf("%s() = %q, %q, want both names to end with a hash suffix", funcName, name1, name2) + } + prefix1 := hashSuffixRegexp.ReplaceAllString(name1, "") + prefix2 := hashSuffixRegexp.ReplaceAllString(name2, "") + if prefix1 != prefix2 { + t.Errorf("%s() = %q, %q, want the same name before the hash suffix", funcName, name1, name2) + } + + // The function is deterministic. + if again := formatFunc(runName1, stageName); again != name1 { + t.Errorf("%s(%q, %q) = %q, then %q, want the same name", funcName, runName1, stageName, name1, again) + } + }) + } + + // The before-stage and after-stage approval requests of the same stage have different names. + beforeStageName := formatBeforeStageApprovalRequestName(runName1, stageName) + afterStageName := formatAfterStageApprovalRequestName(runName1, stageName) + if beforeStageName == afterStageName { + t.Errorf("formatBeforeStageApprovalRequestName(%q, %q) = formatAfterStageApprovalRequestName(%q, %q) = %q, want different names", + runName1, stageName, runName1, stageName, beforeStageName) + } +} + +// TestFormatApprovalRequestNameLengthAndValidity verifies, for all the valid lengths of staged update run names (and +// a few lengths of stage names), that approval request names never exceed the length limit, are left as is when +// they fit, carry a hash suffix when they do not, and are valid Kubernetes object names (DNS subdomains). +// +// The run names have dots at varying positions, so that some of them are truncated right after a dot (which must be +// trimmed for the result to stay a valid name). +func TestFormatApprovalRequestNameLengthAndValidity(t *testing.T) { + formatFuncs := []struct { + name string + formatFunc func(string, string) string + fullNameTmpl string + }{ + { + name: "formatBeforeStageApprovalRequestName", + formatFunc: formatBeforeStageApprovalRequestName, + fullNameTmpl: "%s-before-stage-%s", + }, + { + name: "formatAfterStageApprovalRequestName", + formatFunc: formatAfterStageApprovalRequestName, + fullNameTmpl: "%s-after-stage-%s", + }, + } + + for _, ff := range formatFuncs { + t.Run(ff.name, func(t *testing.T) { + for _, stageNameLength := range []int{1, 10, maxStageNameLength} { + stageName := strings.Repeat("s", stageNameLength) + for dotOffset := 1; dotOffset <= 9; dotOffset++ { + for runNameLength := 1; runNameLength <= maxStagedUpdateRunNameLength; runNameLength++ { + runName := dottedName(runNameLength, dotOffset) + gotName := ff.formatFunc(runName, stageName) + fullName := fmt.Sprintf(ff.fullNameTmpl, runName, stageName) + + if len(gotName) > wantMaxApprovalRequestNameLength { + t.Errorf("%s(%q, %q) = %q (%d characters), want at most %d characters", + ff.name, runName, stageName, gotName, len(gotName), wantMaxApprovalRequestNameLength) + } + if len(fullName) <= wantMaxApprovalRequestNameLength && gotName != fullName { + t.Errorf("%s(%q, %q) = %q, want %q", ff.name, runName, stageName, gotName, fullName) + } + if len(fullName) > wantMaxApprovalRequestNameLength && !hashSuffixRegexp.MatchString(gotName) { + t.Errorf("%s(%q, %q) = %q, want a name with a hash suffix", ff.name, runName, stageName, gotName) + } + if errs := validation.IsDNS1123Subdomain(gotName); len(errs) > 0 { + t.Errorf("%s(%q, %q) = %q, want a valid DNS subdomain name (errors: %v)", ff.name, runName, stageName, gotName, errs) + } + } + } + } + }) + } +} diff --git a/pkg/v1/controllers/stagedupdaterun/utils_integration_test.go b/pkg/v1/controllers/stagedupdaterun/utils_integration_test.go new file mode 100644 index 000000000..7d8bcf45c --- /dev/null +++ b/pkg/v1/controllers/stagedupdaterun/utils_integration_test.go @@ -0,0 +1,1683 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package stagedupdaterun + +import ( + "fmt" + "strconv" + "strings" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" +) + +const ( + configMapNameTemplate = "configmap-%s" + placementPolicyNameTemplate = "policy-%s" + placementBindingNameTemplate = "binding-%s" + stagedUpdateStrategyNameTemplate = "strategy-%s" + stagedUpdateRunNameTemplate = "run-%s" + // Member clusters are numbered from 0 (zero-padded), so that the lexical order of their names matches the + // numerical order of their indexes. + memberClusterNameTemplate = "cluster-%s-%02d" + // The approval requests that test cases create (and delete) only to trigger the processing of a staged update + // run (see triggerStagedUpdateRunReconciliation). + triggerApprovalRequestNameTemplate = "trigger-%s" + + // The resource snapshot name that placement bindings point to before a staged update run rolls out + // a new one; no snapshot of this name needs to exist, as no work generator runs in the test environment. + previousResourceSnapshotName = "previous-resource-snapshot" + + envLabelKey = "env" + envLabelValueCanary = "canary" + envLabelValueProd = "prod" + envLabelValueStaging = "staging" + + regionLabelKey = "region" + regionLabelValueEast = "east" + regionLabelValueWest = "west" + + groupLabelKey = "group" + orderLabelKey = "order" + + canaryStageName = "canary" + prodStageName = "prod" + eastStageName = "east" + emptyStageName = "empty" + restStageName = "rest" + + configMapDataKey = "foo" + configMapDataValue = "bar" + // An alternative ConfigMap data value, used to simulate changes to the selected resources. + altConfigMapDataValue = "baz" +) + +// The namespace of cluster-scoped objects (i.e., none); the test helpers use it to tell cluster-scoped objects +// (e.g., ClusterStagedUpdateRun objects) apart from their namespace-scoped counterparts. +const clusterScopedNS = "" + +const ( + eventuallyDuration = time.Second * 20 + eventuallyInterval = time.Millisecond * 500 + consistentlyDuration = time.Second * 5 + consistentlyInterval = time.Millisecond * 500 + + // The controller persists the progress of a rollout only at the end of a reconciliation, i.e., when the + // processing time budget (the refresh period) runs out if the rollout is blocked; checks on the progress of + // in-flight rollouts use a longer timeout that covers a budget expiry plus some leeway. + eventuallyDurationForBudgetExpiry = time.Second * 20 + // Checks that span multiple processing time budget expiries (i.e., multiple reconciliations of an in-flight + // rollout) use a longer duration. + consistentlyDurationAcrossBudgetExpiries = 2*refreshPeriodForTesting + 2*time.Second + + // The API requires the per-cluster wait time limit to be at least one minute; checks that wait for a member + // cluster to time out use a longer timeout that covers the limit plus some leeway. + eventuallyDurationForClusterTimeout = time.Minute * 2 +) + +// The reasons that test cases use when they simulate placement binding status changes. +const ( + markedAsSynchronizedReason = "MarkedAsSynchronized" + markedAsNotSynchronizedReason = "MarkedAsNotSynchronized" + markedAsAvailableReason = "MarkedAsAvailable" + markedAsUnavailableReason = "MarkedAsUnavailable" +) + +// The reasons that test cases use when they approve or reject approval requests, as a user would do. +const ( + markedAsApprovedReason = "MarkedAsApproved" + markedAsRejectedReason = "MarkedAsRejected" +) + +// A finalizer that test cases add to objects, so that the objects stay around (with a deletion timestamp set) +// after being deleted. +const testFinalizer = "kubefleet.dev/integration-test" + +var ( + ignoreFieldConditionLTTMsg = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message") + + // The option for comparing the status of a staged update run that has started rolling out, where the stage + // timestamps are set as the rollout progresses; test cases check the timestamps separately. + ignoreFieldsPerStageStatusTimestamps = cmpopts.IgnoreFields(rolloutv1alpha1.PerStageStatus{}, "StartedTimestamp", "CompletedTimestamp") + + // The options for comparing the layout of the stages in a staged update run (i.e., the stages, their clusters + // and tasks, and the resolved max concurrency), with all the progress information ignored. + ignoreFieldsPerStageStatusProgress = cmpopts.IgnoreFields(rolloutv1alpha1.PerStageStatus{}, "Conditions", "StartedTimestamp", "CompletedTimestamp") + ignoreFieldsPerClusterStatusProgress = cmpopts.IgnoreFields(rolloutv1alpha1.PerClusterStatus{}, "Conditions") + ignoreFieldsPerStageTaskProgress = cmpopts.IgnoreFields(rolloutv1alpha1.PerStageTaskStatus{}, "Conditions", "ApprovalRequestName") + + lessFuncCondition = func(a, b metav1.Condition) bool { + return a.Type < b.Type + } + lessFuncObjectRef = func(a, b placementv1alpha1.ObjectReference) bool { + return fmt.Sprintf("%s/%s/%s", a.Kind, a.Namespace, a.Name) < fmt.Sprintf("%s/%s/%s", b.Kind, b.Namespace, b.Name) + } +) + +// placementPolicyKind returns the kind of the placement policy of the scope that the given namespace implies. +func placementPolicyKind(policyNSName string) string { + if policyNSName == clusterScopedNS { + return placementv1alpha1.ClusterPlacementPolicyKind + } + return placementv1alpha1.PlacementPolicyKind +} + +// stagedUpdateRunKind returns the kind of the staged update run of the scope that the given namespace implies. +func stagedUpdateRunKind(runNSName string) string { + if runNSName == clusterScopedNS { + return rolloutv1alpha1.ClusterStagedUpdateRunKind + } + return rolloutv1alpha1.StagedUpdateRunKind +} + +// newPlacementPolicyObj returns a placement policy object of the appropriate scope with only the namespace and name +// set: a ClusterPlacementPolicy object if the namespace is empty, or a PlacementPolicy object otherwise. +func newPlacementPolicyObj(policyNSName, policyName string) placementv1alpha1.PlacementPolicyAccessor { + if policyNSName == clusterScopedNS { + return &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: policyName, + }, + } + } + return &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: policyNSName, + Name: policyName, + }, + } +} + +// newPlacementBindingObj returns a placement binding object of the appropriate scope with only the namespace and name +// set: a ClusterPlacementBinding object if the namespace is empty, or a PlacementBinding object otherwise. +func newPlacementBindingObj(bindingNS, bindingName string) placementv1alpha1.PlacementBindingAccessor { + if bindingNS == clusterScopedNS { + return &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: bindingName, + }, + } + } + return &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: bindingNS, + Name: bindingName, + }, + } +} + +// newPlacementResourceSnapshotObj returns a placement resource snapshot object of the appropriate scope with only the +// namespace and name set: a ClusterPlacementResourceSnapshot object if the namespace is empty, or a +// PlacementResourceSnapshot object otherwise. +func newPlacementResourceSnapshotObj(snapshotNS, snapshotName string) placementv1alpha1.PlacementResourceSnapshotAccessor { + if snapshotNS == clusterScopedNS { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + }, + } + } + return &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: snapshotNS, + Name: snapshotName, + }, + } +} + +// newStagedUpdateStrategyObj returns a staged update strategy object of the appropriate scope with only the namespace +// and name set: a ClusterStagedUpdateStrategy object if the namespace is empty, or a StagedUpdateStrategy object +// otherwise. +func newStagedUpdateStrategyObj(strategyNS, strategyName string) rolloutv1alpha1.StagedUpdateStrategyAccessor { + if strategyNS == clusterScopedNS { + return &rolloutv1alpha1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: strategyName, + }, + } + } + return &rolloutv1alpha1.StagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: strategyNS, + Name: strategyName, + }, + } +} + +// newStagedUpdateRunObj returns a staged update run object of the appropriate scope with only the namespace and name +// set: a ClusterStagedUpdateRun object if the namespace is empty, or a StagedUpdateRun object otherwise. +func newStagedUpdateRunObj(runNSName, runName string) rolloutv1alpha1.StagedUpdateRunAccessor { + if runNSName == clusterScopedNS { + return &rolloutv1alpha1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: runName, + }, + } + } + return &rolloutv1alpha1.StagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: runNSName, + Name: runName, + }, + } +} + +// newApprovalRequestObj returns an approval request object of the appropriate scope with only the namespace and name +// set: a ClusterApprovalRequest object if the namespace is empty, or an ApprovalRequest object otherwise. +func newApprovalRequestObj(requestNS, requestName string) rolloutv1alpha1.ApprovalRequestAccessor { + if requestNS == clusterScopedNS { + return &rolloutv1alpha1.ClusterApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{ + Name: requestName, + }, + } + } + return &rolloutv1alpha1.ApprovalRequest{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: requestNS, + Name: requestName, + }, + } +} + +// newConfigMapObj returns a ConfigMap object in the app namespace with only the namespace and name set. +func newConfigMapObj(configMapName string) *corev1.ConfigMap { + return &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: appNamespaceName, + Name: configMapName, + }, + } +} + +// createConfigMap creates a ConfigMap object of the given name in the app namespace, for placement policies to select. +func createConfigMap(configMapName string) *corev1.ConfigMap { + configMap := newConfigMapObj(configMapName) + configMap.Data = map[string]string{ + configMapDataKey: configMapDataValue, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the ConfigMap") + return configMap +} + +// createPlacementPolicy creates a placement policy that selects a ConfigMap (by name) in the app namespace; it returns +// the policy as written to the API server. +// +// If the namespace is empty, a ClusterPlacementPolicy object is created; otherwise, a PlacementPolicy object is +// created, which must be in the app namespace to select the ConfigMap. +func createPlacementPolicy(policyNSName, policyName, configMapName string) placementv1alpha1.PlacementPolicyAccessor { + resourceSelector := placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMapName, + } + if policyNSName == clusterScopedNS { + // A cluster placement policy must specify the namespace of the namespaced resources it selects. + resourceSelector.Namespace = appNamespaceName + } + + policy := newPlacementPolicyObj(policyNSName, policyName) + policy.SetSpec(placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{resourceSelector}, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create the placement policy") + return policy +} + +// createMemberCluster creates a member cluster object of the given name with the given labels. +func createMemberCluster(clusterName string, labels map[string]string) *clusterv1beta1.MemberCluster { + memberCluster := &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: clusterName, + Labels: labels, + }, + Spec: clusterv1beta1.MemberClusterSpec{ + Identity: rbacv1.Subject{ + Kind: "ServiceAccount", + Name: clusterName, + Namespace: "fleet-system", + }, + }, + } + Expect(hubClient.Create(ctx, memberCluster)).To(Succeed(), "Failed to create the member cluster") + return memberCluster +} + +// createPlacementBinding creates a placement binding that binds a placement resource snapshot of a placement policy +// to a member cluster, as the placement policy controller would do; it returns the binding as written to the API server. +// +// If the namespace is empty, a ClusterPlacementBinding object is created; otherwise, a PlacementBinding object +// is created. +func createPlacementBinding(bindingNS, bindingName, policyName, clusterName, snapshotName string) placementv1alpha1.PlacementBindingAccessor { + binding := newPlacementBindingObj(bindingNS, bindingName) + binding.SetSpec(placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: policyName, + ClusterName: clusterName, + ResourceSnapshotName: snapshotName, + }) + Expect(hubClient.Create(ctx, binding)).To(Succeed(), "Failed to create the placement binding") + return binding +} + +// memberClusterAndBindingNames returns the names of the given number of member clusters, along with the names of +// the placement bindings that bind a placement policy to them (at matching indexes). +func memberClusterAndBindingNames(randStr string, count int) (memberClusterNames, bindingNames []string) { + memberClusterNames = make([]string, 0, count) + bindingNames = make([]string, 0, count) + for idx := 0; idx < count; idx++ { + memberClusterName := fmt.Sprintf(memberClusterNameTemplate, randStr, idx) + memberClusterNames = append(memberClusterNames, memberClusterName) + bindingNames = append(bindingNames, fmt.Sprintf(placementBindingNameTemplate, memberClusterName)) + } + return memberClusterNames, bindingNames +} + +// createMemberClustersAndBindings creates a member cluster with the given labels for each of the given names, along +// with a placement binding that binds the placement policy to the member cluster (pointing to the previous resource +// snapshot); it returns the created objects. +// +// The objects are created in the reverse order of the names, so that the order of the clusters in a stage is never +// a by-product of the creation order. +func createMemberClustersAndBindings( + bindingNS, policyName string, + memberClusterNames, bindingNames []string, + memberClusterLabels []map[string]string, +) []client.Object { + objs := make([]client.Object, 0, 2*len(memberClusterNames)) + for idx := len(memberClusterNames) - 1; idx >= 0; idx-- { + memberCluster := createMemberCluster(memberClusterNames[idx], memberClusterLabels[idx]) + binding := createPlacementBinding(bindingNS, bindingNames[idx], policyName, memberClusterNames[idx], previousResourceSnapshotName) + objs = append(objs, memberCluster, binding) + } + return objs +} + +// memberClusterAndBindingObjs returns objects (with only the namespace and name set) for the given member clusters +// and their placement bindings, e.g., for cleanup. +func memberClusterAndBindingObjs(bindingNS string, memberClusterNames, bindingNames []string) []client.Object { + objs := make([]client.Object, 0, 2*len(memberClusterNames)) + for idx := range memberClusterNames { + objs = append(objs, + newPlacementBindingObj(bindingNS, bindingNames[idx]), + &clusterv1beta1.MemberCluster{ObjectMeta: metav1.ObjectMeta{Name: memberClusterNames[idx]}}, + ) + } + return objs +} + +// createStagedUpdateStrategy creates a staged update strategy with the given stages; it returns the strategy as +// written to the API server. +// +// If the namespace is empty, a ClusterStagedUpdateStrategy object is created; otherwise, a StagedUpdateStrategy +// object is created. +func createStagedUpdateStrategy(strategyNS, strategyName string, stages []rolloutv1alpha1.Stage) rolloutv1alpha1.StagedUpdateStrategyAccessor { + strategy := newStagedUpdateStrategyObj(strategyNS, strategyName) + strategy.SetSpec(rolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: stages, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create the staged update strategy") + return strategy +} + +// createStagedUpdateRun creates a staged update run with the given spec; it returns the run as written to the +// API server. +// +// If the namespace is empty, a ClusterStagedUpdateRun object is created; otherwise, a StagedUpdateRun object +// is created. +func createStagedUpdateRun(runNSName, runName string, spec rolloutv1alpha1.StagedUpdateRunSpec) rolloutv1alpha1.StagedUpdateRunAccessor { + run := newStagedUpdateRunObj(runNSName, runName) + run.SetSpec(spec) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create the staged update run") + return run +} + +// createPlacementResourceSnapshot creates a placement resource snapshot (with no resources) of the given index and +// sub-index, controlled by the given placement policy; it returns the snapshot as written to the API server. +// +// The snapshot has the labels and the controller owner reference that the staged update run controller checks when +// a staged update run names the snapshot to roll out explicitly. It does not have the annotations that the placement +// resource snapshot manager adds (the resource contents hash and the owner), which the manager relies on when it +// looks up the latest snapshot and when it garbage collects snapshots; use it only with staged update runs that +// name their snapshot explicitly. +// +// The snapshot is created in the namespace of the placement policy (i.e., it is a ClusterPlacementResourceSnapshot +// object if the placement policy is cluster-scoped). The count of snapshots sharing the same index is set (as a +// label) only on the primary snapshot (sub-index 0). +func createPlacementResourceSnapshot( + snapshotName string, + ownerPolicy placementv1alpha1.PlacementPolicyAccessor, + index, subIndex, subIndexedSnapshotCount int, +) placementv1alpha1.PlacementResourceSnapshotAccessor { + snapshot := newPlacementResourceSnapshotObj(ownerPolicy.GetNamespace(), snapshotName) + labels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: ownerPolicy.GetName(), + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: strconv.Itoa(index), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: strconv.Itoa(subIndex), + } + if subIndex == 0 { + labels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = strconv.Itoa(subIndexedSnapshotCount) + } + snapshot.SetLabels(labels) + snapshot.SetOwnerReferences([]metav1.OwnerReference{ + *metav1.NewControllerRef(ownerPolicy, placementv1alpha1.GroupVersion.WithKind(placementPolicyKind(ownerPolicy.GetNamespace()))), + }) + Expect(hubClient.Create(ctx, snapshot)).To(Succeed(), "Failed to create the placement resource snapshot") + return snapshot +} + +// updateConfigMapData updates the data of a ConfigMap object in the app namespace. +func updateConfigMapData(configMapName string, data map[string]string) { + Eventually(func() error { + configMap := newConfigMapObj(configMapName) + if err := hubClient.Get(ctx, client.ObjectKeyFromObject(configMap), configMap); err != nil { + return fmt.Errorf("failed to retrieve the ConfigMap: %w", err) + } + configMap.Data = data + return hubClient.Update(ctx, configMap) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the ConfigMap") +} + +// updateMemberClusterLabels replaces the labels of a member cluster with the given ones. +func updateMemberClusterLabels(clusterName string, labels map[string]string) { + Eventually(func() error { + memberCluster := &clusterv1beta1.MemberCluster{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: clusterName}, memberCluster); err != nil { + return fmt.Errorf("failed to retrieve the member cluster: %w", err) + } + memberCluster.Labels = labels + return hubClient.Update(ctx, memberCluster) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the labels of the member cluster") +} + +// updateStagedUpdateStrategyStages replaces the stages of a staged update strategy with the given ones. +func updateStagedUpdateStrategyStages(strategyNS, strategyName string, stages []rolloutv1alpha1.Stage) { + Eventually(func() error { + strategy := newStagedUpdateStrategyObj(strategyNS, strategyName) + if err := hubClient.Get(ctx, client.ObjectKeyFromObject(strategy), strategy); err != nil { + return fmt.Errorf("failed to retrieve the staged update strategy: %w", err) + } + strategy.GetSpec().Stages = stages + return hubClient.Update(ctx, strategy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the stages of the staged update strategy") +} + +// updateStagedUpdateRunSuspended suspends or resumes a staged update run. +func updateStagedUpdateRunSuspended(runNSName, runName string, suspended bool) { + Eventually(func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + run.GetSpec().Suspended = suspended + return hubClient.Update(ctx, run) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the suspended flag of the staged update run") +} + +// markPlacementBindingAsAvailable marks a placement binding as synchronized and available, as the work generator +// would do once the resources in the binding have been applied and become available on the member cluster. +// +// See setPlacementBindingConditions for how the function avoids conflicts with the controller. +func markPlacementBindingAsAvailable(bindingNS, bindingName, wantSnapshotName string) { + setPlacementBindingConditions(bindingNS, bindingName, wantSnapshotName, metav1.ConditionTrue, metav1.ConditionTrue, 0) +} + +// setPlacementBindingConditions sets the Synchronized and Available conditions of a placement binding to the given +// statuses, as the work generator would do. +// +// The function first waits until the binding points to the given resource snapshot, i.e., the staged update run +// has rolled out to the member cluster, and then sets the conditions as observed at the generation of the binding at +// that point plus the given offset (e.g., -1 for a stale observation); this way, the status update does not conflict +// with the update the controller makes to the binding spec. +func setPlacementBindingConditions( + bindingNS, bindingName, wantSnapshotName string, + synchronizedStatus, availableStatus metav1.ConditionStatus, + observedGenerationOffset int64, +) { + synchronizedReason := markedAsSynchronizedReason + if synchronizedStatus != metav1.ConditionTrue { + synchronizedReason = markedAsNotSynchronizedReason + } + availableReason := markedAsAvailableReason + if availableStatus != metav1.ConditionTrue { + availableReason = markedAsUnavailableReason + } + + Eventually(func() error { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); err != nil { + return fmt.Errorf("failed to retrieve the placement binding: %w", err) + } + if gotSnapshotName := binding.GetSpec().ResourceSnapshotName; gotSnapshotName != wantSnapshotName { + return fmt.Errorf("placement binding %s points to resource snapshot %s, want %s", bindingName, gotSnapshotName, wantSnapshotName) + } + + observedGeneration := binding.GetGeneration() + observedGenerationOffset + bindingStatus := binding.GetStatus() + meta.SetStatusCondition(&bindingStatus.Conditions, metav1.Condition{ + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: synchronizedStatus, + Reason: synchronizedReason, + ObservedGeneration: observedGeneration, + }) + meta.SetStatusCondition(&bindingStatus.Conditions, metav1.Condition{ + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: availableStatus, + Reason: availableReason, + ObservedGeneration: observedGeneration, + }) + return hubClient.Status().Update(ctx, binding) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to set the conditions of the placement binding") +} + +// updatePlacementBindingSuspended suspends or resumes a placement binding; it waits until the cache-backed client +// (which the controller shares) observes the change. +func updatePlacementBindingSuspended(bindingNS, bindingName string, suspended bool) { + updatePlacementBindingSpec(bindingNS, bindingName, func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.Suspended = suspended + }) +} + +// updatePlacementBindingSpec updates the spec of a placement binding with the given function, e.g., to simulate +// changes that other parties make to the binding; it waits until the cache-backed client (which the controller +// shares) observes the update. +func updatePlacementBindingSpec(bindingNS, bindingName string, mutate func(spec *placementv1alpha1.PlacementBindingSpec)) { + key := client.ObjectKey{Namespace: bindingNS, Name: bindingName} + var updatedGeneration int64 + Eventually(func() error { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, key, binding); err != nil { + return fmt.Errorf("failed to retrieve the placement binding: %w", err) + } + mutate(binding.GetSpec()) + if err := hubClient.Update(ctx, binding); err != nil { + return fmt.Errorf("failed to update the placement binding: %w", err) + } + updatedGeneration = binding.GetGeneration() + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the spec of the placement binding") + + Eventually(func() error { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, key, binding); err != nil { + return fmt.Errorf("failed to retrieve the placement binding: %w", err) + } + if binding.GetGeneration() < updatedGeneration { + return fmt.Errorf("the cache has yet to observe the update to placement binding %s", bindingName) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the update to the placement binding in the cache") +} + +// approveApprovalRequest approves an approval request (cluster-scoped if the namespace is empty), as a user would +// do, by setting its Approved condition to True. +func approveApprovalRequest(requestNS, requestName string) { + setApprovalRequestApproved(requestNS, requestName, metav1.ConditionTrue) +} + +// rejectApprovalRequest rejects an approval request (cluster-scoped if the namespace is empty), as a user would do, +// by setting its Approved condition to False. +func rejectApprovalRequest(requestNS, requestName string) { + setApprovalRequestApproved(requestNS, requestName, metav1.ConditionFalse) +} + +// setApprovalRequestApproved sets the Approved condition of an approval request (cluster-scoped if the namespace is +// empty) to the given status, observed at the current generation of the request, and waits until the cache (which +// the controller shares) observes the new status; this way, test cases that check the approval request right +// afterwards (which read from the cache too) do not see a stale copy. +func setApprovalRequestApproved(requestNS, requestName string, status metav1.ConditionStatus) { + Eventually(func() error { + request := newApprovalRequestObj(requestNS, requestName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: requestNS, Name: requestName}, request); err != nil { + return fmt.Errorf("failed to retrieve the approval request: %w", err) + } + approvedCond := approvalRequestApprovedCond(status) + approvedCond.ObservedGeneration = request.GetGeneration() + meta.SetStatusCondition(&request.GetStatus().Conditions, approvedCond) + return hubClient.Status().Update(ctx, request) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to set the Approved condition of the approval request") + + Eventually(func() error { + request, err := getApprovalRequest(requestNS, requestName) + if err != nil { + return err + } + approvedCond := meta.FindStatusCondition(request.GetStatus().Conditions, rolloutv1alpha1.ApprovalRequestCondTypeApproved) + if approvedCond == nil || approvedCond.Status != status { + return fmt.Errorf("the Approved condition of the approval request is %+v, want status %s", approvedCond, status) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the Approved condition of the approval request in the cache") +} + +// createManualApprovalRequest creates an approval request (cluster-scoped if the namespace is empty) for the given +// stage of a staged update run, as a user (rather than the controller) would do, i.e., with no owner references. +func createManualApprovalRequest(requestNS, requestName, runName, stageName string) rolloutv1alpha1.ApprovalRequestAccessor { + request := newApprovalRequestObj(requestNS, requestName) + request.SetSpec(rolloutv1alpha1.ApprovalRequestSpec{ + StagedUpdateRunName: runName, + StageName: stageName, + }) + Expect(hubClient.Create(ctx, request)).To(Succeed(), "Failed to create the approval request") + return request +} + +// triggerStagedUpdateRunReconciliation makes the controller process a staged update run (cluster-scoped if the +// namespace is empty) again, without changing the run itself. The controller does not watch most of the objects +// that test cases change (e.g., placement policies and bindings), but it does process a staged update run when an +// approval request that references the run is deleted; the function creates such an approval request (with the +// given name, for the given stage) and deletes it, and returns once the cache has observed the deletion. +func triggerStagedUpdateRunReconciliation(runNSName, runName, stageName, requestName string) { + request := createManualApprovalRequest(runNSName, requestName, runName, stageName) + waitForObjectsToBeCached(request) + Expect(hubClient.Delete(ctx, request)).To(Succeed(), "Failed to delete the approval request") + Eventually(objectRemovedActual(request), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the deletion of the approval request in the cache") +} + +// getApprovalRequest retrieves an approval request (cluster-scoped if the namespace is empty). +func getApprovalRequest(requestNS, requestName string) (rolloutv1alpha1.ApprovalRequestAccessor, error) { + request := newApprovalRequestObj(requestNS, requestName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: requestNS, Name: requestName}, request); err != nil { + return nil, fmt.Errorf("failed to retrieve the approval request: %w", err) + } + return request, nil +} + +// markObjectForDeletion adds the test finalizer to an object and then deletes it, so that the object stays around +// with a deletion timestamp set; it waits until the cache-backed client (which the controller shares) observes the +// deletion timestamp. +// +// Test cases must call removeTestFinalizer on the object as part of the cleanup. +func markObjectForDeletion(obj client.Object) { + key := client.ObjectKeyFromObject(obj) + Eventually(func() error { + if err := hubClient.Get(ctx, key, obj); err != nil { + return fmt.Errorf("failed to retrieve object %T %s: %w", obj, key, err) + } + controllerutil.AddFinalizer(obj, testFinalizer) + return hubClient.Update(ctx, obj) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the test finalizer to object %T %s", obj, key) + + Expect(hubClient.Delete(ctx, obj)).To(Succeed(), "Failed to delete object %T %s", obj, key) + + Eventually(func() error { + objCopy := obj.DeepCopyObject().(client.Object) + if err := hubClient.Get(ctx, key, objCopy); err != nil { + return fmt.Errorf("failed to retrieve object %T %s: %w", obj, key, err) + } + if objCopy.GetDeletionTimestamp() == nil { + return fmt.Errorf("object %T %s has not been marked for deletion yet", obj, key) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the deletion timestamp on object %T %s", obj, key) +} + +// removeTestFinalizer removes the test finalizer from an object (if the object still exists), so that the deletion +// of the object can complete. +func removeTestFinalizer(obj client.Object) { + key := client.ObjectKeyFromObject(obj) + Eventually(func() error { + objCopy := obj.DeepCopyObject().(client.Object) + if err := hubClient.Get(ctx, key, objCopy); err != nil { + if apierrors.IsNotFound(err) { + return nil + } + return fmt.Errorf("failed to retrieve object %T %s: %w", obj, key, err) + } + if !controllerutil.ContainsFinalizer(objCopy, testFinalizer) { + return nil + } + controllerutil.RemoveFinalizer(objCopy, testFinalizer) + return hubClient.Update(ctx, objCopy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the test finalizer from object %T %s", obj, key) +} + +// waitForObjectsToBeCached waits until the cache-backed client (which the controller shares) observes all the given +// objects. +// +// The controller reads placement policies, staged update strategies, placement bindings, and member clusters from +// the cache, and treats some of them being absent as a terminal initialization failure; test cases should call this +// function on their fixtures before creating a staged update run, so that the run is never initialized against a +// cache that has yet to catch up. +func waitForObjectsToBeCached(objs ...client.Object) { + for _, obj := range objs { + Eventually(func() error { + // Read into a copy so that the object passed in is not modified. + objCopy := obj.DeepCopyObject().(client.Object) + return hubClient.Get(ctx, client.ObjectKeyFromObject(obj), objCopy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe object %T %s in the cache", obj, client.ObjectKeyFromObject(obj)) + } +} + +// getStagedUpdateRun retrieves a staged update run (cluster-scoped if the namespace is empty). +func getStagedUpdateRun(runNSName, runName string) (rolloutv1alpha1.StagedUpdateRunAccessor, error) { + run := newStagedUpdateRunObj(runNSName, runName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: runNSName, Name: runName}, run); err != nil { + return nil, fmt.Errorf("failed to retrieve the staged update run: %w", err) + } + return run, nil +} + +// primaryPlacementResourceSnapshotName returns the name of the primary placement resource snapshot of the given +// index for a placement policy. +func primaryPlacementResourceSnapshotName(policy placementv1alpha1.PlacementPolicyAccessor, index int) string { + return uniquename.PrimaryPlacementResourceSnapshotName(policy.GetName(), policy.GetUID(), index) +} + +// secondaryPlacementResourceSnapshotName returns the name of the secondary placement resource snapshot of the given +// index and sub-index for a placement policy. +func secondaryPlacementResourceSnapshotName(policy placementv1alpha1.PlacementPolicyAccessor, index, subIndex int) string { + return uniquename.SecondaryPlacementResourceSnapshotName(policy.GetName(), policy.GetUID(), index, subIndex) +} + +// wantPerClusterStatuses builds the per-cluster statuses (with no conditions) of the member clusters at the given +// indexes, in the given order, along with their placement bindings. +func wantPerClusterStatuses(memberClusterNames, bindingNames []string, idxs ...int) []rolloutv1alpha1.PerClusterStatus { + perClusterStatuses := make([]rolloutv1alpha1.PerClusterStatus, 0, len(idxs)) + for _, idx := range idxs { + perClusterStatuses = append(perClusterStatuses, wantPerClusterStatus(memberClusterNames, bindingNames, idx)) + } + return perClusterStatuses +} + +// wantPerClusterStatus builds the per-cluster status of the member cluster at the given index (along with its +// placement binding), with the given conditions. +func wantPerClusterStatus(memberClusterNames, bindingNames []string, idx int, conds ...metav1.Condition) rolloutv1alpha1.PerClusterStatus { + return rolloutv1alpha1.PerClusterStatus{ + ClusterName: memberClusterNames[idx], + PlacementBindingName: bindingNames[idx], + Conditions: conds, + } +} + +// The builders below return the conditions that the controller sets on staged update runs, with only the type, +// the status, and the reason set; the status checkers fill in the observed generations, and ignore the last +// transition times and the messages. + +// stagedUpdateRunInitializedCond returns the Initialized condition of a successfully initialized staged update run. +func stagedUpdateRunInitializedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeInitialized, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunInitializedCondReasonPreppedResourceSnapshotAndAllStages, + } +} + +// stagedUpdateRunStartedCond returns the Started condition of a staged update run. +func stagedUpdateRunStartedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeStarted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunStartedCondReasonUpdateStarted, + } +} + +// stagedUpdateRunSuspendedCond returns the Suspended condition of a suspended staged update run. +func stagedUpdateRunSuspendedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeSuspended, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunSuspendedCondReasonSuspended, + } +} + +// stagedUpdateRunResumedCond returns the Suspended condition of a staged update run that has been resumed. +func stagedUpdateRunResumedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeSuspended, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunSuspendedCondReasonResumed, + } +} + +// stagedUpdateRunCompletedCond returns the Completed condition of a staged update run with the given reason. +func stagedUpdateRunCompletedCond(reason string) metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: reason, + } +} + +// perStageStartedCond returns the Started condition of a stage. +func perStageStartedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerStageCondTypeStarted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerStageStartedCondReasonUpdateStarted, + } +} + +// perStageCompletedCond returns the Completed condition of a stage with the given reason. +func perStageCompletedCond(reason string) metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: reason, + } +} + +// perClusterStartedCond returns the Started condition of a cluster. +func perClusterStartedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunPerClusterStartedCondReasonUpdateStarted, + } +} + +// perClusterCompletedCond returns the Completed condition of a cluster with the given reason. +func perClusterCompletedCond(reason string) metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, + Status: metav1.ConditionTrue, + Reason: reason, + } +} + +// perClusterUpdateSucceededConds returns the conditions of a cluster that has been successfully updated. +func perClusterUpdateSucceededConds() []metav1.Condition { + return []metav1.Condition{ + perClusterStartedCond(), + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded), + } +} + +// perClusterUpdateFailedConds returns the conditions of a cluster that has failed to be updated after the update +// started. +func perClusterUpdateFailedConds() []metav1.Condition { + return []metav1.Condition{ + perClusterStartedCond(), + perClusterCompletedCond(rolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed), + } +} + +// perStageTaskTimedWaitStartedCond returns the TimedWaitStarted condition of a TimedWait stage task. +func perStageTaskTimedWaitStartedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeTimedWaitStarted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunTaskTimedWaitStartedCondReasonTimerStarted, + } +} + +// perStageTaskWaitTimeElapsedCond returns the WaitTimeElapsed condition of a TimedWait stage task. +func perStageTaskWaitTimeElapsedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeWaitTimeElapsed, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunTaskWaitTimeElapsedCondReasonTimerElapsed, + } +} + +// perStageTaskApprovalRequestCreatedCond returns the ApprovalRequestCreated condition of an Approval stage task whose +// approval request has been created. +func perStageTaskApprovalRequestCreatedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunTaskApprovalRequestCreatedCondReasonCreated, + } +} + +// perStageTaskApprovalRequestApprovedCond returns the ApprovalRequestApproved condition of an Approval stage task +// whose approval request has been approved. +func perStageTaskApprovalRequestApprovedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestApproved, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.StagedUpdateRunTaskApprovalRequestApprovedCondReasonApproved, + } +} + +// perStageTaskApprovalRequestOwnershipMismatchCond returns the ApprovalRequestCreated condition of an Approval stage +// task whose approval request exists but is not owned by the staged update run. +func perStageTaskApprovalRequestOwnershipMismatchCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunTaskApprovalRequestCreatedCondReasonOwnershipMismatch, + } +} + +// approvalRequestApprovedCond returns the Approved condition of an approval request with the given status, as the +// test cases set it. +func approvalRequestApprovedCond(status metav1.ConditionStatus) metav1.Condition { + reason := markedAsApprovedReason + if status != metav1.ConditionTrue { + reason = markedAsRejectedReason + } + return metav1.Condition{ + Type: rolloutv1alpha1.ApprovalRequestCondTypeApproved, + Status: status, + Reason: reason, + } +} + +// approvalRequestApprovalAcceptedCond returns the ApprovalAccepted condition that the controller sets on an approval +// request once it accepts the approval. +func approvalRequestApprovalAcceptedCond() metav1.Condition { + return metav1.Condition{ + Type: rolloutv1alpha1.ApprovalRequestCondTypeApprovalAccepted, + Status: metav1.ConditionTrue, + Reason: rolloutv1alpha1.ApprovalRequestApprovalAcceptedCondReasonAccepted, + } +} + +// wantApprovalTaskStatus builds the status of an Approval stage task with the given approval request name and +// conditions. +func wantApprovalTaskStatus(approvalRequestName string, conds ...metav1.Condition) rolloutv1alpha1.PerStageTaskStatus { + return rolloutv1alpha1.PerStageTaskStatus{ + Type: rolloutv1alpha1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestName, + Conditions: conds, + } +} + +// wantTimedWaitTaskStatus builds the status of a TimedWait stage task with the given wait time and conditions. +func wantTimedWaitTaskStatus(waitTime time.Duration, conds ...metav1.Condition) rolloutv1alpha1.PerStageTaskStatus { + return rolloutv1alpha1.PerStageTaskStatus{ + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: waitTime}, + Conditions: conds, + } +} + +// perStageSucceededConds returns the conditions of a stage that has completed successfully. +func perStageSucceededConds() []metav1.Condition { + return []metav1.Condition{ + perStageStartedCond(), + perStageCompletedCond(rolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded), + } +} + +// wantStagedUpdateRunStatusInitializedAndSuspended builds the status that a staged update run created with the +// suspended flag set is expected to have, once it has been initialized with the given resource snapshot and stages. +func wantStagedUpdateRunStatusInitializedAndSuspended(snapshotName string, stages []rolloutv1alpha1.PerStageStatus) *rolloutv1alpha1.StagedUpdateRunStatus { + return &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{stagedUpdateRunInitializedCond(), stagedUpdateRunSuspendedCond()}, + ResourceSnapshotNameToRollout: snapshotName, + Stages: stages, + } +} + +// setObservedGenerationOnAllConditions sets the observed generation of all the conditions in a staged update run +// status (run-level, per-stage, per-cluster, and per-task ones) to the given generation. +func setObservedGenerationOnAllConditions(status *rolloutv1alpha1.StagedUpdateRunStatus, generation int64) { + setObsGen := func(conds []metav1.Condition) { + for idx := range conds { + conds[idx].ObservedGeneration = generation + } + } + + setObsGen(status.Conditions) + for stageIdx := range status.Stages { + stage := &status.Stages[stageIdx] + setObsGen(stage.Conditions) + for clusterIdx := range stage.Clusters { + setObsGen(stage.Clusters[clusterIdx].Conditions) + } + for taskIdx := range stage.BeforeStageTasks { + setObsGen(stage.BeforeStageTasks[taskIdx].Conditions) + } + for taskIdx := range stage.AfterStageTasks { + setObsGen(stage.AfterStageTasks[taskIdx].Conditions) + } + } +} + +// stagedUpdateRunStatusUpdatedActual verifies that the status of a staged update run matches the given one; the +// observed generations of all the conditions are expected to match the current generation of the run. +func stagedUpdateRunStatusUpdatedActual(runNSName, runName string, wantStatus *rolloutv1alpha1.StagedUpdateRunStatus) func() error { + return stagedUpdateRunStatusMatchesActual(runNSName, runName, wantStatus, true) +} + +// stagedUpdateRunRolloutStatusActual is the same as stagedUpdateRunStatusUpdatedActual, except that it ignores the +// timestamps of the stages, which are set as a rollout progresses; see stagedUpdateRunStagesCompletedInOrderActual +// for checking the timestamps. +func stagedUpdateRunRolloutStatusActual(runNSName, runName string, wantStatus *rolloutv1alpha1.StagedUpdateRunStatus) func() error { + return stagedUpdateRunStatusMatchesActual(runNSName, runName, wantStatus, true, ignoreFieldsPerStageStatusTimestamps) +} + +// stagedUpdateRunRolloutStatusAsIsActual is the same as stagedUpdateRunRolloutStatusActual, except that it compares +// the observed generations of the conditions as given in the expected status (see observedAt), rather than expecting +// all of them to match the current generation of the run; this is useful once the generation of a run has changed +// (e.g., after it is suspended), as the controller refreshes the observed generations of some conditions only. +func stagedUpdateRunRolloutStatusAsIsActual(runNSName, runName string, wantStatus *rolloutv1alpha1.StagedUpdateRunStatus) func() error { + return stagedUpdateRunStatusMatchesActual(runNSName, runName, wantStatus, false, ignoreFieldsPerStageStatusTimestamps) +} + +// observedAt returns a copy of the given condition, observed at the given generation. +func observedAt(generation int64, cond metav1.Condition) metav1.Condition { + cond.ObservedGeneration = generation + return cond +} + +// stagedUpdateRunStatusMatchesActual verifies that the status of a staged update run matches the given one, with +// the given additional comparison options. If setObservedGenerations is true, the observed generations of all the +// conditions are expected to match the current generation of the run; otherwise, they are compared as given. +func stagedUpdateRunStatusMatchesActual( + runNSName, runName string, + wantStatus *rolloutv1alpha1.StagedUpdateRunStatus, + setObservedGenerations bool, + additionalCmpOpts ...cmp.Option, +) func() error { + return func() error { + // Retrieve the staged update run. + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + + // Prepare the expected status; update the conditions with the observed generation if applicable. + wantStatusWithObsGen := wantStatus.DeepCopy() + if setObservedGenerations { + setObservedGenerationOnAllConditions(wantStatusWithObsGen, run.GetGeneration()) + } + + // Check that the staged update run status has been updated as expected. + cmpOpts := append([]cmp.Option{ + ignoreFieldConditionLTTMsg, + cmpopts.SortSlices(lessFuncCondition), + cmpopts.EquateEmpty(), + }, additionalCmpOpts...) + if diff := cmp.Diff(*run.GetStatus(), *wantStatusWithObsGen, cmpOpts...); diff != "" { + return fmt.Errorf("staged update run status diff (-got +want):\n%s", diff) + } + return nil + } +} + +// stagedUpdateRunStageLastedAtLeastActual verifies that a stage of a staged update run has started and completed, +// and lasted at least the given duration, as its timestamps show. +// +// Note that the timestamps have a precision of seconds; as the controller computes elapsed wait times against +// persisted (i.e., truncated) timestamps as well, a stage that waits for a given duration still lasts at least that +// long by its timestamps. +func stagedUpdateRunStageLastedAtLeastActual(runNSName, runName, stageName string, minDuration time.Duration) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + + for _, stage := range run.GetStatus().Stages { + if stage.StageName != stageName { + continue + } + if stage.StartedTimestamp == nil || stage.CompletedTimestamp == nil { + return fmt.Errorf("stage %s has not started or has not completed yet (started: %v, completed: %v)", + stageName, stage.StartedTimestamp, stage.CompletedTimestamp) + } + if lasted := stage.CompletedTimestamp.Sub(stage.StartedTimestamp.Time); lasted < minDuration { + return fmt.Errorf("stage %s lasted %v, want at least %v", stageName, lasted, minDuration) + } + return nil + } + return fmt.Errorf("stage %s is not found in the staged update run", stageName) + } +} + +// stagedUpdateRunStagesCompletedInOrderActual verifies that all the stages of a staged update run have started and +// completed, one after another, as their timestamps show. +// +// Note that the timestamps have a precision of seconds; a stage might start in the same second as the previous one +// completes. +func stagedUpdateRunStagesCompletedInOrderActual(runNSName, runName string) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + + var prevStageCompletedTimestamp *metav1.Time + for _, stage := range run.GetStatus().Stages { + if stage.StartedTimestamp == nil || stage.CompletedTimestamp == nil { + return fmt.Errorf("stage %s has not started or has not completed yet (started: %v, completed: %v)", + stage.StageName, stage.StartedTimestamp, stage.CompletedTimestamp) + } + if stage.CompletedTimestamp.Before(stage.StartedTimestamp) { + return fmt.Errorf("stage %s completed (%v) before it started (%v)", + stage.StageName, stage.CompletedTimestamp, stage.StartedTimestamp) + } + if prevStageCompletedTimestamp != nil && stage.StartedTimestamp.Before(prevStageCompletedTimestamp) { + return fmt.Errorf("stage %s started (%v) before the previous stage completed (%v)", + stage.StageName, stage.StartedTimestamp, prevStageCompletedTimestamp) + } + prevStageCompletedTimestamp = stage.CompletedTimestamp + } + return nil + } +} + +// stagedUpdateRunFailedToInitializeActual verifies that a staged update run has failed to initialize, i.e., its status +// has only the Initialized condition set to False, and the message of the condition contains the given substring +// (so that test cases can tell the causes of failures apart). +func stagedUpdateRunFailedToInitializeActual(runNSName, runName, wantMsgSubstr string) func() error { + wantStatus := &rolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + { + Type: rolloutv1alpha1.StagedUpdateRunCondTypeInitialized, + Status: metav1.ConditionFalse, + Reason: rolloutv1alpha1.StagedUpdateRunInitializedCondReasonFailed, + }, + }, + } + statusUpdatedActual := stagedUpdateRunStatusUpdatedActual(runNSName, runName, wantStatus) + + return func() error { + if err := statusUpdatedActual(); err != nil { + return err + } + + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + initCond := meta.FindStatusCondition(run.GetStatus().Conditions, rolloutv1alpha1.StagedUpdateRunCondTypeInitialized) + if initCond == nil || !strings.Contains(initCond.Message, wantMsgSubstr) { + return fmt.Errorf("the Initialized condition (%+v) does not have a message that contains %q", initCond, wantMsgSubstr) + } + return nil + } +} + +// stagedUpdateRunConditionActual verifies that a staged update run has the given run-level condition, observed at +// the current generation of the run. +func stagedUpdateRunConditionActual(runNSName, runName string, wantCond metav1.Condition) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + + wantCondWithObsGen := wantCond.DeepCopy() + wantCondWithObsGen.ObservedGeneration = run.GetGeneration() + gotCond := meta.FindStatusCondition(run.GetStatus().Conditions, wantCond.Type) + if diff := cmp.Diff(gotCond, wantCondWithObsGen, ignoreFieldConditionLTTMsg); diff != "" { + return fmt.Errorf("staged update run condition %s diff (-got +want):\n%s", wantCond.Type, diff) + } + return nil + } +} + +// stagedUpdateRunStageLayoutActual verifies the layout of the stages in a staged update run, i.e., the stages, their +// clusters and tasks, and the resolved max concurrency; the progress information (conditions, timestamps, and +// approval request names) is ignored. +func stagedUpdateRunStageLayoutActual(runNSName, runName string, wantStages []rolloutv1alpha1.PerStageStatus) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + + if diff := cmp.Diff( + run.GetStatus().Stages, wantStages, + ignoreFieldsPerStageStatusProgress, + ignoreFieldsPerClusterStatusProgress, + ignoreFieldsPerStageTaskProgress, + cmpopts.EquateEmpty(), + ); diff != "" { + return fmt.Errorf("staged update run stages diff (-got +want):\n%s", diff) + } + return nil + } +} + +// stagedUpdateRunCleanupFinalizerRemovedActual verifies that a staged update run does not have the cleanup finalizer. +func stagedUpdateRunCleanupFinalizerRemovedActual(runNSName, runName string) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + if controllerutil.ContainsFinalizer(run, stagedUpdateRunCleanupFinalizer) { + return fmt.Errorf("the cleanup finalizer has not been removed yet") + } + return nil + } +} + +// stagedUpdateRunCleanupFinalizerAddedActual verifies that a staged update run has the cleanup finalizer. +func stagedUpdateRunCleanupFinalizerAddedActual(runNSName, runName string) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + if !controllerutil.ContainsFinalizer(run, stagedUpdateRunCleanupFinalizer) { + return fmt.Errorf("the cleanup finalizer has not been added yet") + } + return nil + } +} + +// wantStagedUpdateRunObjectRef returns the reference to a staged update run (cluster-scoped if the namespace is +// empty) that the controller registers in the binding manager claim of a placement policy. +func wantStagedUpdateRunObjectRef(runNSName, runName string) placementv1alpha1.ObjectReference { + return placementv1alpha1.ObjectReference{ + APIGroup: rolloutv1alpha1.GroupVersion.Group, + APIVersion: rolloutv1alpha1.GroupVersion.Version, + Kind: stagedUpdateRunKind(runNSName), + Namespace: runNSName, + Name: runName, + } +} + +// bindingManagerRoleHeldActual verifies that the staged update run controller holds the binding manager role of a +// placement policy, on behalf of exactly the given staged update runs (in any order). +func bindingManagerRoleHeldActual(policyNSName, policyName string, wantObjectRefs []placementv1alpha1.ObjectReference) func() error { + return bindingManagerActual(policyNSName, policyName, &placementv1alpha1.BindingManager{ + ControllerName: controllerName, + ObjectRefs: wantObjectRefs, + }) +} + +// bindingManagerActual verifies that the binding manager of a placement policy matches the given one (nil if no one +// holds the role); the object references may come in any order. +func bindingManagerActual(policyNSName, policyName string, wantBindingManager *placementv1alpha1.BindingManager) func() error { + return func() error { + policy := newPlacementPolicyObj(policyNSName, policyName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: policyNSName, Name: policyName}, policy); err != nil { + return fmt.Errorf("failed to retrieve the placement policy: %w", err) + } + + if diff := cmp.Diff( + policy.GetStatus().BindingManager, wantBindingManager, + cmpopts.SortSlices(lessFuncObjectRef), + ); diff != "" { + return fmt.Errorf("binding manager diff (-got +want):\n%s", diff) + } + return nil + } +} + +// updatePlacementPolicyBindingManager sets the binding manager of a placement policy to the given one (nil to clear +// it), e.g., to simulate another controller holding (or releasing) the role. +// +// Note that the function does not wait for the cache-backed client (which the controller shares) to observe the +// change, as the controller may act on it right away (e.g., claim a released role). +func updatePlacementPolicyBindingManager(policyNSName, policyName string, bindingManager *placementv1alpha1.BindingManager) { + Eventually(func() error { + policy := newPlacementPolicyObj(policyNSName, policyName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: policyNSName, Name: policyName}, policy); err != nil { + return fmt.Errorf("failed to retrieve the placement policy: %w", err) + } + policy.GetStatus().BindingManager = bindingManager.DeepCopy() + return hubClient.Status().Update(ctx, policy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the binding manager of the placement policy") +} + +// wantStagedUpdateRunOwnerRefs returns the owner references that the controller sets on the objects (e.g., approval +// requests) that a staged update run (cluster-scoped if the namespace is empty) owns. +func wantStagedUpdateRunOwnerRefs(runNSName, runName string, runUID types.UID) []metav1.OwnerReference { + return []metav1.OwnerReference{ + { + APIVersion: rolloutv1alpha1.GroupVersion.String(), + Kind: stagedUpdateRunKind(runNSName), + Name: runName, + UID: runUID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } +} + +// stagedUpdateRunConditionMessageActual verifies that the run-level condition of the given type of a staged update +// run has a message that contains the given substring. +func stagedUpdateRunConditionMessageActual(runNSName, runName, condType, wantMsgSubstr string) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + cond := meta.FindStatusCondition(run.GetStatus().Conditions, condType) + if cond == nil || !strings.Contains(cond.Message, wantMsgSubstr) { + return fmt.Errorf("the %s condition (%+v) does not have a message that contains %q", condType, cond, wantMsgSubstr) + } + return nil + } +} + +// bindingManagerRoleReleasedActual verifies that no one holds the binding manager role of a placement policy. +func bindingManagerRoleReleasedActual(policyNSName, policyName string) func() error { + return func() error { + policy := newPlacementPolicyObj(policyNSName, policyName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: policyNSName, Name: policyName}, policy); err != nil { + return fmt.Errorf("failed to retrieve the placement policy: %w", err) + } + if bindingManager := policy.GetStatus().BindingManager; bindingManager != nil { + return fmt.Errorf("the binding manager role is still held: %+v", *bindingManager) + } + return nil + } +} + +// placementBindingResourceSnapshotActual verifies that a placement binding points to the given resource snapshot. +func placementBindingResourceSnapshotActual(bindingNS, bindingName, wantSnapshotName string) func() error { + return func() error { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); err != nil { + return fmt.Errorf("failed to retrieve the placement binding: %w", err) + } + if gotSnapshotName := binding.GetSpec().ResourceSnapshotName; gotSnapshotName != wantSnapshotName { + return fmt.Errorf("placement binding %s points to resource snapshot %s, want %s", bindingName, gotSnapshotName, wantSnapshotName) + } + return nil + } +} + +// placementBindingsResourceSnapshotActual verifies that all the given placement bindings point to the given resource +// snapshot. +func placementBindingsResourceSnapshotActual(bindingNS string, bindingNames []string, wantSnapshotName string) func() error { + return func() error { + for _, bindingName := range bindingNames { + if err := placementBindingResourceSnapshotActual(bindingNS, bindingName, wantSnapshotName)(); err != nil { + return err + } + } + return nil + } +} + +// placementBindingGenerationActual verifies that a placement binding is at the given generation, e.g., that its +// spec has not been updated since it was created. +func placementBindingGenerationActual(bindingNS, bindingName string, wantGeneration int64) func() error { + return func() error { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); err != nil { + return fmt.Errorf("failed to retrieve the placement binding: %w", err) + } + if gotGeneration := binding.GetGeneration(); gotGeneration != wantGeneration { + return fmt.Errorf("placement binding %s is at generation %d, want %d", bindingName, gotGeneration, wantGeneration) + } + return nil + } +} + +// getPerClusterCondition retrieves the condition of the given type (nil if absent) of a cluster in a staged update run. +func getPerClusterCondition(runNSName, runName, clusterName, condType string) (*metav1.Condition, error) { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return nil, err + } + for _, stage := range run.GetStatus().Stages { + for _, cluster := range stage.Clusters { + if cluster.ClusterName == clusterName { + return meta.FindStatusCondition(cluster.Conditions, condType), nil + } + } + } + return nil, fmt.Errorf("cluster %s is not found in the staged update run", clusterName) +} + +// perClusterConditionMessageActual verifies that the condition of the given type of a cluster in a staged update run +// has a message that contains the given substring (so that test cases can tell the causes of outcomes apart). +func perClusterConditionMessageActual(runNSName, runName, clusterName, condType, wantMsgSubstr string) func() error { + return func() error { + cond, err := getPerClusterCondition(runNSName, runName, clusterName, condType) + if err != nil { + return err + } + if cond == nil || !strings.Contains(cond.Message, wantMsgSubstr) { + return fmt.Errorf("the %s condition (%+v) of cluster %s does not have a message that contains %q", condType, cond, clusterName, wantMsgSubstr) + } + return nil + } +} + +// perStageConditionMessageActual verifies that the condition of the given type of a stage in a staged update run +// has a message that contains the given substring. +func perStageConditionMessageActual(runNSName, runName, stageName, condType, wantMsgSubstr string) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + for _, stage := range run.GetStatus().Stages { + if stage.StageName != stageName { + continue + } + cond := meta.FindStatusCondition(stage.Conditions, condType) + if cond == nil || !strings.Contains(cond.Message, wantMsgSubstr) { + return fmt.Errorf("the %s condition (%+v) of stage %s does not have a message that contains %q", condType, cond, stageName, wantMsgSubstr) + } + return nil + } + return fmt.Errorf("stage %s is not found in the staged update run", stageName) + } +} + +// approvalRequestCreatedActual verifies that an approval request has been created for the given stage of a staged +// update run, i.e., its spec references the run and the stage, and the run is its controller (owner). +// +// The approval request is in the namespace of the staged update run (i.e., it is a ClusterApprovalRequest object if +// the staged update run is cluster-scoped). +func approvalRequestCreatedActual(runNSName, runName, requestName, wantStageName string) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + + request := newApprovalRequestObj(runNSName, requestName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: runNSName, Name: requestName}, request); err != nil { + return fmt.Errorf("failed to retrieve the approval request: %w", err) + } + + wantSpec := rolloutv1alpha1.ApprovalRequestSpec{ + StagedUpdateRunName: runName, + StageName: wantStageName, + } + if diff := cmp.Diff(*request.GetSpec(), wantSpec); diff != "" { + return fmt.Errorf("approval request spec diff (-got +want):\n%s", diff) + } + + wantOwnerRefs := wantStagedUpdateRunOwnerRefs(runNSName, runName, run.GetUID()) + if diff := cmp.Diff(request.GetOwnerReferences(), wantOwnerRefs); diff != "" { + return fmt.Errorf("approval request owner references diff (-got +want):\n%s", diff) + } + return nil + } +} + +// approvalRequestOwnerRefsActual verifies that an approval request (cluster-scoped if the namespace is empty) exists +// with exactly the given owner references. +func approvalRequestOwnerRefsActual(requestNS, requestName string, wantOwnerRefs []metav1.OwnerReference) func() error { + return func() error { + request, err := getApprovalRequest(requestNS, requestName) + if err != nil { + return err + } + if diff := cmp.Diff(request.GetOwnerReferences(), wantOwnerRefs); diff != "" { + return fmt.Errorf("approval request owner references diff (-got +want):\n%s", diff) + } + return nil + } +} + +// approvalRequestApprovalAcceptedActual verifies that the controller has accepted the approval of an approval request +// (cluster-scoped if the namespace is empty), i.e., the request has both the Approved condition (as set by the test +// case) and the ApprovalAccepted condition set to True, observed at its current generation. +func approvalRequestApprovalAcceptedActual(requestNS, requestName string) func() error { + return approvalRequestConditionsActual(requestNS, requestName, []metav1.Condition{ + approvalRequestApprovedCond(metav1.ConditionTrue), + approvalRequestApprovalAcceptedCond(), + }) +} + +// approvalRequestConditionsActual verifies that an approval request (cluster-scoped if the namespace is empty) has +// exactly the given conditions, observed at its current generation. +func approvalRequestConditionsActual(requestNS, requestName string, wantConds []metav1.Condition) func() error { + return func() error { + request, err := getApprovalRequest(requestNS, requestName) + if err != nil { + return err + } + + wantCondsWithObsGen := make([]metav1.Condition, 0, len(wantConds)) + for _, cond := range wantConds { + cond.ObservedGeneration = request.GetGeneration() + wantCondsWithObsGen = append(wantCondsWithObsGen, cond) + } + if diff := cmp.Diff( + request.GetStatus().Conditions, wantCondsWithObsGen, + ignoreFieldConditionLTTMsg, + cmpopts.SortSlices(lessFuncCondition), + cmpopts.EquateEmpty(), + ); diff != "" { + return fmt.Errorf("approval request conditions diff (-got +want):\n%s", diff) + } + return nil + } +} + +// approvalRequestAbsentActual verifies that an approval request (cluster-scoped if the namespace is empty) does not +// exist. +func approvalRequestAbsentActual(requestNS, requestName string) func() error { + return func() error { + request := newApprovalRequestObj(requestNS, requestName) + err := hubClient.Get(ctx, client.ObjectKey{Namespace: requestNS, Name: requestName}, request) + switch { + case err == nil: + return fmt.Errorf("approval request %s still exists", requestName) + case !apierrors.IsNotFound(err): + return fmt.Errorf("failed to retrieve approval request %s: %w", requestName, err) + } + return nil + } +} + +// approvalRequestReplacedActual verifies that an approval request (cluster-scoped if the namespace is empty) exists, +// and is not the object of the given UID, i.e., it has been deleted and re-created. +func approvalRequestReplacedActual(requestNS, requestName string, oldUID types.UID) func() error { + return func() error { + request, err := getApprovalRequest(requestNS, requestName) + if err != nil { + return err + } + if request.GetUID() == oldUID { + return fmt.Errorf("approval request %s has not been replaced (UID: %s)", requestName, oldUID) + } + return nil + } +} + +// perStageTaskTimedWaitLastedAtLeastActual verifies that the after-stage TimedWait task of a stage in a staged update +// run has started and elapsed, at least the given duration apart (as the last transition times of its conditions +// show), and that the stage has completed no earlier than the wait time elapsed. +func perStageTaskTimedWaitLastedAtLeastActual(runNSName, runName, stageName string, minDuration time.Duration) func() error { + return func() error { + run, err := getStagedUpdateRun(runNSName, runName) + if err != nil { + return err + } + + for _, stage := range run.GetStatus().Stages { + if stage.StageName != stageName { + continue + } + for _, task := range stage.AfterStageTasks { + if task.Type != rolloutv1alpha1.StageTaskTypeTimedWait { + continue + } + startedCond := meta.FindStatusCondition(task.Conditions, rolloutv1alpha1.StagedUpdateRunTaskCondTypeTimedWaitStarted) + elapsedCond := meta.FindStatusCondition(task.Conditions, rolloutv1alpha1.StagedUpdateRunTaskCondTypeWaitTimeElapsed) + if startedCond == nil || elapsedCond == nil { + return fmt.Errorf("the timed wait task of stage %s has not started or its wait time has not elapsed yet (started: %+v, elapsed: %+v)", + stageName, startedCond, elapsedCond) + } + if waited := elapsedCond.LastTransitionTime.Sub(startedCond.LastTransitionTime.Time); waited < minDuration { + return fmt.Errorf("the timed wait task of stage %s waited for %v, want at least %v", stageName, waited, minDuration) + } + if stage.CompletedTimestamp == nil || stage.CompletedTimestamp.Before(&elapsedCond.LastTransitionTime) { + return fmt.Errorf("stage %s has not completed after the wait time elapsed (completed: %v, elapsed: %v)", + stageName, stage.CompletedTimestamp, elapsedCond.LastTransitionTime) + } + return nil + } + return fmt.Errorf("stage %s has no after-stage timed wait task", stageName) + } + return fmt.Errorf("stage %s is not found in the staged update run", stageName) + } +} + +// primaryPlacementResourceSnapshotCreatedActual verifies that a primary placement resource snapshot of the given +// index has been created for a placement policy, as the only snapshot of its index, and is owned by the policy. +func primaryPlacementResourceSnapshotCreatedActual(policy placementv1alpha1.PlacementPolicyAccessor, index int) func() error { + return func() error { + // Retrieve the placement resource snapshot. + snapshotName := primaryPlacementResourceSnapshotName(policy, index) + snapshot := newPlacementResourceSnapshotObj(policy.GetNamespace(), snapshotName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: policy.GetNamespace(), Name: snapshotName}, snapshot); err != nil { + return fmt.Errorf("failed to retrieve the placement resource snapshot: %w", err) + } + + // Check the labels. + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: policy.GetName(), + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: fmt.Sprintf("%d", index), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(snapshot.GetLabels(), wantLabels); diff != "" { + return fmt.Errorf("placement resource snapshot labels diff (-got +want):\n%s", diff) + } + + // Check the owner references. + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementPolicyKind(policy.GetNamespace()), + Name: policy.GetName(), + UID: policy.GetUID(), + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + if diff := cmp.Diff(snapshot.GetOwnerReferences(), wantOwnerRefs); diff != "" { + return fmt.Errorf("placement resource snapshot owner references diff (-got +want):\n%s", diff) + } + return nil + } +} + +// objectRemovedActual verifies that an object is gone. +func objectRemovedActual(obj client.Object) func() error { + return func() error { + objCopy := obj.DeepCopyObject().(client.Object) + err := hubClient.Get(ctx, client.ObjectKeyFromObject(obj), objCopy) + switch { + case err == nil: + return fmt.Errorf("object %T %s still exists", obj, client.ObjectKeyFromObject(obj)) + case !apierrors.IsNotFound(err): + return fmt.Errorf("failed to retrieve object %T %s: %w", obj, client.ObjectKeyFromObject(obj), err) + } + return nil + } +} + +// removeObjects deletes the given objects, and waits until all of them are gone. +// +// The envtest environment does not run the garbage collector; test cases must remove all the objects they create +// (including those created by the controller on their behalf) explicitly. +// +// Approval requests are removed last, only after all the other objects (staged update runs in particular) are gone: +// the deletion of an approval request makes the controller process the staged update run that the request +// references, and a run that is still around (as far as the cache is concerned) might have its approval request +// re-created, which would then never go away. +func removeObjects(objs ...client.Object) { + approvalRequests := make([]client.Object, 0, len(objs)) + others := make([]client.Object, 0, len(objs)) + for _, obj := range objs { + switch obj.(type) { + case *rolloutv1alpha1.ApprovalRequest, *rolloutv1alpha1.ClusterApprovalRequest: + approvalRequests = append(approvalRequests, obj) + default: + others = append(others, obj) + } + } + + deleteObjectsAndWaitForRemoval(others) + deleteObjectsAndWaitForRemoval(approvalRequests) +} + +// deleteObjectsAndWaitForRemoval deletes the given objects, and waits until all of them are gone. All the deletions +// are issued before any wait, so that the cleanup does not take one polling interval per object. +func deleteObjectsAndWaitForRemoval(objs []client.Object) { + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T %s", obj, client.ObjectKeyFromObject(obj)) + } + for _, obj := range objs { + Eventually(objectRemovedActual(obj), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove object %T %s", obj, client.ObjectKeyFromObject(obj)) + } +} diff --git a/pkg/v1/controllers/workapplier/apply.go b/pkg/v1/controllers/workapplier/apply.go new file mode 100644 index 000000000..e9d066a3d --- /dev/null +++ b/pkg/v1/controllers/workapplier/apply.go @@ -0,0 +1,674 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/validation" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/jsonmergepatch" + "k8s.io/apimachinery/pkg/util/mergepatch" + "k8s.io/apimachinery/pkg/util/strategicpatch" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + "k8s.io/klog/v2" + "k8s.io/kubectl/pkg/util/deployment" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/utils/resource" + "go.goms.io/fleet/pkg/v1/utils/ownerreferences" +) + +var builtInScheme = runtime.NewScheme() + +func init() { + // This is a trick that allows KubeFleet to check if a resource is a K8s built-in one. + _ = clientgoscheme.AddToScheme(builtInScheme) +} + +// applyInDryRunMode dry-runs an apply op. +func (r *Reconciler) applyInDryRunMode( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, +) (*unstructured.Unstructured, error) { + // In this method, KubeFleet will always use forced server-side apply + // w/o optimistic lock for diff calculation. + // + // This is OK as partial comparison concerns only fields that are currently present + // in the manifest object, and KubeFleet will clear out system managed and read-only fields + // before the comparison. + // + // Note that full comparison can be carried out directly without involving the apply op. + return r.serverSideApply(ctx, gvr, manifestObj, inMemberClusterObj, true, false, true) +} + +func (r *Reconciler) apply( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (*unstructured.Unstructured, error) { + gvr := manifestProcessingState.gvr + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + expectedAppliedWorkOwnerRef := manifestProcessingState.ownedBy + + // Create a sanitized copy of the manifest object. + // + // TO-DO (chenyu1): this processing step can be dropped if the KubeFleet hub agent can guarantee that all manifest + // objects have already been sanitized. + manifestObjCopy := sanitizeManifestObject(manifestProcessingState.manifestObj) + + // Compute the hash of the manifest object. + // + // Originally the manifest hash is kept only if three-way merge patch (client side apply) + // is used; with the new drift detection and takeover capabilities, the manifest hash + // will always be kept regardless of the apply method in use, as it is needed for + // drift detection purposes. + // + // Note that certain fields have been removed from the manifest object in the hash computation + // process. + if err := setLastAppliedManifestHashAnnotation(manifestObjCopy); err != nil { + return nil, errors.Wraps(err, "failed to set manifest hash annotation") + } + + // Validate owner references. + // + // As previously mentioned, with the new capabilities, at this point of the workflow, + // KubeFleet has been added as an owner for the object. Still, to guard against cases where + // co-ownership is turned on then off or the addition of new owner references + // in the manifest, KubeFleet will still perform a validation round. + if err := validateOwnerRefs(manifestProcessingState); err != nil { + return nil, errors.Wraps(err, "failed to validate owner references") + } + + // Add the owner reference information. + ownerreferences.Own(manifestObjCopy, expectedAppliedWorkOwnerRef) + + // If three-way merge patch is used, set the KubeFleet-specific last applied annotation. + // Note that this op might not complete due to the last applied annotation being too large; + // this is not recognized as an error and KubeFleet will switch to server-side apply instead. + isLastAppliedAnnotationSet := false + if syncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodClientSideApply { + var err error + isLastAppliedAnnotationSet, err = setLastAppliedConfigAnnotation(manifestObjCopy) + if err != nil { + return nil, errors.Wraps(err, "failed to set last applied configuration annotation") + } + + // Note that KubeFleet might choose to skip the last applied annotation due to size limits + // even if no error has occurred. + klog.V(2).InfoS("Completed the last applied annotation setting process", "isSet", isLastAppliedAnnotationSet, + "GVR", *gvr, "manifestObj", klog.KObj(manifestObjCopy)) + } + + // Create the object if it does not exist in the member cluster. + if inMemberClusterObj == nil { + shouldCreateNS, err := r.shouldCreateNamespaceFor(ctx, manifestObjCopy, syncStrategy) + if err != nil { + return nil, errors.Wraps(err, "failed to determine if the namespace should be created") + } + if shouldCreateNS { + if err := r.createNamespaceFor(ctx, manifestObjCopy); err != nil { + return nil, errors.Wraps(err, "failed to create namespace for the manifest object") + } + } + return r.createManifestObject(ctx, gvr, manifestObjCopy) + } + + // Run the apply op. Note that KubeFleet will always attempt to apply the manifest, even if + // the manifest object hash does not change. + + // Optimistic lock is enabled when the sync strategy dictates that a drift should be reported + // rather than overwritten (i.e., the WhenDrifted field is set to ReportError); this helps + // KubeFleet guard against cases where inadvertent changes are being made in an untimely manner + // (i.e., changes are made when the KubeFleet agent is preparing an apply op). + // + // Note that if the sync strategy dictates that drifts should be overwritten (i.e., + // the WhenDrifted field is set to ApplyAnyway), KubeFleet will not enable optimistic lock. This + // is consistent with the behavior before the drift detection and takeover experience + // is added. + isOptimisticLockEnabled := shouldEnableOptimisticLock(syncStrategy) + + switch { + case syncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodClientSideApply && isLastAppliedAnnotationSet: + // The sync strategy dictates that three-way merge patch + // (client-side apply) should be used, and the last applied annotation + // has been set. + klog.V(2).InfoS("Using three-way merge patch to apply the manifest object", + "GVR", *gvr, "manifestObj", klog.KObj(manifestObjCopy)) + return r.threeWayMergePatch(ctx, gvr, manifestObjCopy, inMemberClusterObj, isOptimisticLockEnabled, false) + case syncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodClientSideApply: + // The sync strategy dictates that three-way merge patch + // (client-side apply) should be used, but the last applied annotation + // cannot be set. KubeFleet will fall back to server-side apply. + klog.V(2).InfoS("Falling back to server-side apply as the last applied annotation cannot be set", + "GVR", *gvr, "manifestObj", klog.KObj(manifestObjCopy)) + return r.serverSideApply( + ctx, + gvr, manifestObjCopy, inMemberClusterObj, + // When falling back to SSA, always disable force apply ops (this is also the default + // behavior). + // + // Note that the work applier might still enable force apply ops if it finds that + // self-conflicts might occur. + false, isOptimisticLockEnabled, false, + ) + case syncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodServerSideApply: + // The sync strategy dictates that server-side apply should be used. + // + // The server-side apply options are not defaulted by the API server when absent. + forceConflicts := syncStrategy.ServerSideApplyOptions != nil && syncStrategy.ServerSideApplyOptions.ForceConflicts + klog.V(2).InfoS("Using server-side apply to apply the manifest object", + "GVR", *gvr, "manifestObj", klog.KObj(manifestObjCopy)) + return r.serverSideApply( + ctx, + gvr, manifestObjCopy, inMemberClusterObj, + forceConflicts, isOptimisticLockEnabled, false, + ) + default: + // An unexpected apply method has been set. Normally this will never run as the built-in + // validation would block invalid values. + return nil, errors.NewUnexpectedError(nil, "an unexpected apply method is found", "applyMethod", syncStrategy.ApplyMethod) + } +} + +// shouldEnableOptimisticLock checks if optimistic lock should be enabled given a sync strategy. +func shouldEnableOptimisticLock(syncStrategy *placementv1alpha1.SyncStrategy) bool { + // Optimistic lock is enabled if drifts are to be reported rather than overwritten. + return syncStrategy.WhenDrifted == placementv1alpha1.WhenDriftedOptionReportError +} + +// sanitizeManifestObject sanitizes the manifest object before applying it. +// +// The sanitization logic here is consistent with that of the CRP controller, sans the API server +// specific parts; see also the generateRawContent function in the respective controller. +// +// Note that this function returns a copy of the manifest object; the original object will be left +// untouched. +func sanitizeManifestObject(manifestObj *unstructured.Unstructured) *unstructured.Unstructured { + // Create a deep copy of the object. + manifestObjCopy := manifestObj.DeepCopy() + + // Remove certain labels and annotations. + if annotations := manifestObjCopy.GetAnnotations(); annotations != nil { + // Remove the two KubeFleet reserved annotations. This is normally not set by users. + delete(annotations, placementv1alpha1.LastAppliedManifestHashAnnotationKey) + delete(annotations, placementv1alpha1.LastAppliedConfigAnnotationKey) + + // Remove the last applied configuration set by kubectl. + delete(annotations, corev1.LastAppliedConfigAnnotation) + + // Remove the revision annotation set by deployment controller. + delete(annotations, deployment.RevisionAnnotation) + + if len(annotations) == 0 { + manifestObjCopy.SetAnnotations(nil) + } else { + manifestObjCopy.SetAnnotations(annotations) + } + } + + // Remove certain system-managed fields. + manifestObjCopy.SetOwnerReferences(nil) + manifestObjCopy.SetManagedFields(nil) + + // Remove the read-only fields. + manifestObjCopy.SetCreationTimestamp(metav1.Time{}) + manifestObjCopy.SetDeletionTimestamp(nil) + manifestObjCopy.SetDeletionGracePeriodSeconds(nil) + manifestObjCopy.SetGeneration(0) + manifestObjCopy.SetResourceVersion("") + manifestObjCopy.SetSelfLink("") + manifestObjCopy.SetUID("") + + // Remove the status field. + unstructured.RemoveNestedField(manifestObjCopy.Object, "status") + + // Note: in the KubeFleet hub agent logic, the system also handles the Service and Job objects + // in a special way, so as to remove certain fields that are set by the hub cluster API + // server automatically; for the KubeFleet member agent logic here, however, KubeFleet assumes + // that if these fields are set, users must have set them on purpose, and they should not + // be removed. The difference comes to the fact that the KubeFleet member agent sanitization + // logic concerns only the enveloped objects, which are free from any hub cluster API + // server manipulation anyway. + + return manifestObjCopy +} + +// setLastAppliedManifestHashAnnotation computes the hash of the provided manifest and sets an annotation of the +// hash on the provided unstructured object. +func setLastAppliedManifestHashAnnotation(manifestObj *unstructured.Unstructured) error { + cleanedManifestObj := discardFieldsIrrelevantInComparisonFrom(manifestObj) + manifestObjHash, err := resource.HashOf(cleanedManifestObj.Object) + if err != nil { + return errors.NewUnexpectedError(err, "failed to compute the hash of the manifest object") + } + + annotations := manifestObj.GetAnnotations() + if annotations == nil { + annotations = map[string]string{} + } + annotations[placementv1alpha1.LastAppliedManifestHashAnnotationKey] = manifestObjHash + manifestObj.SetAnnotations(annotations) + return nil +} + +// validateOwnerRefs validates the owner references of an applied manifest, checking +// if an apply op can be performed on the object. +func validateOwnerRefs( + manifestProcessingState *manifestProcessingState, +) error { + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + expectedAppliedWorkOwnerRef := manifestProcessingState.ownedBy + isCoOwnershipAllowed := syncStrategy.WhenOwnedByOthers == placementv1alpha1.WhenOwnedByOthersOptionShareOwnership + + manifestObjOwnerRefs := manifestProcessingState.manifestObj.GetOwnerReferences() + + // If the manifest object already features some owner reference(s), but co-ownership is + // disallowed, the validation fails. + // + // This is just a sanity check; normally the branch will never get triggered as KubeFleet would + // perform sanitization on the manifest object before applying it, which removes all owner + // references. + if len(manifestObjOwnerRefs) > 0 && !isCoOwnershipAllowed { + return errors.NewUnexpectedError(nil, "manifest is set to have owner references but co-ownership is disallowed") + } + + // Do a sanity check to verify that no AppliedWork object is directly added as an owner + // in the manifest object. Normally the branch will never get triggered as KubeFleet would + // perform sanitization on the manifest object before applying it, which removes all owner + // references. + for idx := range manifestObjOwnerRefs { + ownerRef := &manifestObjOwnerRefs[idx] + if ownerRef.APIVersion == placementv1alpha1.GroupVersion.String() && ownerRef.Kind == placementv1alpha1.AppliedWorkKind { + return errors.NewUnexpectedError(nil, "an appliedWork object is unexpectedly added as an owner in the manifest object") + } + } + + if inMemberClusterObj == nil { + // The manifest object has never been applied yet; no need to do further validation. + return nil + } + inMemberClusterObjOwnerRefs := inMemberClusterObj.GetOwnerReferences() + + // If the live object is co-owned but co-ownership is no longer allowed, the validation fails. + if len(inMemberClusterObjOwnerRefs) > 1 && !isCoOwnershipAllowed { + return errors.NewUserError(nil, "object is co-owned by multiple objects but co-ownership has been disallowed", + "observedOwners", inMemberClusterObjOwnerRefs) + } + + // Note that at this point of execution, one of the owner references is guaranteed to be the + // expected AppliedWork object. For safety reasons, KubeFleet will still do a sanity check. + found := false + for idx := range inMemberClusterObjOwnerRefs { + if ownerreferences.AreEqual(&inMemberClusterObjOwnerRefs[idx], expectedAppliedWorkOwnerRef) { + found = true + break + } + } + if !found { + return errors.NewUnexpectedError(nil, "object is not owned by the expected appliedWork object") + } + + // If the object is already owned by another appliedWork object, the validation fails. + // + // Normally this branch will never get executed as KubeFleet would refuse to take over an object + // that has been owned by another appliedWork object. + if isPlacedByKubeFleetInDuplicate(inMemberClusterObjOwnerRefs, expectedAppliedWorkOwnerRef) { + return errors.NewUnexpectedError(nil, "object is already owned by another appliedWork object") + } + + return nil +} + +// setLastAppliedConfigAnnotation sets the last applied annotation on the provided manifest object. +func setLastAppliedConfigAnnotation(manifestObj *unstructured.Unstructured) (bool, error) { + annotations := manifestObj.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string) + } + + // Remove the last applied annotation just in case. + delete(annotations, placementv1alpha1.LastAppliedConfigAnnotationKey) + manifestObj.SetAnnotations(annotations) + + lastAppliedManifestJSONBytes, err := manifestObj.MarshalJSON() + if err != nil { + return false, errors.NewUnexpectedError(err, "failed to marshal the manifest object into JSON") + } + annotations[placementv1alpha1.LastAppliedConfigAnnotationKey] = string(lastAppliedManifestJSONBytes) + isLastAppliedAnnotationSet := true + + if err := validation.ValidateAnnotationsSize(annotations); err != nil { + // If the annotation size exceeds the limit, KubeFleet will set the annotation to an empty string. + annotations[placementv1alpha1.LastAppliedConfigAnnotationKey] = "" + isLastAppliedAnnotationSet = false + } + + manifestObj.SetAnnotations(annotations) + return isLastAppliedAnnotationSet, nil +} + +// shouldCreateNamespaceFor checks if a namespace must be created in the member cluster before the manifest +// object itself can be applied. +func (r *Reconciler) shouldCreateNamespaceFor(ctx context.Context, manifestObj *unstructured.Unstructured, syncStrategy *placementv1alpha1.SyncStrategy) (bool, error) { + if syncStrategy.WhenNamespaceDoesNotExist != placementv1alpha1.WhenNamespaceDoesNotExistOptionCreateNamespace { + // The sync strategy forbids KubeFleet from creating the namespace. + return false, nil + } + + ns := manifestObj.GetNamespace() + if len(ns) == 0 { + // The manifest object is cluster-scoped; no namespace is needed. + return false, nil + } + + err := r.spokeClient.Get(ctx, types.NamespacedName{Name: ns}, &corev1.Namespace{}) + switch { + case err == nil: + return false, nil + case apierrors.IsNotFound(err): + klog.V(2).InfoS("The namespace of the manifest object does not exist in the member cluster", "namespace", ns) + return true, nil + default: + // An unexpected error has occurred. + return false, errors.NewAPIServerError(err, + "failed to check if the namespace of the manifest object exists in the member cluster", true, "namespace", ns) + } +} + +// createNamespaceFor creates the namespace that a manifest object resides in. +// +// The namespace itself is not managed by KubeFleet; it is left behind when the placement is removed. +func (r *Reconciler) createNamespaceFor(ctx context.Context, manifestObj *unstructured.Unstructured) error { + ns := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: manifestObj.GetNamespace(), + }, + } + + err := r.spokeClient.Create(ctx, ns, client.FieldOwner(fieldManagerName)) + switch { + case err == nil: + klog.V(2).InfoS("Created the namespace for the manifest object", + "namespace", ns.Name, "manifestObj", klog.KObj(manifestObj)) + return nil + case apierrors.IsAlreadyExists(err): + // The namespace has been created; this is not considered an error. + klog.V(2).InfoS("The namespace for the manifest object already exists in the member cluster", + "namespace", ns.Name, "manifestObj", klog.KObj(manifestObj)) + return nil + default: + // false as this is a write op. + return errors.NewAPIServerError(err, "failed to create the namespace for the manifest object", false, + "namespace", ns.Name) + } +} + +// createManifestObject creates the manifest object in the member cluster. +func (r *Reconciler) createManifestObject( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObject *unstructured.Unstructured, +) (*unstructured.Unstructured, error) { + createOpts := metav1.CreateOptions{ + FieldManager: fieldManagerName, + } + createdObj, err := r.spokeDynamicClient.Resource(*gvr).Namespace(manifestObject.GetNamespace()).Create(ctx, manifestObject, createOpts) + if err != nil { + // false as the dynamic client is non-caching. + return nil, errors.NewAPIServerError(err, "failed to create the manifest object", false) + } + klog.V(2).InfoS("Created the manifest object", "GVR", *gvr, "manifestObj", klog.KObj(createdObj)) + return createdObj, nil +} + +// threeWayMergePatch uses three-way merge patch to apply the manifest object. +func (r *Reconciler) threeWayMergePatch( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, + optimisticLock, dryRun bool, +) (*unstructured.Unstructured, error) { + // Enable optimistic lock by forcing the resource version field to be added to the + // JSON merge patch. Optimistic lock is always enabled in the dry run mode. + if optimisticLock || dryRun { + curResourceVer := inMemberClusterObj.GetResourceVersion() + if len(curResourceVer) == 0 { + return nil, errors.NewUnexpectedError(nil, "failed to enable optimistic lock: resource version is empty on the object from the member cluster", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + } + + // Add the resource version to the manifest object. + manifestObj.SetResourceVersion(curResourceVer) + + // Remove the resource version from the object in the member cluster. + inMemberClusterObj.SetResourceVersion("") + } + + // Create a three-way merge patch. + patch, err := buildThreeWayMergePatch(manifestObj, inMemberClusterObj) + if err != nil { + return nil, errors.Wraps(err, "failed to create three-way merge patch") + } + data, err := patch.Data(manifestObj) + if err != nil { + // KubeFleet uses raw patch; this branch should never run. + return nil, errors.NewUnexpectedError(err, "failed to get patch data") + } + + // Use three-way merge (similar to kubectl client side apply) to patch the object in the + // member cluster. + // + // This will: + // * Remove fields that are present in the last applied configuration but not in the + // manifest object. + // * Create fields that are present in the manifest object but not in the object from the member cluster. + // * Update fields that are present in both the manifest object and the object from the member cluster. + patchOpts := metav1.PatchOptions{ + FieldManager: fieldManagerName, + } + if dryRun { + patchOpts.DryRun = []string{metav1.DryRunAll} + } + patchedObj, err := r.spokeDynamicClient. + Resource(*gvr).Namespace(manifestObj.GetNamespace()). + Patch(ctx, manifestObj.GetName(), patch.Type(), data, patchOpts) + if err != nil { + // false as the dynamic client is non-caching. + return nil, errors.NewAPIServerError(err, "failed to patch the manifest object", false) + } + return patchedObj, nil +} + +// threeWayMergePatch creates a patch by computing a three-way diff based on +// the manifest object, the live object, and the last applied configuration as kept in +// the annotations. +func buildThreeWayMergePatch(manifestObj, liveObj *unstructured.Unstructured) (client.Patch, error) { + // Marshal the manifest object into JSON bytes. + manifestObjJSONBytes, err := manifestObj.MarshalJSON() + if err != nil { + return nil, err + } + // Marshal the live object into JSON bytes. + liveObjJSONBytes, err := liveObj.MarshalJSON() + if err != nil { + return nil, err + } + // Retrieve the last applied configuration from the annotations. This can be an empty string. + lastAppliedObjJSONBytes := getLastAppliedConfigAnnotation(liveObj) + + var patchType types.PatchType + var patchData []byte + var lookupPatchMeta strategicpatch.LookupPatchMeta + + versionedObject, err := builtInScheme.New(liveObj.GetObjectKind().GroupVersionKind()) + switch { + case runtime.IsNotRegisteredError(err): + // use JSONMergePatch for custom resources + // because StrategicMergePatch doesn't support custom resources + patchType = types.MergePatchType + preconditions := []mergepatch.PreconditionFunc{ + mergepatch.RequireKeyUnchanged("apiVersion"), + mergepatch.RequireKeyUnchanged("kind"), + mergepatch.RequireMetadataKeyUnchanged("name"), + } + patchData, err = jsonmergepatch.CreateThreeWayJSONMergePatch( + lastAppliedObjJSONBytes, manifestObjJSONBytes, liveObjJSONBytes, preconditions...) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to create three-way JSON merge patch") + } + case err != nil: + return nil, err + default: + // use StrategicMergePatch for K8s built-in resources + patchType = types.StrategicMergePatchType + lookupPatchMeta, err = strategicpatch.NewPatchMetaFromStruct(versionedObject) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to create patch meta from struct (strategic merge patch)") + } + patchData, err = strategicpatch.CreateThreeWayMergePatch(lastAppliedObjJSONBytes, manifestObjJSONBytes, liveObjJSONBytes, lookupPatchMeta, true) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to create three-way strategic merge patch") + } + } + return client.RawPatch(patchType, patchData), nil +} + +// getLastAppliedConfigAnnotation returns the last applied annotation of a manifest object. +func getLastAppliedConfigAnnotation(inMemberClusterObj *unstructured.Unstructured) []byte { + annotations := inMemberClusterObj.GetAnnotations() + if annotations == nil { + // The last applied annotation is not found in the live object; normally this should not + // happen, but KubeFleet can still handle this situation. + klog.V(2).InfoS("No annotations in the live object", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return nil + } + + lastAppliedManifestJSONStr, found := annotations[placementv1alpha1.LastAppliedConfigAnnotationKey] + if !found { + // The last applied annotation is not found in the live object; normally this should not + // happen, but KubeFleet can still handle this situation. + klog.V(2).InfoS("The last applied annotation is not found in the live object", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return nil + } + + return []byte(lastAppliedManifestJSONStr) +} + +// serverSideApply uses server-side apply to apply the manifest object. +func (r *Reconciler) serverSideApply( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, + force, optimisticLock, dryRun bool, +) (*unstructured.Unstructured, error) { + // Enable optimistic lock by forcing the resource version field to be added to the + // JSON merge patch. Optimistic lock is always disabled in the dry run mode. + if optimisticLock && !dryRun { + curResourceVer := inMemberClusterObj.GetResourceVersion() + if len(curResourceVer) == 0 { + return nil, errors.NewUnexpectedError(nil, "failed to enable optimistic lock: resource version is empty on the object from the member cluster", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + } + + // Add the resource version to the manifest object. + manifestObj.SetResourceVersion(curResourceVer) + + // Remove the resource version from the object in the member cluster. + inMemberClusterObj.SetResourceVersion("") + } + + // Check if forced server-side apply is needed even if it is not turned on by the user. + // + // Note (chenyu1): This is added to addresses cases where Kubernetes might register + // KubeFleet (the member agent) as an Update typed field manager for the object, which blocks + // the same agent itself from performing a server-side apply due to conflicts, + // as Kubernetes considers Update typed and Apply typed field managers to be different + // entities, despite having the same identifier. In these cases, users will see their + // first apply attempt being successful, yet any subsequent update would fail due to + // conflicts. There are also a few other similar cases that are solved by this check; + // see the inner comments for specifics. + if shouldUseForcedServerSideApply(inMemberClusterObj) { + force = true + } + + // Use server-side apply to apply the manifest object. + // + // See the Kubernetes documentation on structured merged diff for the exact behaviors. + applyOpts := metav1.ApplyOptions{ + FieldManager: fieldManagerName, + Force: force, + } + if dryRun { + applyOpts.DryRun = []string{metav1.DryRunAll} + } + appliedObj, err := r.spokeDynamicClient. + Resource(*gvr).Namespace(manifestObj.GetNamespace()). + Apply(ctx, manifestObj.GetName(), manifestObj, applyOpts) + if err != nil { + return nil, errors.NewAPIServerError(err, "failed to apply the manifest object: an error is returned by the API server", false) + } + return appliedObj, nil +} + +// shouldUseForcedServerSideApply checks if forced server-side apply should be used even if +// the force option is not turned on. +func shouldUseForcedServerSideApply(inMemberClusterObj *unstructured.Unstructured) bool { + managedFields := inMemberClusterObj.GetManagedFields() + for idx := range managedFields { + mf := &managedFields[idx] + // fieldManagerName is the field manager name used by KubeFleet; its presence + // suggests that some (not necessarily all) fields are managed by KubeFleet. + // + // `before-first-apply` is a field manager name used by Kubernetes to "properly" + // track field managers between non-apply and apply ops. Specifically, this + // manager is added when an object is being applied, but Kubernetes finds + // that the object does not have any managed field specified. + // + // Note (chenyu1): unfortunately this name is not exposed as a public variable. See + // the Kubernetes source code for more information. + if mf.Manager != fieldManagerName && mf.Manager != "before-first-apply" { + // There exists a field manager this is neither KubeFleet nor the `before-first-apply` + // field manager, which suggests that the object (or at least some of its fields) + // is managed by another entity. KubeFleet will not enable forced server-side apply in + // this case and let user decide if forced apply is needed. + klog.V(2).InfoS("Found a field manager that is neither KubeFleet nor the `before-first-apply` field manager; KubeFleet will not enable forced server-side apply unless explicitly requested", + "fieldManager", mf.Manager, + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return false + } + } + + // All field managers are either KubeFleet or the `before-first-apply` field manager; + // use forced server-side apply to avoid confusing self-conflicts. This would + // allow KubeFleet to (correctly) assume ownership of managed fields. + klog.V(2).InfoS("All field managers are either KubeFleet or the `before-first-apply` field manager; KubeFleet will enable forced server-side apply", + "GVK", inMemberClusterObj.GroupVersionKind(), "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return true +} diff --git a/pkg/v1/controllers/workapplier/apply_integration_test.go b/pkg/v1/controllers/workapplier/apply_integration_test.go new file mode 100644 index 000000000..5eff9bfd8 --- /dev/null +++ b/pkg/v1/controllers/workapplier/apply_integration_test.go @@ -0,0 +1,722 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("apply ops", func() { + Context("client-side apply (manifest updates and changes made by third parties)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var originalConfigMap *corev1.ConfigMap + var updatedConfigMap *corev1.ConfigMap + var updatedConfigMapJSON []byte + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the original manifest. + originalConfigMap = configMap.DeepCopy() + originalConfigMap.Namespace = nsName + originalConfigMap.Labels = map[string]string{ + "label-kept": "v1", + "label-removed": "v1", + } + originalConfigMap.Annotations = map[string]string{ + "annotation-kept": "v1", + } + originalConfigMap.Data = map[string]string{ + configMapDataKey: "bar", + "data-kept": "v1", + } + originalConfigMapJSON := marshalK8sObjJSON(originalConfigMap) + + // Prepare the updated manifest, which changes a label value, removes a label, adds an annotation, and + // changes a data value. + updatedConfigMap = originalConfigMap.DeepCopy() + updatedConfigMap.Labels = map[string]string{ + "label-kept": "v2", + } + updatedConfigMap.Annotations = map[string]string{ + "annotation-kept": "v1", + "annotation-added": "v2", + } + updatedConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + "data-kept": "v1", + } + updatedConfigMapJSON = marshalK8sObjJSON(updatedConfigMap) + + // Create a new work object with no sync strategy; by default, the work applier uses client-side apply + // and overwrites drifts. + createWorkObject(workName, memberReservedNSName1, nil, originalConfigMapJSON) + }) + + It("should apply the manifest", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + wantConfigMap := originalConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map") + }) + + It("can make changes to the config map as a third party", func() { + // Add a label and a data key that the manifest does not specify, and change a data value that the + // manifest specifies. + updateConfigMapInMemberCluster(memberClient1, nsName, configMapName, func(configMap *corev1.ConfigMap) { + configMap.Labels[memberOnlyLabelKey] = memberOnlyLabelValue + configMap.Data[memberOnlyLabelKey] = memberOnlyLabelValue + configMap.Data[configMapDataKey] = "changed-by-third-party" + }) + }) + + It("should revert the change on the field that the manifest specifies, and keep the other changes", func() { + // The change in the member cluster triggers no event for the work object; the change is reverted at the + // next requeue of the work object. + wantConfigMap := originalConfigMap.DeepCopy() + wantConfigMap.Labels[memberOnlyLabelKey] = memberOnlyLabelValue + wantConfigMap.Data[memberOnlyLabelKey] = memberOnlyLabelValue + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to revert the change on the field that the manifest specifies") + }) + + It("can update the manifest", func() { + updateWorkObject(workName, memberReservedNSName1, nil, updatedConfigMapJSON) + }) + + It("should apply the updated manifest, and keep the changes made by the third party", func() { + // The three-way merge removes the label that the original manifest specifies but the updated one does + // not, while leaving the label and the data key added by the third party alone. + wantConfigMap := updatedConfigMap.DeepCopy() + wantConfigMap.Labels[memberOnlyLabelKey] = memberOnlyLabelValue + wantConfigMap.Data[memberOnlyLabelKey] = memberOnlyLabelValue + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the updated manifest") + }) + + It("should update the work object status", func() { + // The work object conditions observe the latest generation of the work object. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("client-side apply falling back to server-side apply (oversized objects)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var oversizedConfigMap *corev1.ConfigMap + var updatedOversizedConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare a config map that is too large for its last applied configuration to fit in an annotation + // (the total size of all annotations on an object is capped at 256 KiB). + oversizedConfigMap = configMap.DeepCopy() + oversizedConfigMap.Namespace = nsName + oversizedConfigMap.Data = map[string]string{ + configMapDataKey: "bar", + "large": strings.Repeat("a", 300*1024), + } + oversizedConfigMapJSON := marshalK8sObjJSON(oversizedConfigMap) + + updatedOversizedConfigMap = oversizedConfigMap.DeepCopy() + updatedOversizedConfigMap.Data[configMapDataKey] = "baz" + + // Create a new work object with no sync strategy; by default, the work applier uses client-side apply. + createWorkObject(workName, memberReservedNSName1, nil, oversizedConfigMapJSON) + }) + + It("should apply the manifest", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + wantConfigMap := oversizedConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectWithDataDigestActual := configMapObjectWithDataDigestActual(memberClient1, wantConfigMap) + Eventually(configMapObjectWithDataDigestActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map") + }) + + It("should leave the last applied configuration annotation empty", func() { + objectAnnotationActual := objectAnnotationActual(memberClient1, oversizedConfigMap, placementv1alpha1.LastAppliedConfigAnnotationKey, "") + Expect(objectAnnotationActual()).To(Succeed(), "Failed to verify the last applied configuration annotation") + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can update the manifest", func() { + updateWorkObject(workName, memberReservedNSName1, nil, marshalK8sObjJSON(updatedOversizedConfigMap)) + }) + + It("should apply the updated manifest", func() { + wantConfigMap := updatedOversizedConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectWithDataDigestActual := configMapObjectWithDataDigestActual(memberClient1, wantConfigMap) + Eventually(configMapObjectWithDataDigestActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the updated config map") + }) + + It("should have applied the updated manifest via server-side apply", func() { + // Had the work applier used client-side apply (a patch op), the changed field would have been managed + // by an Update operation instead. + fieldManagersActual := fieldManagersActual(memberClient1, oversizedConfigMap, []string{"f:data", fmt.Sprintf("f:%s", configMapDataKey)}, []string{ + fmt.Sprintf("%s/%s", fieldManagerName, metav1.ManagedFieldsOperationApply), + }) + Eventually(fieldManagersActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to verify the field managers of the changed field") + }) + + It("should update the work object status (updated manifest)", func() { + // The work object conditions observe the latest generation of the work object. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("server-side apply (field ownership conflicts)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + var regularConfigMapJSON []byte + // The object created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON = marshalK8sObjJSON(regularConfigMap) + + // Create the config map in the member cluster before the work object, with a different value for the + // data key that the manifest specifies; the field is managed by another field manager. + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + Expect(memberClient1.Create(ctx, preExistingConfigMap, client.FieldOwner(thirdPartyFieldManagerName))).To(Succeed(), "Failed to create the pre-existing config map") + + // Create a new work object that uses server-side apply without forcing conflicts (the server-side apply + // options are absent), and that always takes over pre-existing objects. + syncStrategy := &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodServerSideApply, + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should update the work object status (conflict)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToApply), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should report the conflict as the cause of the failure", func() { + manifestAppliedCondMessageActual := manifestAppliedCondMessageActual(memberReservedNSName1, workName, 0, fmt.Sprintf("conflict with %q", thirdPartyFieldManagerName)) + Eventually(manifestAppliedCondMessageActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the conflict as the cause of the failure") + }) + + It("should take over the config map, but not apply the manifest", func() { + // The check spans at least one periodic requeue of the work object. + wantConfigMap := preExistingConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Consistently(configMapObjectActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The config map has been changed unexpectedly") + }) + + It("can update the work object to force conflicts", func() { + syncStrategy := &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodServerSideApply, + ServerSideApplyOptions: &placementv1alpha1.ServerSideApplyOptions{ + ForceConflicts: true, + }, + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + } + updateWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should apply the manifest", func() { + wantConfigMap := regularConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map") + + Expect(objectNotRecreatedActual(memberClient1, preExistingConfigMap)()).To(Succeed(), "The config map has been re-created instead of being taken over") + }) + + It("should take over the ownership of the conflicting field", func() { + fieldManagersActual := fieldManagersActual(memberClient1, preExistingConfigMap, []string{"f:data", fmt.Sprintf("f:%s", configMapDataKey)}, []string{ + fmt.Sprintf("%s/%s", fieldManagerName, metav1.ManagedFieldsOperationApply), + }) + Eventually(fieldManagersActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the ownership of the conflicting field") + }) + + It("should update the work object status (applied)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("namespace auto-creation (default)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The namespace does not exist in the member cluster at first; the work applier creates it. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var appliedConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Prepare the manifest. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a new work object with no sync strategy; by default, the work applier creates the namespace of + // a manifest if the namespace does not exist yet. + createWorkObject(workName, memberReservedNSName1, nil, regularConfigMapJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should create the namespace, but not manage it", func() { + namespaceNotManagedActual := namespaceNotManagedActual(memberClient1, nsName) + Eventually(namespaceNotManagedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the namespace") + }) + + It("should apply the manifest", func() { + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map") + + appliedConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, appliedConfigMap)).To(Succeed(), "Failed to retrieve the config map") + }) + + It("should update the work object status", func() { + // The namespace is not reported, as it is not a manifest. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + // The namespace is not reported as an applied resource. + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: appliedConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("can delete the work object", func() { + deleteWorkObject(workName, memberReservedNSName1) + }) + + It("should remove the work object", func() { + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + }) + + It("should leave the namespace as it is", func() { + namespaceNotManagedActual := namespaceNotManagedActual(memberClient1, nsName) + Consistently(namespaceNotManagedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The namespace has been changed after the work object is deleted") + }) + + AfterAll(func() { + // The work object might still exist if the test case fails before the work object is deleted. + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete the work object") + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/availability_tracker.go b/pkg/v1/controllers/workapplier/availability_tracker.go new file mode 100644 index 000000000..57a804da9 --- /dev/null +++ b/pkg/v1/controllers/workapplier/availability_tracker.go @@ -0,0 +1,328 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + + appv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + policyv1 "k8s.io/api/policy/v1" + apiextensionshelpers "k8s.io/apiextensions-apiserver/pkg/apihelpers" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/component-helpers/apps/poddisruptionbudget" + "k8s.io/klog/v2" + + "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/errors" +) + +// trackInMemberClusterObjAvailability tracks the availability of applied objects in the member cluster. +func (r *Reconciler) trackInMemberClusterObjAvailability(ctx context.Context, manifestProcessingStates []*manifestProcessingState) error { + // Track the availability of all the applied objects in the member cluster in parallel. + // + // This is concurrency-safe as the processing state slice has been pre-allocated. + + // Prepare a child context. + // Cancel the child context anyway to avoid leaks. + childCtx, cancel := context.WithCancel(ctx) + defer cancel() + + doWork := func(pieces int) { + state := manifestProcessingStates[pieces] + ownerWorkObj := state.fromWorkObj + primaryWorkObj := state.fromPrimaryWorkObject + + if !isManifestObjectApplied(state.applyRes) { + // The manifest object has not been applied yet. No availability check is needed. + state.availabilityCheckRes = AvailabilityResultTypeSkipped + + // Note that some of the objects might have failed the pre-processing stage and do not + // even have a GVR or a manifest object. + if state.gvr != nil && state.manifestObj != nil { + klog.V(2).InfoS("The manifest object is not applied yet, skipping the availability check", + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, + "inMemberClusterObj", klog.KObj(state.inMemberClusterObj), + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(primaryWorkObj)) + } else { + klog.V(2).InfoS("The manifest object is not applied yet, skipping the availability check", + "ordinal", pieces, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(primaryWorkObj)) + } + return + } + + availabilityCheckRes, err := trackInMemberClusterObjAvailabilityByGVR(state.gvr, state.inMemberClusterObj) + if err != nil { + // An unexpected error has occurred during the availability check. + state.availabilityCheckErr = err + state.availabilityCheckRes = AvailabilityResultTypeFailed + klog.ErrorS(err, "Failed to track the availability of the applied object in the member cluster", + errors.Args(err, + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(primaryWorkObj))...) + return + } + state.availabilityCheckRes = availabilityCheckRes + klog.V(2).InfoS("Tracked availability of a resource", + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(primaryWorkObj), + "availabilityCheckRes", availabilityCheckRes) + } + + // Run the availability check in parallel. + r.parallelizer.ParallelizeUntil(childCtx, len(manifestProcessingStates), doWork, "trackInMemberClusterObjAvailability") + + // Unlike some other steps in the reconciliation loop, the availability checking step does not end + // with a contextual API call; consequently, if the context has been cancelled during this step, + // some checks might not run at all, and passing such states to the next step may trigger + // unexpected behaviors. To address this, at the end of this step the work applier checks for context + // cancellation directly. + if err := ctx.Err(); err != nil { + klog.V(2).InfoS("Availability checking has been interrupted as the main context has been cancelled") + return errors.NewTransientError(err, "availability checking has been interrupted") + } + return nil +} + +// isManifestObjectApplied returns whether an apply result type indicates that the manifest object +// has been applied. +func isManifestObjectApplied(applyRes ApplyResultType) bool { + return applyRes == ApplyResTypeApplied || applyRes == ApplyResTypeAppliedWithFailedDriftDetection +} + +// trackInMemberClusterObjAvailabilityByGVR tracks the availability of an object in the member cluster based +// on its GVR. +func trackInMemberClusterObjAvailabilityByGVR( + gvr *schema.GroupVersionResource, + inMemberClusterObj *unstructured.Unstructured, +) (AvailabilityCheckResultType, error) { + switch *gvr { + case utils.DeploymentGVR: + return trackDeploymentAvailability(inMemberClusterObj) + case utils.StatefulSetGVR: + return trackStatefulSetAvailability(inMemberClusterObj) + case utils.DaemonSetGVR: + return trackDaemonSetAvailability(inMemberClusterObj) + case utils.ServiceGVR: + return trackServiceAvailability(inMemberClusterObj) + case utils.CustomResourceDefinitionGVR: + return trackCRDAvailability(inMemberClusterObj) + case utils.PodDisruptionBudgetGVR: + return trackPDBAvailability(inMemberClusterObj) + default: + if isDataResource(*gvr) { + klog.V(2).InfoS("The object from the member cluster is a data object, consider it to be immediately available", + "GVR", *gvr, "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Cannot determine the availability of the object from the member cluster; untrack its availability", + "GVR", *gvr, "inMemberClusterObj", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotTrackable, nil + } +} + +// trackDeploymentAvailability tracks the availability of a deployment in the member cluster. +func trackDeploymentAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var deploy appv1.Deployment + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &deploy); err != nil { + // Normally this branch should never run. + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a deployment") + } + + // Check if the deployment is available. + requiredReplicas := int32(1) + if deploy.Spec.Replicas != nil { + requiredReplicas = *deploy.Spec.Replicas + } + if deploy.Status.ObservedGeneration == deploy.Generation && + requiredReplicas == deploy.Status.AvailableReplicas && + requiredReplicas == deploy.Status.UpdatedReplicas && + deploy.Status.UnavailableReplicas == 0 { + klog.V(2).InfoS("Deployment is available", "deployment", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Deployment is not ready yet, will check later to see if it becomes available", "deployment", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// trackStatefulSetAvailability tracks the availability of a stateful set in the member cluster. +func trackStatefulSetAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var statefulSet appv1.StatefulSet + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &statefulSet); err != nil { + // Normally this branch should never run. + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a stateful set") + } + + // Check if the stateful set is available. + // + // A statefulSet is available if all if its replicas are available and the current replica count + // is equal to the updated replica count, which implies that all replicas are up to date. + requiredReplicas := int32(1) + if statefulSet.Spec.Replicas != nil { + requiredReplicas = *statefulSet.Spec.Replicas + } + if statefulSet.Status.ObservedGeneration == statefulSet.Generation && + statefulSet.Status.AvailableReplicas == requiredReplicas && + statefulSet.Status.CurrentReplicas == statefulSet.Status.UpdatedReplicas && + statefulSet.Status.CurrentRevision == statefulSet.Status.UpdateRevision { + klog.V(2).InfoS("StatefulSet is available", "statefulSet", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Stateful set is not ready yet, will check later to see if it becomes available", "statefulSet", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// trackDaemonSetAvailability tracks the availability of a daemon set in the member cluster. +func trackDaemonSetAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var daemonSet appv1.DaemonSet + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &daemonSet); err != nil { + // Normally this branch should never run. + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a daemon set") + } + + // Check if the daemonSet is available. + // + // A daemonSet is available if all if its desired replicas (the count of which is equal to + // the number of applicable nodes in the cluster) are available and the current replica count + // is equal to the updated replica count, which implies that all replicas are up to date. + if daemonSet.Status.ObservedGeneration == daemonSet.Generation && + daemonSet.Status.NumberAvailable == daemonSet.Status.DesiredNumberScheduled && + daemonSet.Status.CurrentNumberScheduled == daemonSet.Status.UpdatedNumberScheduled { + klog.V(2).InfoS("DaemonSet is available", "daemonSet", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Daemon set is not ready yet, will check later to see if it becomes available", "daemonSet", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// trackServiceAvailability tracks the availability of a service in the member cluster. +func trackServiceAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var svc corev1.Service + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &svc); err != nil { + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a service") + } + switch svc.Spec.Type { + case "": + fallthrough // The default service type is ClusterIP. + case corev1.ServiceTypeClusterIP: + fallthrough + case corev1.ServiceTypeNodePort: + // KubeFleet considers a ClusterIP or NodePort service to be available if it has at least one + // IP assigned. + if len(svc.Spec.ClusterIPs) > 0 && len(svc.Spec.ClusterIPs[0]) > 0 { + klog.V(2).InfoS("Service is available", "service", klog.KObj(inMemberClusterObj), "serviceType", svc.Spec.Type) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Service is not ready yet, will check later to see if it becomes available", "service", klog.KObj(inMemberClusterObj), "serviceType", svc.Spec.Type) + return AvailabilityResultTypeNotYetAvailable, nil + case corev1.ServiceTypeLoadBalancer: + // KubeFleet considers a loadBalancer service to be available if it has at least one load + // balancer IP or hostname assigned. + if len(svc.Status.LoadBalancer.Ingress) > 0 && + (len(svc.Status.LoadBalancer.Ingress[0].IP) > 0 || len(svc.Status.LoadBalancer.Ingress[0].Hostname) > 0) { + klog.V(2).InfoS("Service is available", "service", klog.KObj(inMemberClusterObj), "serviceType", svc.Spec.Type) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Service is not ready yet, will check later to see if it becomes available", "service", klog.KObj(inMemberClusterObj), "serviceType", svc.Spec.Type) + return AvailabilityResultTypeNotYetAvailable, nil + } + + // KubeFleet does not know how to track the availability of an externalName service. + klog.V(2).InfoS("Cannot determine the availability of external name services; untrack its availability", "service", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotTrackable, nil +} + +// trackCRDAvailability tracks the availability of a custom resource definition in the member cluster. +func trackCRDAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var crd apiextensionsv1.CustomResourceDefinition + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &crd); err != nil { + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a custom resource definition") + } + + // If both conditions are True, the CRD has become available. + if apiextensionshelpers.IsCRDConditionTrue(&crd, apiextensionsv1.Established) && apiextensionshelpers.IsCRDConditionTrue(&crd, apiextensionsv1.NamesAccepted) { + klog.V(2).InfoS("CustomResourceDefinition is available", "customResourceDefinition", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + + klog.V(2).InfoS("Custom resource definition is not ready yet, will check later to see if it becomes available", + "customResourceDefinition", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// trackPDBAvailability tracks the availability of a pod disruption budget in the member cluster. +func trackPDBAvailability(inMemberClusterObj *unstructured.Unstructured) (AvailabilityCheckResultType, error) { + var pdb policyv1.PodDisruptionBudget + if err := runtime.DefaultUnstructuredConverter.FromUnstructured(inMemberClusterObj.Object, &pdb); err != nil { + return AvailabilityResultTypeFailed, errors.NewUnexpectedError(err, "failed to convert the unstructured object to a pod disruption budget") + } + // Check if conditions are up-to-date. + if poddisruptionbudget.ConditionsAreUpToDate(&pdb) { + klog.V(2).InfoS("PodDisruptionBudget is available", "podDisruptionBudget", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeAvailable, nil + } + klog.V(2).InfoS("Still need to wait for PodDisruptionBudget to be available", "podDisruptionBudget", klog.KObj(inMemberClusterObj)) + return AvailabilityResultTypeNotYetAvailable, nil +} + +// isDataResource checks if the resource is a data resource; such resources are +// available immediately after creation. +func isDataResource(gvr schema.GroupVersionResource) bool { + switch gvr { + case utils.NamespaceGVR: + return true + case utils.SecretGVR: + return true + case utils.ConfigMapGVR: + return true + case utils.RoleGVR: + return true + case utils.ClusterRoleGVR: + return true + case utils.RoleBindingGVR: + return true + case utils.ClusterRoleBindingGVR: + return true + case utils.ServiceAccountGVR: + return true + case utils.NetworkPolicyGVR: + return true + case utils.CSIDriverGVR: + return true + case utils.CSINodeGVR: + return true + case utils.StorageClassGVR: + return true + case utils.CSIStorageCapacityGVR: + return true + case utils.ControllerRevisionGVR: + return true + case utils.IngressClassGVR: + return true + case utils.LimitRangeGVR: + return true + case utils.ResourceQuotaGVR: + return true + case utils.PriorityClassGVR: + return true + } + return false +} diff --git a/pkg/v1/controllers/workapplier/backoff.go b/pkg/v1/controllers/workapplier/backoff.go new file mode 100644 index 000000000..4275230fa --- /dev/null +++ b/pkg/v1/controllers/workapplier/backoff.go @@ -0,0 +1,445 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + "sync" + "time" + + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/utils/resource" +) + +const ( + attemptsWithFixedDelayUpperBound = 40 + attemptsWithFixedDelayLowerBound = 1 + fixedDelaySecondsLowerBound = 2 + initialSlowBackoffDelaySecondsLowerBound = 2 + maxFastBackoffDelaySecondsUpperBound = 3600 // 1 hour + + // Set caps on the bases to make sure that delay would be monotonically growing and + // no overflowing would occur. + exponentialBaseLowerBound = 1.05 + exponentialBaseUpperBound = 100.0 + + // With the current set of extremums (attemptsWithFixedDelayUpperBound = 40, + // initialSlowBackoffDelaySecondsLowerBound = 2, maxFastBackoffDelaySecondsUpperBound = 3600, + // exponentialBaseLowerBound = 1.05), + // the number of requeues needed for the requeue delay to reach to the maximum value + // would be 195. The rate limiter has no need to track number of requeues beyond + // this value. + maxNumOfRequeuesToTrack = 200 +) + +const ( + gcTriggeringForgetThreshold = 100000 +) + +const ( + manifestProcessingStateStrTmpl = "%s,%s" + linkedWorkObjGenStrTmpl = "%s,%d" +) + +// The parameters of the default requeue rate limiter; see init for the reasoning behind each value. +const ( + defaultAttemptsWithFixedDelay = 1 + defaultFixedDelaySeconds = 5 + defaultExponentialBaseForSlowBackoff = 1.2 + defaultInitialSlowBackoffDelaySeconds = 2 + defaultMaxSlowBackoffDelaySeconds = 15 + defaultExponentialBaseForFastBackoff = 1.5 + defaultMaxFastBackoffDelaySeconds = 900 + defaultSkipToFastBackoffWhenAllAppliedManifestsAreAvailable = true +) + +var defaultRequeueRateLimiter *MultiStageRequeueWithExponentialBackoffRateLimiter + +func init() { + var err error + defaultRequeueRateLimiter, err = NewMultiStageRequeueWithExponentialBackoffRateLimiter( + // Allow 1 attempt of fixed delay; this helps give objects a bit of headroom to become available. + defaultAttemptsWithFixedDelay, + // Use a fixed delay of 5 seconds for the first attempt. + // + // Important (chenyu1): before the introduction of the requeue rate limiter, the work + // applier uses static requeue intervals, specifically 5 seconds (if the work object is unavailable), + // and 15 seconds (if the work object is available). There are a number of test cases that + // implicitly assume this behavior (e.g., a test case might expect that the availability check completes + // w/in 10 seconds), which is why the rate limiter uses the 5 seconds fast requeue delay by default. + // If you need to change this value and see that some test cases begin to fail, update the test + // cases accordingly. + defaultFixedDelaySeconds, + // Then switch to slow exponential backoff with a base of 1.2 with an initial delay of 2 seconds + // and a cap of 15 seconds (12 requeues in total, ~90 seconds in total). + // This is to allow fast checkups in cases where objects are not yet available. + defaultExponentialBaseForSlowBackoff, + defaultInitialSlowBackoffDelaySeconds, + defaultMaxSlowBackoffDelaySeconds, + // Eventually, switch to a fast exponential backoff with a base of 1.5 with an initial delay of 15 seconds + // and a cap of 15 minutes (10 requeues in total, ~42 minutes in total). + defaultExponentialBaseForFastBackoff, + defaultMaxFastBackoffDelaySeconds, + // Allow skipping to the fast exponential backoff stage if all the applied manifests are available. + defaultSkipToFastBackoffWhenAllAppliedManifestsAreAvailable, + // When the linked Work objects do not change and the processing result remains the same (e.g., some + // manifests stay unavailable), the requeue pattern is essentially: + // * 1 attempt of requeue with fixed delays (5 seconds); then + // * 12 attempts of requeues with slow exponential backoff (factor of 1.2, ~90 seconds in total); then + // * 10 attempts of requeues with fast exponential backoff (factor of 1.5, ~42 minutes in total); + // * afterwards, requeue with a delay of 15 minutes indefinitely. + ) + if err != nil { + klog.Fatalf("Failed to initialize default requeue rate limiter: %v", err) + } +} + +// NewDefaultRequeueRateLimiterWithMaxDelay creates a rate limiter that requeues the same way as the default one, +// except that no requeue delay exceeds the given maximum. Values at or above the default caps leave the default +// behavior unchanged. +// +// It is meant for test environments, where changes made only on the member cluster side (e.g., drifts) need to +// be picked up quickly. +func NewDefaultRequeueRateLimiterWithMaxDelay(maxDelaySeconds int) (*MultiStageRequeueWithExponentialBackoffRateLimiter, error) { + return NewMultiStageRequeueWithExponentialBackoffRateLimiter( + defaultAttemptsWithFixedDelay, + min(defaultFixedDelaySeconds, maxDelaySeconds), + defaultExponentialBaseForSlowBackoff, + min(defaultInitialSlowBackoffDelaySeconds, maxDelaySeconds), + min(defaultMaxSlowBackoffDelaySeconds, maxDelaySeconds), + defaultExponentialBaseForFastBackoff, + min(defaultMaxFastBackoffDelaySeconds, maxDelaySeconds), + defaultSkipToFastBackoffWhenAllAppliedManifestsAreAvailable, + ) +} + +// MultiStageRequeueWithExponentialBackoffRateLimiter is a rate limiter that allows requeues of various +// delays in multiple stages; it helps to control the rate of requeues for Work objects while keeping +// the system as responsive as possible. +// +// It will: +// - first, requeue with a fixed delay for a limited number of attempts; +// this is a stage where the work objects are expected to be unavailable (e.g., it might be known +// already that an application takes a while to start up). +// - then, switch to exponential backoff at a slow rate if the processing results stay consistent, +// until the delay period reaches a cap; +// this is a stage where the work objects are expected to become available, and the system will +// requeue (relatively) fast for a more responsive experience. +// - at last, switch to exponential backoff at a fast rate with a cap, if the processing results +// remain consistent; +// this is a stage where the system no longer expects the work objects to have a status change +// and chooses to requeue (relatively) slowly to avoid overwhelming the system. +// +// When all the applied manifests are available, the rate limiter can be set to skip to the fast +// backoff stage to save unnecessary fast requeues. +// +// Note that the implementation tracks the names and generations of all the linked work objects, along +// with the processing results, so that a spec change on any linked work object, a change to the set +// of linked work objects, and/or a processing result change will reset the requeue counter. +// +// TO-DO (chenyu1): the current implementation tracks processing results as strings, which incurs +// additional overhead when doing comparison (albeit small); evaluate if we need to switch to a more +// performant representation. +type MultiStageRequeueWithExponentialBackoffRateLimiter struct { + mu sync.Mutex + requeueCounter map[types.NamespacedName]int + lastRequeueDelayTracker map[types.NamespacedName]time.Duration + lastTrackedLinkedWorksGenHash map[types.NamespacedName]string + lastTrackedProcessingResultHash map[types.NamespacedName]string + forgets int + + attemptsWithFixedDelay int + fixedDelay time.Duration + + exponentialBaseForSlowBackoff float64 + initialSlowBackoffDelay time.Duration + maxSlowBackoffDelay time.Duration + + exponentialBaseForFastBackoff float64 + maxFastBackoffDelay time.Duration + + skipToFastBackoffWhenAllAppliedManifestsAreAvailable bool +} + +// NewMultiStageRequeueWithExponentialBackoffRateLimiter creates a MultiStageRequeueWithExponentialBackoffRateLimiter. +func NewMultiStageRequeueWithExponentialBackoffRateLimiter( + attemptsWithFixedDelay int, + fixedDelaySeconds int, + exponentialBaseForSlowBackoff float64, + initialSlowBackoffDelaySeconds int, + maxSlowBackoffDelaySeconds int, + exponentialBaseForFastBackoff float64, + maxFastBackoffDelaySeconds int, + skipToFastBackoffWhenAllAppliedManifestsAreAvailable bool, +) (*MultiStageRequeueWithExponentialBackoffRateLimiter, error) { + // Validate the input parameters. + if !(attemptsWithFixedDelay >= attemptsWithFixedDelayLowerBound && + attemptsWithFixedDelay <= attemptsWithFixedDelayUpperBound) { + return nil, kferrors.NewUserError(nil, "attemptsWithFixedDelay is not set to a valid value", + "observedValue", attemptsWithFixedDelay, + "lowerBound", attemptsWithFixedDelayLowerBound, "upperBound", attemptsWithFixedDelayUpperBound) + } + if fixedDelaySeconds < fixedDelaySecondsLowerBound { + return nil, kferrors.NewUserError(nil, "fixedDelaySeconds is below the minimum allowed value", + "observedValue", fixedDelaySeconds, + "lowerBound", fixedDelaySecondsLowerBound) + } + if fixedDelaySeconds > maxFastBackoffDelaySeconds { + return nil, kferrors.NewUserError(nil, "fixedDelaySeconds exceeds maxFastBackoffDelaySeconds", + "observedValue", fixedDelaySeconds, + "upperBound", maxFastBackoffDelaySeconds) + } + + if !(exponentialBaseForSlowBackoff >= exponentialBaseLowerBound && + exponentialBaseForSlowBackoff <= exponentialBaseForFastBackoff && + exponentialBaseForFastBackoff <= exponentialBaseUpperBound) { + return nil, kferrors.NewUserError(nil, + "exponentialBaseForSlowBackoff and exponentialBaseForFastBackoff are not set to valid values", + "observedValueForSlowBackoff", exponentialBaseForSlowBackoff, + "observedValueForFastBackoff", exponentialBaseForFastBackoff, + "lowerBound", exponentialBaseLowerBound, + "upperBound", exponentialBaseUpperBound) + } + + if !(initialSlowBackoffDelaySeconds >= initialSlowBackoffDelaySecondsLowerBound && + initialSlowBackoffDelaySeconds <= maxSlowBackoffDelaySeconds && + maxSlowBackoffDelaySeconds <= maxFastBackoffDelaySeconds && + maxFastBackoffDelaySeconds <= maxFastBackoffDelaySecondsUpperBound) { + return nil, kferrors.NewUserError(nil, + "initialSlowBackoffDelaySeconds, maxSlowBackoffDelaySeconds, and maxFastBackoffDelaySeconds are not set to valid values", + "observedInitialSlowBackoffDelaySeconds", initialSlowBackoffDelaySeconds, + "observedMaxSlowBackoffDelaySeconds", maxSlowBackoffDelaySeconds, + "observedMaxFastBackoffDelaySeconds", maxFastBackoffDelaySeconds, + "lowerBoundForInitialSlowBackoffDelaySeconds", initialSlowBackoffDelaySecondsLowerBound, + "upperBoundForMaxFastBackoffDelaySeconds", maxFastBackoffDelaySecondsUpperBound) + } + + return &MultiStageRequeueWithExponentialBackoffRateLimiter{ + requeueCounter: make(map[types.NamespacedName]int), + lastRequeueDelayTracker: make(map[types.NamespacedName]time.Duration), + lastTrackedLinkedWorksGenHash: make(map[types.NamespacedName]string), + lastTrackedProcessingResultHash: make(map[types.NamespacedName]string), + attemptsWithFixedDelay: attemptsWithFixedDelay, + fixedDelay: time.Duration(fixedDelaySeconds) * time.Second, + exponentialBaseForSlowBackoff: exponentialBaseForSlowBackoff, + initialSlowBackoffDelay: time.Duration(initialSlowBackoffDelaySeconds) * time.Second, + maxSlowBackoffDelay: time.Duration(maxSlowBackoffDelaySeconds) * time.Second, + exponentialBaseForFastBackoff: exponentialBaseForFastBackoff, + maxFastBackoffDelay: time.Duration(maxFastBackoffDelaySeconds) * time.Second, + skipToFastBackoffWhenAllAppliedManifestsAreAvailable: skipToFastBackoffWhenAllAppliedManifestsAreAvailable, + }, nil +} + +// When returns the duration to wait before requeuing the item. +// +// The implementation borrows from the workqueue package's various rate limiter implementations. +func (r *MultiStageRequeueWithExponentialBackoffRateLimiter) When( + linkedWorks []*placementv1alpha1.Work, manifestProcessingStates []*manifestProcessingState) time.Duration { + r.mu.Lock() + defer r.mu.Unlock() + + primaryWork := linkedWorks[0] + primaryWorkNamespacedName := types.NamespacedName{ + Namespace: primaryWork.Namespace, + Name: primaryWork.Name, + } + + curLinkedWorksGenHash, genHashErr := computeLinkedWorksObjectGenHash(linkedWorks) + curProcessingResHash, resHashErr := computeProcessingResultHash(manifestProcessingStates) + + // Check if the Work object has been processed before, and check if the rate limiter needs + // to reset the requeue counter. + lastTrackedLinkedWorksGenHash := r.lastTrackedLinkedWorksGenHash[primaryWorkNamespacedName] + lastTrackedProcessingResHash := r.lastTrackedProcessingResultHash[primaryWorkNamespacedName] + + // Reset the requeue counter under the following conditions: + // * The set of linked work objects, or the generation of any linked work object, has changed + // since last observation. + // * Cannot compute the hash of the linked work object generations or the processing result; + // normally this should never occur. + // * The processing result has changed since last observation. + if lastTrackedLinkedWorksGenHash != curLinkedWorksGenHash || + genHashErr != nil || + lastTrackedProcessingResHash != curProcessingResHash || + resHashErr != nil { + r.requeueCounter[primaryWorkNamespacedName] = 0 + r.lastRequeueDelayTracker[primaryWorkNamespacedName] = 0 + } + + // Always update the last tracked linked work object generation hash and processing result hash + // for the Work object, regardless of whether the requeue counter is reset or not. + r.lastTrackedLinkedWorksGenHash[primaryWorkNamespacedName] = curLinkedWorksGenHash + r.lastTrackedProcessingResultHash[primaryWorkNamespacedName] = curProcessingResHash + + numRequeues := r.requeueCounter[primaryWorkNamespacedName] + lastRequeueDelayWithBackoff := r.lastRequeueDelayTracker[primaryWorkNamespacedName] + var requeueDelay time.Duration + + allProcessedManifestsAvailable := areAllProcessedManifestsAvailable(manifestProcessingStates) + + switch { + case numRequeues < r.attemptsWithFixedDelay: + // Requeue with a fixed delay for the first few attempts. + requeueDelay = r.fixedDelay + case numRequeues == r.attemptsWithFixedDelay: + // Start to requeue with backoff. + requeueDelay = r.initialSlowBackoffDelay + case lastRequeueDelayWithBackoff >= r.maxFastBackoffDelay: + // The requeue delay has reached the cap; no need to backoff anymore. + requeueDelay = r.maxFastBackoffDelay + case r.skipToFastBackoffWhenAllAppliedManifestsAreAvailable && + allProcessedManifestsAvailable: + // All the applied manifests are available, and the rate limiter is configured to skip to the fast + // backoff stage under such conditions. + requeueDelay = time.Duration(float64(lastRequeueDelayWithBackoff) * r.exponentialBaseForFastBackoff) + case lastRequeueDelayWithBackoff >= r.maxSlowBackoffDelay: + // The requeue delay has reached the cap for the slow backoff stage; + // start to fast back off. + requeueDelay = time.Duration(float64(lastRequeueDelayWithBackoff) * r.exponentialBaseForFastBackoff) + default: + // Continue to slow back off. + requeueDelay = time.Duration(float64(lastRequeueDelayWithBackoff) * r.exponentialBaseForSlowBackoff) + if requeueDelay > r.maxSlowBackoffDelay { + // Backing off slowly this time will break the cap for the slow backoff stage; + // switch to fast back off. + requeueDelay = time.Duration(float64(lastRequeueDelayWithBackoff) * r.exponentialBaseForFastBackoff) + } + } + + // Cap on the requeue delay. + if requeueDelay > r.maxFastBackoffDelay { + requeueDelay = r.maxFastBackoffDelay + } + + // Update the requeue counter and last requeue delay tracker. + if numRequeues+1 < maxNumOfRequeuesToTrack { + r.requeueCounter[primaryWorkNamespacedName] = numRequeues + 1 + } + r.lastRequeueDelayTracker[primaryWorkNamespacedName] = requeueDelay + return requeueDelay +} + +// Forget untracks a work object from the rate limiter. +func (r *MultiStageRequeueWithExponentialBackoffRateLimiter) Forget(work *placementv1alpha1.Work) { + r.mu.Lock() + defer r.mu.Unlock() + + // Reset the trackers for the item. + namespacedName := types.NamespacedName{ + Namespace: work.Namespace, + Name: work.Name, + } + delete(r.requeueCounter, namespacedName) + delete(r.lastRequeueDelayTracker, namespacedName) + delete(r.lastTrackedLinkedWorksGenHash, namespacedName) + delete(r.lastTrackedProcessingResultHash, namespacedName) + + r.forgets++ + if r.forgets >= gcTriggeringForgetThreshold { + // As Go does not GC used space in maps; we manually rebuild the maps to free up memory. + r.garbageCollect() + r.forgets = 0 + } +} + +// computeLinkedWorksObjectGenHash returns the hash of the names and generations of a set of linked work objects. +func computeLinkedWorksObjectGenHash(linkedWorks []*placementv1alpha1.Work) (string, error) { + // The order of linked work objects is stable (the primary work object first, followed by the + // other linked work objects sorted by their names). + linkedWorkObjGens := make([]string, 0, len(linkedWorks)) + for _, linkedWork := range linkedWorks { + linkedWorkObjGens = append(linkedWorkObjGens, + fmt.Sprintf(linkedWorkObjGenStrTmpl, linkedWork.Name, linkedWork.Generation)) + } + + linkedWorkObjGenHash, err := resource.HashOf(linkedWorkObjGens) + if err != nil { + wrappedErr := kferrors.NewUnexpectedError(err, "failed to marshal linked work object generations as JSON") + return "", wrappedErr + } + return linkedWorkObjGenHash, nil +} + +// computeProcessingResultHash returns the hash of the result of a work object processing attempt, +// specifically the apply result and the availability check result of each manifest. +// +// Note (chenyu1): there exists a corner case where even though the processing result for a specific +// manifest remains unchanged, the cause of the result have changed (e.g., an apply op first failed +// due to an API server error, then it failed again because a webhook denied it). At this moment +// the rate limiter does not distinguish between the two cases. +func computeProcessingResultHash(manifestProcessingStates []*manifestProcessingState) (string, error) { + // The order of manifests is stable in a bundle. + manifestProcessingResults := make([]string, 0, len(manifestProcessingStates)) + for _, manifestProcessingState := range manifestProcessingStates { + manifestProcessingResults = append(manifestProcessingResults, + fmt.Sprintf(manifestProcessingStateStrTmpl, manifestProcessingState.applyRes, manifestProcessingState.availabilityCheckRes)) + } + + processingResHash, err := resource.HashOf(manifestProcessingResults) + if err != nil { + wrappedErr := kferrors.NewUnexpectedError(err, "failed to marshal processing results as JSON") + return "", wrappedErr + } + return processingResHash, nil +} + +// areAllProcessedManifestsAvailable returns whether all the processed manifests are available. +// +// A manifest is considered to be available if its availability check result is either Available or +// NotTrackable; this is consistent with how the Available condition is set on a Work object. If no +// manifest has been processed, all processed manifests are considered to be available, also +// consistent with the Available condition behavior. +func areAllProcessedManifestsAvailable(manifestProcessingStates []*manifestProcessingState) bool { + for _, manifestProcessingState := range manifestProcessingStates { + if !isAppliedObjectAvailable(manifestProcessingState.availabilityCheckRes) { + return false + } + } + return true +} + +func (r *MultiStageRequeueWithExponentialBackoffRateLimiter) garbageCollect() { + rebuiltRequeueCounter := make(map[types.NamespacedName]int, len(r.requeueCounter)) + for k, v := range r.requeueCounter { + rebuiltRequeueCounter[k] = v + } + r.requeueCounter = rebuiltRequeueCounter + + rebuiltLastRequeueDelayTracker := make(map[types.NamespacedName]time.Duration, len(r.lastRequeueDelayTracker)) + for k, v := range r.lastRequeueDelayTracker { + rebuiltLastRequeueDelayTracker[k] = v + } + r.lastRequeueDelayTracker = rebuiltLastRequeueDelayTracker + + rebuiltLastTrackedLinkedWorksGenHash := make(map[types.NamespacedName]string, len(r.lastTrackedLinkedWorksGenHash)) + for k, v := range r.lastTrackedLinkedWorksGenHash { + rebuiltLastTrackedLinkedWorksGenHash[k] = v + } + r.lastTrackedLinkedWorksGenHash = rebuiltLastTrackedLinkedWorksGenHash + + rebuiltLastTrackedProcessingResultHash := make(map[types.NamespacedName]string, len(r.lastTrackedProcessingResultHash)) + for k, v := range r.lastTrackedProcessingResultHash { + rebuiltLastTrackedProcessingResultHash[k] = v + } + r.lastTrackedProcessingResultHash = rebuiltLastTrackedProcessingResultHash + + r.forgets = 0 +} diff --git a/pkg/v1/controllers/workapplier/backoff_test.go b/pkg/v1/controllers/workapplier/backoff_test.go new file mode 100644 index 000000000..7e1b46490 --- /dev/null +++ b/pkg/v1/controllers/workapplier/backoff_test.go @@ -0,0 +1,1522 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + "math" + "reflect" + "slices" + "sync" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" +) + +const ( + backoffTestAltNSName = "fleet-member-experimental-2" + + // The SHA-256 checksum of `[]`, i.e., the JSON representation of an empty string slice. + emptyJSONListSHA256 = "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945" + + // Account for the precision loss when multiplying time.Duration values with float exponential bases. + delayToleranceSeconds = 0.01 +) + +var ( + backoffTestPrimaryWorkName = fmt.Sprintf(workNameTemplate, "1") + backoffTestSecondaryWorkName = fmt.Sprintf(workNameTemplate, "1-1") + backoffTestOtherWorkName = fmt.Sprintf(workNameTemplate, "2") + + // The configuration of the default requeue rate limiter. + defaultRateLimiterCfg = rateLimiterConfig{ + attemptsWithFixedDelay: 1, + fixedDelaySeconds: 5, + exponentialBaseForSlowBackoff: 1.2, + initialSlowBackoffDelaySeconds: 2, + maxSlowBackoffDelaySeconds: 15, + exponentialBaseForFastBackoff: 1.5, + maxFastBackoffDelaySeconds: 900, + skipToFastBackoffWhenAllAppliedManifestsAreAvailable: true, + } + + // A configuration that yields easy-to-follow requeue delays: 5 (fixed), 10 (initial slow backoff), + // 20 (slow backoff), 100 (fast backoff, as 40 breaks the slow backoff cap), 200 (capped). + simpleRateLimiterCfg = rateLimiterConfig{ + attemptsWithFixedDelay: 1, + fixedDelaySeconds: 5, + exponentialBaseForSlowBackoff: 2, + initialSlowBackoffDelaySeconds: 10, + maxSlowBackoffDelaySeconds: 30, + exponentialBaseForFastBackoff: 5, + maxFastBackoffDelaySeconds: 200, + } + + rateLimiterCmpOpts = []cmp.Option{ + cmp.AllowUnexported(MultiStageRequeueWithExponentialBackoffRateLimiter{}), + cmpopts.IgnoreFields(MultiStageRequeueWithExponentialBackoffRateLimiter{}, "mu"), + } +) + +// rateLimiterConfig is the set of parameters for creating a MultiStageRequeueWithExponentialBackoffRateLimiter. +type rateLimiterConfig struct { + attemptsWithFixedDelay int + fixedDelaySeconds int + exponentialBaseForSlowBackoff float64 + initialSlowBackoffDelaySeconds int + maxSlowBackoffDelaySeconds int + exponentialBaseForFastBackoff float64 + maxFastBackoffDelaySeconds int + skipToFastBackoffWhenAllAppliedManifestsAreAvailable bool +} + +// requeueStep is a single call to the When method, along with the requeue delay it should return. +type requeueStep struct { + linkedWorks []*placementv1alpha1.Work + states []*manifestProcessingState + wantDelaySeconds float64 +} + +// trackedRequeueState is the per-work object state that the rate limiter tracks. +type trackedRequeueState struct { + requeues int + lastDelay time.Duration + linkedWorksGenHash string + processingResultHash string +} + +func newRateLimiterForTest(t *testing.T, cfg rateLimiterConfig) *MultiStageRequeueWithExponentialBackoffRateLimiter { + t.Helper() + rateLimiter, err := NewMultiStageRequeueWithExponentialBackoffRateLimiter( + cfg.attemptsWithFixedDelay, + cfg.fixedDelaySeconds, + cfg.exponentialBaseForSlowBackoff, + cfg.initialSlowBackoffDelaySeconds, + cfg.maxSlowBackoffDelaySeconds, + cfg.exponentialBaseForFastBackoff, + cfg.maxFastBackoffDelaySeconds, + cfg.skipToFastBackoffWhenAllAppliedManifestsAreAvailable, + ) + if err != nil { + t.Fatalf("NewMultiStageRequeueWithExponentialBackoffRateLimiter(%+v) = %v, want no error", cfg, err) + } + return rateLimiter +} + +func withTrackedRequeueStates( + rateLimiter *MultiStageRequeueWithExponentialBackoffRateLimiter, + trackedStates map[types.NamespacedName]trackedRequeueState, + forgets int, +) *MultiStageRequeueWithExponentialBackoffRateLimiter { + for key, state := range trackedStates { + rateLimiter.requeueCounter[key] = state.requeues + rateLimiter.lastRequeueDelayTracker[key] = state.lastDelay + rateLimiter.lastTrackedLinkedWorksGenHash[key] = state.linkedWorksGenHash + rateLimiter.lastTrackedProcessingResultHash[key] = state.processingResultHash + } + rateLimiter.forgets = forgets + return rateLimiter +} + +func workForBackoffTest(namespace, name string, generation int64) *placementv1alpha1.Work { + return &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: name, + Generation: generation, + }, + } +} + +// workWithLabelsAndStatusForBackoffTest returns a work object with some labels and status conditions set, +// which (unlike the name and the generation) should not affect the requeue delays. +func workWithLabelsAndStatusForBackoffTest(namespace, name string, generation int64) *placementv1alpha1.Work { + work := workForBackoffTest(namespace, name, generation) + work.Labels = map[string]string{"foo": "bar"} + work.Status = placementv1alpha1.WorkStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + }, + }, + } + return work +} + +func stateWith(applyRes ApplyResultType, availabilityCheckRes AvailabilityCheckResultType) *manifestProcessingState { + return &manifestProcessingState{ + applyRes: applyRes, + availabilityCheckRes: availabilityCheckRes, + } +} + +// requeueSteps builds a sequence of requeue steps that share the same inputs. +func requeueSteps( + linkedWorks []*placementv1alpha1.Work, + states []*manifestProcessingState, + wantDelaysSeconds ...float64, +) []requeueStep { + steps := make([]requeueStep, 0, len(wantDelaysSeconds)) + for _, wantDelaySeconds := range wantDelaysSeconds { + steps = append(steps, requeueStep{ + linkedWorks: linkedWorks, + states: states, + wantDelaySeconds: wantDelaySeconds, + }) + } + return steps +} + +// verifyRequeueSteps calls the When method for each step in order and verifies the returned delays. +// +// As the rate limiter is stateful, the verification stops at the first mismatch. +func verifyRequeueSteps(t *testing.T, rateLimiter *MultiStageRequeueWithExponentialBackoffRateLimiter, steps []requeueStep) { + t.Helper() + for idx, step := range steps { + gotDelay := rateLimiter.When(step.linkedWorks, step.states) + if !cmp.Equal(gotDelay.Seconds(), step.wantDelaySeconds, cmpopts.EquateApprox(0, delayToleranceSeconds)) { + t.Fatalf("When() (step #%d) = %v, want %vs", idx+1, gotDelay, step.wantDelaySeconds) + } + } +} + +// TestNewMultiStageRequeueWithExponentialBackoffRateLimiter tests the +// NewMultiStageRequeueWithExponentialBackoffRateLimiter function. +func TestNewMultiStageRequeueWithExponentialBackoffRateLimiter(t *testing.T) { + typicalCfg := rateLimiterConfig{ + attemptsWithFixedDelay: 10, + fixedDelaySeconds: 5, + exponentialBaseForSlowBackoff: 1.2, + initialSlowBackoffDelaySeconds: 2, + maxSlowBackoffDelaySeconds: 15, + exponentialBaseForFastBackoff: 1.5, + maxFastBackoffDelaySeconds: 60, + skipToFastBackoffWhenAllAppliedManifestsAreAvailable: true, + } + // mutated returns a copy of the typical configuration with one or more parameters changed. + mutated := func(mutateFn func(cfg *rateLimiterConfig)) rateLimiterConfig { + cfg := typicalCfg + mutateFn(&cfg) + return cfg + } + + testCases := []struct { + name string + cfg rateLimiterConfig + wantRateLimiter *MultiStageRequeueWithExponentialBackoffRateLimiter + wantErred bool + }{ + { + name: "typical parameters", + cfg: typicalCfg, + wantRateLimiter: &MultiStageRequeueWithExponentialBackoffRateLimiter{ + requeueCounter: map[types.NamespacedName]int{}, + lastRequeueDelayTracker: map[types.NamespacedName]time.Duration{}, + lastTrackedLinkedWorksGenHash: map[types.NamespacedName]string{}, + lastTrackedProcessingResultHash: map[types.NamespacedName]string{}, + attemptsWithFixedDelay: 10, + fixedDelay: 5 * time.Second, + exponentialBaseForSlowBackoff: 1.2, + initialSlowBackoffDelay: 2 * time.Second, + maxSlowBackoffDelay: 15 * time.Second, + exponentialBaseForFastBackoff: 1.5, + maxFastBackoffDelay: 60 * time.Second, + skipToFastBackoffWhenAllAppliedManifestsAreAvailable: true, + }, + }, + { + name: "all parameters at their lower bounds", + cfg: rateLimiterConfig{ + attemptsWithFixedDelay: 1, + fixedDelaySeconds: 2, + exponentialBaseForSlowBackoff: 1.05, + initialSlowBackoffDelaySeconds: 2, + maxSlowBackoffDelaySeconds: 2, + exponentialBaseForFastBackoff: 1.05, + maxFastBackoffDelaySeconds: 2, + }, + wantRateLimiter: &MultiStageRequeueWithExponentialBackoffRateLimiter{ + requeueCounter: map[types.NamespacedName]int{}, + lastRequeueDelayTracker: map[types.NamespacedName]time.Duration{}, + lastTrackedLinkedWorksGenHash: map[types.NamespacedName]string{}, + lastTrackedProcessingResultHash: map[types.NamespacedName]string{}, + attemptsWithFixedDelay: 1, + fixedDelay: 2 * time.Second, + exponentialBaseForSlowBackoff: 1.05, + initialSlowBackoffDelay: 2 * time.Second, + maxSlowBackoffDelay: 2 * time.Second, + exponentialBaseForFastBackoff: 1.05, + maxFastBackoffDelay: 2 * time.Second, + }, + }, + { + name: "all parameters at their upper bounds", + cfg: rateLimiterConfig{ + attemptsWithFixedDelay: 40, + fixedDelaySeconds: 3600, + exponentialBaseForSlowBackoff: 100, + initialSlowBackoffDelaySeconds: 3600, + maxSlowBackoffDelaySeconds: 3600, + exponentialBaseForFastBackoff: 100, + maxFastBackoffDelaySeconds: 3600, + }, + wantRateLimiter: &MultiStageRequeueWithExponentialBackoffRateLimiter{ + requeueCounter: map[types.NamespacedName]int{}, + lastRequeueDelayTracker: map[types.NamespacedName]time.Duration{}, + lastTrackedLinkedWorksGenHash: map[types.NamespacedName]string{}, + lastTrackedProcessingResultHash: map[types.NamespacedName]string{}, + attemptsWithFixedDelay: 40, + fixedDelay: 3600 * time.Second, + exponentialBaseForSlowBackoff: 100, + initialSlowBackoffDelay: 3600 * time.Second, + maxSlowBackoffDelay: 3600 * time.Second, + exponentialBaseForFastBackoff: 100, + maxFastBackoffDelay: 3600 * time.Second, + }, + }, + { + name: "fixed delay equal to max fast backoff delay", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.fixedDelaySeconds = 60 }), + wantRateLimiter: &MultiStageRequeueWithExponentialBackoffRateLimiter{ + requeueCounter: map[types.NamespacedName]int{}, + lastRequeueDelayTracker: map[types.NamespacedName]time.Duration{}, + lastTrackedLinkedWorksGenHash: map[types.NamespacedName]string{}, + lastTrackedProcessingResultHash: map[types.NamespacedName]string{}, + attemptsWithFixedDelay: 10, + fixedDelay: 60 * time.Second, + exponentialBaseForSlowBackoff: 1.2, + initialSlowBackoffDelay: 2 * time.Second, + maxSlowBackoffDelay: 15 * time.Second, + exponentialBaseForFastBackoff: 1.5, + maxFastBackoffDelay: 60 * time.Second, + skipToFastBackoffWhenAllAppliedManifestsAreAvailable: true, + }, + }, + { + name: "attempts with fixed delay (zero)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.attemptsWithFixedDelay = 0 }), + wantErred: true, + }, + { + name: "attempts with fixed delay (negative)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.attemptsWithFixedDelay = -1 }), + wantErred: true, + }, + { + name: "attempts with fixed delay (too many)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.attemptsWithFixedDelay = 41 }), + wantErred: true, + }, + { + name: "fixed delay (too small)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.fixedDelaySeconds = 1 }), + wantErred: true, + }, + { + name: "fixed delay (zero)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.fixedDelaySeconds = 0 }), + wantErred: true, + }, + { + name: "fixed delay (above max fast backoff delay)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.fixedDelaySeconds = 61 }), + wantErred: true, + }, + { + name: "exponential base for slow backoff (too small)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.exponentialBaseForSlowBackoff = 1.04 }), + wantErred: true, + }, + { + name: "exponential base for slow backoff (NaN)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.exponentialBaseForSlowBackoff = math.NaN() }), + wantErred: true, + }, + { + name: "exponential base for slow backoff (above the base for fast backoff)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.exponentialBaseForSlowBackoff = 2.0 }), + wantErred: true, + }, + { + name: "exponential base for fast backoff (too large)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.exponentialBaseForFastBackoff = 100.01 }), + wantErred: true, + }, + { + name: "exponential base for fast backoff (NaN)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.exponentialBaseForFastBackoff = math.NaN() }), + wantErred: true, + }, + { + name: "initial slow backoff delay (too small)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.initialSlowBackoffDelaySeconds = 1 }), + wantErred: true, + }, + { + name: "initial slow backoff delay (above max slow backoff delay)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.initialSlowBackoffDelaySeconds = 16 }), + wantErred: true, + }, + { + name: "max slow backoff delay (above max fast backoff delay)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.maxSlowBackoffDelaySeconds = 61 }), + wantErred: true, + }, + { + name: "max fast backoff delay (too large)", + cfg: mutated(func(cfg *rateLimiterConfig) { cfg.maxFastBackoffDelaySeconds = 3601 }), + wantErred: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + rateLimiter, err := NewMultiStageRequeueWithExponentialBackoffRateLimiter( + tc.cfg.attemptsWithFixedDelay, + tc.cfg.fixedDelaySeconds, + tc.cfg.exponentialBaseForSlowBackoff, + tc.cfg.initialSlowBackoffDelaySeconds, + tc.cfg.maxSlowBackoffDelaySeconds, + tc.cfg.exponentialBaseForFastBackoff, + tc.cfg.maxFastBackoffDelaySeconds, + tc.cfg.skipToFastBackoffWhenAllAppliedManifestsAreAvailable, + ) + if tc.wantErred { + if err == nil || kferrors.Category(err) != kferrors.ErrCategoryUser { + t.Fatalf("NewMultiStageRequeueWithExponentialBackoffRateLimiter(%+v) = %v, want a user error", tc.cfg, err) + } + if rateLimiter != nil { + t.Errorf("NewMultiStageRequeueWithExponentialBackoffRateLimiter(%+v) = %+v, want nil", tc.cfg, rateLimiter) + } + return + } + if err != nil { + t.Fatalf("NewMultiStageRequeueWithExponentialBackoffRateLimiter(%+v) = %v, want no error", tc.cfg, err) + } + + // Note that the comparison does not equate nil and empty maps; the trackers must be initialized. + if diff := cmp.Diff(rateLimiter, tc.wantRateLimiter, rateLimiterCmpOpts...); diff != "" { + t.Errorf("NewMultiStageRequeueWithExponentialBackoffRateLimiter(%+v) mismatches (-got, +want):\n%s", tc.cfg, diff) + } + }) + } +} + +// TestDefaultRequeueRateLimiter tests the configuration of the default requeue rate limiter. +func TestDefaultRequeueRateLimiter(t *testing.T) { + if defaultRequeueRateLimiter == nil { + t.Fatalf("defaultRequeueRateLimiter = nil, want a rate limiter") + } + + wantRateLimiter := &MultiStageRequeueWithExponentialBackoffRateLimiter{ + attemptsWithFixedDelay: defaultRateLimiterCfg.attemptsWithFixedDelay, + fixedDelay: time.Duration(defaultRateLimiterCfg.fixedDelaySeconds) * time.Second, + exponentialBaseForSlowBackoff: defaultRateLimiterCfg.exponentialBaseForSlowBackoff, + initialSlowBackoffDelay: time.Duration(defaultRateLimiterCfg.initialSlowBackoffDelaySeconds) * time.Second, + maxSlowBackoffDelay: time.Duration(defaultRateLimiterCfg.maxSlowBackoffDelaySeconds) * time.Second, + exponentialBaseForFastBackoff: defaultRateLimiterCfg.exponentialBaseForFastBackoff, + maxFastBackoffDelay: time.Duration(defaultRateLimiterCfg.maxFastBackoffDelaySeconds) * time.Second, + skipToFastBackoffWhenAllAppliedManifestsAreAvailable: defaultRateLimiterCfg.skipToFastBackoffWhenAllAppliedManifestsAreAvailable, + } + // The default rate limiter is shared with the integration tests in the same package; compare + // only the configuration, which is never written after initialization. + if diff := cmp.Diff( + defaultRequeueRateLimiter, wantRateLimiter, + cmp.AllowUnexported(MultiStageRequeueWithExponentialBackoffRateLimiter{}), + cmpopts.IgnoreFields(MultiStageRequeueWithExponentialBackoffRateLimiter{}, + "mu", "requeueCounter", "lastRequeueDelayTracker", + "lastTrackedLinkedWorksGenHash", "lastTrackedProcessingResultHash", "forgets"), + ); diff != "" { + t.Errorf("defaultRequeueRateLimiter mismatches (-got, +want):\n%s", diff) + } +} + +// TestNewDefaultRequeueRateLimiterWithMaxDelay tests that the capped rate limiter keeps the default parameters +// and lowers only those that exceed the cap. +func TestNewDefaultRequeueRateLimiterWithMaxDelay(t *testing.T) { + // wantCapped returns the wanted rate limiter for the given (already capped) delays in seconds. + wantCapped := func(fixedDelaySeconds, initialSlowBackoffDelaySeconds, maxSlowBackoffDelaySeconds, maxFastBackoffDelaySeconds int) *MultiStageRequeueWithExponentialBackoffRateLimiter { + return &MultiStageRequeueWithExponentialBackoffRateLimiter{ + requeueCounter: map[types.NamespacedName]int{}, + lastRequeueDelayTracker: map[types.NamespacedName]time.Duration{}, + lastTrackedLinkedWorksGenHash: map[types.NamespacedName]string{}, + lastTrackedProcessingResultHash: map[types.NamespacedName]string{}, + attemptsWithFixedDelay: defaultRateLimiterCfg.attemptsWithFixedDelay, + fixedDelay: time.Duration(fixedDelaySeconds) * time.Second, + exponentialBaseForSlowBackoff: defaultRateLimiterCfg.exponentialBaseForSlowBackoff, + initialSlowBackoffDelay: time.Duration(initialSlowBackoffDelaySeconds) * time.Second, + maxSlowBackoffDelay: time.Duration(maxSlowBackoffDelaySeconds) * time.Second, + exponentialBaseForFastBackoff: defaultRateLimiterCfg.exponentialBaseForFastBackoff, + maxFastBackoffDelay: time.Duration(maxFastBackoffDelaySeconds) * time.Second, + skipToFastBackoffWhenAllAppliedManifestsAreAvailable: defaultRateLimiterCfg.skipToFastBackoffWhenAllAppliedManifestsAreAvailable, + } + } + + testCases := []struct { + name string + maxDelaySeconds int + wantRateLimiter *MultiStageRequeueWithExponentialBackoffRateLimiter + wantErred bool + }{ + { + name: "cap below the fixed delay", + maxDelaySeconds: 3, + wantRateLimiter: wantCapped(3, 2, 3, 3), + }, + { + name: "cap at the fixed delay", + maxDelaySeconds: 5, + wantRateLimiter: wantCapped(5, 2, 5, 5), + }, + { + name: "cap between the slow and fast backoff caps", + maxDelaySeconds: 60, + wantRateLimiter: wantCapped(5, 2, 15, 60), + }, + { + name: "cap above the defaults", + maxDelaySeconds: 3600, + wantRateLimiter: wantCapped( + defaultRateLimiterCfg.fixedDelaySeconds, + defaultRateLimiterCfg.initialSlowBackoffDelaySeconds, + defaultRateLimiterCfg.maxSlowBackoffDelaySeconds, + defaultRateLimiterCfg.maxFastBackoffDelaySeconds, + ), + }, + { + name: "cap below the minimum fixed delay", + maxDelaySeconds: 1, + wantErred: true, + }, + { + name: "negative cap", + maxDelaySeconds: -5, + wantErred: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + rateLimiter, err := NewDefaultRequeueRateLimiterWithMaxDelay(tc.maxDelaySeconds) + if tc.wantErred { + if err == nil || kferrors.Category(err) != kferrors.ErrCategoryUser { + t.Fatalf("NewDefaultRequeueRateLimiterWithMaxDelay(%d) = %v, want a user error", tc.maxDelaySeconds, err) + } + return + } + if err != nil { + t.Fatalf("NewDefaultRequeueRateLimiterWithMaxDelay(%d) = %v, want no error", tc.maxDelaySeconds, err) + } + if diff := cmp.Diff(rateLimiter, tc.wantRateLimiter, rateLimiterCmpOpts...); diff != "" { + t.Errorf("NewDefaultRequeueRateLimiterWithMaxDelay(%d) mismatches (-got, +want):\n%s", tc.maxDelaySeconds, diff) + } + }) + } +} + +// TestWhenWithFullSequences tests the When method with full requeue sequences. +func TestWhenWithFullSequences(t *testing.T) { + linkedWorks := []*placementv1alpha1.Work{ + workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1), + } + states := []*manifestProcessingState{ + stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable), + } + + testCases := []struct { + name string + cfg rateLimiterConfig + wantDelaysSeconds []float64 + }{ + { + // This sequence matches the requeue pattern documented with the default rate limiter. + name: "default configuration", + cfg: defaultRateLimiterCfg, + wantDelaysSeconds: []float64{ + // 1 attempt with fixed delay. + 5, + // 12 attempts with slow backoff (base 1.2). + 2, 2.4, 2.88, 3.456, 4.1472, 4.9766, 5.9720, 7.1664, 8.5996, 10.3196, 12.3835, 14.8602, + // 10 attempts with fast backoff (base 1.5), as 14.8602 * 1.2 breaks the slow backoff cap (15). + 22.2903, 33.4354, 50.1531, 75.2296, 112.8444, 169.2666, 253.8999, 380.8498, 571.2748, 856.9121, + // Capped. + 900, 900, + }, + }, + { + name: "multiple attempts with fixed delay, full slow and fast backoff stages", + cfg: rateLimiterConfig{ + attemptsWithFixedDelay: 2, + fixedDelaySeconds: 5, + exponentialBaseForSlowBackoff: 1.2, + initialSlowBackoffDelaySeconds: 2, + maxSlowBackoffDelaySeconds: 15, + exponentialBaseForFastBackoff: 1.5, + maxFastBackoffDelaySeconds: 60, + }, + wantDelaysSeconds: []float64{ + 5, 5, + 2, 2.4, 2.88, 3.456, 4.1472, 4.9766, 5.9720, 7.1664, 8.5996, 10.3196, 12.3835, 14.8602, + 22.2903, 33.4354, 50.1531, + // 50.1531 * 1.5 = 75.2296, capped at 60. + 60, 60, + }, + }, + { + name: "no slow backoff stage", + cfg: rateLimiterConfig{ + attemptsWithFixedDelay: 1, + fixedDelaySeconds: 5, + exponentialBaseForSlowBackoff: 2, + initialSlowBackoffDelaySeconds: 2, + maxSlowBackoffDelaySeconds: 2, + exponentialBaseForFastBackoff: 5, + maxFastBackoffDelaySeconds: 20, + }, + // The initial slow backoff delay reaches the slow backoff cap; fast backoff starts immediately + // (2 * 5 = 10), then 10 * 5 = 50 is capped at 20. + wantDelaysSeconds: []float64{5, 2, 10, 20, 20}, + }, + { + name: "no fast backoff stage", + cfg: rateLimiterConfig{ + attemptsWithFixedDelay: 1, + fixedDelaySeconds: 5, + exponentialBaseForSlowBackoff: 2, + initialSlowBackoffDelaySeconds: 2, + maxSlowBackoffDelaySeconds: 10, + exponentialBaseForFastBackoff: 5, + maxFastBackoffDelaySeconds: 10, + }, + // 8 * 2 = 16 breaks the slow backoff cap; 8 * 5 = 40 is capped at 10. + wantDelaysSeconds: []float64{5, 2, 4, 8, 10, 10}, + }, + { + name: "no backoff at all", + cfg: rateLimiterConfig{ + attemptsWithFixedDelay: 1, + fixedDelaySeconds: 5, + exponentialBaseForSlowBackoff: 2, + initialSlowBackoffDelaySeconds: 10, + maxSlowBackoffDelaySeconds: 10, + exponentialBaseForFastBackoff: 5, + maxFastBackoffDelaySeconds: 10, + }, + wantDelaysSeconds: []float64{5, 10, 10, 10}, + }, + { + name: "slow backoff lands exactly on the slow backoff cap", + cfg: rateLimiterConfig{ + attemptsWithFixedDelay: 1, + fixedDelaySeconds: 5, + exponentialBaseForSlowBackoff: 2, + initialSlowBackoffDelaySeconds: 2, + maxSlowBackoffDelaySeconds: 8, + exponentialBaseForFastBackoff: 4, + maxFastBackoffDelaySeconds: 100, + }, + // 4 * 2 = 8 does not break the slow backoff cap; afterwards, fast backoff starts (8 * 4 = 32), + // then 32 * 4 = 128 is capped at 100. + wantDelaysSeconds: []float64{5, 2, 4, 8, 32, 100, 100}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + rateLimiter := newRateLimiterForTest(t, tc.cfg) + verifyRequeueSteps(t, rateLimiter, requeueSteps(linkedWorks, states, tc.wantDelaysSeconds...)) + }) + } +} + +// TestWhenWithLinkedWorksAndProcessingResultChanges tests the When method when the linked work objects +// and/or the processing results change. +func TestWhenWithLinkedWorksAndProcessingResultChanges(t *testing.T) { + primaryWorkGen1 := workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1) + primaryWorkGen2 := workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 2) + secondaryWorkGen1 := workForBackoffTest(memberReservedNSName1, backoffTestSecondaryWorkName, 1) + secondaryWorkGen2 := workForBackoffTest(memberReservedNSName1, backoffTestSecondaryWorkName, 2) + primaryWorkGen1WithStatus := workWithLabelsAndStatusForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1) + + notYetAvailableStates := []*manifestProcessingState{ + stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable), + } + availableStates := []*manifestProcessingState{ + stateWith(ApplyResTypeApplied, AvailabilityResultTypeAvailable), + } + notTrackableStates := []*manifestProcessingState{ + stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotTrackable), + } + failedStates := []*manifestProcessingState{ + stateWith(ApplyResTypeFailedToApply, AvailabilityResultTypeSkipped), + stateWith(ApplyResTypeApplied, AvailabilityResultTypeFailed), + stateWith(ApplyResTypeFoundDrifts, AvailabilityResultTypeSkipped), + } + // The same processing results as failedStates, with different errors and diffs. + failedStatesWithDetails := []*manifestProcessingState{ + { + applyRes: ApplyResTypeFailedToApply, + availabilityCheckRes: AvailabilityResultTypeSkipped, + applyErr: fmt.Errorf("failed to apply the manifest"), + }, + { + applyRes: ApplyResTypeApplied, + availabilityCheckRes: AvailabilityResultTypeFailed, + availabilityCheckErr: fmt.Errorf("failed to check the availability"), + }, + { + applyRes: ApplyResTypeFoundDrifts, + availabilityCheckRes: AvailabilityResultTypeSkipped, + diffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "1", + ValueInHub: "2", + }, + }, + }, + } + + testCases := []struct { + name string + steps []requeueStep + }{ + { + name: "primary work object generation change", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, notYetAvailableStates, 5, 10, 20), + // Reset. + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen2}, notYetAvailableStates, 5, 10), + ), + }, + { + name: "secondary work object generation change", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1, secondaryWorkGen1}, notYetAvailableStates, 5, 10, 20, 100), + // Reset. + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1, secondaryWorkGen2}, notYetAvailableStates, 5, 10), + ), + }, + { + name: "linked work object added", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, notYetAvailableStates, 5, 10), + // Reset. + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1, secondaryWorkGen1}, notYetAvailableStates, 5, 10), + ), + }, + { + name: "linked work object removed", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1, secondaryWorkGen1}, notYetAvailableStates, 5, 10), + // Reset. + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, notYetAvailableStates, 5, 10), + ), + }, + { + name: "processing result change", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, notYetAvailableStates, 5, 10, 20, 100, 200), + // Reset. + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, availableStates, 5, 10), + ), + }, + { + name: "both generation and processing result change", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, notYetAvailableStates, 5, 10, 20), + // Reset. + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen2}, availableStates, 5, 10), + ), + }, + { + name: "availability check result change between Available and NotTrackable", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, availableStates, 5, 10, 20), + // Reset, even though both results count as available. + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, notTrackableStates, 5), + ), + }, + { + name: "error and diff changes only (no reset)", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, failedStates, 5, 10, 20), + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, failedStatesWithDetails, 100), + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, failedStates, 200), + ), + }, + { + name: "work object label and status changes only (no reset)", + steps: slices.Concat( + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1}, notYetAvailableStates, 5, 10, 20), + requeueSteps([]*placementv1alpha1.Work{primaryWorkGen1WithStatus}, notYetAvailableStates, 100), + ), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + rateLimiter := newRateLimiterForTest(t, simpleRateLimiterCfg) + verifyRequeueSteps(t, rateLimiter, tc.steps) + }) + } +} + +// TestWhenWithSkipToFastBackoff tests the When method with the skip to fast backoff option. +func TestWhenWithSkipToFastBackoff(t *testing.T) { + linkedWorks := []*placementv1alpha1.Work{ + workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1), + } + available := stateWith(ApplyResTypeApplied, AvailabilityResultTypeAvailable) + notTrackable := stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotTrackable) + notYetAvailable := stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable) + failedToCheck := stateWith(ApplyResTypeApplied, AvailabilityResultTypeFailed) + skipped := stateWith(ApplyResTypeFailedToApply, AvailabilityResultTypeSkipped) + + skipEnabledCfg := simpleRateLimiterCfg + skipEnabledCfg.skipToFastBackoffWhenAllAppliedManifestsAreAvailable = true + + testCases := []struct { + name string + cfg rateLimiterConfig + steps []requeueStep + }{ + { + // The skip never applies in the fixed delay stage or at the start of the slow backoff stage; + // afterwards, 10 * 5 = 50, then 50 * 5 = 250 is capped at 200. + name: "all manifests available", + cfg: skipEnabledCfg, + steps: requeueSteps(linkedWorks, []*manifestProcessingState{available, available}, 5, 10, 50, 200, 200), + }, + { + name: "all manifests not trackable", + cfg: skipEnabledCfg, + steps: requeueSteps(linkedWorks, []*manifestProcessingState{notTrackable, notTrackable}, 5, 10, 50, 200, 200), + }, + { + name: "mixed available and not trackable manifests", + cfg: skipEnabledCfg, + steps: requeueSteps(linkedWorks, []*manifestProcessingState{available, notTrackable}, 5, 10, 50, 200, 200), + }, + { + // With no manifests, all processed manifests are considered to be available. + name: "no manifests", + cfg: skipEnabledCfg, + steps: requeueSteps(linkedWorks, []*manifestProcessingState{}, 5, 10, 50, 200, 200), + }, + { + name: "one manifest not yet available", + cfg: skipEnabledCfg, + steps: requeueSteps(linkedWorks, []*manifestProcessingState{available, notYetAvailable}, 5, 10, 20, 100, 200), + }, + { + name: "one manifest failed availability check", + cfg: skipEnabledCfg, + steps: requeueSteps(linkedWorks, []*manifestProcessingState{available, failedToCheck}, 5, 10, 20, 100, 200), + }, + { + name: "one manifest skipped availability check (failed to apply)", + cfg: skipEnabledCfg, + steps: requeueSteps(linkedWorks, []*manifestProcessingState{available, skipped}, 5, 10, 20, 100, 200), + }, + { + name: "all manifests available, skip disabled", + cfg: simpleRateLimiterCfg, + steps: requeueSteps(linkedWorks, []*manifestProcessingState{available, available}, 5, 10, 20, 100, 200), + }, + { + name: "manifests become available", + cfg: skipEnabledCfg, + steps: slices.Concat( + requeueSteps(linkedWorks, []*manifestProcessingState{notYetAvailable}, 5, 10, 20), + // Reset, as the processing result has changed. + requeueSteps(linkedWorks, []*manifestProcessingState{available}, 5, 10, 50, 200), + ), + }, + { + name: "manifests become unavailable", + cfg: skipEnabledCfg, + steps: slices.Concat( + requeueSteps(linkedWorks, []*manifestProcessingState{available}, 5, 10, 50), + // Reset, as the processing result has changed. + requeueSteps(linkedWorks, []*manifestProcessingState{notYetAvailable}, 5, 10, 20, 100), + ), + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + rateLimiter := newRateLimiterForTest(t, tc.cfg) + verifyRequeueSteps(t, rateLimiter, tc.steps) + }) + } +} + +// TestWhenWithMultipleWorks tests the When method with multiple work objects tracked at the same time. +func TestWhenWithMultipleWorks(t *testing.T) { + workA := workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1) + workB := workForBackoffTest(memberReservedNSName1, backoffTestOtherWorkName, 1) + // A work object with the same name as work A, but in a different namespace. + workC := workForBackoffTest(backoffTestAltNSName, backoffTestPrimaryWorkName, 1) + states := []*manifestProcessingState{ + stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable), + } + + rateLimiter := newRateLimiterForTest(t, simpleRateLimiterCfg) + verifyRequeueSteps(t, rateLimiter, slices.Concat( + requeueSteps([]*placementv1alpha1.Work{workA}, states, 5, 10), + requeueSteps([]*placementv1alpha1.Work{workB}, states, 5), + requeueSteps([]*placementv1alpha1.Work{workA}, states, 20), + requeueSteps([]*placementv1alpha1.Work{workC}, states, 5), + requeueSteps([]*placementv1alpha1.Work{workB}, states, 10), + requeueSteps([]*placementv1alpha1.Work{workC}, states, 10), + requeueSteps([]*placementv1alpha1.Work{workA}, states, 100), + )) + + keyA := types.NamespacedName{Namespace: memberReservedNSName1, Name: backoffTestPrimaryWorkName} + keyB := types.NamespacedName{Namespace: memberReservedNSName1, Name: backoffTestOtherWorkName} + keyC := types.NamespacedName{Namespace: backoffTestAltNSName, Name: backoffTestPrimaryWorkName} + wantRequeueCounter := map[types.NamespacedName]int{ + keyA: 4, + keyB: 2, + keyC: 2, + } + if diff := cmp.Diff(rateLimiter.requeueCounter, wantRequeueCounter); diff != "" { + t.Errorf("requeueCounter mismatches (-got, +want):\n%s", diff) + } + wantLastRequeueDelayTracker := map[types.NamespacedName]time.Duration{ + keyA: 100 * time.Second, + keyB: 10 * time.Second, + keyC: 10 * time.Second, + } + if diff := cmp.Diff(rateLimiter.lastRequeueDelayTracker, wantLastRequeueDelayTracker); diff != "" { + t.Errorf("lastRequeueDelayTracker mismatches (-got, +want):\n%s", diff) + } +} + +// TestWhenWithRequeueCounterCap tests the When method when the requeue counter reaches its cap. +func TestWhenWithRequeueCounterCap(t *testing.T) { + linkedWorks := []*placementv1alpha1.Work{ + workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1), + } + states := []*manifestProcessingState{ + stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable), + } + key := types.NamespacedName{Namespace: memberReservedNSName1, Name: backoffTestPrimaryWorkName} + wantRequeueCounter := maxNumOfRequeuesToTrack - 1 + + t.Run("cap is above the fixed delay and initial slow backoff stages", func(t *testing.T) { + // The requeue counter is only consulted in the fixed delay stage and at the start of the slow backoff + // stage; capping it must never send a work object back to these stages. + if maxNumOfRequeuesToTrack <= attemptsWithFixedDelayUpperBound+1 { + t.Fatalf("maxNumOfRequeuesToTrack = %d, want > %d", maxNumOfRequeuesToTrack, attemptsWithFixedDelayUpperBound+1) + } + }) + + t.Run("counter stops at the cap", func(t *testing.T) { + rateLimiter := newRateLimiterForTest(t, defaultRateLimiterCfg) + var gotDelay time.Duration + for range maxNumOfRequeuesToTrack + 50 { + gotDelay = rateLimiter.When(linkedWorks, states) + } + + wantDelay := time.Duration(defaultRateLimiterCfg.maxFastBackoffDelaySeconds) * time.Second + if gotDelay != wantDelay { + t.Errorf("When() = %v, want %v", gotDelay, wantDelay) + } + if gotRequeueCounter := rateLimiter.requeueCounter[key]; gotRequeueCounter != wantRequeueCounter { + t.Errorf("requeueCounter[%v] = %d, want %d", key, gotRequeueCounter, wantRequeueCounter) + } + }) + + t.Run("backoff continues at the cap", func(t *testing.T) { + linkedWorksGenHash, err := computeLinkedWorksObjectGenHash(linkedWorks) + if err != nil { + t.Fatalf("computeLinkedWorksObjectGenHash() = %v, want no error", err) + } + processingResultHash, err := computeProcessingResultHash(states) + if err != nil { + t.Fatalf("computeProcessingResultHash() = %v, want no error", err) + } + rateLimiter := withTrackedRequeueStates( + newRateLimiterForTest(t, defaultRateLimiterCfg), + map[types.NamespacedName]trackedRequeueState{ + key: { + requeues: wantRequeueCounter, + lastDelay: 20 * time.Second, + linkedWorksGenHash: linkedWorksGenHash, + processingResultHash: processingResultHash, + }, + }, + 0, + ) + + // 20 seconds is above the slow backoff cap (15 seconds); fast backoff continues (20 * 1.5 = 30). + gotDelay := rateLimiter.When(linkedWorks, states) + if wantDelay := 30 * time.Second; gotDelay != wantDelay { + t.Errorf("When() = %v, want %v", gotDelay, wantDelay) + } + if gotRequeueCounter := rateLimiter.requeueCounter[key]; gotRequeueCounter != wantRequeueCounter { + t.Errorf("requeueCounter[%v] = %d, want %d", key, gotRequeueCounter, wantRequeueCounter) + } + }) +} + +// TestForget tests the Forget method. +func TestForget(t *testing.T) { + work1 := workForBackoffTest(memberReservedNSName1, fmt.Sprintf(workNameTemplate, "1"), 1) + work2 := workForBackoffTest(memberReservedNSName1, fmt.Sprintf(workNameTemplate, "2"), 1) + work3 := workForBackoffTest(memberReservedNSName1, fmt.Sprintf(workNameTemplate, "3"), 1) + key1 := types.NamespacedName{Namespace: work1.Namespace, Name: work1.Name} + key2 := types.NamespacedName{Namespace: work2.Namespace, Name: work2.Name} + key3 := types.NamespacedName{Namespace: work3.Namespace, Name: work3.Name} + trackedState1 := trackedRequeueState{ + requeues: 1, + lastDelay: 5 * time.Second, + linkedWorksGenHash: "gen-hash-1", + processingResultHash: "res-hash-1", + } + trackedState2 := trackedRequeueState{ + requeues: 5, + lastDelay: 20 * time.Second, + linkedWorksGenHash: "gen-hash-2", + processingResultHash: "res-hash-2", + } + trackedState3 := trackedRequeueState{ + requeues: 9, + lastDelay: 200 * time.Second, + linkedWorksGenHash: "gen-hash-3", + processingResultHash: "res-hash-3", + } + + testCases := []struct { + name string + trackedStates map[types.NamespacedName]trackedRequeueState + forgets int + work *placementv1alpha1.Work + wantTrackedStates map[types.NamespacedName]trackedRequeueState + wantForgets int + }{ + { + name: "forget tracked work", + trackedStates: map[types.NamespacedName]trackedRequeueState{ + key1: trackedState1, + key2: trackedState2, + key3: trackedState3, + }, + work: work2, + wantTrackedStates: map[types.NamespacedName]trackedRequeueState{ + key1: trackedState1, + key3: trackedState3, + }, + wantForgets: 1, + }, + { + name: "forget untracked work", + trackedStates: map[types.NamespacedName]trackedRequeueState{ + key1: trackedState1, + key3: trackedState3, + }, + forgets: 10, + work: work2, + wantTrackedStates: map[types.NamespacedName]trackedRequeueState{ + key1: trackedState1, + key3: trackedState3, + }, + wantForgets: 11, + }, + { + name: "forget tracked work, triggering garbage collection", + trackedStates: map[types.NamespacedName]trackedRequeueState{ + key1: trackedState1, + key2: trackedState2, + key3: trackedState3, + }, + forgets: gcTriggeringForgetThreshold - 1, + work: work2, + wantTrackedStates: map[types.NamespacedName]trackedRequeueState{ + key1: trackedState1, + key3: trackedState3, + }, + wantForgets: 0, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + rateLimiter := withTrackedRequeueStates(newRateLimiterForTest(t, simpleRateLimiterCfg), tc.trackedStates, tc.forgets) + rateLimiter.Forget(tc.work) + + wantRateLimiter := withTrackedRequeueStates(newRateLimiterForTest(t, simpleRateLimiterCfg), tc.wantTrackedStates, tc.wantForgets) + if diff := cmp.Diff(rateLimiter, wantRateLimiter, rateLimiterCmpOpts...); diff != "" { + t.Errorf("Forget() mismatches (-got, +want):\n%s", diff) + } + + // After the work object is forgotten, its requeues start over with the fixed delay. + gotDelay := rateLimiter.When( + []*placementv1alpha1.Work{tc.work}, + []*manifestProcessingState{stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable)}, + ) + if wantDelay := time.Duration(simpleRateLimiterCfg.fixedDelaySeconds) * time.Second; gotDelay != wantDelay { + t.Errorf("When() after Forget() = %v, want %v", gotDelay, wantDelay) + } + }) + } +} + +// TestGarbageCollect tests the garbageCollect method. +func TestGarbageCollect(t *testing.T) { + testCases := []struct { + name string + trackedStates map[types.NamespacedName]trackedRequeueState + forgets int + }{ + { + name: "populated trackers", + trackedStates: map[types.NamespacedName]trackedRequeueState{ + {Namespace: memberReservedNSName1, Name: backoffTestPrimaryWorkName}: { + requeues: 3, + lastDelay: 20 * time.Second, + linkedWorksGenHash: "gen-hash-1", + processingResultHash: "res-hash-1", + }, + {Namespace: backoffTestAltNSName, Name: backoffTestOtherWorkName}: { + requeues: 7, + lastDelay: 200 * time.Second, + linkedWorksGenHash: "gen-hash-2", + processingResultHash: "res-hash-2", + }, + }, + forgets: 42, + }, + { + name: "empty trackers", + trackedStates: map[types.NamespacedName]trackedRequeueState{}, + forgets: 7, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + rateLimiter := withTrackedRequeueStates(newRateLimiterForTest(t, simpleRateLimiterCfg), tc.trackedStates, tc.forgets) + oldTrackers := []any{ + rateLimiter.requeueCounter, + rateLimiter.lastRequeueDelayTracker, + rateLimiter.lastTrackedLinkedWorksGenHash, + rateLimiter.lastTrackedProcessingResultHash, + } + + rateLimiter.garbageCollect() + + // Note that the comparison does not equate nil and empty maps; the rebuilt trackers must be initialized. + wantRateLimiter := withTrackedRequeueStates(newRateLimiterForTest(t, simpleRateLimiterCfg), tc.trackedStates, 0) + if diff := cmp.Diff(rateLimiter, wantRateLimiter, rateLimiterCmpOpts...); diff != "" { + t.Errorf("garbageCollect() mismatches (-got, +want):\n%s", diff) + } + + newTrackers := []any{ + rateLimiter.requeueCounter, + rateLimiter.lastRequeueDelayTracker, + rateLimiter.lastTrackedLinkedWorksGenHash, + rateLimiter.lastTrackedProcessingResultHash, + } + for idx := range oldTrackers { + if reflect.ValueOf(oldTrackers[idx]).Pointer() == reflect.ValueOf(newTrackers[idx]).Pointer() { + t.Errorf("garbageCollect() did not rebuild tracker #%d, want a new map", idx) + } + } + + // The rate limiter should remain usable after garbage collection. + gotDelay := rateLimiter.When( + []*placementv1alpha1.Work{workForBackoffTest(memberReservedNSName1, backoffTestOtherWorkName, 1)}, + []*manifestProcessingState{stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable)}, + ) + if wantDelay := time.Duration(simpleRateLimiterCfg.fixedDelaySeconds) * time.Second; gotDelay != wantDelay { + t.Errorf("When() after garbageCollect() = %v, want %v", gotDelay, wantDelay) + } + }) + } +} + +// TestComputeLinkedWorksObjectGenHash tests the computeLinkedWorksObjectGenHash function. +func TestComputeLinkedWorksObjectGenHash(t *testing.T) { + primaryWorkGen1 := workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1) + secondaryWorkGen1 := workForBackoffTest(memberReservedNSName1, backoffTestSecondaryWorkName, 1) + + t.Run("no linked work objects", func(t *testing.T) { + for _, linkedWorks := range [][]*placementv1alpha1.Work{nil, {}} { + gotHash, err := computeLinkedWorksObjectGenHash(linkedWorks) + if err != nil { + t.Fatalf("computeLinkedWorksObjectGenHash(%v) = %v, want no error", linkedWorks, err) + } + if gotHash != emptyJSONListSHA256 { + t.Errorf("computeLinkedWorksObjectGenHash(%v) = %s, want %s", linkedWorks, gotHash, emptyJSONListSHA256) + } + } + }) + + primaryWorkGen1WithStatus := workWithLabelsAndStatusForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1) + + testCases := []struct { + name string + linkedWorksA []*placementv1alpha1.Work + linkedWorksB []*placementv1alpha1.Work + wantSameHash bool + }{ + { + name: "same names and generations (different objects)", + linkedWorksA: []*placementv1alpha1.Work{primaryWorkGen1, secondaryWorkGen1}, + linkedWorksB: []*placementv1alpha1.Work{ + workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 1), + workForBackoffTest(memberReservedNSName1, backoffTestSecondaryWorkName, 1), + }, + wantSameHash: true, + }, + { + name: "labels and status do not count", + linkedWorksA: []*placementv1alpha1.Work{primaryWorkGen1}, + linkedWorksB: []*placementv1alpha1.Work{primaryWorkGen1WithStatus}, + wantSameHash: true, + }, + { + // Linked work objects always share the same namespace. + name: "namespace does not count", + linkedWorksA: []*placementv1alpha1.Work{primaryWorkGen1}, + linkedWorksB: []*placementv1alpha1.Work{workForBackoffTest(backoffTestAltNSName, backoffTestPrimaryWorkName, 1)}, + wantSameHash: true, + }, + { + name: "primary work object generation change", + linkedWorksA: []*placementv1alpha1.Work{primaryWorkGen1}, + linkedWorksB: []*placementv1alpha1.Work{workForBackoffTest(memberReservedNSName1, backoffTestPrimaryWorkName, 2)}, + }, + { + name: "secondary work object generation change", + linkedWorksA: []*placementv1alpha1.Work{primaryWorkGen1, secondaryWorkGen1}, + linkedWorksB: []*placementv1alpha1.Work{ + primaryWorkGen1, + workForBackoffTest(memberReservedNSName1, backoffTestSecondaryWorkName, 2), + }, + }, + { + name: "linked work object added", + linkedWorksA: []*placementv1alpha1.Work{primaryWorkGen1}, + linkedWorksB: []*placementv1alpha1.Work{primaryWorkGen1, secondaryWorkGen1}, + }, + { + name: "linked work object order change", + linkedWorksA: []*placementv1alpha1.Work{primaryWorkGen1, secondaryWorkGen1}, + linkedWorksB: []*placementv1alpha1.Work{secondaryWorkGen1, primaryWorkGen1}, + }, + { + name: "name change with the same generation", + linkedWorksA: []*placementv1alpha1.Work{primaryWorkGen1}, + linkedWorksB: []*placementv1alpha1.Work{workForBackoffTest(memberReservedNSName1, backoffTestOtherWorkName, 1)}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + hashA, err := computeLinkedWorksObjectGenHash(tc.linkedWorksA) + if err != nil { + t.Fatalf("computeLinkedWorksObjectGenHash(A) = %v, want no error", err) + } + hashB, err := computeLinkedWorksObjectGenHash(tc.linkedWorksB) + if err != nil { + t.Fatalf("computeLinkedWorksObjectGenHash(B) = %v, want no error", err) + } + if gotSameHash := hashA == hashB; gotSameHash != tc.wantSameHash { + t.Errorf("computeLinkedWorksObjectGenHash(A) == computeLinkedWorksObjectGenHash(B): %t, want %t (A: %s, B: %s)", + gotSameHash, tc.wantSameHash, hashA, hashB) + } + }) + } +} + +// TestComputeProcessingResultHash tests the computeProcessingResultHash function. +func TestComputeProcessingResultHash(t *testing.T) { + t.Run("no manifests", func(t *testing.T) { + for _, states := range [][]*manifestProcessingState{nil, {}} { + gotHash, err := computeProcessingResultHash(states) + if err != nil { + t.Fatalf("computeProcessingResultHash(%v) = %v, want no error", states, err) + } + if gotHash != emptyJSONListSHA256 { + t.Errorf("computeProcessingResultHash(%v) = %s, want %s", states, gotHash, emptyJSONListSHA256) + } + } + }) + + failedToApply := stateWith(ApplyResTypeFailedToApply, AvailabilityResultTypeSkipped) + available := stateWith(ApplyResTypeApplied, AvailabilityResultTypeAvailable) + notTrackable := stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotTrackable) + notYetAvailable := stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable) + + testCases := []struct { + name string + statesA []*manifestProcessingState + statesB []*manifestProcessingState + wantSameHash bool + }{ + { + name: "apply error does not count", + statesA: []*manifestProcessingState{failedToApply}, + statesB: []*manifestProcessingState{ + { + applyRes: ApplyResTypeFailedToApply, + availabilityCheckRes: AvailabilityResultTypeSkipped, + applyErr: fmt.Errorf("failed to apply the manifest"), + }, + }, + wantSameHash: true, + }, + { + name: "availability check error does not count", + statesA: []*manifestProcessingState{stateWith(ApplyResTypeApplied, AvailabilityResultTypeFailed)}, + statesB: []*manifestProcessingState{ + { + applyRes: ApplyResTypeApplied, + availabilityCheckRes: AvailabilityResultTypeFailed, + availabilityCheckErr: fmt.Errorf("failed to check the availability"), + }, + }, + wantSameHash: true, + }, + { + name: "diffs do not count", + statesA: []*manifestProcessingState{stateWith(ApplyResTypeFoundDrifts, AvailabilityResultTypeSkipped)}, + statesB: []*manifestProcessingState{ + { + applyRes: ApplyResTypeFoundDrifts, + availabilityCheckRes: AvailabilityResultTypeSkipped, + diffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "1", + ValueInHub: "2", + }, + }, + }, + }, + wantSameHash: true, + }, + { + name: "manifest identity does not count", + statesA: []*manifestProcessingState{available}, + statesB: []*manifestProcessingState{ + { + id: &placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + Name: configMapName, + Namespace: ns.Name, + }, + idStr: "GV=/v1, Kind=ConfigMap, Namespace=ns-1, Name=configmap-1", + fromWorkObj: workForBackoffTest(memberReservedNSName1, backoffTestSecondaryWorkName, 1), + applyRes: ApplyResTypeApplied, + availabilityCheckRes: AvailabilityResultTypeAvailable, + }, + }, + wantSameHash: true, + }, + { + // This is a case that normally should not occur. + name: "zero-value state vs. no manifests", + statesA: []*manifestProcessingState{}, + statesB: []*manifestProcessingState{{}}, + }, + { + name: "availability check result change (NotYetAvailable vs. Available)", + statesA: []*manifestProcessingState{notYetAvailable}, + statesB: []*manifestProcessingState{available}, + }, + { + name: "availability check result change (Available vs. NotTrackable)", + statesA: []*manifestProcessingState{available}, + statesB: []*manifestProcessingState{notTrackable}, + }, + { + name: "apply result change", + statesA: []*manifestProcessingState{stateWith(ApplyResTypeDecodingErred, AvailabilityResultTypeSkipped)}, + statesB: []*manifestProcessingState{failedToApply}, + }, + { + name: "manifest order change", + statesA: []*manifestProcessingState{failedToApply, available, notTrackable}, + statesB: []*manifestProcessingState{available, failedToApply, notTrackable}, + }, + { + name: "manifest added", + statesA: []*manifestProcessingState{available}, + statesB: []*manifestProcessingState{available, available}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + hashA, err := computeProcessingResultHash(tc.statesA) + if err != nil { + t.Fatalf("computeProcessingResultHash(A) = %v, want no error", err) + } + hashB, err := computeProcessingResultHash(tc.statesB) + if err != nil { + t.Fatalf("computeProcessingResultHash(B) = %v, want no error", err) + } + if gotSameHash := hashA == hashB; gotSameHash != tc.wantSameHash { + t.Errorf("computeProcessingResultHash(A) == computeProcessingResultHash(B): %t, want %t (A: %s, B: %s)", + gotSameHash, tc.wantSameHash, hashA, hashB) + } + }) + } +} + +// TestAreAllProcessedManifestsAvailable tests the areAllProcessedManifestsAvailable function. +func TestAreAllProcessedManifestsAvailable(t *testing.T) { + available := stateWith(ApplyResTypeApplied, AvailabilityResultTypeAvailable) + notTrackable := stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotTrackable) + notYetAvailable := stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable) + + testCases := []struct { + name string + states []*manifestProcessingState + want bool + }{ + { + name: "nil", + states: nil, + want: true, + }, + { + name: "no manifests", + states: []*manifestProcessingState{}, + want: true, + }, + { + name: "available", + states: []*manifestProcessingState{available}, + want: true, + }, + { + name: "not trackable", + states: []*manifestProcessingState{notTrackable}, + want: true, + }, + { + name: "not yet available", + states: []*manifestProcessingState{notYetAvailable}, + want: false, + }, + { + name: "failed to check availability", + states: []*manifestProcessingState{stateWith(ApplyResTypeApplied, AvailabilityResultTypeFailed)}, + want: false, + }, + { + name: "availability check skipped", + states: []*manifestProcessingState{stateWith(ApplyResTypeFailedToApply, AvailabilityResultTypeSkipped)}, + want: false, + }, + { + name: "no availability check result", + states: []*manifestProcessingState{stateWith(ApplyResTypePreparingToProcess, "")}, + want: false, + }, + { + name: "mixed available and not trackable", + states: []*manifestProcessingState{available, notTrackable}, + want: true, + }, + { + name: "mixed available and not yet available", + states: []*manifestProcessingState{available, notYetAvailable}, + want: false, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := areAllProcessedManifestsAvailable(tc.states); got != tc.want { + t.Errorf("areAllProcessedManifestsAvailable() = %t, want %t", got, tc.want) + } + }) + } +} + +// TestRateLimiterConcurrentAccess tests that the rate limiter can be used concurrently. +// +// Run with the race detector on for this test to be meaningful. +func TestRateLimiterConcurrentAccess(t *testing.T) { + const ( + concurrentWorkerCount = 16 + iterationsPerWorker = 100 + forgetEveryNIterations = 10 + workCount = 4 + ) + + works := make([]*placementv1alpha1.Work, 0, workCount) + for idx := range workCount { + works = append(works, workForBackoffTest(memberReservedNSName1, fmt.Sprintf(workNameTemplate, fmt.Sprint(idx)), 1)) + } + states := []*manifestProcessingState{ + stateWith(ApplyResTypeApplied, AvailabilityResultTypeNotYetAvailable), + } + + rateLimiter := newRateLimiterForTest(t, defaultRateLimiterCfg) + var wg sync.WaitGroup + for workerIdx := range concurrentWorkerCount { + wg.Go(func() { + for iterIdx := range iterationsPerWorker { + // Workers share work objects with each other. + work := works[(workerIdx+iterIdx)%workCount] + rateLimiter.When([]*placementv1alpha1.Work{work}, states) + if (iterIdx+1)%forgetEveryNIterations == 0 { + rateLimiter.Forget(work) + } + } + }) + } + wg.Wait() + + wantForgets := concurrentWorkerCount * iterationsPerWorker / forgetEveryNIterations + if rateLimiter.forgets != wantForgets { + t.Errorf("forgets = %d, want %d", rateLimiter.forgets, wantForgets) + } + + // All trackers should always track the same set of work objects. + wantKeys := sets.KeySet(rateLimiter.requeueCounter) + if diff := cmp.Diff(sets.KeySet(rateLimiter.lastRequeueDelayTracker), wantKeys); diff != "" { + t.Errorf("lastRequeueDelayTracker keys mismatch (-got, +want):\n%s", diff) + } + if diff := cmp.Diff(sets.KeySet(rateLimiter.lastTrackedLinkedWorksGenHash), wantKeys); diff != "" { + t.Errorf("lastTrackedLinkedWorksGenHash keys mismatch (-got, +want):\n%s", diff) + } + if diff := cmp.Diff(sets.KeySet(rateLimiter.lastTrackedProcessingResultHash), wantKeys); diff != "" { + t.Errorf("lastTrackedProcessingResultHash keys mismatch (-got, +want):\n%s", diff) + } +} diff --git a/pkg/v1/controllers/workapplier/cleanup.go b/pkg/v1/controllers/workapplier/cleanup.go new file mode 100644 index 000000000..f1bab14dc --- /dev/null +++ b/pkg/v1/controllers/workapplier/cleanup.go @@ -0,0 +1,353 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + utilerrors "k8s.io/apimachinery/pkg/util/errors" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" + "go.goms.io/fleet/pkg/v1/utils/ownerreferences" +) + +const ( + DefaultCleanupBaseDelay = 2 * time.Second + DefaultCleanupMaxDelay = 30 * time.Second +) + +// cleanupWhenBindingDeleted performs cleanup ops when the placement binding that spawns all the linked +// work objects has been deleted, based on the given SyncStrategy. +func (r *Reconciler) cleanupWhenBindingDeleted(ctx context.Context, + primaryWork *placementv1alpha1.Work) (*time.Duration, error) { + primaryWorkCopy := primaryWork.DeepCopy() + primaryWorkNamespacedName := types.NamespacedName{Namespace: primaryWork.Namespace, Name: primaryWork.Name} + setDefaultSyncStrategy(primaryWorkCopy) + + deletionPolicy := metav1.DeletePropagationForeground + if primaryWorkCopy.Spec.SyncStrategy.WhenPlacementDeleted == placementv1alpha1.WhenPlacementDeletedOptionOrphanResources { + deletionPolicy = metav1.DeletePropagationOrphan + } + + // Retrieve all works that derive from the same placement policy. + // + // Note that this list comes from the cache; if the cache has become stale, and there exists a work object + // that is owned by the placement policy but not yet reflected in the cache, it is guaranteed that the work applier + // has never processed it before; as a result, the work object should have no finalizer and needs no cleanup. + ownerNS := primaryWork.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + ownerName := primaryWork.GetLabels()[placementv1alpha1.WorkOwnedByPlacementBindingLabelKey] + if ownerName == "" { + return nil, errors.NewUnexpectedError(nil, "no owner placement binding found for the primary work object", "observedOwnerNS", ownerNS) + } + ownedByFieldVal := fmt.Sprintf(fieldindexers.WorkOwnedByPlacementBindingCustomFieldValFmt, ownerNS, ownerName) + workList := &placementv1alpha1.WorkList{} + if err := r.hubClient.List(ctx, workList, + client.InNamespace(primaryWork.Namespace), + client.MatchingFields{fieldindexers.WorkOwnedByPlacementBindingCustomFieldName: ownedByFieldVal}, + ); err != nil { + return nil, errors.NewAPIServerError(err, "failed to list the work objects owned by the placement policy", true, + "ownerPlacementPolicy", ownedByFieldVal) + } + works := workList.Items + + // All the manifests across the linked work objects are owned by the appliedWork object of the primary work. + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{Name: primaryWork.Name}, + } + err := r.spokeClient.Get(ctx, client.ObjectKey{Name: primaryWork.Name}, appliedWork) + switch { + case apierrors.IsNotFound(err): + // The primary appliedWork object is absent; no manifest cleanup is needed. + klog.V(2).InfoS("The primary appliedWork object is not found; no manifest cleanup is needed", "appliedWork", klog.KObj(appliedWork)) + case err != nil: + // An unexpected error has occurred. + return nil, errors.NewAPIServerError(err, "failed to retrieve the appliedWork object", true, "appliedWork", klog.KObj(appliedWork)) + case appliedWork.DeletionTimestamp.IsZero(): + // The appliedWork object exists and has not been marked for deletion yet; delete it. + if err := r.spokeClient.Delete(ctx, appliedWork, &client.DeleteOptions{PropagationPolicy: &deletionPolicy}); err != nil && !apierrors.IsNotFound(err) { + // An unexpected error has occurred. + return nil, errors.NewAPIServerError(err, "failed to delete the appliedWork object", false, + "appliedWork", klog.KObj(appliedWork), "propagationPolicy", deletionPolicy) + } + klog.V(2).InfoS("The primary appliedWork object has been marked for deletion; wait for the deletion to complete", + "appliedWork", klog.KObj(appliedWork), "propagationPolicy", deletionPolicy) + return ptr.To(r.cleanupRequeueRateLimiter.When(primaryWorkNamespacedName)), nil + default: + // The appliedWork object has been marked for deletion; wait for the deletion to complete. + if time.Since(appliedWork.DeletionTimestamp.Time) < r.cleanupWaitTime { + // Keep waiting. + return ptr.To(r.cleanupRequeueRateLimiter.When(primaryWorkNamespacedName)), nil + } + + if _, found := appliedWork.Annotations[appliedWorkForcedDeletedAnnotationKey]; !found { + klog.V(2).InfoS("The primary appliedWork object has been stuck in the pending deletion state; force-remove KubeFleet owner reference from applied manifests as a last resort", + "appliedWork", klog.KObj(appliedWork)) + // The appliedWork object has been stuck in the pending deletion state for longer than the cleanup wait time; + // manually update owner references from all the applied manifests so that they no longer block the deletion + // of the appliedWork object. + if err := r.manuallyUpdateOwnerReferencesOnManifests(ctx, appliedWork, works); err != nil { + return nil, errors.Wraps(err, "failed to manually update owner references from manifests to unblock cleanup") + } + + // Add the force-deleted annotation to the appliedWork object. + appliedWorkPatch := client.MergeFrom(appliedWork.DeepCopy()) + annotations := appliedWork.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string) + } + annotations[appliedWorkForcedDeletedAnnotationKey] = "true" + appliedWork.SetAnnotations(annotations) + if err := r.spokeClient.Patch(ctx, appliedWork, appliedWorkPatch); err != nil && !apierrors.IsNotFound(err) { + return nil, errors.NewAPIServerError(err, "failed to add the force-deleted annotation to the appliedWork object", false, + "appliedWork", klog.KObj(appliedWork)) + } + } + + if time.Since(appliedWork.DeletionTimestamp.Time) < 2*r.cleanupWaitTime { + // Keep waiting. + return ptr.To(r.cleanupRequeueRateLimiter.When(primaryWorkNamespacedName)), nil + } + + // If the appliedWork object is still stuck in the pending deletion state after us manually removing the + // owner references from all the applied manifests, the work applier will NOT keep + // waiting for the appliedWork object to disappear. Users should keep an eye on the GC progress (the + // API server/GC controller might not be fast enough) themselves, and manual intervention might be needed + // if the situation persists. + klog.Warning("The appliedWork object has been stuck in the pending deletion state for too long, and KubeFleet will proceed with the cleanup without further waiting; manual intervention might be needed.") + } + + // Remove the cleanup finalizer from non-primary work objects first. + for idx := range works { + work := &works[idx] + if work.Name == primaryWork.Name || !controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + continue + } + + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{Name: work.Name}, + } + // No need to specify the propagation policy; these appliedWork objects do not own any child resources. + if err := r.spokeClient.Delete(ctx, appliedWork); err != nil && !apierrors.IsNotFound(err) { + return nil, errors.NewAPIServerError(err, "failed to delete the appliedWork object", false, + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + } + klog.V(2).InfoS("Cleaned up the appliedWork object", "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + + controllerutil.RemoveFinalizer(work, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, work); err != nil { + return nil, errors.NewAPIServerError(err, "failed to remove the cleanup finalizer from the work", false, + "work", klog.KObj(work)) + } + klog.V(2).InfoS("Removed the cleanup finalizer from the work", "work", klog.KObj(work)) + } + + // Set the requeue rate limiters to forget about the work. + r.cleanupRequeueRateLimiter.Forget(primaryWorkNamespacedName) + r.periodicRequeueRateLimiter.Forget(primaryWork) + + // Remove the cleanup finalizer from the primary work object. + if controllerutil.ContainsFinalizer(primaryWork, workApplierCleanupFinalizer) { + controllerutil.RemoveFinalizer(primaryWork, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, primaryWork); err != nil { + return nil, errors.NewAPIServerError(err, "failed to remove the cleanup finalizer from the primary work", false, + "work", klog.KObj(primaryWork)) + } + klog.V(2).InfoS("Removed the cleanup finalizer from the primary work", "work", klog.KObj(primaryWork)) + } + + return nil, nil +} + +// shouldCleanUpManifest checks if the object derived from a manifest might have been applied to the member cluster, +// based on the manifest's last recorded status; if so, the object should be cleaned up once the manifest is no +// longer present. +// +// The check is permissive by design: any manifest that can be identified and has been processed (i.e., it has an +// Applied condition) is considered, regardless of its apply result. This is safe as the cleanup process always +// verifies object ownership before taking any action. +func shouldCleanUpManifest(manifestStatus *placementv1alpha1.PerManifestStatus) bool { + // Skip manifests that cannot be identified in the member cluster; attempting to clean them up would always fail. + id := &manifestStatus.Identifier + if id.APIVersion == "" || id.Kind == "" || id.Resource == "" || id.Name == "" { + return false + } + + appliedCond := meta.FindStatusCondition(manifestStatus.Conditions, placementv1alpha1.WorkCondTypeApplied) + // There are three possibilities: + // + // a) the manifest has no applied condition; it is guaranteed that the object has never been applied (recall + // the controller's write-ahead process). + // b) the manifest has a True applied condition; it might have been applied, cleanup is needed. + // c) the manifest has a False applied condition; there are certain cases where cleanup might not be necessary, + // however, as there might be reconciliation attempts that haven't been properly tracked (e.g., failed status + // updates), here the work applier will always err on the side of cleaning up the manifest. + // + // Note that additional checks will be performed before the cleanup runs. + return appliedCond != nil +} + +func (r *Reconciler) manuallyUpdateOwnerReferencesOnManifests(ctx context.Context, + appliedWork *placementv1alpha1.AppliedWork, + works []placementv1alpha1.Work) error { + appliedWorkOwnerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "AppliedWork", + Name: appliedWork.Name, + UID: appliedWork.UID, + } + + // Loop through all work objects, identify all manifests that might have been applied + // in the status, and add them as owner reference removal candidates. + manifestsToRemoveOwnerRef := []placementv1alpha1.ManifestIdentifier{} + for workIdx := range works { + work := &works[workIdx] + for manifestIdx := range work.Status.Manifests { + manifestStatus := &work.Status.Manifests[manifestIdx] + if !shouldCleanUpManifest(manifestStatus) { + continue + } + + manifestsToRemoveOwnerRef = append(manifestsToRemoveOwnerRef, manifestStatus.Identifier) + } + } + + // Update the owner references of all the identified manifests so that they no longer block appliedWork object's + // deletion. + errs := make([]error, len(manifestsToRemoveOwnerRef)) + removeOwnerRefFromManifest := func(piece int) { + identifier := manifestsToRemoveOwnerRef[piece] + gvr := schema.GroupVersionResource{ + Group: identifier.APIGroup, + Version: identifier.APIVersion, + Resource: identifier.Resource, + } + manifestRef := klog.KRef(identifier.Namespace, identifier.Name) + manifestObj, err := r.spokeDynamicClient.Resource(gvr).Namespace(identifier.Namespace).Get(ctx, identifier.Name, metav1.GetOptions{}) + if err != nil { + if apierrors.IsNotFound(err) { + return + } + errs[piece] = errors.NewAPIServerError(err, "failed to retrieve a manifest while unblocking appliedWork deletion", false, + "gvr", gvr, "manifestObj", manifestRef, "appliedWork", klog.KObj(appliedWork)) + return + } + + ownerRefs := manifestObj.GetOwnerReferences() + if len(ownerRefs) == 0 { + // The applied manifest is no longer owned by KubeFleet; no further action is needed. + return + } + updated := false + for ownerIdx := range ownerRefs { + if ownerreferences.AreEqual(&ownerRefs[ownerIdx], appliedWorkOwnerRef) && ptr.Deref(ownerRefs[ownerIdx].BlockOwnerDeletion, false) { + // Set BlockOwnerDeletion to false to allow the appliedWork object to be deleted without being blocked + // by this manifest. + ownerRefs[ownerIdx].BlockOwnerDeletion = ptr.To(false) + updated = true + } + } + if !updated { + return + } + + manifestObj.SetOwnerReferences(ownerRefs) + if _, err := r.spokeDynamicClient.Resource(gvr).Namespace(identifier.Namespace).Update(ctx, manifestObj, metav1.UpdateOptions{}); err != nil && !apierrors.IsNotFound(err) { + errs[piece] = errors.NewAPIServerError(err, "failed to unblock a manifest owner reference", false, + "gvr", gvr, "manifestObj", manifestRef, "appliedWork", klog.KObj(appliedWork)) + } + } + r.parallelizer.ParallelizeUntil(ctx, len(manifestsToRemoveOwnerRef), removeOwnerRefFromManifest, "manuallyRemoveOwnerReferencesFromManifests") + + if aggregatedErrs := utilerrors.NewAggregate(errs); aggregatedErrs != nil { + return errors.Wraps(nil, "failed to update owner references on manifests", + "appliedWork", klog.KObj(appliedWork), "errs", aggregatedErrs) + } + return nil +} + +func (r *Reconciler) cleanupLeftOverWorks(ctx context.Context, + leftOverWorks []*placementv1alpha1.Work, seenManifIDs sets.Set[string], appliedWorkOwnerRef *metav1.OwnerReference) error { + for idx := range leftOverWorks { + leftOverWork := leftOverWorks[idx] + if !controllerutil.ContainsFinalizer(leftOverWork, workApplierCleanupFinalizer) { + // The left-over work does not have the cleanup finalizer; no cleanup is needed. + continue + } + + // Identify manifests that are no longer seen and need to be removed. + manifestsToDelete := make([]placementv1alpha1.ManifestIdentifier, 0) + for manifestIdx := range leftOverWork.Status.Manifests { + manifestStatus := &leftOverWork.Status.Manifests[manifestIdx] + if seenManifIDs.Has(formatManifestIdentifierStr(&manifestStatus.Identifier)) { + // There exists a corner case where a manifest, though present on a left-over work, is still present + // on one of the linked work objects; in this case, such manifests would not be removed. + continue + } + + // Check if the manifest might have been applied before; if so, attempt to remove it. + if shouldCleanUpManifest(manifestStatus) { + manifestsToDelete = append(manifestsToDelete, manifestStatus.Identifier) + } + } + + errs := make([]error, len(manifestsToDelete)) + removeManifest := func(piece int) { + manifest := manifestsToDelete[piece] + if err := r.removeOneLeftOverManifest(ctx, manifest, appliedWorkOwnerRef); err != nil { + errs[piece] = errors.Wraps(err, "failed to remove a left-over manifest", "manifestId", manifest, "work", klog.KObj(leftOverWork)) + } + } + r.parallelizer.ParallelizeUntil(ctx, len(manifestsToDelete), removeManifest, "cleanupLeftOverWorks") + aggregatedErrs := utilerrors.NewAggregate(errs) + if aggregatedErrs != nil { + return errors.Wraps(nil, "failed to remove manifests on a left-over work", "work", klog.KObj(leftOverWork), "errs", aggregatedErrs) + } + + // Clean up the appliedWork object associated with the left-over work. + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{Name: leftOverWork.Name}, + } + if err := r.spokeClient.Delete(ctx, appliedWork); err != nil && !apierrors.IsNotFound(err) { + return errors.NewAPIServerError(err, "failed to delete the appliedWork object associated with the left-over work", false, + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(leftOverWork)) + } + klog.V(2).InfoS("Cleaned up the appliedWork object associated with the left-over work", + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(leftOverWork)) + + controllerutil.RemoveFinalizer(leftOverWork, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, leftOverWork); err != nil { + return errors.NewAPIServerError(err, "failed to remove the cleanup finalizer from the left-over work", false, + "work", klog.KObj(leftOverWork)) + } + klog.V(2).InfoS("Cleaned up a left-over work", "work", klog.KObj(leftOverWork)) + } + return nil +} diff --git a/pkg/v1/controllers/workapplier/cleanup_integration_test.go b/pkg/v1/controllers/workapplier/cleanup_integration_test.go new file mode 100644 index 000000000..5dcaea7de --- /dev/null +++ b/pkg/v1/controllers/workapplier/cleanup_integration_test.go @@ -0,0 +1,781 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("cleanup", func() { + Context("orphan resources upon placement deletion (linked work objects)", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and the + // same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + primaryConfigMapName := "configmap-primary" + secondaryConfigMapName := "configmap-secondary" + + var appliedWorkOwnerRef *metav1.OwnerReference + // The config maps as applied in the member cluster. + var appliedPrimaryConfigMap *corev1.ConfigMap + var appliedSecondaryConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work objects do not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + primaryConfigMap := configMap.DeepCopy() + primaryConfigMap.Namespace = nsName + primaryConfigMap.Name = primaryConfigMapName + primaryConfigMapJSON := marshalK8sObjJSON(primaryConfigMap) + + secondaryConfigMap := configMap.DeepCopy() + secondaryConfigMap.Namespace = nsName + secondaryConfigMap.Name = secondaryConfigMapName + secondaryConfigMapJSON := marshalK8sObjJSON(secondaryConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. The secondary work object features + // no sync strategy, as the work applier always uses the one on the primary work object. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + secondaryConfigMapJSON, + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object, which orphans the applied resources upon placement deletion. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenPlacementDeleted: placementv1alpha1.WhenPlacementDeletedOptionOrphanResources, + } + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + syncStrategy, + primaryConfigMapJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + It("should apply the manifests", func() { + // All the manifests across the linked work objects are owned by the appliedWork object of the primary + // work object. + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + + secondaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, secondaryWorkName) + Eventually(secondaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the secondary work object") + + secondaryFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the secondary work object") + + primaryConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, primaryConfigMapName, appliedWorkOwnerRef) + Eventually(primaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map from the primary work object") + + secondaryConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, secondaryConfigMapName, appliedWorkOwnerRef) + Eventually(secondaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map from the secondary work object") + + appliedPrimaryConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: primaryConfigMapName}, appliedPrimaryConfigMap)).To(Succeed(), "Failed to retrieve the config map from the primary work object") + appliedSecondaryConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: secondaryConfigMapName}, appliedSecondaryConfigMap)).To(Succeed(), "Failed to retrieve the config map from the secondary work object") + }) + + It("can delete the primary work object", func() { + deleteWorkObject(primaryWorkName, memberReservedNSName1) + }) + + It("should start deleting the primary appliedWork object with the orphan propagation policy", func() { + appliedWorkBeingDeletedWithOrphanActual := appliedWorkBeingDeletedWithOrphanActual(memberClient1, primaryWorkName) + Eventually(appliedWorkBeingDeletedWithOrphanActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start deleting the primary appliedWork object with the orphan propagation policy") + }) + + It("should unblock the appliedWork deletion on all applied objects", func() { + // As there is no GC controller in the test environment, the appliedWork object stays in the pending + // deletion state; the work applier unblocks the deletion after cleanupWaitTime, and marks the + // appliedWork object as forcibly deleted. + for _, configMapName := range []string{primaryConfigMapName, secondaryConfigMapName} { + configMapOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: configMapName}}, appliedWorkOwnerRef) + Eventually(configMapOwnerRefUnblockedActual, cleanupWaitTime+eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the config map %s", configMapName) + } + + appliedWorkForcedDeletedActual := objectAnnotationActual(memberClient1, &placementv1alpha1.AppliedWork{ObjectMeta: metav1.ObjectMeta{Name: primaryWorkName}}, appliedWorkForcedDeletedAnnotationKey, "true") + Eventually(appliedWorkForcedDeletedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the primary appliedWork object as forcibly deleted") + }) + + It("should remove the cleanup finalizer from the secondary work object", func() { + // The work applier proceeds with the cleanup after 2 * cleanupWaitTime. + secondaryFinalizerRemovedActual := workFinalizerRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryFinalizerRemovedActual, 2*cleanupWaitTime+eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer from the secondary work object") + }) + + It("should remove the secondary appliedWork object", func() { + secondaryAppliedWorkAbsentActual := appliedWorkAbsentActual(memberClient1, secondaryWorkName) + Eventually(secondaryAppliedWorkAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + }) + + It("should remove the primary work object", func() { + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + }) + + It("should leave the primary appliedWork object to the GC controller", func() { + // The work applier stops waiting for the appliedWork object after 2 * cleanupWaitTime. + appliedWorkBeingDeletedWithOrphanActual := appliedWorkBeingDeletedWithOrphanActual(memberClient1, primaryWorkName) + Expect(appliedWorkBeingDeletedWithOrphanActual()).To(Succeed(), "The primary appliedWork object is no longer pending deletion with the orphan propagation policy") + }) + + It("should leave the applied objects in place", func() { + Expect(objectNotRecreatedActual(memberClient1, appliedPrimaryConfigMap)()).To(Succeed(), "The config map from the primary work object has been deleted") + Expect(objectNotRecreatedActual(memberClient1, appliedSecondaryConfigMap)()).To(Succeed(), "The config map from the secondary work object has been deleted") + }) + + AfterAll(func() { + // There is no GC controller in the hub cluster test environment to delete the secondary work object along + // with the primary one; delete both of them (the primary work object might still exist if the test case + // fails before it is deleted). + for _, workName := range []string{primaryWorkName, secondaryWorkName} { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete work object %s", workName) + } + + // Remove the config maps manually, as there is no GC controller in the test environment. + for _, configMapName := range []string{primaryConfigMapName, secondaryConfigMapName} { + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map %s", configMapName) + } + + // Simulate the GC controller by removing the orphan/foreground deletion finalizers from the appliedWork + // objects. + for _, workName := range []string{primaryWorkName, secondaryWorkName} { + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object %s", workName) + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object %s", workName) + } + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("leftover objects no longer owned by KubeFleet", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // The config map that stays in the work object throughout the test case. + keptConfigMapName := "configmap-kept" + + var appliedWorkOwnerRef *metav1.OwnerReference + var keptConfigMapJSON []byte + var appliedKeptConfigMap *corev1.ConfigMap + // The config map as it is after the KubeFleet owner reference has been removed. + var unownedConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + keptConfigMap := configMap.DeepCopy() + keptConfigMap.Namespace = nsName + keptConfigMap.Name = keptConfigMapName + keptConfigMapJSON = marshalK8sObjJSON(keptConfigMap) + + // Create a new work object with no sync strategy; by default, the work applier does not take over + // objects that it does not own. + createWorkObject(workName, memberReservedNSName1, nil, regularConfigMapJSON, keptConfigMapJSON) + }) + + It("should apply the manifests", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + configMapAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(configMapAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map") + + keptConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, keptConfigMapName, appliedWorkOwnerRef) + Eventually(keptConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the kept config map") + + appliedKeptConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: keptConfigMapName}, appliedKeptConfigMap)).To(Succeed(), "Failed to retrieve the kept config map") + }) + + It("can remove the KubeFleet owner reference from the config map", func() { + // This mimics a user taking the object away from KubeFleet; deleting and re-creating the object (without + // the owner reference) leads to the same result. + unownedConfigMap = updateConfigMapInMemberCluster(memberClient1, nsName, configMapName, func(configMap *corev1.ConfigMap) { + configMap.OwnerReferences = nil + }) + }) + + It("should report that the config map is not taken over", func() { + // The change in the member cluster triggers no event for the work object; it is picked up at the next + // requeue of the work object. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeNotTakenOver), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: keptConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report that the config map is not taken over") + }) + + It("can remove the manifest from the work object", func() { + updateWorkObject(workName, memberReservedNSName1, nil, keptConfigMapJSON) + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: keptConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should leave the config map as it is", func() { + // The work applier no longer owns the config map; the leftover cleanup must not delete it. + configMapUntouchedActual := objectUntouchedActual(memberClient1, unownedConfigMap) + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The config map has been changed or deleted") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: keptConfigMapName, + Namespace: nsName, + }, + UID: appliedKeptConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config maps manually, as there is no GC controller in the test environment. + for _, name := range []string{configMapName, keptConfigMapName} { + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, name) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map %s", name) + } + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("leftover objects never taken over", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // The config map that stays in the work object throughout the test case. + keptConfigMapName := "configmap-kept" + + var keptConfigMapJSON []byte + // The object created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + keptConfigMap := configMap.DeepCopy() + keptConfigMap.Namespace = nsName + keptConfigMap.Name = keptConfigMapName + keptConfigMapJSON = marshalK8sObjJSON(keptConfigMap) + + // Create the config map in the member cluster before the work object, as if a user or another + // controller has created it. + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + // Create a new work object with no sync strategy; by default, the work applier does not take over + // pre-existing objects. + createWorkObject(workName, memberReservedNSName1, nil, regularConfigMapJSON, keptConfigMapJSON) + }) + + It("should report that the pre-existing config map is not taken over", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeNotTakenOver), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: keptConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report that the pre-existing config map is not taken over") + }) + + It("can remove the manifest from the work object", func() { + updateWorkObject(workName, memberReservedNSName1, nil, keptConfigMapJSON) + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: keptConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should leave the pre-existing config map as it is", func() { + // The work applier has never owned the config map; the leftover cleanup must not delete it. + configMapUntouchedActual := objectUntouchedActual(memberClient1, preExistingConfigMap) + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The pre-existing config map has been changed or deleted") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config maps manually, as there is no GC controller in the test environment. + for _, name := range []string{configMapName, keptConfigMapName} { + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, name) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map %s", name) + } + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("leftover objects co-owned with other sources", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // The config map that stays in the work object throughout the test case. + keptConfigMapName := "configmap-kept" + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + var keptConfigMapJSON []byte + var appliedKeptConfigMap *corev1.ConfigMap + var syncStrategy *placementv1alpha1.SyncStrategy + // The object created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + keptConfigMap := configMap.DeepCopy() + keptConfigMap.Namespace = nsName + keptConfigMap.Name = keptConfigMapName + keptConfigMapJSON = marshalK8sObjJSON(keptConfigMap) + + // Create the config map in the member cluster before the work object, as if another controller has + // created (and owns) it. + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.OwnerReferences = []metav1.OwnerReference{externalOwnerRef} + preExistingConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + // Create a new work object that takes over pre-existing objects as a co-owner. + syncStrategy = &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: placementv1alpha1.WhenOwnedByOthersOptionShareOwnership, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON, keptConfigMapJSON) + }) + + It("should apply the manifests", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + // The pre-existing config map is co-owned by KubeFleet and the other source. + wantConfigMap := regularConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{externalOwnerRef, *appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the config map as a co-owner") + + keptConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, keptConfigMapName, appliedWorkOwnerRef) + Eventually(keptConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the kept config map") + + appliedKeptConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: keptConfigMapName}, appliedKeptConfigMap)).To(Succeed(), "Failed to retrieve the kept config map") + }) + + It("can remove the manifest of the co-owned config map from the work object", func() { + updateWorkObject(workName, memberReservedNSName1, syncStrategy, keptConfigMapJSON) + }) + + It("should drop the KubeFleet ownership of the config map, but not delete it", func() { + // Only the owner reference to the appliedWork object is removed; the other owner, and the data applied + // while KubeFleet co-owned the object, are left alone. + wantConfigMap := regularConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{externalOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to drop the KubeFleet ownership of the config map") + Consistently(configMapObjectActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The config map has been changed or deleted after KubeFleet dropped its ownership") + + Expect(objectNotRecreatedActual(memberClient1, preExistingConfigMap)()).To(Succeed(), "The config map has been re-created") + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: keptConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: keptConfigMapName, + Namespace: nsName, + }, + UID: appliedKeptConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config maps manually, as there is no GC controller in the test environment. + for _, name := range []string{configMapName, keptConfigMapName} { + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, name) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map %s", name) + } + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/cleanup_test.go b/pkg/v1/controllers/workapplier/cleanup_test.go new file mode 100644 index 000000000..2dda1c76f --- /dev/null +++ b/pkg/v1/controllers/workapplier/cleanup_test.go @@ -0,0 +1,128 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "testing" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// TestShouldCleanUpManifest tests the shouldCleanUpManifest function. +func TestShouldCleanUpManifest(t *testing.T) { + identifier := placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: "configmap-1", + Namespace: "ns-1", + } + appliedCond := func(status metav1.ConditionStatus, reason string) []metav1.Condition { + return []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: status, + Reason: reason, + }, + } + } + + // Any manifest that can be identified and has been processed should be considered for cleanup, regardless of + // its apply result. + for _, applyResType := range allApplyResTypes.UnsortedList() { + t.Run(string(applyResType), func(t *testing.T) { + // The Applied condition is set to True only for successful apply ops. + status := metav1.ConditionFalse + if applyResType == ApplyResTypeApplied || applyResType == ApplyResTypeAppliedWithFailedDriftDetection { + status = metav1.ConditionTrue + } + manifestStatus := &placementv1alpha1.PerManifestStatus{ + Identifier: identifier, + Conditions: appliedCond(status, string(applyResType)), + } + want := true + if got := shouldCleanUpManifest(manifestStatus); got != want { + t.Errorf("shouldCleanUpManifest(%v) = %v, want %v", manifestStatus, got, want) + } + }) + } + + testCases := []struct { + name string + manifestStatus *placementv1alpha1.PerManifestStatus + want bool + }{ + { + name: "no applied condition", + manifestStatus: &placementv1alpha1.PerManifestStatus{ + Identifier: identifier, + Conditions: []metav1.Condition{}, + }, + want: false, + }, + { + name: "unknown apply result type", + manifestStatus: &placementv1alpha1.PerManifestStatus{ + Identifier: identifier, + Conditions: appliedCond(metav1.ConditionFalse, "UnknownApplyResultType"), + }, + want: true, + }, + { + name: "identifier without resource", + manifestStatus: &placementv1alpha1.PerManifestStatus{ + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "malformed", + APIVersion: "v10", + Kind: "Unknown", + Name: "configmap-1", + Namespace: "ns-1", + }, + Conditions: appliedCond(metav1.ConditionFalse, string(ApplyResTypeDecodingErred)), + }, + want: false, + }, + { + name: "identifier without name", + manifestStatus: &placementv1alpha1.PerManifestStatus{ + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Namespace: "ns-1", + }, + Conditions: appliedCond(metav1.ConditionTrue, string(ApplyResTypeApplied)), + }, + want: false, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + if got := shouldCleanUpManifest(tc.manifestStatus); got != tc.want { + t.Errorf("shouldCleanUpManifest(%v) = %v, want %v", tc.manifestStatus, got, tc.want) + } + }) + } +} diff --git a/pkg/v1/controllers/workapplier/controller.go b/pkg/v1/controllers/workapplier/controller.go new file mode 100644 index 000000000..ab4b8f440 --- /dev/null +++ b/pkg/v1/controllers/workapplier/controller.go @@ -0,0 +1,395 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "sync/atomic" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/builder" + "sigs.k8s.io/controller-runtime/pkg/client" + ctrloption "sigs.k8s.io/controller-runtime/pkg/controller" + "sigs.k8s.io/controller-runtime/pkg/predicate" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + parallelizerutil "go.goms.io/fleet/pkg/utils/parallelizer" +) + +const ( + controllerName = "work-applier-v1" + fieldManagerName = "kubefleet-member-agent" + + workApplierCleanupFinalizer = "placement.kubefleet.dev/work-cleanup" + + appliedWorkForcedDeletedAnnotationKey = "placement.kubefleet.dev/applied-work-forced-deleted" +) + +type ApplyResultType string + +const ( + // The result type for manifests that are to be processed. + ApplyResTypePreparingToProcess ApplyResultType = "PreparingToProcess" + + // The result types for apply op failures. + ApplyResTypeDecodingErred ApplyResultType = "DecodingErred" + ApplyResTypeFoundGenerateName ApplyResultType = "FoundGenerateName" + ApplyResTypeNoName ApplyResultType = "NoNameSpecified" + ApplyResTypeDuplicated ApplyResultType = "Duplicated" + ApplyResTypeFailedToFindObjInMemberCluster ApplyResultType = "FailedToFindObjInMemberCluster" + ApplyResTypeOwnedByOtherKubeFleetAPIObjects ApplyResultType = "OwnedByOtherKubeFleetAPIObjects" + ApplyResTypeFailedToTakeOver ApplyResultType = "FailedToTakeOver" + ApplyResTypeNotTakenOver ApplyResultType = "NotTakenOver" + ApplyResTypeFailedToRunDriftDetection ApplyResultType = "FailedToRunDriftDetection" + ApplyResTypeFoundDrifts ApplyResultType = "FoundDrifts" + ApplyResTypeFoundDriftsInDegradedMode ApplyResultType = "FoundDriftsInDegradedMode" + ApplyResTypeFailedToApply ApplyResultType = "FailedToApply" + + // The result type and description for successful apply ops. + ApplyResTypeApplied ApplyResultType = "Applied" + // The apply op has succeeded, but KubeFleet cannot tell whether the object has drifted. + ApplyResTypeAppliedWithFailedDriftDetection ApplyResultType = "AppliedWithFailedDriftDetection" +) + +var allApplyResTypes = sets.New( + ApplyResTypePreparingToProcess, + ApplyResTypeDecodingErred, + ApplyResTypeFoundGenerateName, + ApplyResTypeNoName, + ApplyResTypeDuplicated, + ApplyResTypeFailedToFindObjInMemberCluster, + ApplyResTypeOwnedByOtherKubeFleetAPIObjects, + ApplyResTypeFailedToTakeOver, + ApplyResTypeNotTakenOver, + ApplyResTypeFailedToRunDriftDetection, + ApplyResTypeFoundDrifts, + ApplyResTypeFoundDriftsInDegradedMode, + ApplyResTypeFailedToApply, + ApplyResTypeApplied, + ApplyResTypeAppliedWithFailedDriftDetection, +) + +// The messages reported on the Applied condition of a manifest. +const ( + ApplyResTypeAppliedDescription = "The manifest has been applied successfully" + ApplyResTypeAppliedWithFailedDriftDetectionDescription = "The manifest has been applied successfully, but KubeFleet cannot determine whether the object has drifted" + ApplyResTypeFailedToApplyDescription = "Failed to apply the manifest (error: %s)" +) + +type AvailabilityCheckResultType string + +const ( + // The result type for availability check being skipped. + AvailabilityResultTypeSkipped AvailabilityCheckResultType = "Skipped" + + // The result type for availability check failures. + AvailabilityResultTypeFailed AvailabilityCheckResultType = "FailedToCheck" + + // The result types for completed availability checks. + AvailabilityResultTypeAvailable AvailabilityCheckResultType = "Available" + AvailabilityResultTypeNotYetAvailable AvailabilityCheckResultType = "Unavailable" + AvailabilityResultTypeNotTrackable AvailabilityCheckResultType = "NotTrackable" +) + +// The messages reported on the Available condition of a manifest. +const ( + AvailabilityResultTypeAvailableDescription = "The manifest is available" + AvailabilityResultTypeNotYetAvailableDescription = "The manifest is not yet available; KubeFleet will check again later" + AvailabilityResultTypeNotTrackableDescription = "The manifest's availability is not trackable; KubeFleet assumes that the applied manifest is available" + AvailabilityResultTypeFailedDescription = "Failed to track the availability of the applied manifest (error: %s)" +) + +type manifestProcessingState struct { + // The manifest data in its raw form (not yet decoded). + manifest *placementv1alpha1.Manifest + + // The manifest identifier. + // + // If the manifest data cannot be decoded as a Kubernetes API object at all, the identifier + // will feature only the ordinal of the manifest data (its rank in the list of the resources). + // If the manifest data can be decoded as a Kubernetes API object, but the API is not available + // on the member cluster, the resource field of the identifier will be empty. + id *placementv1alpha1.ManifestIdentifier + // A string representation of the resource identifier (sans the resources field). + // This is only populated if the manifest data can be successfully decoded. + // + // It is of the format `GV=[API_GROUP]/[VERSION], Kind=[KIND], Namespace=[NAMESPACE], Name=[NAME]`, + // where [API_GROUP], [VERSION], [KIND], [NAMESPACE], and [NAME] are the API group/version/kind of the + // manifest object, and its owner namespace (if applicable)/name, respectively. + idStr string + + // The manifest data, decoded as a Kubernetes API object. + manifestObj *unstructured.Unstructured + + // The object in the member cluster that corresponds to the manifest object. + inMemberClusterObj *unstructured.Unstructured + + // The GVR of the manifest object. + gvr *schema.GroupVersionResource + + // The result of the apply operation. + applyRes ApplyResultType + // The result of the availability check operation. + availabilityCheckRes AvailabilityCheckResultType + // The error that occurred during the apply operation, if any. + applyErr error + // The error that occurred during the availability check operation, if any. + availabilityCheckErr error + // The diffs detected in the apply operation. + diffs []placementv1alpha1.PatchDetail + // A link back to the work object that includes the manifest. + fromWorkObj *placementv1alpha1.Work + // A link back to the primary work object. + fromPrimaryWorkObject *placementv1alpha1.Work + // The expected owner reference for the manifest object. + ownedBy *metav1.OwnerReference +} + +type workObjectProcessingState struct { + // The work object being processed. + work *placementv1alpha1.Work + + // The corresponding appliedWork object for the work object. + appliedWork *placementv1alpha1.AppliedWork + // The owner reference to use for all the manifests within the work object when they are applied. + appliedWorkOwnerRef *metav1.OwnerReference + + // The processing state of each manifest within the work object. + manifestProcessingStates []*manifestProcessingState +} + +type Reconciler struct { + hubClient client.Client + hubUncachedReader client.Reader + spokeClient client.Client + spokeDynamicClient dynamic.Interface + + workNSName string + + restMapper meta.RESTMapper + + parallelizer parallelizerutil.Parallelizer + + concurrentReconciles int + + cleanupRequeueRateLimiter workqueue.TypedRateLimiter[types.NamespacedName] + periodicRequeueRateLimiter *MultiStageRequeueWithExponentialBackoffRateLimiter + + cleanupWaitTime time.Duration + + ready atomic.Bool +} + +func New(workNSName string, + hubClient client.Client, hubUncachedReader client.Reader, + spokeClient client.Client, spokeDynamicClient dynamic.Interface, + restMapper meta.RESTMapper, + parallelizer parallelizerutil.Parallelizer, + concurrentReconciles int, + cleanupRequeueRateLimiter workqueue.TypedRateLimiter[types.NamespacedName], + periodicRequeueRateLimiter *MultiStageRequeueWithExponentialBackoffRateLimiter, + cleanupWaitTime time.Duration, +) *Reconciler { + if cleanupRequeueRateLimiter == nil { + cleanupRequeueRateLimiter = workqueue.NewTypedItemExponentialFailureRateLimiter[types.NamespacedName](DefaultCleanupBaseDelay, DefaultCleanupMaxDelay) + } + if periodicRequeueRateLimiter == nil { + periodicRequeueRateLimiter = defaultRequeueRateLimiter + } + + return &Reconciler{ + workNSName: workNSName, + hubClient: hubClient, + hubUncachedReader: hubUncachedReader, + spokeDynamicClient: spokeDynamicClient, + spokeClient: spokeClient, + restMapper: restMapper, + parallelizer: parallelizer, + concurrentReconciles: concurrentReconciles, + cleanupRequeueRateLimiter: cleanupRequeueRateLimiter, + periodicRequeueRateLimiter: periodicRequeueRateLimiter, + cleanupWaitTime: cleanupWaitTime, + } +} + +func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { + if !r.ready.Load() { + klog.V(2).InfoS("Work applier is not yet ready to start; the member agent might still be connecting to the hub cluster", + "work", req.NamespacedName) + return ctrl.Result{RequeueAfter: time.Second * 5}, nil + } + + startTime := time.Now() + klog.V(2).InfoS("Reconciliation starts", "work", req.NamespacedName, "controller", controllerName) + defer func() { + latency := time.Since(startTime).Milliseconds() + klog.V(2).InfoS("Reconciliation ends", "work", req.NamespacedName, "latency", latency, "controller", controllerName) + }() + + // Retrieve the work object. + work := &placementv1alpha1.Work{} + err := r.hubClient.Get(ctx, req.NamespacedName, work) + switch { + case apierrors.IsNotFound(err): + klog.V(2).InfoS("The work object cannot be found", "work", req.NamespacedName, "controller", controllerName) + return ctrl.Result{}, nil + case err != nil: + wrappedErr := errors.NewAPIServerError(err, "", true, "work", req.NamespacedName, "controller", controllerName) + klog.ErrorS(err, "Failed to retrieve the work", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Check if the work object is a primary one, i.e., it has the count annotation set. + _, found := work.GetAnnotations()[placementv1alpha1.LinkedWorkCountAnnotationKey] + if !found { + klog.V(2).InfoS("The work object is not a primary one; skipping reconciliation", "work", req.NamespacedName, "controller", controllerName) + return ctrl.Result{}, nil + } + + // Clean things up if the work object has been marked for deletion. + if !work.DeletionTimestamp.IsZero() { + requeueAfter, err := r.cleanupWhenBindingDeleted(ctx, work) + if err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to clean up linked work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + if requeueAfter != nil { + return ctrl.Result{RequeueAfter: *requeueAfter}, nil + } + return ctrl.Result{}, nil + } + + // Retrieve all linked work objects. + linkedWorks, leftOverWorks, err := r.retrieveLinkedAndLeftOverWorks(ctx, work) + if err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to retrieve linked work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Add cleanup finalizer to all linked work objects. + if err := r.addCleanupFinalizerTo(ctx, linkedWorks); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to add cleanup finalizer to linked work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Ensure an appliedWork object for each work object. + // + // The appliedWork object that corresponds to the primary work object will be set as the owner for all + // applied manifests across the linked work objects. + // + // Note that the appliedWork objects are returned in the same order as their corresponding work objects in the + // input. + appliedWorks, err := r.ensureAppliedWorks(ctx, linkedWorks) + if err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to ensure appliedWork objects for linked work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Prepare the processing states for the work objects and their manifests. + workObjProcessingStates, manifestProcessingStates := prepareWorkObjectAndManifestProcessingStates(linkedWorks, appliedWorks) + + // Pre-process the manifests to apply. + // + // In this step, the work applier will: + // a) decode the manifests; and + // b) write ahead the manifest processing attempts; and + // c) remove any applied manifests left over from previous runs. + seenManifestIDs, err := r.preProcessWorkObjects(ctx, workObjProcessingStates) + if err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to pre-process work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Before processing the manifests in the linked work objects, handle the left-over work objects: check if + // there is any applied manifest on them that needs to be cleaned up, and drop the cleanup finalizer + // from these work objects. + if err := r.cleanupLeftOverWorks(ctx, leftOverWorks, seenManifestIDs, workObjProcessingStates[0].appliedWorkOwnerRef); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to clean up left-over work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Set default values in the work object spec to avoid additional validation logic in the later steps. + setDefaultSyncStrategy(work) + + // Process the manifests. + // + // In this step, the work applier will: + // a) find if there has been a corresponding object in the member cluster for each manifest; + // b) take over the object if applicable; + // c) check for diffs/drifts if applicable; + // e) apply each manifest. + if err := r.processManifests(ctx, manifestProcessingStates); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to process manifests", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Track the availability information. + if err := r.trackInMemberClusterObjAvailability(ctx, manifestProcessingStates); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to check for object availability", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Refresh the status of the Work object. + if err := r.refreshWorkStatus(ctx, workObjProcessingStates); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to refresh work object status", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Refresh the status of the AppliedWork object. + if err := r.refreshAppliedWorkStatus(ctx, workObjProcessingStates); err != nil { + wrappedErr := errors.Wraps(err, "", "primaryWork", klog.KObj(work), "controller", controllerName) + klog.ErrorS(err, "Failed to refresh appliedWork object status", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + + // Requeue the work object with a delay based on the given multi-stage requeue with exponential backoff + // rate limiter. + requeueAfter := r.periodicRequeueRateLimiter.When(linkedWorks, manifestProcessingStates) + klog.V(2).InfoS("Requeue the work object for re-processing", "requeueAfter", requeueAfter, "work", klog.KObj(work), "controller", controllerName) + return ctrl.Result{RequeueAfter: requeueAfter}, nil +} + +func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + Named(controllerName). + WithOptions(ctrloption.Options{ + MaxConcurrentReconciles: r.concurrentReconciles, + }). + Watches(&placementv1alpha1.Work{}, &workEventHandler, builder.WithPredicates(predicate.GenerationChangedPredicate{})). + Complete(r) +} diff --git a/pkg/v1/controllers/workapplier/core_integration_test.go b/pkg/v1/controllers/workapplier/core_integration_test.go new file mode 100644 index 000000000..81550e780 --- /dev/null +++ b/pkg/v1/controllers/workapplier/core_integration_test.go @@ -0,0 +1,3056 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("single work object", func() { + Context("core ops (apply manifests)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularDeployJSON) + }) + + It("should add cleanup finalizer to the work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the work object") + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the manifests", func() { + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + // Prepare the status information. + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + UID: regularDeploy.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Ensure applied manifest has been removed. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("garbage collection (delete work object)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularDeployJSON) + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object status update, etc.) are + // omitted in this test node, as the behaviors have been verified elsewhere. + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object. + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can delete the work object", func() { + deleteWorkObject(workName, memberReservedNSName1) + }) + + It("should start deleting the appliedWork object in the foreground", func() { + appliedWorkBeingDeletedActual := appliedWorkBeingDeletedInForegroundActual(memberClient1, workName) + Eventually(appliedWorkBeingDeletedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start deleting the appliedWork object in the foreground") + }) + + It("should remove the work object", func() { + // As there is no GC controller in the test environment, the appliedWork object will be stuck in the + // pending deletion state; the work applier will unblock the deletion after cleanupWaitTime, and + // proceed to remove the cleanup finalizer after 2 * cleanupWaitTime. + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, 2*cleanupWaitTime+eventuallyDuration, eventuallyInterval*2).Should(Succeed(), "Failed to remove the work object") + }) + + It("should unblock the appliedWork deletion on all applied objects", func() { + // The owner references stay (they are removed by the GC controller in a real cluster), but they should + // no longer block the deletion of the appliedWork object. + nsOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}}, appliedWorkOwnerRef) + Eventually(nsOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the namespace object") + + deployOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: deployName}}, appliedWorkOwnerRef) + Eventually(deployOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the deployment object") + }) + + It("should delete the appliedWork object", func() { + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + }) + + AfterAll(func() { + // Remove the deployment object manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("leftover cleanup (remove manifests from work object)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularNSJSON []byte + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularDeployJSON) + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object. + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can update the work object to remove the deployment manifest", func() { + updateWorkObject(workName, memberReservedNSName1, nil, regularNSJSON) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should remove the deployment object", func() { + regularDeployAbsentActual := regularDeployAbsentActual(memberClient1, nsName, deployName) + Eventually(regularDeployAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) +}) + +var _ = Describe("multiple work objects", func() { + Context("core ops (apply manifests)", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and + // the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + // + // As KubeFleet does, the primary work object is set as the controller owner of all secondary work objects, + // so that events on secondary work objects trigger reconciliations of the primary work object. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularConfigMapJSON, + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, regularDeployJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + It("should add cleanup finalizer to all the work objects", func() { + primaryWorkFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the primary work object") + + secondaryWorkFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the secondary work object") + }) + + It("should prepare an appliedWork object for each work object", func() { + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + + secondaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, secondaryWorkName) + Eventually(secondaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the secondary work object") + + // All the manifests across the linked work objects are owned by the appliedWork object of the + // primary work object. + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + }) + + It("should apply the manifests", func() { + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + + // Ensure that the ConfigMap object has been applied as expected. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, regularConfigMap)).To(Succeed(), "Failed to retrieve the ConfigMap object") + }) + + It("should update the work object statuses (deployment not yet available)", func() { + // The primary work object is not yet available, as the deployment has not become available. + primaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeNotYetAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, primaryWorkConds, primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // The secondary work object is available. + // + // Note that the ordinals of the manifests are relative to their own work object. + secondaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, secondaryWorkConds, secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("should update the appliedWork object statuses", func() { + // Each appliedWork object reports the resources applied from its own work object. + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + UID: regularDeploy.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object statuses (all available)", func() { + // The primary work object becomes available. + primaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, primaryWorkConds, primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // The secondary work object stays available. + secondaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, secondaryWorkConds, secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + AfterAll(func() { + // Delete the work objects. + // + // As there is no GC controller in the test environment (and the secondary work object is not owned by + // the primary one in this test), both work objects are deleted explicitly. + deleteWorkObject(primaryWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Ensure that the appliedWork objects have been removed. + primaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(primaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + + secondaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, secondaryWorkName) + Eventually(secondaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + + // Ensure that the work objects have been removed. + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("garbage collection (delete primary work object)", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and + // the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + // + // As KubeFleet does, the primary work object is set as the controller owner of all secondary work objects, + // so that events on secondary work objects trigger reconciliations of the primary work object. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularConfigMapJSON, + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, regularDeployJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object of the primary work object; all the manifests + // across the linked work objects are owned by this appliedWork object. + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + // Ensure that the ConfigMap object has been applied as expected. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the two work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("can delete the primary work object", func() { + deleteWorkObject(primaryWorkName, memberReservedNSName1) + }) + + It("should start deleting the primary appliedWork object in the foreground", func() { + appliedWorkBeingDeletedActual := appliedWorkBeingDeletedInForegroundActual(memberClient1, primaryWorkName) + Eventually(appliedWorkBeingDeletedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start deleting the primary appliedWork object in the foreground") + }) + + It("should remove the primary work object", func() { + // As there is no GC controller in the test environment, the primary appliedWork object will be stuck in + // the pending deletion state; the work applier will unblock the deletion after cleanupWaitTime, and + // proceed to remove the cleanup finalizers after 2 * cleanupWaitTime. + workRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(workRemovedActual, 2*cleanupWaitTime+eventuallyDuration, eventuallyInterval*2).Should(Succeed(), "Failed to remove the primary work object") + }) + + It("should remove the cleanup finalizer from the secondary work object", func() { + // In a real cluster the secondary work object is owned by the primary one and will be garbage collected; + // here only the finalizer removal is verified. + workFinalizerRemovedActual := workFinalizerRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(workFinalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer from the secondary work object") + }) + + It("should remove the secondary appliedWork object", func() { + // The secondary appliedWork object owns no resources, and is deleted by the work applier directly. + appliedWorkAbsentActual := appliedWorkAbsentActual(memberClient1, secondaryWorkName) + Eventually(appliedWorkAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + }) + + It("should unblock the primary appliedWork deletion on all applied objects", func() { + // The owner references stay (they are removed by the GC controller in a real cluster), but they should + // no longer block the deletion of the primary appliedWork object. + nsOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}}, appliedWorkOwnerRef) + Eventually(nsOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the namespace object") + + deployOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: deployName}}, appliedWorkOwnerRef) + Eventually(deployOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the deployment object") + + configMapOwnerRefUnblockedActual := appliedWorkOwnerRefUnblockedActual(memberClient1, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: configMapName}}, appliedWorkOwnerRef) + Eventually(configMapOwnerRefUnblockedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unblock the appliedWork deletion on the config map object") + }) + + It("should delete the primary appliedWork object", func() { + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + }) + + AfterAll(func() { + // Delete the secondary work object manually, as there is no GC controller in the test environment. + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("leftover cleanup (remove and shuffle manifests from linked work objects)", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and + // the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The name of the new primary placement resource snapshot, as created by a rollout. + newPrimarySnapshotName := fmt.Sprintf("%s-new", primarySnapshotName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularConfigMap *corev1.ConfigMap + var regularNSJSON []byte + var regularDeployJSON []byte + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare a deployment object. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON = marshalK8sObjJSON(regularDeploy) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + // + // As KubeFleet does, the primary work object is set as the controller owner of all secondary work objects, + // so that events on secondary work objects trigger reconciliations of the primary work object. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularConfigMapJSON, + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, regularDeployJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object of the primary work object; all the manifests + // across the linked work objects are owned by this appliedWork object. + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the Deployment object has been applied as expected. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + + // Ensure that the ConfigMap object has been applied as expected. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the two work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("can update the primary work object to keep only the namespace manifest (rollout in progress)", func() { + // Mimic a rollout: the primary placement resource snapshot has been bumped, and the primary work object + // is refreshed first; the deployment manifest is to be moved to the secondary work object, which has not + // been refreshed yet. + updateLinkedWorkObject(primaryWorkName, memberReservedNSName1, newPrimarySnapshotName, nil, regularNSJSON) + }) + + It("should not remove the deployment object", func() { + // The linked work objects are now in an inconsistent state (the secondary work object is still linked + // to the old primary placement resource snapshot); the work applier should hold off any processing, + // including the removal of the deployment object. + Consistently(func() error { + gotDeploy := &appsv1.Deployment{} + if err := memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, gotDeploy); err != nil { + return fmt.Errorf("failed to retrieve the Deployment object: %w", err) + } + if !gotDeploy.DeletionTimestamp.IsZero() { + return fmt.Errorf("deployment object has been marked for deletion") + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The deployment object has been removed unexpectedly") + }) + + It("can update the secondary work object to add the deployment manifest and drop the config map manifest", func() { + updateLinkedWorkObject(secondaryWorkName, memberReservedNSName1, newPrimarySnapshotName, nil, regularDeployJSON) + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + // The update on the secondary work object triggers a reconciliation of the primary work object via the + // owner reference. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // Note that the ordinals of the manifests are relative to their own work object. + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("should update the appliedWork object statuses", func() { + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + // The deployment object is the same one as before (i.e., it has not been re-created). + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + UID: regularDeploy.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + }) + + It("should keep the deployment object", func() { + // The deployment object should still be owned by the appliedWork object of the primary work object. + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to keep the deployment object") + + gotDeploy := &appsv1.Deployment{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, gotDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + Expect(gotDeploy.UID).To(Equal(regularDeploy.UID), "The deployment object has been re-created unexpectedly") + }) + + It("should remove the config map object", func() { + regularConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + }) + + AfterAll(func() { + // Delete the work objects. + // + // As there is no GC controller in the test environment (and the secondary work object is not owned by + // the primary one in this test), both work objects are deleted explicitly. + deleteWorkObject(primaryWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Ensure that the appliedWork objects have been removed. + primaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(primaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + + secondaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, secondaryWorkName) + Eventually(secondaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + + // Ensure that the work objects have been removed. + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("scaling up the number of work objects", Ordered, func() { + // The two work objects are linked to each other at first: they share the same owner placement policy/binding + // and the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. A rollout later bumps the linked work count to three, and a third work object joins the set. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + thirdWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The name of the new primary placement resource snapshot, as created by a rollout. + newPrimarySnapshotName := fmt.Sprintf("%s-new", primarySnapshotName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // For simplicity reasons, each work object features only one config map. + primaryConfigMapName := "configmap-primary" + secondaryConfigMapName := "configmap-secondary" + thirdConfigMapName := "configmap-third" + // The generation of the existing work objects when they were last processed before the rollout. + // + // The work objects are only updated by the work applier with finalizer and status changes before the + // rollout, neither of which bumps their generations. + lastProcessedWorkGeneration := int64(1) + + var appliedWorkOwnerRef *metav1.OwnerReference + var primaryConfigMap *corev1.ConfigMap + var secondaryConfigMap *corev1.ConfigMap + var thirdConfigMap *corev1.ConfigMap + // The config map objects as they appear in the new primary placement resource snapshot. + var updatedPrimaryConfigMap *corev1.ConfigMap + var updatedSecondaryConfigMap *corev1.ConfigMap + var updatedPrimaryConfigMapJSON []byte + var updatedSecondaryConfigMapJSON []byte + var thirdConfigMapJSON []byte + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as none of the work objects in this test case + // features the namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the config map objects. + primaryConfigMap = configMap.DeepCopy() + primaryConfigMap.Namespace = nsName + primaryConfigMap.Name = primaryConfigMapName + primaryConfigMapJSON := marshalK8sObjJSON(primaryConfigMap) + + secondaryConfigMap = configMap.DeepCopy() + secondaryConfigMap.Namespace = nsName + secondaryConfigMap.Name = secondaryConfigMapName + secondaryConfigMapJSON := marshalK8sObjJSON(secondaryConfigMap) + + // Prepare the config map objects as they appear in the new primary placement resource snapshot: the + // data of the existing config maps is updated, and a new config map is added. + updatedPrimaryConfigMap = primaryConfigMap.DeepCopy() + updatedPrimaryConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + updatedPrimaryConfigMapJSON = marshalK8sObjJSON(updatedPrimaryConfigMap) + + updatedSecondaryConfigMap = updatedPrimaryConfigMap.DeepCopy() + updatedSecondaryConfigMap.Name = secondaryConfigMapName + updatedSecondaryConfigMapJSON = marshalK8sObjJSON(updatedSecondaryConfigMap) + + thirdConfigMap = updatedPrimaryConfigMap.DeepCopy() + thirdConfigMap.Name = thirdConfigMapName + thirdConfigMapJSON = marshalK8sObjJSON(thirdConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + // + // As KubeFleet does, the primary work object is set as the controller owner of all secondary work objects, + // so that events on secondary work objects trigger reconciliations of the primary work object. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + secondaryConfigMapJSON, + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + primaryConfigMapJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object of the primary work object; all the manifests + // across the linked work objects are owned by this appliedWork object. + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + + // Ensure that the config map objects have been applied as expected. + primaryConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, primaryConfigMapName, appliedWorkOwnerRef) + Eventually(primaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the primary work object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: primaryConfigMapName}, primaryConfigMap)).To(Succeed(), "Failed to retrieve the config map object from the primary work object") + + secondaryConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, secondaryConfigMapName, appliedWorkOwnerRef) + Eventually(secondaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the secondary work object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: secondaryConfigMapName}, secondaryConfigMap)).To(Succeed(), "Failed to retrieve the config map object from the secondary work object") + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // Note that the ordinals of the manifests are relative to their own work object. + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("should update the appliedWork object statuses", func() { + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + UID: primaryConfigMap.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + UID: secondaryConfigMap.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + }) + + It("can update the primary work object to bump the linked work count (rollout in progress)", func() { + // Mimic a rollout: the primary placement resource snapshot has been bumped, which now requires three + // work objects; the primary work object is refreshed first, while the secondary work object has not + // been refreshed yet, and the third work object has not been created yet. + updatePrimaryLinkedWorkObject(primaryWorkName, memberReservedNSName1, newPrimarySnapshotName, "3", nil, updatedPrimaryConfigMapJSON) + }) + + It("should not process the work objects (primary work object refreshed only)", func() { + // The linked work objects are now in an inconsistent state (the secondary work object is still linked + // to the old primary placement resource snapshot, and the third work object is absent); the work applier + // should hold off any processing. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + // All the conditions on the primary work object should become stale. + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + primaryWorkStatusStaleActual := workStatusStaleActual(memberReservedNSName1, primaryWorkName, lastProcessedWorkGeneration, workConds, primaryWorkManifestStatuses) + + // The secondary work object has not been refreshed yet; its status should stay the same. + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUnchangedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + + // The applied config map objects should stay the same. + primaryConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, primaryConfigMapName, appliedWorkOwnerRef) + secondaryConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, secondaryConfigMapName, appliedWorkOwnerRef) + + // Run all the checks in one Consistently block, so that they are verified over the same period of time. + Consistently(func() error { + if err := primaryWorkStatusStaleActual(); err != nil { + return fmt.Errorf("primary work object: %w", err) + } + if err := secondaryWorkStatusUnchangedActual(); err != nil { + return fmt.Errorf("secondary work object: %w", err) + } + if err := primaryConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the primary work object: %w", err) + } + if err := secondaryConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the secondary work object: %w", err) + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been processed unexpectedly") + }) + + It("can update the secondary work object (rollout in progress)", func() { + updateLinkedWorkObject(secondaryWorkName, memberReservedNSName1, newPrimarySnapshotName, nil, updatedSecondaryConfigMapJSON) + }) + + It("should not process the work objects (primary and secondary work objects refreshed only)", func() { + // The linked work objects are still in an inconsistent state (the third work object is absent); the + // work applier should hold off any processing. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + // All the conditions on the primary work object should stay stale. + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + primaryWorkStatusStaleActual := workStatusStaleActual(memberReservedNSName1, primaryWorkName, lastProcessedWorkGeneration, workConds, primaryWorkManifestStatuses) + + // All the conditions on the secondary work object should become stale as well. + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusStaleActual := workStatusStaleActual(memberReservedNSName1, secondaryWorkName, lastProcessedWorkGeneration, workConds, secondaryWorkManifestStatuses) + + // The applied config map objects should stay the same. + primaryConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, primaryConfigMapName, appliedWorkOwnerRef) + secondaryConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, secondaryConfigMapName, appliedWorkOwnerRef) + + // Run all the checks in one Consistently block, so that they are verified over the same period of time. + Consistently(func() error { + if err := primaryWorkStatusStaleActual(); err != nil { + return fmt.Errorf("primary work object: %w", err) + } + if err := secondaryWorkStatusStaleActual(); err != nil { + return fmt.Errorf("secondary work object: %w", err) + } + if err := primaryConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the primary work object: %w", err) + } + if err := secondaryConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the secondary work object: %w", err) + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been processed unexpectedly") + }) + + It("can create the third work object (rollout completed)", func() { + thirdWork := buildWorkObject( + thirdWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, newPrimarySnapshotName, + nil, + thirdConfigMapJSON, + ) + setPrimaryWorkAsOwner(thirdWork, primaryWorkName) + Expect(hubClient.Create(ctx, thirdWork)).To(Succeed(), "Failed to create the third work object") + }) + + It("should apply the manifests", func() { + // The creation of the third work object triggers a reconciliation of the primary work object via the + // owner reference. + primaryConfigMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, updatedPrimaryConfigMap, appliedWorkOwnerRef) + Eventually(primaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the primary work object") + + secondaryConfigMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, updatedSecondaryConfigMap, appliedWorkOwnerRef) + Eventually(secondaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the secondary work object") + + thirdConfigMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, thirdConfigMap, appliedWorkOwnerRef) + Eventually(thirdConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the third work object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: thirdConfigMapName}, thirdConfigMap)).To(Succeed(), "Failed to retrieve the config map object from the third work object") + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + + thirdWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: thirdConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + thirdWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, thirdWorkName, append([]metav1.Condition{}, workConds...), thirdWorkManifestStatuses) + Eventually(thirdWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the third work object status") + }) + + It("should update the appliedWork object statuses", func() { + // The existing config map objects are the same ones as before (i.e., they have not been re-created). + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + UID: primaryConfigMap.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + UID: secondaryConfigMap.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + + thirdAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: thirdConfigMapName, + Namespace: nsName, + }, + UID: thirdConfigMap.UID, + }, + } + thirdAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, thirdWorkName, thirdAppliedResources) + Eventually(thirdAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the third appliedWork object status") + }) + + AfterAll(func() { + // Delete the work objects. + // + // As there is no GC controller in the test environment (and the secondary/third work objects are not + // owned by the primary one in this test), all work objects are deleted explicitly. + deleteWorkObject(primaryWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + deleteWorkObject(thirdWorkName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + primaryConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, primaryConfigMapName) + Eventually(primaryConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object from the primary work object") + + secondaryConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, secondaryConfigMapName) + Eventually(secondaryConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object from the secondary work object") + + thirdConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, thirdConfigMapName) + Eventually(thirdConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object from the third work object") + + // Ensure that the appliedWork objects have been removed. + primaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(primaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + + secondaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, secondaryWorkName) + Eventually(secondaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + + thirdAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, thirdWorkName) + Eventually(thirdAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the third appliedWork object") + + // Ensure that the work objects have been removed. + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + thirdWorkRemovedActual := workRemovedActual(memberReservedNSName1, thirdWorkName) + Eventually(thirdWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the third work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("scaling down the number of work objects", Ordered, func() { + // The three work objects are linked to each other at first: they share the same owner placement policy/binding + // and the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. A rollout later drops the linked work count to two, and the third work object leaves the set. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + thirdWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The name of the new primary placement resource snapshot, as created by a rollout. + newPrimarySnapshotName := fmt.Sprintf("%s-new", primarySnapshotName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // For simplicity reasons, each work object features only one config map. + primaryConfigMapName := "configmap-primary" + secondaryConfigMapName := "configmap-secondary" + thirdConfigMapName := "configmap-third" + // The generation of the existing work objects when they were last processed before the rollout. + // + // The work objects are only updated by the work applier with finalizer and status changes before the + // rollout, neither of which bumps their generations. + lastProcessedWorkGeneration := int64(1) + + var appliedWorkOwnerRef *metav1.OwnerReference + var primaryConfigMap *corev1.ConfigMap + var secondaryConfigMap *corev1.ConfigMap + var thirdConfigMap *corev1.ConfigMap + // The config map objects as they appear in the new primary placement resource snapshot. + var updatedPrimaryConfigMap *corev1.ConfigMap + var updatedSecondaryConfigMap *corev1.ConfigMap + var updatedPrimaryConfigMapJSON []byte + var updatedSecondaryConfigMapJSON []byte + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as none of the work objects in this test case + // features the namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the config map objects. + primaryConfigMap = configMap.DeepCopy() + primaryConfigMap.Namespace = nsName + primaryConfigMap.Name = primaryConfigMapName + primaryConfigMapJSON := marshalK8sObjJSON(primaryConfigMap) + + secondaryConfigMap = configMap.DeepCopy() + secondaryConfigMap.Namespace = nsName + secondaryConfigMap.Name = secondaryConfigMapName + secondaryConfigMapJSON := marshalK8sObjJSON(secondaryConfigMap) + + thirdConfigMap = configMap.DeepCopy() + thirdConfigMap.Namespace = nsName + thirdConfigMap.Name = thirdConfigMapName + thirdConfigMapJSON := marshalK8sObjJSON(thirdConfigMap) + + // Prepare the config map objects as they appear in the new primary placement resource snapshot: the + // data of the remaining config maps is updated, and the third config map is dropped. + updatedPrimaryConfigMap = primaryConfigMap.DeepCopy() + updatedPrimaryConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + updatedPrimaryConfigMapJSON = marshalK8sObjJSON(updatedPrimaryConfigMap) + + updatedSecondaryConfigMap = updatedPrimaryConfigMap.DeepCopy() + updatedSecondaryConfigMap.Name = secondaryConfigMapName + updatedSecondaryConfigMapJSON = marshalK8sObjJSON(updatedSecondaryConfigMap) + + // Create the secondary and third work objects first, so that the work applier can find all the linked + // work objects as soon as it starts processing the primary work object. + // + // As KubeFleet does, the primary work object is set as the controller owner of all secondary work objects, + // so that events on secondary work objects trigger reconciliations of the primary work object. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + secondaryConfigMapJSON, + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + thirdWork := buildWorkObject( + thirdWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + thirdConfigMapJSON, + ) + setPrimaryWorkAsOwner(thirdWork, primaryWorkName) + Expect(hubClient.Create(ctx, thirdWork)).To(Succeed(), "Failed to create the third work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + primaryConfigMapJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "3" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should apply the manifests", func() { + // Retrieve the owner reference of the appliedWork object of the primary work object; all the manifests + // across the linked work objects are owned by this appliedWork object. + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + + // Ensure that the config map objects have been applied as expected. + primaryConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, primaryConfigMapName, appliedWorkOwnerRef) + Eventually(primaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the primary work object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: primaryConfigMapName}, primaryConfigMap)).To(Succeed(), "Failed to retrieve the config map object from the primary work object") + + secondaryConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, secondaryConfigMapName, appliedWorkOwnerRef) + Eventually(secondaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the secondary work object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: secondaryConfigMapName}, secondaryConfigMap)).To(Succeed(), "Failed to retrieve the config map object from the secondary work object") + + thirdConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, thirdConfigMapName, appliedWorkOwnerRef) + Eventually(thirdConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the third work object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: thirdConfigMapName}, thirdConfigMap)).To(Succeed(), "Failed to retrieve the config map object from the third work object") + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // Note that the ordinals of the manifests are relative to their own work object. + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + + thirdWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: thirdConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + thirdWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, thirdWorkName, append([]metav1.Condition{}, workConds...), thirdWorkManifestStatuses) + Eventually(thirdWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the third work object status") + }) + + It("should update the appliedWork object statuses", func() { + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + UID: primaryConfigMap.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + UID: secondaryConfigMap.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + + thirdAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: thirdConfigMapName, + Namespace: nsName, + }, + UID: thirdConfigMap.UID, + }, + } + thirdAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, thirdWorkName, thirdAppliedResources) + Eventually(thirdAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the third appliedWork object status") + }) + + It("can update the primary work object to drop the linked work count (rollout in progress)", func() { + // Mimic a rollout: the primary placement resource snapshot has been bumped, which now requires only two + // work objects; the primary work object is refreshed first, while the secondary work object has not + // been refreshed yet, and the third work object has not been deleted yet. + updatePrimaryLinkedWorkObject(primaryWorkName, memberReservedNSName1, newPrimarySnapshotName, "2", nil, updatedPrimaryConfigMapJSON) + }) + + It("should not process the work objects (primary work object refreshed only)", func() { + // The linked work objects are now in an inconsistent state (the secondary work object is still linked + // to the old primary placement resource snapshot); the work applier should hold off any processing. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + // All the conditions on the primary work object should become stale. + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + primaryWorkStatusStaleActual := workStatusStaleActual(memberReservedNSName1, primaryWorkName, lastProcessedWorkGeneration, workConds, primaryWorkManifestStatuses) + + // The secondary and third work objects have not been refreshed yet; their statuses should stay the same. + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUnchangedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + + thirdWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: thirdConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + thirdWorkStatusUnchangedActual := workStatusUpdated(memberReservedNSName1, thirdWorkName, append([]metav1.Condition{}, workConds...), thirdWorkManifestStatuses) + + // The applied config map objects should stay the same. + primaryConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, primaryConfigMapName, appliedWorkOwnerRef) + secondaryConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, secondaryConfigMapName, appliedWorkOwnerRef) + thirdConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, thirdConfigMapName, appliedWorkOwnerRef) + + // Run all the checks in one Consistently block, so that they are verified over the same period of time. + Consistently(func() error { + if err := primaryWorkStatusStaleActual(); err != nil { + return fmt.Errorf("primary work object: %w", err) + } + if err := secondaryWorkStatusUnchangedActual(); err != nil { + return fmt.Errorf("secondary work object: %w", err) + } + if err := thirdWorkStatusUnchangedActual(); err != nil { + return fmt.Errorf("third work object: %w", err) + } + if err := primaryConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the primary work object: %w", err) + } + if err := secondaryConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the secondary work object: %w", err) + } + if err := thirdConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the third work object: %w", err) + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been processed unexpectedly") + }) + + It("can update the secondary work object (rollout in progress)", func() { + updateLinkedWorkObject(secondaryWorkName, memberReservedNSName1, newPrimarySnapshotName, nil, updatedSecondaryConfigMapJSON) + }) + + It("should not process the work objects (primary and secondary work objects refreshed only)", func() { + // The linked work objects are still in an inconsistent state (the third work object, which is no longer + // linked to the current primary placement resource snapshot, has not been marked for deletion yet); the + // work applier should hold off any processing. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + // All the conditions on the primary work object should stay stale. + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + primaryWorkStatusStaleActual := workStatusStaleActual(memberReservedNSName1, primaryWorkName, lastProcessedWorkGeneration, workConds, primaryWorkManifestStatuses) + + // All the conditions on the secondary work object should become stale as well. + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusStaleActual := workStatusStaleActual(memberReservedNSName1, secondaryWorkName, lastProcessedWorkGeneration, workConds, secondaryWorkManifestStatuses) + + // The third work object has not been deleted yet; its status should stay the same. + thirdWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: thirdConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + thirdWorkStatusUnchangedActual := workStatusUpdated(memberReservedNSName1, thirdWorkName, append([]metav1.Condition{}, workConds...), thirdWorkManifestStatuses) + + // The applied config map objects should stay the same. + primaryConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, primaryConfigMapName, appliedWorkOwnerRef) + secondaryConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, secondaryConfigMapName, appliedWorkOwnerRef) + thirdConfigMapUnchangedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, thirdConfigMapName, appliedWorkOwnerRef) + + // Run all the checks in one Consistently block, so that they are verified over the same period of time. + Consistently(func() error { + if err := primaryWorkStatusStaleActual(); err != nil { + return fmt.Errorf("primary work object: %w", err) + } + if err := secondaryWorkStatusStaleActual(); err != nil { + return fmt.Errorf("secondary work object: %w", err) + } + if err := thirdWorkStatusUnchangedActual(); err != nil { + return fmt.Errorf("third work object: %w", err) + } + if err := primaryConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the primary work object: %w", err) + } + if err := secondaryConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the secondary work object: %w", err) + } + if err := thirdConfigMapUnchangedActual(); err != nil { + return fmt.Errorf("config map from the third work object: %w", err) + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been processed unexpectedly") + }) + + It("can delete the third work object (rollout completed)", func() { + // The third work object has the cleanup finalizer; it will stay in the pending deletion state until + // the work applier cleans it up as a left-over work object. + deleteWorkObject(thirdWorkName, memberReservedNSName1) + }) + + It("should apply the manifests", func() { + // The deletion of the third work object triggers a reconciliation of the primary work object via the + // owner reference. + primaryConfigMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, updatedPrimaryConfigMap, appliedWorkOwnerRef) + Eventually(primaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the primary work object") + + secondaryConfigMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, updatedSecondaryConfigMap, appliedWorkOwnerRef) + Eventually(secondaryConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the secondary work object") + }) + + It("should remove the config map object from the third work object", func() { + regularConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, nsName, thirdConfigMapName) + Eventually(regularConfigMapAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object from the third work object") + }) + + It("should remove the third appliedWork object", func() { + // The third appliedWork object owns no resources, and is deleted by the work applier directly. + appliedWorkAbsentActual := appliedWorkAbsentActual(memberClient1, thirdWorkName) + Eventually(appliedWorkAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the third appliedWork object") + }) + + It("should remove the third work object", func() { + workRemovedActual := workRemovedActual(memberReservedNSName1, thirdWorkName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the third work object") + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("should update the appliedWork object statuses", func() { + // The remaining config map objects are the same ones as before (i.e., they have not been re-created). + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: primaryConfigMapName, + Namespace: nsName, + }, + UID: primaryConfigMap.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondaryConfigMapName, + Namespace: nsName, + }, + UID: secondaryConfigMap.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + }) + + AfterAll(func() { + // Delete the remaining work objects. + // + // As there is no GC controller in the test environment (and the secondary work object is not owned by + // the primary one in this test), both work objects are deleted explicitly. + deleteWorkObject(primaryWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + primaryConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, primaryConfigMapName) + Eventually(primaryConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object from the primary work object") + + secondaryConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, secondaryConfigMapName) + Eventually(secondaryConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object from the secondary work object") + + // Ensure that the appliedWork objects have been removed. + primaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(primaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + + secondaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, secondaryWorkName) + Eventually(secondaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + + // Ensure that the work objects have been removed. + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/defaults.go b/pkg/v1/controllers/workapplier/defaults.go new file mode 100644 index 000000000..96124ef37 --- /dev/null +++ b/pkg/v1/controllers/workapplier/defaults.go @@ -0,0 +1,47 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// setDefaultSyncStrategy fills in a default sync strategy on a work object that carries none. +// +// The individual fields of a sync strategy are defaulted by the API server; this only covers the case where +// the sync strategy itself is absent, which leaves those defaults unapplied. +func setDefaultSyncStrategy(work *placementv1alpha1.Work) { + if work.Spec.SyncStrategy != nil { + if work.Spec.SyncStrategy.ApplyMethod == placementv1alpha1.ApplyMethodServerSideApply && + work.Spec.SyncStrategy.ServerSideApplyOptions == nil { + work.Spec.SyncStrategy.ServerSideApplyOptions = &placementv1alpha1.ServerSideApplyOptions{ + ForceConflicts: false, + } + } + return + } + + work.Spec.SyncStrategy = &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodClientSideApply, + WhenOwnedByOthers: placementv1alpha1.WhenOwnedByOthersOptionReportError, + WhenDrifted: placementv1alpha1.WhenDriftedOptionApplyAnyway, + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionReportError, + WhenPlacementDeleted: placementv1alpha1.WhenPlacementDeletedOptionCleanUpResources, + WhenNamespaceDoesNotExist: placementv1alpha1.WhenNamespaceDoesNotExistOptionCreateNamespace, + ComparisonOption: placementv1alpha1.ComparisonOptionPartialComparison, + } +} diff --git a/pkg/v1/controllers/workapplier/drift_integration_test.go b/pkg/v1/controllers/workapplier/drift_integration_test.go new file mode 100644 index 000000000..5f8de320c --- /dev/null +++ b/pkg/v1/controllers/workapplier/drift_integration_test.go @@ -0,0 +1,1200 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("drift detection", func() { + Context("apply anyway (default), partial comparison", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularDeploy *appsv1.Deployment + + BeforeAll(func() { + // Prepare the manifests. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a new work object with no sync strategy; by default, the work applier overwrites drifts and + // uses partial comparison. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularDeployJSON) + }) + + It("should apply the manifests", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can introduce a drift on the deployment and add a label that the manifest does not specify", func() { + updateDeploymentInMemberCluster(memberClient1, nsName, deployName, func(deploy *appsv1.Deployment) { + deploy.Spec.Replicas = ptr.To(int32(2)) + deploy.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + }) + }) + + It("should overwrite the drift", func() { + // The change in the member cluster triggers no event for the work object; the drift is overwritten at the + // next requeue of the work object, with no change on the work object itself. + wantDeploy := regularDeploy.DeepCopy() + wantDeploy.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + deploymentObjectActual := deploymentObjectActual(memberClient1, wantDeploy) + Eventually(deploymentObjectActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to overwrite the drift on the deployment") + }) + + It("should keep the label that the manifest does not specify", func() { + objectLabelsActual := objectLabelsActual(memberClient1, regularDeploy, map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + }) + Consistently(objectLabelsActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The label that the manifest does not specify has been changed") + }) + + It("can mark the deployment as available", func() { + // The deployment has been updated twice (by the drift, and by the work applier) since it was last marked + // as available. + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status (no diff reported)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 3, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 3, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the deployment manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("report error, partial comparison (drift appears, changes, and is reverted)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + // The deployment as it is after the drift has been introduced. + var driftedDeploy *appsv1.Deployment + // The timestamp when the drift was first observed. + var firstDiffedTimestamp metav1.Time + + // The status of the namespace manifest stays the same throughout the test case. + nsManifestStatus := func() placementv1alpha1.PerManifestStatus { + return placementv1alpha1.PerManifestStatus{ + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + } + } + deployManifestIdentifier := func() placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + } + } + + BeforeAll(func() { + // Prepare the manifests. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a new work object that reports drifts instead of overwriting them; the API server fills in the + // defaults for the other fields in the sync strategy (partial comparison). + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenDrifted: placementv1alpha1.WhenDriftedOptionReportError, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularNSJSON, regularDeployJSON) + }) + + It("should apply the manifests", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: deployManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can introduce a drift on the deployment", func() { + driftedDeploy = updateDeploymentInMemberCluster(memberClient1, nsName, deployName, func(deploy *appsv1.Deployment) { + deploy.Spec.Replicas = ptr.To(int32(2)) + }) + }) + + It("should report the drift", func() { + // The change in the member cluster triggers no event for the work object; the drift is found at the next + // requeue of the work object. The manifest is no longer applied, and its Available condition is removed. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: deployManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFoundDrifts), + ObservedGeneration: 2, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(2)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "2", + ValueInHub: "1", + }, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the drift") + + firstDiffedTimestamp = retrieveManifestFirstDiffedTimestamp(memberReservedNSName1, workName, 1) + }) + + It("should not overwrite the drift", func() { + // The check spans at least one periodic requeue of the work object. + deployUntouchedActual := objectUntouchedActual(memberClient1, driftedDeploy) + Consistently(deployUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The drifted deployment has been modified") + }) + + It("can change the drift on the deployment", func() { + updateDeploymentInMemberCluster(memberClient1, nsName, deployName, func(deploy *appsv1.Deployment) { + deploy.Spec.Replicas = ptr.To(int32(3)) + }) + }) + + It("should report the changed drift", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: deployManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFoundDrifts), + ObservedGeneration: 3, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(3)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "3", + ValueInHub: "1", + }, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the changed drift") + }) + + It("should keep the timestamp when the drift was first observed", func() { + // The diff details have been rewritten since the timestamp was captured, but the object has stayed + // drifted all along. + gotTimestamp := retrieveManifestFirstDiffedTimestamp(memberReservedNSName1, workName, 1) + Expect(gotTimestamp.Equal(&firstDiffedTimestamp)).To(BeTrue(), "first diffed timestamp = %v, want %v", gotTimestamp, firstDiffedTimestamp) + }) + + It("can revert the drift on the deployment", func() { + updateDeploymentInMemberCluster(memberClient1, nsName, deployName, func(deploy *appsv1.Deployment) { + deploy.Spec.Replicas = ptr.To(int32(1)) + }) + }) + + It("should report the manifest as applied again (deployment not yet available)", func() { + // The deployment has been updated three times since it was last marked as available. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: deployManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 4, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeNotYetAvailable), + ObservedGeneration: 4, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the manifest as applied again") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status (all available)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: deployManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 4, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 4, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the deployment manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("report error, full comparison", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + // The config map as it is after a label has been added in the member cluster. + var labeledConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster; the work object does not feature the namespace + // manifest, as full comparison always finds diffs on the fields that the API server populates on + // namespaces (e.g., the finalizers), which the work applier would then keep reporting as drifts. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a new work object that reports drifts instead of overwriting them, and that compares all the + // fields, including those that the manifest does not specify. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenDrifted: placementv1alpha1.WhenDriftedOptionReportError, + ComparisonOption: placementv1alpha1.ComparisonOptionFullComparison, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should apply the manifest", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + }) + + It("should update the work object status", func() { + // No diff is reported. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can add a label that the manifest does not specify to the config map", func() { + labeledConfigMap = updateConfigMapInMemberCluster(memberClient1, nsName, configMapName, func(configMap *corev1.ConfigMap) { + configMap.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + }) + }) + + It("should report the label as a drift", func() { + // The change in the member cluster triggers no event for the work object; the drift is found at the next + // requeue of the work object. The manifest is no longer applied, and its Available condition is removed. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFoundDrifts), + ObservedGeneration: 0, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: fmt.Sprintf("/metadata/labels/%s", memberOnlyLabelKey), + ValueInMember: memberOnlyLabelValue, + }, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the drift") + }) + + It("should not overwrite the drift", func() { + // The check spans at least one periodic requeue of the work object. + configMapUntouchedActual := objectUntouchedActual(memberClient1, labeledConfigMap) + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The drifted config map has been modified") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("report error, partial comparison (manifest updated after a drift)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // The image that the updated manifest specifies. + updatedImage := "nginx:1.27" + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNSJSON []byte + var updatedDeploy *appsv1.Deployment + var syncStrategy *placementv1alpha1.SyncStrategy + + // The status of the namespace manifest stays the same throughout the test case. + nsManifestStatus := func() placementv1alpha1.PerManifestStatus { + return placementv1alpha1.PerManifestStatus{ + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + } + } + deployManifestIdentifier := func() placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + } + } + + BeforeAll(func() { + // Prepare the manifests. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare the updated deployment manifest; it changes the image, but keeps the replica count. + updatedDeploy = regularDeploy.DeepCopy() + updatedDeploy.Spec.Template.Spec.Containers[0].Image = updatedImage + + // Create a new work object that reports drifts instead of overwriting them. + syncStrategy = &placementv1alpha1.SyncStrategy{ + WhenDrifted: placementv1alpha1.WhenDriftedOptionReportError, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularNSJSON, regularDeployJSON) + }) + + It("should apply the manifests", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + }) + + It("can introduce a drift on the deployment", func() { + updateDeploymentInMemberCluster(memberClient1, nsName, deployName, func(deploy *appsv1.Deployment) { + deploy.Spec.Replicas = ptr.To(int32(2)) + }) + }) + + It("should report the drift", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: deployManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFoundDrifts), + ObservedGeneration: 2, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(2)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "2", + ValueInHub: "1", + }, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the drift") + }) + + It("can update the deployment manifest", func() { + updateWorkObject(workName, memberReservedNSName1, syncStrategy, regularNSJSON, marshalK8sObjJSON(updatedDeploy)) + }) + + It("should apply the updated manifest and overwrite the drift", func() { + // Drift detection runs only if the manifest has not changed since it was last applied; the updated + // manifest is applied regardless of the drift, which brings the replica count back to the value that + // the manifest specifies. + wantDeploy := updatedDeploy.DeepCopy() + wantDeploy.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + deploymentObjectActual := deploymentObjectActual(memberClient1, wantDeploy) + Eventually(deploymentObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the updated manifest") + }) + + It("should update the work object status (no drift reported)", func() { + // The deployment has been updated twice (by the drift, and by the work applier) since it was created, and + // has never been marked as available. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: deployManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 3, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeNotYetAvailable), + ObservedGeneration: 3, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the deployment manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("apply anyway, full comparison (diffs reported after apply ops)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + + // The status of the namespace manifest stays the same throughout the test case: the namespace manifest does + // not specify any finalizer, while the API server adds the finalizer for the namespace on its own; full + // comparison reports this as a diff, which the apply ops cannot resolve. + nsManifestStatus := func() placementv1alpha1.PerManifestStatus { + return placementv1alpha1.PerManifestStatus{ + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/finalizers", + ValueInMember: "[kubernetes]", + }, + }, + }, + } + } + configMapManifestIdentifier := func() placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + } + } + + BeforeAll(func() { + // Prepare the manifests. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a new work object that overwrites drifts (the default), and that compares all the fields, + // including those that the manifest does not specify. + syncStrategy := &placementv1alpha1.SyncStrategy{ + ComparisonOption: placementv1alpha1.ComparisonOptionFullComparison, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularNSJSON, regularConfigMapJSON) + }) + + It("should apply the manifests", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + }) + + It("should update the work object status", func() { + // The manifests are applied; only the namespace manifest has diffs. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: configMapManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can add a label that the manifest does not specify to the config map", func() { + updateConfigMapInMemberCluster(memberClient1, nsName, configMapName, func(configMap *corev1.ConfigMap) { + configMap.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + }) + }) + + It("should report the label as a diff while keeping the manifest applied", func() { + // The change in the member cluster triggers no event for the work object; the diff is found at the next + // requeue of the work object. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + nsManifestStatus(), + { + Identifier: configMapManifestIdentifier(), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: fmt.Sprintf("/metadata/labels/%s", memberOnlyLabelKey), + ValueInMember: memberOnlyLabelValue, + }, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the diff") + }) + + It("should keep the label that the manifest does not specify", func() { + // The apply ops (client-side apply) do not remove fields that the manifest has never specified. + wantConfigMap := regularConfigMap.DeepCopy() + wantConfigMap.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Consistently(configMapObjectActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The label that the manifest does not specify has been changed") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/eventhandler.go b/pkg/v1/controllers/workapplier/eventhandler.go new file mode 100644 index 000000000..ff84830ec --- /dev/null +++ b/pkg/v1/controllers/workapplier/eventhandler.go @@ -0,0 +1,93 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/handler" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +var ( + // workEventHandler maps every work object event to a reconcile request for the corresponding primary work + // object, as the work applier always processes all the linked work objects via their primary work object. + workEventHandler = handler.TypedFuncs[client.Object, reconcile.Request]{ + CreateFunc: func(_ context.Context, evt event.TypedCreateEvent[client.Object], queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + enqueuePrimaryWork(evt.Object, "create", queue) + }, + UpdateFunc: func(_ context.Context, evt event.TypedUpdateEvent[client.Object], queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + enqueuePrimaryWork(evt.ObjectNew, "update", queue) + }, + DeleteFunc: func(_ context.Context, evt event.TypedDeleteEvent[client.Object], queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + enqueuePrimaryWork(evt.Object, "delete", queue) + }, + GenericFunc: func(_ context.Context, evt event.TypedGenericEvent[client.Object], queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + enqueuePrimaryWork(evt.Object, "generic", queue) + }, + } +) + +// enqueuePrimaryWork enqueues a reconcile request for the primary work object associated with the given work object. +// +// A primary work object (one with the linked work count annotation) is enqueued directly; a secondary work object +// is resolved to its primary work object via its controller owner reference. +func enqueuePrimaryWork(work client.Object, eventType string, queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + if work == nil { + wrappedErr := errors.NewUnexpectedError(nil, "received a work object event with no object", "eventType", eventType) + klog.ErrorS(wrappedErr, "Failed to process a work object event", errors.Args(wrappedErr)...) + return + } + + if _, isPrimary := work.GetAnnotations()[placementv1alpha1.LinkedWorkCountAnnotationKey]; isPrimary { + klog.V(2).InfoS("Enqueueing a primary work object", "work", klog.KObj(work), "eventType", eventType) + queue.Add(reconcile.Request{NamespacedName: types.NamespacedName{ + Namespace: work.GetNamespace(), + Name: work.GetName(), + }}) + return + } + + ownerRef := metav1.GetControllerOfNoCopy(work) + if ownerRef == nil || + ownerRef.APIVersion != placementv1alpha1.GroupVersion.String() || + ownerRef.Kind != placementv1alpha1.WorkKind || + ownerRef.Name == "" { + wrappedErr := errors.NewUnexpectedError(nil, + "the work object is neither a primary work object nor a secondary work object with a valid owner work reference", + "work", klog.KObj(work), "eventType", eventType, "ownerRefs", work.GetOwnerReferences()) + klog.ErrorS(wrappedErr, "Failed to process a work object event", errors.Args(wrappedErr)...) + return + } + + // Owner references can only point to objects in the same namespace. + klog.V(2).InfoS("Enqueueing the owner primary work object of a secondary work object", + "work", klog.KObj(work), "primaryWork", klog.KRef(work.GetNamespace(), ownerRef.Name), "eventType", eventType) + queue.Add(reconcile.Request{NamespacedName: types.NamespacedName{ + Namespace: work.GetNamespace(), + Name: ownerRef.Name, + }}) +} diff --git a/pkg/v1/controllers/workapplier/eventhandler_test.go b/pkg/v1/controllers/workapplier/eventhandler_test.go new file mode 100644 index 000000000..bf4dc8e08 --- /dev/null +++ b/pkg/v1/controllers/workapplier/eventhandler_test.go @@ -0,0 +1,269 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "testing" + + "github.com/google/go-cmp/cmp" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/util/workqueue" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + eventHandlerTestNSName = "fleet-member-experimental" + eventHandlerTestPrimaryWorkName = "work-1" + eventHandlerTestSecondaryWorkName = "work-1-1" +) + +func eventHandlerTestPrimaryWork() *placementv1alpha1.Work { + return &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: eventHandlerTestNSName, + Name: eventHandlerTestPrimaryWorkName, + Annotations: map[string]string{ + placementv1alpha1.LinkedWorkCountAnnotationKey: "2", + }, + }, + } +} + +func eventHandlerTestSecondaryWork(ownerRef *metav1.OwnerReference) *placementv1alpha1.Work { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: eventHandlerTestNSName, + Name: eventHandlerTestSecondaryWorkName, + }, + } + if ownerRef != nil { + work.OwnerReferences = []metav1.OwnerReference{*ownerRef} + } + return work +} + +func eventHandlerTestOwnerRef(mutate func(*metav1.OwnerReference)) *metav1.OwnerReference { + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.WorkKind, + Name: eventHandlerTestPrimaryWorkName, + UID: "owner-uid", + Controller: ptr.To(true), + } + if mutate != nil { + mutate(ownerRef) + } + return ownerRef +} + +// drainQueue returns all the requests currently in the queue. +func drainQueue(queue workqueue.TypedRateLimitingInterface[reconcile.Request]) []reconcile.Request { + reqs := []reconcile.Request{} + for queue.Len() > 0 { + req, _ := queue.Get() + reqs = append(reqs, req) + queue.Done(req) + } + return reqs +} + +func newTestQueue() workqueue.TypedRateLimitingInterface[reconcile.Request] { + return workqueue.NewTypedRateLimitingQueue(workqueue.DefaultTypedControllerRateLimiter[reconcile.Request]()) +} + +func primaryWorkRequest() reconcile.Request { + return reconcile.Request{NamespacedName: types.NamespacedName{ + Namespace: eventHandlerTestNSName, + Name: eventHandlerTestPrimaryWorkName, + }} +} + +// TestEnqueuePrimaryWork tests the enqueuePrimaryWork function. +func TestEnqueuePrimaryWork(t *testing.T) { + testCases := []struct { + name string + work client.Object + wantReqs []reconcile.Request + }{ + { + name: "primary work", + work: eventHandlerTestPrimaryWork(), + wantReqs: []reconcile.Request{primaryWorkRequest()}, + }, + { + name: "primary work with an owner reference", + work: func() client.Object { + work := eventHandlerTestPrimaryWork() + work.OwnerReferences = []metav1.OwnerReference{*eventHandlerTestOwnerRef(func(o *metav1.OwnerReference) { + o.Name = "some-other-work" + })} + return work + }(), + wantReqs: []reconcile.Request{primaryWorkRequest()}, + }, + { + name: "secondary work with a valid controller owner reference", + work: eventHandlerTestSecondaryWork(eventHandlerTestOwnerRef(nil)), + wantReqs: []reconcile.Request{primaryWorkRequest()}, + }, + { + name: "nil work", + work: nil, + wantReqs: []reconcile.Request{}, + }, + { + name: "work with no annotation and no owner reference", + work: eventHandlerTestSecondaryWork(nil), + wantReqs: []reconcile.Request{}, + }, + { + name: "work with a non-controller owner reference", + work: eventHandlerTestSecondaryWork(eventHandlerTestOwnerRef(func(o *metav1.OwnerReference) { + o.Controller = ptr.To(false) + })), + wantReqs: []reconcile.Request{}, + }, + { + name: "work with an owner reference with a wrong API version", + work: eventHandlerTestSecondaryWork(eventHandlerTestOwnerRef(func(o *metav1.OwnerReference) { + o.APIVersion = "placement.kubefleet.dev/v1beta1" + })), + wantReqs: []reconcile.Request{}, + }, + { + name: "work with an owner reference with a wrong kind", + work: eventHandlerTestSecondaryWork(eventHandlerTestOwnerRef(func(o *metav1.OwnerReference) { + o.Kind = placementv1alpha1.AppliedWorkKind + })), + wantReqs: []reconcile.Request{}, + }, + { + name: "work with an owner reference with an empty name", + work: eventHandlerTestSecondaryWork(eventHandlerTestOwnerRef(func(o *metav1.OwnerReference) { + o.Name = "" + })), + wantReqs: []reconcile.Request{}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + queue := newTestQueue() + defer queue.ShutDown() + + enqueuePrimaryWork(tc.work, "test", queue) + + gotReqs := drainQueue(queue) + if diff := cmp.Diff(gotReqs, tc.wantReqs); diff != "" { + t.Errorf("enqueuePrimaryWork() requests diff (-got, +want):\n%s", diff) + } + }) + } +} + +// TestWorkEventHandler tests the workEventHandler event handler funcs. +func TestWorkEventHandler(t *testing.T) { + ctx := context.Background() + + events := []struct { + name string + // handle invokes the handler func for the event type with the given object. + handle func(obj client.Object, queue workqueue.TypedRateLimitingInterface[reconcile.Request]) + }{ + { + name: "create", + handle: func(obj client.Object, queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + workEventHandler.Create(ctx, event.TypedCreateEvent[client.Object]{Object: obj}, queue) + }, + }, + { + name: "update", + handle: func(obj client.Object, queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + // The old object is deliberately a different work object; only the new one should be used. + workEventHandler.Update(ctx, event.TypedUpdateEvent[client.Object]{ + ObjectOld: &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: eventHandlerTestNSName, + Name: "stale-work", + Annotations: map[string]string{ + placementv1alpha1.LinkedWorkCountAnnotationKey: "1", + }, + }, + }, + ObjectNew: obj, + }, queue) + }, + }, + { + name: "delete", + handle: func(obj client.Object, queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + workEventHandler.Delete(ctx, event.TypedDeleteEvent[client.Object]{Object: obj}, queue) + }, + }, + { + name: "generic", + handle: func(obj client.Object, queue workqueue.TypedRateLimitingInterface[reconcile.Request]) { + workEventHandler.Generic(ctx, event.TypedGenericEvent[client.Object]{Object: obj}, queue) + }, + }, + } + + objects := []struct { + name string + obj client.Object + wantReqs []reconcile.Request + }{ + { + name: "primary work", + obj: eventHandlerTestPrimaryWork(), + wantReqs: []reconcile.Request{primaryWorkRequest()}, + }, + { + name: "secondary work", + obj: eventHandlerTestSecondaryWork(eventHandlerTestOwnerRef(nil)), + wantReqs: []reconcile.Request{primaryWorkRequest()}, + }, + { + name: "nil work", + obj: nil, + wantReqs: []reconcile.Request{}, + }, + } + + for _, evt := range events { + for _, o := range objects { + t.Run(evt.name+"/"+o.name, func(t *testing.T) { + queue := newTestQueue() + defer queue.ShutDown() + + evt.handle(o.obj, queue) + + gotReqs := drainQueue(queue) + if diff := cmp.Diff(gotReqs, o.wantReqs); diff != "" { + t.Errorf("workEventHandler %s requests diff (-got, +want):\n%s", evt.name, diff) + } + }) + } + } +} diff --git a/pkg/v1/controllers/workapplier/failures_integration_test.go b/pkg/v1/controllers/workapplier/failures_integration_test.go new file mode 100644 index 000000000..040b4690a --- /dev/null +++ b/pkg/v1/controllers/workapplier/failures_integration_test.go @@ -0,0 +1,3119 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + "time" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + autoscalingv1 "k8s.io/api/autoscaling/v1" + autoscalingv2 "k8s.io/api/autoscaling/v2" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +const ( + // The data value of the config map in a duplicated manifest; it differs from the one in the config map fixture. + duplicatedConfigMapDataVal = "qux" +) + +var _ = Describe("partial failures", func() { + Context("decoding errors mixed with valid manifests", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + malformedObjName := "malformed-1" + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularDeploy *appsv1.Deployment + var regularConfigMap *corev1.ConfigMap + var regularNSJSON []byte + var regularDeployJSON []byte + var regularConfigMapJSON []byte + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare an object of an unknown API group. + // + // The hub API server requires that each manifest has the API version and kind fields set; here the + // object is built from the config map fixture, with the type information overwritten. + malformedObj := configMap.DeepCopy() + malformedObj.TypeMeta = metav1.TypeMeta{ + APIVersion: "malformed/v10", + Kind: "Unknown", + } + malformedObj.Namespace = nsName + malformedObj.Name = malformedObjName + malformedObjJSON := marshalK8sObjJSON(malformedObj) + + // Prepare a deployment object of a known API group, but of an API version that is not served. + unservedVersionDeploy := deploy.DeepCopy() + unservedVersionDeploy.APIVersion = "apps/v1beta1" + unservedVersionDeploy.Namespace = nsName + unservedVersionDeploy.Name = deployName + unservedVersionDeployJSON := marshalK8sObjJSON(unservedVersionDeploy) + + // Prepare a deployment object of a served API version, which is used to fix the manifest above later. + regularDeploy = deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + regularDeployJSON = marshalK8sObjJSON(regularDeploy) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON = marshalK8sObjJSON(regularConfigMap) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, malformedObjJSON, unservedVersionDeployJSON, regularConfigMapJSON) + }) + + It("should add cleanup finalizer to the work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the work object") + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the valid manifests", func() { + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the ConfigMap object has been applied as expected. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, regularConfigMap)).To(Succeed(), "Failed to retrieve the ConfigMap object") + }) + + It("should not create any object for the manifests that cannot be decoded", func() { + // In particular, the work applier should not silently fall back to a served API version for the + // deployment manifest. + regularDeployAbsentActual := regularDeployAbsentActual(memberClient1, nsName, deployName) + Consistently(regularDeployAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The deployment object has been created unexpectedly") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + // Note that the manifests that cannot be REST mapped have an empty resource field in their + // identifiers, and no Available condition. + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "malformed", + APIVersion: "v10", + Kind: "Unknown", + Resource: "", + Name: malformedObjName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDecodingErred), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 2, + APIGroup: "apps", + APIVersion: "v1beta1", + Kind: "Deployment", + Resource: "", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDecodingErred), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 3, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + // Only the applied manifests are reported. + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 3, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("can update the work object to fix the manifests that cannot be decoded", func() { + // Drop the manifest of the unknown API group, and switch the deployment manifest to a served API version. + updateWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularDeployJSON, regularConfigMapJSON) + }) + + It("should apply the deployment manifest", func() { + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, regularDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + }) + + It("should keep the config map object", func() { + // The ordinal of the config map manifest has changed (3 -> 2); as the work applier looks for left-over + // manifests without ordinals, the config map object should not be deleted, re-created, or overwritten. + configMapStableActual := configMapStableActual(memberClient1, nsName, configMapName, regularConfigMap.UID, configMap.Data) + Consistently(configMapStableActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The config map object has been deleted, re-created, or overwritten unexpectedly") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 2, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + // The config map object is the same one as before (i.e., it has not been re-created). + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + UID: regularDeploy.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 2, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("manifests with generate name", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + generateNamePrefix := "cm-gen-" + secondConfigMapName := "configmap-2" + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularConfigMap *corev1.ConfigMap + var secondConfigMap *corev1.ConfigMap + var regularNSJSON []byte + var regularConfigMapJSON []byte + var secondConfigMapJSON []byte + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare a config map object with only the generate name set. + generateNameOnlyConfigMap := configMap.DeepCopy() + generateNameOnlyConfigMap.Namespace = nsName + generateNameOnlyConfigMap.Name = "" + generateNameOnlyConfigMap.GenerateName = generateNamePrefix + generateNameOnlyConfigMapJSON := marshalK8sObjJSON(generateNameOnlyConfigMap) + + // Prepare a config map object with both the name and the generate name set; it is rejected as well. + nameAndGenerateNameConfigMap := configMap.DeepCopy() + nameAndGenerateNameConfigMap.Namespace = nsName + nameAndGenerateNameConfigMap.Name = configMapName + nameAndGenerateNameConfigMap.GenerateName = generateNamePrefix + nameAndGenerateNameConfigMapJSON := marshalK8sObjJSON(nameAndGenerateNameConfigMap) + + // Prepare config map objects with only the name set, which are used to fix the manifests above later. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON = marshalK8sObjJSON(regularConfigMap) + + secondConfigMap = configMap.DeepCopy() + secondConfigMap.Namespace = nsName + secondConfigMap.Name = secondConfigMapName + secondConfigMapJSON = marshalK8sObjJSON(secondConfigMap) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, generateNameOnlyConfigMapJSON, nameAndGenerateNameConfigMapJSON) + }) + + It("should add cleanup finalizer to the work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the work object") + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the manifest without a generate name", func() { + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + }) + + It("should not create any object for the manifests with a generate name", func() { + configMapsWithPrefixAbsentActual := configMapsWithPrefixAbsentActual(memberClient1, nsName, generateNamePrefix) + Consistently(configMapsWithPrefixAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "A config map object has been created from the manifest with only the generate name unexpectedly") + + regularConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, nsName, configMapName) + Consistently(regularConfigMapAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "A config map object has been created from the manifest with both the name and the generate name unexpectedly") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + // Note that the manifests with a generate name can be REST mapped, so the resource field in their + // identifiers is populated; the name field of the manifest with only the generate name, however, is empty. + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: "", + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFoundGenerateName), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 2, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFoundGenerateName), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + // Only the applied manifest is reported. + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("can update the work object to remove the generate names", func() { + updateWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, secondConfigMapJSON, regularConfigMapJSON) + }) + + It("should apply the manifests", func() { + secondConfigMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, secondConfigMap, appliedWorkOwnerRef) + Eventually(secondConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the second config map object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: secondConfigMapName}, secondConfigMap)).To(Succeed(), "Failed to retrieve the second ConfigMap object") + + firstConfigMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, regularConfigMap, appliedWorkOwnerRef) + Eventually(firstConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, regularConfigMap)).To(Succeed(), "Failed to retrieve the ConfigMap object") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondConfigMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 2, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: secondConfigMapName, + Namespace: nsName, + }, + UID: secondConfigMap.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 2, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + firstConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(firstConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + secondConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, secondConfigMapName) + Eventually(secondConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the second config map object") + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("manifests with no name or mismatching scopes", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + noNamespaceConfigMapName := "configmap-no-namespace" + namespacedNSName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a config map object with neither the name nor the generate name set. + noNameConfigMap := configMap.DeepCopy() + noNameConfigMap.Namespace = nsName + noNameConfigMap.Name = "" + noNameConfigMap.GenerateName = "" + noNameConfigMapJSON := marshalK8sObjJSON(noNameConfigMap) + + // Prepare a config map object (namespace-scoped) with no namespace set. + noNamespaceConfigMap := configMap.DeepCopy() + noNamespaceConfigMap.Namespace = "" + noNamespaceConfigMap.Name = noNamespaceConfigMapName + noNamespaceConfigMapJSON := marshalK8sObjJSON(noNamespaceConfigMap) + + // Prepare a namespace object (cluster-scoped) with a namespace set. + namespacedNS := ns.DeepCopy() + namespacedNS.Namespace = nsName + namespacedNS.Name = namespacedNSName + namespacedNSJSON := marshalK8sObjJSON(namespacedNS) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, noNameConfigMapJSON, noNamespaceConfigMapJSON, namespacedNSJSON) + }) + + It("should add cleanup finalizer to the work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the work object") + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the valid manifest", func() { + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + }) + + It("should not create any object for the rejected manifests", func() { + Consistently(func() error { + configMapList := &corev1.ConfigMapList{} + if err := memberClient1.List(ctx, configMapList, client.InNamespace(nsName)); err != nil { + return fmt.Errorf("failed to list ConfigMap objects: %w", err) + } + for idx := range configMapList.Items { + // The root CA config map might be present in the namespace (if the test environment + // publishes it); it is not created by the work applier. + if configMapList.Items[idx].Name != "kube-root-ca.crt" { + return fmt.Errorf("found an unexpected ConfigMap object (%s)", configMapList.Items[idx].Name) + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "A config map object has been created from the manifest with no name unexpectedly") + + noNamespaceConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, "default", noNamespaceConfigMapName) + Consistently(noNamespaceConfigMapAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "A config map object has been created from the manifest with no namespace unexpectedly") + + namespacedNSAbsentActual := namespaceAbsentActual(memberClient1, namespacedNSName) + Consistently(namespacedNSAbsentActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "A namespace object has been created from the manifest with a namespace set unexpectedly") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + // Note that the manifest with no name can be REST mapped, so the resource field in its identifier is + // populated; its name field, however, is empty. The manifests with mismatching scopes have an empty + // resource field in their identifiers, as the scope check is part of the decoding process. + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: "", + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeNoName), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 2, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "", + Name: noNamespaceConfigMapName, + Namespace: "", + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDecodingErred), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 3, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "", + Name: namespacedNSName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDecodingErred), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + // Only the applied manifest is reported. + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("duplicated manifests within one work object", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // The work object stays unavailable (and thus in the slow backoff stage of the requeue rate limiter) as long + // as the duplicate is present; each of the two windows below covers at least two requeues of the work object. + duplicateNeverAppliedCheckDuration := time.Second * 15 + manifestStatusesStableCheckDuration := time.Second * 20 + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularConfigMap *corev1.ConfigMap + var duplicatedConfigMap *corev1.ConfigMap + var regularNSJSON []byte + var duplicatedConfigMapJSON []byte + + // The last transition times of the conditions on the statuses of the two copies of the duplicated manifest. + var firstCopyAppliedCondLTT metav1.Time + var firstCopyAvailableCondLTT metav1.Time + var secondCopyAppliedCondLTT metav1.Time + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Prepare a duplicate of the config map object, with different data. + duplicatedConfigMap = regularConfigMap.DeepCopy() + duplicatedConfigMap.Data = map[string]string{ + configMapDataKey: duplicatedConfigMapDataVal, + } + duplicatedConfigMapJSON = marshalK8sObjJSON(duplicatedConfigMap) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularConfigMapJSON, duplicatedConfigMapJSON) + }) + + It("should add cleanup finalizer to the work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the work object") + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should apply the first occurrence of the duplicated manifest", func() { + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the ConfigMap object has been applied from the first occurrence. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, regularConfigMap)).To(Succeed(), "Failed to retrieve the ConfigMap object") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 2, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDuplicated), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("should never apply the duplicate", func() { + configMapStableActual := configMapStableActual(memberClient1, nsName, configMapName, regularConfigMap.UID, configMap.Data) + Consistently(configMapStableActual, duplicateNeverAppliedCheckDuration, consistentlyInterval).Should(Succeed(), "The duplicate has overwritten the config map object unexpectedly") + }) + + It("should keep the statuses of both copies of the duplicated manifest stable", func() { + // Each copy of the duplicated manifest should get its own previous status back upon each requeue, so + // that the work object status is not rewritten every time. + // + // Note that the work object must not be nudged here: a spec change triggers the write-ahead process, + // which drops the status of the duplicate, and the conditions of the duplicate would then be rebuilt. + firstCopyAppliedCondLTT = retrieveManifestCondLTT(memberReservedNSName1, workName, 1, placementv1alpha1.ManifestCondTypeApplied) + firstCopyAvailableCondLTT = retrieveManifestCondLTT(memberReservedNSName1, workName, 1, placementv1alpha1.ManifestCondTypeAvailable) + secondCopyAppliedCondLTT = retrieveManifestCondLTT(memberReservedNSName1, workName, 2, placementv1alpha1.ManifestCondTypeApplied) + + firstCopyAppliedCondStableActual := manifestCondLTTActual(memberReservedNSName1, workName, 1, placementv1alpha1.ManifestCondTypeApplied, firstCopyAppliedCondLTT) + firstCopyAvailableCondStableActual := manifestCondLTTActual(memberReservedNSName1, workName, 1, placementv1alpha1.ManifestCondTypeAvailable, firstCopyAvailableCondLTT) + secondCopyAppliedCondStableActual := manifestCondLTTActual(memberReservedNSName1, workName, 2, placementv1alpha1.ManifestCondTypeApplied, secondCopyAppliedCondLTT) + Consistently(func() error { + if err := firstCopyAppliedCondStableActual(); err != nil { + return fmt.Errorf("first copy: %w", err) + } + if err := firstCopyAvailableCondStableActual(); err != nil { + return fmt.Errorf("first copy: %w", err) + } + if err := secondCopyAppliedCondStableActual(); err != nil { + return fmt.Errorf("second copy: %w", err) + } + return nil + }, manifestStatusesStableCheckDuration, consistentlyInterval).Should(Succeed(), "The manifest statuses have been rebuilt unexpectedly") + }) + + It("can update the work object to remove the first occurrence of the duplicated manifest", func() { + updateWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, duplicatedConfigMapJSON) + }) + + It("should hand the config map object over to the remaining copy without re-creating it", func() { + // The config map identifier is still present in the work object; as the work applier looks for left-over + // manifests without ordinals, the config map object should not be deleted or re-created. + configMapStableActual := configMapStableActual(memberClient1, nsName, configMapName, regularConfigMap.UID, duplicatedConfigMap.Data) + Eventually(configMapStableActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to hand the config map object over to the remaining copy") + + configMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, duplicatedConfigMap, appliedWorkOwnerRef) + Eventually(configMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the remaining copy of the config map object") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should carry over the conditions of the first copy to the remaining copy", func() { + // Upon the spec change, the write-ahead process should carry over the status of the first copy (the + // one that has been applied), rather than that of the duplicate; consequently, the Applied and + // Available conditions of the remaining copy never transition. + appliedCondLTTActual := manifestCondLTTActual(memberReservedNSName1, workName, 1, placementv1alpha1.ManifestCondTypeApplied, firstCopyAppliedCondLTT) + Expect(appliedCondLTTActual()).To(Succeed(), "The Applied condition has transitioned unexpectedly") + + availableCondLTTActual := manifestCondLTTActual(memberReservedNSName1, workName, 1, placementv1alpha1.ManifestCondTypeAvailable, firstCopyAvailableCondLTT) + Expect(availableCondLTTActual()).To(Succeed(), "The Available condition has transitioned unexpectedly") + }) + + It("should update the appliedWork object status", func() { + // The config map object is the same one as before (i.e., it has not been re-created). + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("duplicated manifests of the same group/kind but different versions within one work object", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + hpaName := "hpa-1" + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularHPA *autoscalingv2.HorizontalPodAutoscaler + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare an HPA object of the autoscaling/v2 API version. + regularHPA = &autoscalingv2.HorizontalPodAutoscaler{ + TypeMeta: metav1.TypeMeta{ + APIVersion: "autoscaling/v2", + Kind: "HorizontalPodAutoscaler", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: hpaName, + }, + Spec: autoscalingv2.HorizontalPodAutoscalerSpec{ + ScaleTargetRef: autoscalingv2.CrossVersionObjectReference{ + APIVersion: "apps/v1", + Kind: "Deployment", + Name: deployName, + }, + MinReplicas: ptr.To(int32(1)), + MaxReplicas: 3, + }, + } + regularHPAJSON := marshalK8sObjJSON(regularHPA) + + // Prepare an HPA object of the same name, but of the autoscaling/v1 API version (and with a different + // spec). Both API versions are served; as the work applier identifies manifests without their API + // versions, the two manifests are duplicates of each other. + v1HPA := &autoscalingv1.HorizontalPodAutoscaler{ + TypeMeta: metav1.TypeMeta{ + APIVersion: "autoscaling/v1", + Kind: "HorizontalPodAutoscaler", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: hpaName, + }, + Spec: autoscalingv1.HorizontalPodAutoscalerSpec{ + ScaleTargetRef: autoscalingv1.CrossVersionObjectReference{ + APIVersion: "apps/v1", + Kind: "Deployment", + Name: deployName, + }, + MinReplicas: ptr.To(int32(1)), + MaxReplicas: 5, + }, + } + v1HPAJSON := marshalK8sObjJSON(v1HPA) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularHPAJSON, v1HPAJSON) + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should apply the first occurrence of the duplicated manifest", func() { + // Retrieve the owner reference of the appliedWork object. + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the HPA object has been applied from the first occurrence (autoscaling/v2). + Eventually(func() error { + gotHPA := &autoscalingv2.HorizontalPodAutoscaler{} + if err := memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: hpaName}, gotHPA); err != nil { + return fmt.Errorf("failed to retrieve the HPA object: %w", err) + } + + wantOwnerRefs := []metav1.OwnerReference{*appliedWorkOwnerRef} + if diff := cmp.Diff(gotHPA.OwnerReferences, wantOwnerRefs); diff != "" { + return fmt.Errorf("owner references diff (-got +want):\n%s", diff) + } + if gotHPA.Spec.MaxReplicas != regularHPA.Spec.MaxReplicas { + return fmt.Errorf("HPA max replicas = %d, want %d", gotHPA.Spec.MaxReplicas, regularHPA.Spec.MaxReplicas) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the HPA object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: hpaName}, regularHPA)).To(Succeed(), "Failed to retrieve the HPA object") + }) + + It("should update the work object status", func() { + // Prepare the status information. + // + // The work object is not available because of the duplicate, even though the availability of the HPA + // object itself cannot be tracked. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "autoscaling", + APIVersion: "v2", + Kind: "HorizontalPodAutoscaler", + Resource: "horizontalpodautoscalers", + Name: hpaName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeNotTrackable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 2, + APIGroup: "autoscaling", + APIVersion: "v1", + Kind: "HorizontalPodAutoscaler", + Resource: "horizontalpodautoscalers", + Name: hpaName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDuplicated), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "autoscaling", + APIVersion: "v2", + Kind: "HorizontalPodAutoscaler", + Resource: "horizontalpodautoscalers", + Name: hpaName, + Namespace: nsName, + }, + UID: regularHPA.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularHPARemovedActual := regularHPARemovedActual(memberClient1, nsName, hpaName) + Eventually(regularHPARemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the HPA object") + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("removing all copies of a duplicated manifest from one work object", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularNSJSON []byte + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare a config map object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Prepare a duplicate of the config map object, with different data. + duplicatedConfigMap := regularConfigMap.DeepCopy() + duplicatedConfigMap.Data = map[string]string{ + configMapDataKey: duplicatedConfigMapDataVal, + } + duplicatedConfigMapJSON := marshalK8sObjJSON(duplicatedConfigMap) + + // Create a new work object with all the manifest JSONs. + createWorkObject(workName, memberReservedNSName1, nil, regularNSJSON, regularConfigMapJSON, duplicatedConfigMapJSON) + }) + + // For simplicity reasons, some steps (finalizer handling, appliedWork object creation, etc.) are omitted in + // this test node, as the behaviors have been verified elsewhere. + + It("should apply the first occurrence of the duplicated manifest", func() { + // Retrieve the owner reference of the appliedWork object. + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the ConfigMap object has been applied from the first occurrence. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + }) + + It("should report both copies of the duplicated manifest in the work object status", func() { + // This guarantees that both copies are present in the work object status before the removal. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 2, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDuplicated), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can update the work object to remove all copies of the duplicated manifest", func() { + updateWorkObject(workName, memberReservedNSName1, nil, regularNSJSON) + }) + + It("should remove the config map object", func() { + regularConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + // + // The config map object should have been removed already; this is kept as a safety net. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("duplicated manifest across the primary and a secondary work object", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and + // the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // Poll frequently when verifying the hand-over, so that a transient deletion of the config map object + // would not go unnoticed. + handOverCheckInterval := time.Millisecond * 250 + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var regularConfigMap *corev1.ConfigMap + var duplicatedConfigMap *corev1.ConfigMap + var regularNSJSON []byte + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON = marshalK8sObjJSON(regularNS) + + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Prepare a duplicate of the config map object, with different data. + duplicatedConfigMap = regularConfigMap.DeepCopy() + duplicatedConfigMap.Data = map[string]string{ + configMapDataKey: duplicatedConfigMapDataVal, + } + duplicatedConfigMapJSON := marshalK8sObjJSON(duplicatedConfigMap) + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + // + // As KubeFleet does, the primary work object is set as the controller owner of all secondary work objects, + // so that events on secondary work objects trigger reconciliations of the primary work object. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + duplicatedConfigMapJSON, + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // Create the primary work object. + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, regularConfigMapJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + It("should add cleanup finalizer to all the work objects", func() { + primaryWorkFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the primary work object") + + secondaryWorkFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the secondary work object") + }) + + It("should prepare an appliedWork object for each work object", func() { + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + + secondaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, secondaryWorkName) + Eventually(secondaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the secondary work object") + + // All the manifests across the linked work objects are owned by the appliedWork object of the + // primary work object. + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + }) + + It("should apply the manifests from the primary work object", func() { + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the ConfigMap object has been applied from the primary work object, which always wins. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, regularConfigMap)).To(Succeed(), "Failed to retrieve the ConfigMap object") + }) + + It("should update the work object statuses", func() { + // The primary work object is applied and available. + primaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, primaryWorkConds, primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + // The secondary work object reports the duplicate; its conditions reflect only its own manifests. + secondaryWorkConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDuplicated), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, secondaryWorkConds, secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("should update the appliedWork object statuses", func() { + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + // Nothing has been applied from the secondary work object. + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, nil) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + }) + + It("should never apply the duplicate from the secondary work object", func() { + configMapStableActual := configMapStableActual(memberClient1, nsName, configMapName, regularConfigMap.UID, configMap.Data) + Consistently(configMapStableActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The duplicate has overwritten the config map object unexpectedly") + }) + + It("can update the primary work object to drop the config map manifest", func() { + // The linked work count and the primary placement resource snapshot stay the same. + updateLinkedWorkObject(primaryWorkName, memberReservedNSName1, primarySnapshotName, nil, regularNSJSON) + }) + + It("should hand the config map object over to the secondary work object without re-creating it", func() { + // The config map identifier is still present in one of the linked work objects; the config map + // object should not be deleted or re-created (the data is not checked here). + configMapNotRecreatedActual := configMapStableActual(memberClient1, nsName, configMapName, regularConfigMap.UID, nil) + Consistently(configMapNotRecreatedActual, consistentlyDuration, handOverCheckInterval).Should(Succeed(), "The config map object has been deleted or re-created unexpectedly") + + configMapStableActual := configMapStableActual(memberClient1, nsName, configMapName, regularConfigMap.UID, duplicatedConfigMap.Data) + Eventually(configMapStableActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to hand the config map object over to the secondary work object") + + // The config map object should still be owned by the appliedWork object of the primary work object. + configMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, duplicatedConfigMap, appliedWorkOwnerRef) + Eventually(configMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from the secondary work object") + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the two work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkName, append([]metav1.Condition{}, workConds...), secondaryWorkManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary work object status") + }) + + It("should update the appliedWork object statuses", func() { + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + // The config map object is the same one as before (i.e., it has not been re-created). + secondaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + secondaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkName, secondaryAppliedResources) + Eventually(secondaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the secondary appliedWork object status") + }) + + AfterAll(func() { + // Delete the work objects. + // + // As there is no GC controller in the test environment (and the secondary work object is not owned by + // the primary one in this test), both work objects are deleted explicitly. + deleteWorkObject(primaryWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Ensure that the appliedWork objects have been removed. + primaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(primaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + + // The secondary appliedWork object owns no resources, and is deleted by the work applier directly. + secondaryAppliedWorkAbsentActual := appliedWorkAbsentActual(memberClient1, secondaryWorkName) + Eventually(secondaryAppliedWorkAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary appliedWork object") + + // Ensure that the work objects have been removed. + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + secondaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkName) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("duplicated manifest across two secondary work objects", Ordered, func() { + // The three work objects are linked to each other: they share the same owner placement policy/binding and + // the same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The secondary work objects are named after the primary one, so that the order of their names is known. + secondaryWorkAName := fmt.Sprintf("%s-a", primaryWorkName) + secondaryWorkBName := fmt.Sprintf("%s-b", primaryWorkName) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + configMapAOnlyName := "configmap-a-only" + // The number of reconciliations to run when verifying that the winner of the duplicate stays the same. + nudgeCount := 5 + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularNS *corev1.Namespace + var configMapA *corev1.ConfigMap + var configMapB *corev1.ConfigMap + var configMapAOnly *corev1.ConfigMap + var configMapAOnlyJSON []byte + var configMapUID types.UID + var configMapAOnlyUID types.UID + + BeforeAll(func() { + // Prepare a namespace object. + regularNS = ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare two copies of a config map object, with different data. + configMapA = configMap.DeepCopy() + configMapA.Namespace = nsName + configMapA.Name = configMapName + configMapA.Data = map[string]string{ + configMapDataKey: "a", + } + configMapAJSON := marshalK8sObjJSON(configMapA) + + configMapB = configMapA.DeepCopy() + configMapB.Data = map[string]string{ + configMapDataKey: "b", + } + configMapBJSON := marshalK8sObjJSON(configMapB) + + // Prepare a different config map object, which replaces the duplicate in secondary work object A later. + configMapAOnly = configMapA.DeepCopy() + configMapAOnly.Name = configMapAOnlyName + configMapAOnlyJSON = marshalK8sObjJSON(configMapAOnly) + + // Create the work objects in an order that does not match the order of their names: secondary work + // object B first, then secondary work object A, and the primary work object at last, so that the work + // applier can find all the linked work objects as soon as it starts processing the primary work object. + // + // As KubeFleet does, the primary work object is set as the controller owner of all secondary work objects, + // so that events on secondary work objects trigger reconciliations of the primary work object. + secondaryWorkB := buildWorkObject( + secondaryWorkBName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + configMapBJSON, + ) + setPrimaryWorkAsOwner(secondaryWorkB, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWorkB)).To(Succeed(), "Failed to create secondary work object B") + + secondaryWorkA := buildWorkObject( + secondaryWorkAName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + configMapAJSON, + ) + setPrimaryWorkAsOwner(secondaryWorkA, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWorkA)).To(Succeed(), "Failed to create secondary work object A") + + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "3" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + It("should add cleanup finalizer to all the work objects", func() { + primaryWorkFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the primary work object") + + secondaryWorkAFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, secondaryWorkAName) + Eventually(secondaryWorkAFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to secondary work object A") + + secondaryWorkBFinalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, secondaryWorkBName) + Eventually(secondaryWorkBFinalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to secondary work object B") + }) + + It("should prepare an appliedWork object for each work object", func() { + primaryAppliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, primaryWorkName) + Eventually(primaryAppliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for the primary work object") + + secondaryAppliedWorkACreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, secondaryWorkAName) + Eventually(secondaryAppliedWorkACreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for secondary work object A") + + secondaryAppliedWorkBCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, secondaryWorkBName) + Eventually(secondaryAppliedWorkBCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for secondary work object B") + + // All the manifests across the linked work objects are owned by the appliedWork object of the + // primary work object. + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, primaryWorkName) + }) + + It("should apply the duplicate from the secondary work object whose name sorts first", func() { + // Ensure that the NS object has been applied as expected. + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the namespace object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, regularNS)).To(Succeed(), "Failed to retrieve the NS object") + + // Ensure that the ConfigMap object has been applied from secondary work object A. + configMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, configMapA, appliedWorkOwnerRef) + Eventually(configMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from secondary work object A") + + gotConfigMap := &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, gotConfigMap)).To(Succeed(), "Failed to retrieve the ConfigMap object") + configMapUID = gotConfigMap.UID + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkAManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkAStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkAName, append([]metav1.Condition{}, workConds...), secondaryWorkAManifestStatuses) + Eventually(secondaryWorkAStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of secondary work object A") + + // Secondary work object B reports the duplicate. + secondaryWorkBConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + secondaryWorkBManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDuplicated), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkBStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkBName, secondaryWorkBConds, secondaryWorkBManifestStatuses) + Eventually(secondaryWorkBStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of secondary work object B") + }) + + It("should update the appliedWork object statuses", func() { + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + secondaryAAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: configMapUID, + }, + } + secondaryAppliedWorkAStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkAName, secondaryAAppliedResources) + Eventually(secondaryAppliedWorkAStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the appliedWork object for secondary work object A") + + // Nothing has been applied from secondary work object B. + secondaryAppliedWorkBStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkBName, nil) + Eventually(secondaryAppliedWorkBStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the appliedWork object for secondary work object B") + }) + + It("should keep the winner of the duplicate stable across reconciliations", func() { + // The linked work objects are listed from a cache, the order of which is random; the work applier + // sorts the secondary work objects by name, so that the same copy wins upon every reconciliation. + // + // Here the primary work object is nudged a fixed number of times; each nudge triggers an immediate + // reconciliation, which lists the linked work objects again. + configMapStableActual := configMapStableActual(memberClient1, nsName, configMapName, configMapUID, configMapA.Data) + + secondaryWorkAConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + secondaryWorkAManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkAStatusUnchangedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkAName, secondaryWorkAConds, secondaryWorkAManifestStatuses) + + secondaryWorkBConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + secondaryWorkBManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeDuplicated), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkBStatusUnchangedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkBName, secondaryWorkBConds, secondaryWorkBManifestStatuses) + + for i := 0; i < nudgeCount; i++ { + nudgeWork(primaryWorkName, memberReservedNSName1) + + primaryWorkProcessedActual := workProcessedAtLatestGenerationActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkProcessedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to process the primary work object (nudge #%d)", i) + + Expect(configMapStableActual()).To(Succeed(), "The winner of the duplicate has changed (nudge #%d)", i) + Expect(secondaryWorkAStatusUnchangedActual()).To(Succeed(), "The status of secondary work object A has changed (nudge #%d)", i) + Expect(secondaryWorkBStatusUnchangedActual()).To(Succeed(), "The status of secondary work object B has changed (nudge #%d)", i) + } + }) + + It("can update secondary work object A to drop the duplicate", func() { + // The spec change on secondary work object A triggers a reconciliation of the primary work object via + // the owner reference. + updateLinkedWorkObject(secondaryWorkAName, memberReservedNSName1, primarySnapshotName, nil, configMapAOnlyJSON) + }) + + It("should hand the config map object over to secondary work object B without re-creating it", func() { + configMapStableActual := configMapStableActual(memberClient1, nsName, configMapName, configMapUID, configMapB.Data) + Eventually(configMapStableActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to hand the config map object over to secondary work object B") + + // The config map object should still be owned by the appliedWork object of the primary work object. + configMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, configMapB, appliedWorkOwnerRef) + Eventually(configMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object from secondary work object B") + }) + + It("should apply the new manifest from secondary work object A", func() { + configMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, configMapAOnly, appliedWorkOwnerRef) + Eventually(configMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the new config map object from secondary work object A") + + gotConfigMap := &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapAOnlyName}, gotConfigMap)).To(Succeed(), "Failed to retrieve the new ConfigMap object") + configMapAOnlyUID = gotConfigMap.UID + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + + primaryWorkManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + // Note that workStatusUpdated sets the observed generations on the given conditions; use a copy here + // as the conditions are shared between the work objects. + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, primaryWorkName, append([]metav1.Condition{}, workConds...), primaryWorkManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary work object status") + + secondaryWorkAManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapAOnlyName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkAStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkAName, append([]metav1.Condition{}, workConds...), secondaryWorkAManifestStatuses) + Eventually(secondaryWorkAStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of secondary work object A") + + secondaryWorkBManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryWorkBStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, secondaryWorkBName, append([]metav1.Condition{}, workConds...), secondaryWorkBManifestStatuses) + Eventually(secondaryWorkBStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of secondary work object B") + }) + + It("should update the appliedWork object statuses", func() { + primaryAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: regularNS.UID, + }, + } + primaryAppliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, primaryWorkName, primaryAppliedResources) + Eventually(primaryAppliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary appliedWork object status") + + secondaryAAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapAOnlyName, + Namespace: nsName, + }, + UID: configMapAOnlyUID, + }, + } + secondaryAppliedWorkAStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkAName, secondaryAAppliedResources) + Eventually(secondaryAppliedWorkAStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the appliedWork object for secondary work object A") + + // The config map object is the same one as before (i.e., it has not been re-created). + secondaryBAppliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: configMapUID, + }, + } + secondaryAppliedWorkBStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, secondaryWorkBName, secondaryBAppliedResources) + Eventually(secondaryAppliedWorkBStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the appliedWork object for secondary work object B") + }) + + AfterAll(func() { + // Delete the work objects. + // + // As there is no GC controller in the test environment (and the secondary work objects are not owned by + // the primary one in this test), all work objects are deleted explicitly. + deleteWorkObject(primaryWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkAName, memberReservedNSName1) + deleteWorkObject(secondaryWorkBName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + configMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(configMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + configMapAOnlyRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapAOnlyName) + Eventually(configMapAOnlyRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new config map object") + + // Ensure that the appliedWork objects have been removed. + primaryAppliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, primaryWorkName) + Eventually(primaryAppliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary appliedWork object") + + // The appliedWork objects for the secondary work objects own no resources, and are deleted by the work + // applier directly. + secondaryAppliedWorkAAbsentActual := appliedWorkAbsentActual(memberClient1, secondaryWorkAName) + Eventually(secondaryAppliedWorkAAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object for secondary work object A") + + secondaryAppliedWorkBAbsentActual := appliedWorkAbsentActual(memberClient1, secondaryWorkBName) + Eventually(secondaryAppliedWorkBAbsentActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object for secondary work object B") + + // Ensure that the work objects have been removed. + primaryWorkRemovedActual := workRemovedActual(memberReservedNSName1, primaryWorkName) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + secondaryWorkARemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkAName) + Eventually(secondaryWorkARemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove secondary work object A") + + secondaryWorkBRemovedActual := workRemovedActual(memberReservedNSName1, secondaryWorkBName) + Eventually(secondaryWorkBRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove secondary work object B") + + // The environment prepared by the envtest package does not support namespace + // deletion; consequently this test suite would not attempt to verify its deletion. + }) + }) + + Context("namespace absent with namespace auto-creation disabled", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The namespace is not created by the work applier, nor by the test case at first. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Prepare a config map object. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Forbid the work applier from creating namespaces; the API server fills in the defaults for the + // other fields in the sync strategy. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenNamespaceDoesNotExist: placementv1alpha1.WhenNamespaceDoesNotExistOptionReportError, + } + + // Create a new work object with the manifest JSON. + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should add cleanup finalizer to the work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the work object") + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should not apply the manifest or create the namespace", func() { + // Keep the window short; the namespace is created outside the work applier later, which can only be + // picked up at the next requeue of the work object. + regularConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, nsName, configMapName) + namespaceAbsentActual := namespaceAbsentActual(memberClient1, nsName) + Consistently(func() error { + if err := regularConfigMapAbsentActual(); err != nil { + return err + } + return namespaceAbsentActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The config map object or the namespace has been created unexpectedly") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToApply), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should report the missing namespace as the cause of the failure", func() { + manifestAppliedCondMessageActual := manifestAppliedCondMessageActual(memberReservedNSName1, workName, 0, fmt.Sprintf("namespaces %q not found", nsName)) + Eventually(manifestAppliedCondMessageActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the missing namespace in the work object status") + }) + + It("should not report any applied resource in the appliedWork object status", func() { + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, nil) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("can create the namespace outside the work applier", func() { + regularNS := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: nsName, + }, + } + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + }) + + It("should apply the manifest", func() { + // Creating the namespace triggers no event for the work object; the manifest is applied at the next + // requeue of the work object. The work object is unavailable and stays in the slow backoff stage at + // this moment, where requeues are at most 15 seconds apart; allow more time for the next attempt to + // happen. + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, regularConfigMap)).To(Succeed(), "Failed to retrieve the ConfigMap object") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: regularConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("should leave the namespace unmanaged", func() { + // The work applier must not adopt an object that it has never placed. + namespaceNotManagedActual := namespaceNotManagedActual(memberClient1, nsName) + Consistently(namespaceNotManagedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The namespace has been adopted by the work applier unexpectedly") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the applied manifests manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Ensure that the appliedWork object has been removed. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/join_leave.go b/pkg/v1/controllers/workapplier/join_leave.go new file mode 100644 index 000000000..327558ddd --- /dev/null +++ b/pkg/v1/controllers/workapplier/join_leave.go @@ -0,0 +1,112 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +// Join sets the work applier to work. +// +// As a member cluster side controller, work applier can only start reconciling work objects after the +// member cluster has successfully joined the fleet. +func (r *Reconciler) Join(_ context.Context) error { + if !r.ready.Load() { + klog.InfoS("Setting the work applier as ready") + } + r.ready.Store(true) + return nil +} + +// Leave performs cleanup ops when a member cluster leaves the fleet. +// +// For availability reasons, when a member cluster leaves the fleet, all the resources are left intact on the member +// cluster. +func (r *Reconciler) Leave(ctx context.Context) error { + if r.ready.Load() { + klog.InfoS("Setting the work applier as not ready") + } + r.ready.Store(false) + + // List all the work objects. + workList := &placementv1alpha1.WorkList{} + if err := r.hubUncachedReader.List(ctx, workList, client.InNamespace(r.workNSName)); err != nil { + return errors.NewAPIServerError(err, "failed to list the work objects", false) + } + + // For each work object, if it has the cleanup finalizer, find the corresponding appliedWork object, and + // delete it with the Orphan propagation policy. + // + // It is OK if the appliedWork object hasn't been created yet. + deletePolicy := metav1.DeletePropagationOrphan + for idx := range workList.Items { + work := &workList.Items[idx] + if !controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + continue + } + + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{Name: work.Name}, + } + if err := r.spokeClient.Delete(ctx, appliedWork, &client.DeleteOptions{PropagationPolicy: &deletePolicy}); err != nil { + if apierrors.IsNotFound(err) { + klog.V(2).InfoS("The appliedWork object does not exist; skipping deletion", + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + continue + } + return errors.NewAPIServerError(err, "failed to delete the appliedWork object", false, + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + } + klog.V(2).InfoS("Deleted the appliedWork object with the Orphan propagation policy", + "appliedWork", klog.KObj(appliedWork), "work", klog.KObj(work)) + } + + // Set the requeue rate limiters to forget about all the work. Secondary work objects are never tracked, and + // forgetting them becomes a no-op. + for idx := range workList.Items { + work := &workList.Items[idx] + r.cleanupRequeueRateLimiter.Forget(types.NamespacedName{Namespace: r.workNSName, Name: work.Name}) + r.periodicRequeueRateLimiter.Forget(work) + } + + // Drop the cleanup finalizer from all the work objects. + for idx := range workList.Items { + work := &workList.Items[idx] + if !controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + continue + } + + controllerutil.RemoveFinalizer(work, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, work); err != nil { + return errors.NewAPIServerError(err, "failed to remove the cleanup finalizer from the work", false, + "work", klog.KObj(work)) + } + } + + klog.V(2).InfoS("Removed the cleanup finalizer from all the work objects", "workCount", len(workList.Items)) + return nil +} diff --git a/pkg/v1/controllers/workapplier/lifecycle_integration_test.go b/pkg/v1/controllers/workapplier/lifecycle_integration_test.go new file mode 100644 index 000000000..2e3afa94d --- /dev/null +++ b/pkg/v1/controllers/workapplier/lifecycle_integration_test.go @@ -0,0 +1,896 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("lifecycle", func() { + Context("owner namespaces of placement bindings (same placement binding name)", Ordered, func() { + // Work A is the only work object of a cluster-scoped placement binding; work B (primary) and work B2 + // (secondary) are the work objects of a namespace-scoped placement binding of the same name. All the work + // objects are linked to a primary placement resource snapshot of the same name, so that the linked work + // counts would not match should the work applier mix the two placement bindings up. + workAName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + workBName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + workB2Name := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, utils.RandStr()) + // The namespace of the namespace-scoped placement policy/binding; it only appears in the labels. + ownerNSName := fmt.Sprintf("owner-ns-%s", utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + configMapAName := "configmap-a" + configMapBName := "configmap-b" + configMapB2Name := "configmap-b2" + + var appliedWorkOwnerRefA *metav1.OwnerReference + var appliedWorkOwnerRefB *metav1.OwnerReference + // The appliedWork object of work B2 before work A is deleted. + var appliedWorkB2 *placementv1alpha1.AppliedWork + + // Each work object features one config map, which is applied and available. + singleConfigMapStatus := func(configMapName string) []placementv1alpha1.PerManifestStatus { + return []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + } + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work objects do not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + configMapJSONs := map[string][]byte{} + for _, name := range []string{configMapAName, configMapBName, configMapB2Name} { + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = name + configMapJSONs[name] = marshalK8sObjJSON(regularConfigMap) + } + + // Create the work objects of the namespace-scoped placement binding; the secondary work object goes + // first, so that the work applier can find all the linked work objects as soon as it starts processing + // the primary work object. + workB2 := buildWorkObject( + workB2Name, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + configMapJSONs[configMapB2Name], + ) + workB2.Labels[placementv1alpha1.WorkOwnerNamespaceLabelKey] = ownerNSName + setPrimaryWorkAsOwner(workB2, workBName) + Expect(hubClient.Create(ctx, workB2)).To(Succeed(), "Failed to create work B2") + + workB := buildWorkObject( + workBName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + configMapJSONs[configMapBName], + ) + workB.Labels[placementv1alpha1.WorkOwnerNamespaceLabelKey] = ownerNSName + workB.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, workB)).To(Succeed(), "Failed to create work B") + + // Create the work object of the cluster-scoped placement binding (an empty owner namespace). + workA := buildWorkObject( + workAName, memberReservedNSName1, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + configMapJSONs[configMapAName], + ) + workA.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = singleWorkLinkedWorkCountAnno + Expect(hubClient.Create(ctx, workA)).To(Succeed(), "Failed to create work A") + }) + + It("should process the work objects of each placement binding on their own", func() { + for _, workName := range []string{workAName, workBName, workB2Name} { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to work %s", workName) + + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for work %s", workName) + } + appliedWorkOwnerRefA = prepareAppliedWorkOwnerRef(memberClient1, workAName) + appliedWorkOwnerRefB = prepareAppliedWorkOwnerRef(memberClient1, workBName) + appliedWorkB2 = &placementv1alpha1.AppliedWork{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: workB2Name}, appliedWorkB2)).To(Succeed(), "Failed to retrieve the appliedWork object of work B2") + + // The config map from work A is owned by the appliedWork object of work A; the config maps from work B + // and work B2 are both owned by the appliedWork object of work B (the primary work object). + configMapAAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapAName, appliedWorkOwnerRefA) + Eventually(configMapAAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map from work A") + + for _, name := range []string{configMapBName, configMapB2Name} { + configMapAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, name, appliedWorkOwnerRefB) + Eventually(configMapAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map %s", name) + } + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + for workName, configMapName := range map[string]string{ + workAName: configMapAName, + workBName: configMapBName, + workB2Name: configMapB2Name, + } { + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, singleConfigMapStatus(configMapName)) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of work %s", workName) + } + }) + + It("can delete work A", func() { + deleteWorkObject(workAName, memberReservedNSName1) + }) + + It("should remove work A", func() { + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workAName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object of work A") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workAName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work A") + }) + + It("should leave the work objects of the namespace-scoped placement binding alone", func() { + // The cleanup of work A concerns only the work objects of the cluster-scoped placement binding; the + // secondary work object of the namespace-scoped placement binding keeps its cleanup finalizer and + // appliedWork object, and the applied config maps stay with the appliedWork object of work B. + // + // Had the cleanup touched work B2, the next periodic reconciliation of work B would add the finalizer + // back and re-create the appliedWork object; the appliedWork object is hence checked by its UID. + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workB2Name) + appliedWorkNotRecreatedActual := objectNotRecreatedActual(memberClient1, appliedWorkB2) + configMapBAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapBName, appliedWorkOwnerRefB) + configMapB2AppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapB2Name, appliedWorkOwnerRefB) + Consistently(func() error { + if err := finalizerAddedActual(); err != nil { + return err + } + if err := appliedWorkNotRecreatedActual(); err != nil { + return err + } + if err := configMapBAppliedActual(); err != nil { + return err + } + return configMapB2AppliedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects of the namespace-scoped placement binding have been changed") + }) + + AfterAll(func() { + // Delete all the work objects (work A might still exist if the test case fails before it is deleted); + // there is no GC controller in the hub cluster test environment to delete secondary work objects along + // with their primary work objects. + for _, workName := range []string{workAName, workBName, workB2Name} { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete work %s", workName) + } + + // Remove the config maps manually, as there is no GC controller in the test environment. + for _, name := range []string{configMapAName, configMapBName, configMapB2Name} { + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, name) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map %s", name) + } + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork objects. + for _, workName := range []string{workAName, workBName, workB2Name} { + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object of work %s", workName) + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work %s", workName) + } + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("leaving and re-joining the fleet", Ordered, func() { + // The test case uses the work applier dedicated to the lifecycle test cases (which watches its own reserved + // namespace), so that leaving the fleet never affects other test cases; no Serial decorator is needed. + // + // Work A uses the default sync strategy, while work B always takes over pre-existing objects. + workAName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + workBName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + configMapAName := "configmap-a" + configMapBName := "configmap-b" + + var configMapA *corev1.ConfigMap + var updatedConfigMapA *corev1.ConfigMap + var updatedConfigMapB *corev1.ConfigMap + var syncStrategyB *placementv1alpha1.SyncStrategy + // The config maps as applied in the member cluster before the member cluster leaves the fleet. + var appliedConfigMapA *corev1.ConfigMap + var appliedConfigMapB *corev1.ConfigMap + // The appliedWork owner references before the member cluster leaves the fleet. + var oldAppliedWorkOwnerRefA *metav1.OwnerReference + var oldAppliedWorkOwnerRefB *metav1.OwnerReference + + BeforeAll(func() { + // Re-join the fleet when the test case completes (or fails), so that the dedicated work applier keeps + // processing work objects for the lifecycle test cases that run afterwards on the same Ginkgo process. + DeferCleanup(func() { + Expect(workApplierLifecycle.Join(ctx)).To(Succeed(), "Failed to re-join the fleet") + }) + + // Create the namespace directly in the member cluster, as the work objects do not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests, and their updated versions. + configMapA = configMap.DeepCopy() + configMapA.Namespace = nsName + configMapA.Name = configMapAName + updatedConfigMapA = configMapA.DeepCopy() + updatedConfigMapA.Data = map[string]string{ + configMapDataKey: "updated", + } + + configMapB := configMap.DeepCopy() + configMapB.Namespace = nsName + configMapB.Name = configMapBName + updatedConfigMapB = configMapB.DeepCopy() + updatedConfigMapB.Data = map[string]string{ + configMapDataKey: "updated", + } + + createWorkObject(workAName, memberReservedNSNameLifecycle, nil, marshalK8sObjJSON(configMapA)) + + syncStrategyB = &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + } + createWorkObject(workBName, memberReservedNSNameLifecycle, syncStrategyB, marshalK8sObjJSON(configMapB)) + }) + + It("should apply the manifests", func() { + for _, workName := range []string{workAName, workBName} { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSNameLifecycle, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for work %s", workName) + } + oldAppliedWorkOwnerRefA = prepareAppliedWorkOwnerRef(memberClient1, workAName) + oldAppliedWorkOwnerRefB = prepareAppliedWorkOwnerRef(memberClient1, workBName) + + configMapAAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapAName, oldAppliedWorkOwnerRefA) + Eventually(configMapAAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map from work A") + configMapBAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapBName, oldAppliedWorkOwnerRefB) + Eventually(configMapBAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map from work B") + + appliedConfigMapA = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapAName}, appliedConfigMapA)).To(Succeed(), "Failed to retrieve the config map from work A") + appliedConfigMapB = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapBName}, appliedConfigMapB)).To(Succeed(), "Failed to retrieve the config map from work B") + }) + + It("can leave the fleet", func() { + // Retry on errors, as the work applier might update the work objects at the same time. + Eventually(func() error { + return workApplierLifecycle.Leave(ctx) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to leave the fleet") + }) + + It("can leave the fleet again", func() { + Expect(workApplierLifecycle.Leave(ctx)).To(Succeed(), "Failed to leave the fleet again") + }) + + It("should remove the cleanup finalizer from the work objects", func() { + for _, workName := range []string{workAName, workBName} { + finalizerRemovedActual := workFinalizerRemovedActual(memberReservedNSNameLifecycle, workName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer from work %s", workName) + } + }) + + It("should delete the appliedWork objects with the orphan propagation policy", func() { + for _, workName := range []string{workAName, workBName} { + appliedWorkBeingDeletedWithOrphanActual := appliedWorkBeingDeletedWithOrphanActual(memberClient1, workName) + Eventually(appliedWorkBeingDeletedWithOrphanActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to delete the appliedWork object of work %s with the orphan propagation policy", workName) + } + }) + + It("can update the work objects", func() { + updateWorkObject(workAName, memberReservedNSNameLifecycle, nil, marshalK8sObjJSON(updatedConfigMapA)) + updateWorkObject(workBName, memberReservedNSNameLifecycle, syncStrategyB, marshalK8sObjJSON(updatedConfigMapB)) + }) + + It("should not process the work objects while away from the fleet", func() { + finalizerARemovedActual := workFinalizerRemovedActual(memberReservedNSNameLifecycle, workAName) + finalizerBRemovedActual := workFinalizerRemovedActual(memberReservedNSNameLifecycle, workBName) + configMapAStableActual := configMapStableActual(memberClient1, nsName, configMapAName, appliedConfigMapA.UID, appliedConfigMapA.Data) + configMapBStableActual := configMapStableActual(memberClient1, nsName, configMapBName, appliedConfigMapB.UID, appliedConfigMapB.Data) + Consistently(func() error { + for _, actual := range []func() error{finalizerARemovedActual, finalizerBRemovedActual, configMapAStableActual, configMapBStableActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been processed while away from the fleet") + }) + + It("can simulate the GC controller", func() { + // The GC controller removes the owner references to the appliedWork objects from the orphaned objects, + // and then removes the orphan finalizer from the appliedWork objects. + for _, name := range []string{configMapAName, configMapBName} { + updateConfigMapInMemberCluster(memberClient1, nsName, name, func(configMap *corev1.ConfigMap) { + configMap.OwnerReferences = nil + }) + } + for _, workName := range []string{workAName, workBName} { + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object of work %s", workName) + } + }) + + It("can re-join the fleet", func() { + Expect(workApplierLifecycle.Join(ctx)).To(Succeed(), "Failed to re-join the fleet") + }) + + It("should add the cleanup finalizer back and prepare new appliedWork objects", func() { + for _, workName := range []string{workAName, workBName} { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSNameLifecycle, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cleanup finalizer back to work %s", workName) + + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSNameLifecycle, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare a new appliedWork object for work %s", workName) + } + + Expect(prepareAppliedWorkOwnerRef(memberClient1, workAName).UID).NotTo(Equal(oldAppliedWorkOwnerRefA.UID), "The appliedWork object of work A has not been re-created") + Expect(prepareAppliedWorkOwnerRef(memberClient1, workBName).UID).NotTo(Equal(oldAppliedWorkOwnerRefB.UID), "The appliedWork object of work B has not been re-created") + }) + + It("should not take over the orphaned object with the default sync strategy (work A)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapAName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeNotTakenOver), + ObservedGeneration: 0, + }, + }, + }, + } + workStatusUpdatedActual := workStatusUpdated(memberReservedNSNameLifecycle, workAName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of work A") + + // The config map keeps the data from the original manifest, and has no owner. + wantConfigMap := configMapA.DeepCopy() + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Consistently(configMapObjectActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The orphaned config map from work A has been changed") + }) + + It("should take over the orphaned object and apply the updated manifest (work B)", func() { + newAppliedWorkOwnerRefB := prepareAppliedWorkOwnerRef(memberClient1, workBName) + wantConfigMap := updatedConfigMapB.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*newAppliedWorkOwnerRefB} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the orphaned config map from work B") + + Expect(objectNotRecreatedActual(memberClient1, appliedConfigMapB)()).To(Succeed(), "The config map from work B has been re-created instead of being taken over") + }) + + It("should update the status of work B", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapBName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + workStatusUpdatedActual := workStatusUpdated(memberReservedNSNameLifecycle, workBName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of work B") + }) + + AfterAll(func() { + // Make sure that the work applier has re-joined the fleet before cleaning up (in case the test case fails + // while away from the fleet); the cleanup relies on the work applier. + Expect(workApplierLifecycle.Join(ctx)).To(Succeed(), "Failed to re-join the fleet") + + for _, workName := range []string{workAName, workBName} { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSNameLifecycle, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete work %s", workName) + } + + // Remove the config maps manually, as there is no GC controller in the test environment. + for _, name := range []string{configMapAName, configMapBName} { + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, name) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map %s", name) + } + + // Simulate the GC controller by removing the orphan/foreground deletion finalizers from the appliedWork + // objects. + for _, workName := range []string{workAName, workBName} { + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object of work %s", workName) + + workRemovedActual := workRemovedActual(memberReservedNSNameLifecycle, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work %s", workName) + } + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("work objects that are not primary work objects", Ordered, func() { + // Neither work object features the linked work count annotation, i.e., neither of them is a primary work + // object. The standalone work object has no owner; the other work object is owned by a primary work object + // that does not exist, as a secondary work object would be. + standaloneWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + nonExistentPrimaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The namespace is never created; were the work objects processed, the work applier would create it. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + standaloneConfigMapName := "configmap-standalone" + secondaryConfigMapName := "configmap-secondary" + + BeforeAll(func() { + // Prepare the manifests. + standaloneConfigMap := configMap.DeepCopy() + standaloneConfigMap.Namespace = nsName + standaloneConfigMap.Name = standaloneConfigMapName + + secondaryConfigMap := configMap.DeepCopy() + secondaryConfigMap.Namespace = nsName + secondaryConfigMap.Name = secondaryConfigMapName + + standaloneWork := buildWorkObject( + standaloneWorkName, memberReservedNSName1, + fmt.Sprintf(placementPolicyNameTemplate, standaloneWorkName), + fmt.Sprintf(placementBindingNameTemplate, standaloneWorkName), + fmt.Sprintf(primarySnapshotNameTemplate, standaloneWorkName), + nil, + marshalK8sObjJSON(standaloneConfigMap), + ) + Expect(hubClient.Create(ctx, standaloneWork)).To(Succeed(), "Failed to create the standalone work object") + + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSName1, + fmt.Sprintf(placementPolicyNameTemplate, nonExistentPrimaryWorkName), + fmt.Sprintf(placementBindingNameTemplate, nonExistentPrimaryWorkName), + fmt.Sprintf(primarySnapshotNameTemplate, nonExistentPrimaryWorkName), + nil, + marshalK8sObjJSON(secondaryConfigMap), + ) + setPrimaryWorkAsOwner(secondaryWork, nonExistentPrimaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + }) + + It("should not process the work objects", func() { + standaloneWorkNotProcessedActual := workNotProcessedActual(memberClient1, memberReservedNSName1, standaloneWorkName) + secondaryWorkNotProcessedActual := workNotProcessedActual(memberClient1, memberReservedNSName1, secondaryWorkName) + standaloneConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, nsName, standaloneConfigMapName) + secondaryConfigMapAbsentActual := regularConfigMapAbsentActual(memberClient1, nsName, secondaryConfigMapName) + namespaceAbsentActual := namespaceAbsentActual(memberClient1, nsName) + Consistently(func() error { + for _, actual := range []func() error{ + standaloneWorkNotProcessedActual, secondaryWorkNotProcessedActual, + standaloneConfigMapAbsentActual, secondaryConfigMapAbsentActual, + namespaceAbsentActual, + } { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been processed") + }) + + It("can delete the work objects", func() { + deleteWorkObject(standaloneWorkName, memberReservedNSName1) + deleteWorkObject(secondaryWorkName, memberReservedNSName1) + }) + + It("should remove the work objects", func() { + // The work objects have no cleanup finalizer; they are removed right away. + for _, workName := range []string{standaloneWorkName, secondaryWorkName} { + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work %s", workName) + } + }) + + AfterAll(func() { + // The work objects might still exist if the test case fails before they are deleted. + for _, workName := range []string{standaloneWorkName, secondaryWorkName} { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete work %s", workName) + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work %s", workName) + } + }) + }) + + Context("malformed primary work objects", Ordered, func() { + // Each work object features the linked work count annotation (i.e., it is a primary work object), but misses + // the information that the work applier needs to find its linked work objects. + noBindingLabelWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + noSnapshotAnnotationWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + invalidLinkedWorkCountWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + workNames := []string{noBindingLabelWorkName, noSnapshotAnnotationWorkName, invalidLinkedWorkCountWorkName} + // The namespace is never created; were the work objects processed, the work applier would create it. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + // Each work object features a config map of its own. + configMapNames := map[string]string{ + noBindingLabelWorkName: "configmap-no-binding-label", + noSnapshotAnnotationWorkName: "configmap-no-snapshot-annotation", + invalidLinkedWorkCountWorkName: "configmap-invalid-linked-work-count", + } + + BeforeAll(func() { + works := make(map[string]*placementv1alpha1.Work, len(workNames)) + for _, workName := range workNames { + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapNames[workName] + + work := buildWorkObject( + workName, memberReservedNSName1, + fmt.Sprintf(placementPolicyNameTemplate, workName), + fmt.Sprintf(placementBindingNameTemplate, workName), + fmt.Sprintf(primarySnapshotNameTemplate, workName), + nil, + marshalK8sObjJSON(regularConfigMap), + ) + work.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = singleWorkLinkedWorkCountAnno + works[workName] = work + } + + delete(works[noBindingLabelWorkName].Labels, placementv1alpha1.WorkOwnedByPlacementBindingLabelKey) + delete(works[noSnapshotAnnotationWorkName].Annotations, placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey) + works[invalidLinkedWorkCountWorkName].Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "one" + + for _, workName := range workNames { + Expect(hubClient.Create(ctx, works[workName])).To(Succeed(), "Failed to create work %s", workName) + } + }) + + It("should not process the work objects", func() { + actuals := []func() error{namespaceAbsentActual(memberClient1, nsName)} + for _, workName := range workNames { + actuals = append(actuals, + workNotProcessedActual(memberClient1, memberReservedNSName1, workName), + regularConfigMapAbsentActual(memberClient1, nsName, configMapNames[workName]), + ) + } + Consistently(func() error { + for _, actual := range actuals { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The malformed work objects have been processed") + }) + + It("can delete the work objects", func() { + for _, workName := range workNames { + deleteWorkObject(workName, memberReservedNSName1) + } + }) + + It("should remove the work objects", func() { + // The work objects have no cleanup finalizer; they are removed right away. + for _, workName := range workNames { + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work %s", workName) + } + }) + + AfterAll(func() { + // The work objects might still exist if the test case fails before they are deleted. + for _, workName := range workNames { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete work %s", workName) + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work %s", workName) + } + }) + }) + + Context("appliedWork objects deleted by an external actor", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + // The appliedWork owner reference and the config map before the appliedWork object is deleted. + var oldAppliedWorkOwnerRef *metav1.OwnerReference + var oldConfigMap *corev1.ConfigMap + // The appliedWork owner reference and the config map after the work applier re-creates them. + var newAppliedWorkOwnerRef *metav1.OwnerReference + var newConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + createWorkObject(workName, memberReservedNSName1, nil, regularConfigMapJSON) + }) + + It("should apply the manifest", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + oldAppliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, oldAppliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map") + + oldConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, oldConfigMap)).To(Succeed(), "Failed to retrieve the config map") + }) + + It("can delete the appliedWork object, as an external actor would", func() { + // With the default (background) propagation policy, the appliedWork object is deleted right away. + appliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + }, + } + Expect(memberClient1.Delete(ctx, appliedWork)).To(Succeed(), "Failed to delete the appliedWork object") + }) + + It("can simulate the GC controller by deleting the dependents of the appliedWork object", func() { + // The work applier might re-create the appliedWork object before the config map is deleted, and report + // the config map, which is still owned by the deleted appliedWork object, as not taken over; the config + // map is re-created at the next requeue of the work object anyway. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map") + }) + + It("should re-create the appliedWork object", func() { + // The deletion triggers no event for the work object; the appliedWork object is re-created at the next + // requeue of the work object. + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the appliedWork object") + + newAppliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + Expect(newAppliedWorkOwnerRef.UID).NotTo(Equal(oldAppliedWorkOwnerRef.UID), "The appliedWork object has not been re-created") + }) + + It("should re-create the config map with the new appliedWork object as its owner", func() { + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, newAppliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the config map") + + newConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, newConfigMap)).To(Succeed(), "Failed to retrieve the config map") + Expect(newConfigMap.UID).NotTo(Equal(oldConfigMap.UID), "The config map has not been re-created") + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the status of the new appliedWork object", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: newConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the new appliedWork object") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/preprocess.go b/pkg/v1/controllers/workapplier/preprocess.go new file mode 100644 index 000000000..90f35f0ac --- /dev/null +++ b/pkg/v1/controllers/workapplier/preprocess.go @@ -0,0 +1,599 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + "sync/atomic" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + utilerrors "k8s.io/apimachinery/pkg/util/errors" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/ownerreferences" +) + +func prepareWorkObjectAndManifestProcessingStates(works []*placementv1alpha1.Work, + appliedWorks []*placementv1alpha1.AppliedWork) ([]*workObjectProcessingState, []*manifestProcessingState) { + // Pre-allocate the slices. + workObjectStates := make([]*workObjectProcessingState, 0, len(works)) + manifestProcessingStates := make([]*manifestProcessingState, 0, len(works[0].Spec.Manifests)) + + primaryWork := works[0] + primaryAppliedWorkOwnerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "AppliedWork", + Name: appliedWorks[0].GetName(), + UID: appliedWorks[0].GetUID(), + BlockOwnerDeletion: ptr.To(true), + } + + for i := range works { + work := works[i] + + perWorkManifestProcessingStates := make([]*manifestProcessingState, 0, len(work.Spec.Manifests)) + for j := range work.Spec.Manifests { + manifest := &work.Spec.Manifests[j] + manifestProcessingState := &manifestProcessingState{ + manifest: manifest, + fromWorkObj: work, + fromPrimaryWorkObject: primaryWork, + // Use the appliedWork object that corresponds to the primary work object as the owner for + // all manifests in this work object once they are applied. + ownedBy: primaryAppliedWorkOwnerRef, + } + perWorkManifestProcessingStates = append(perWorkManifestProcessingStates, manifestProcessingState) + manifestProcessingStates = append(manifestProcessingStates, manifestProcessingState) + } + workObjectState := &workObjectProcessingState{ + work: work, + appliedWork: appliedWorks[i], + appliedWorkOwnerRef: primaryAppliedWorkOwnerRef, + manifestProcessingStates: perWorkManifestProcessingStates, + } + workObjectStates = append(workObjectStates, workObjectState) + } + + return workObjectStates, manifestProcessingStates +} + +func (r *Reconciler) preProcessWorkObjects(ctx context.Context, workObjProcessingStates []*workObjectProcessingState) (sets.Set[string], error) { + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + + // Pre-process the manifests one work object at a time, so that the ordinals assigned to the + // manifest identifiers stay relative to their own work object. + if err := r.preProcessPerWorkObjManifests(ctx, workObjProcessingState.manifestProcessingStates); err != nil { + return nil, kferrors.Wraps(err, "failed to pre-process work object", "work", klog.KObj(workObjProcessingState.work)) + } + + klog.V(2).InfoS("Pre-processed a work object", + "work", klog.KObj(workObjProcessingState.work), + "manifestCount", len(workObjProcessingState.manifestProcessingStates)) + } + + seenManifestIDs := markDuplicatedManifests(workObjProcessingStates) + + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + + // Write-ahead the manifest processing attempts in the work object status so that + // KubeFleet can always track applied manifests, even upon untimely crashes. This method will + // also check for any leftover apply attempts from previous runs and clean them up (if the + // corresponding manifest object has been applied). + if err := r.writeAheadPerWorkObjManifestProcessingAttempts(ctx, workObjProcessingState, seenManifestIDs); err != nil { + return nil, kferrors.Wraps(err, "failed to write ahead manifest processing states", "work", klog.KObj(workObjProcessingState.work)) + } + + klog.V(2).InfoS("Wrote ahead manifest processing states for a work object", + "work", klog.KObj(workObjProcessingState.work)) + } + return seenManifestIDs, nil +} + +func (r *Reconciler) preProcessPerWorkObjManifests(ctx context.Context, manifestProcessingStates []*manifestProcessingState) error { + childCtx, childCancel := context.WithCancel(ctx) + defer childCancel() + + var innerErr atomic.Value + preProcessOneManifest := func(pieces int) { + // At this moment the processing state is just initialized. + processingState := manifestProcessingStates[pieces] + ownerWorkObj := processingState.fromWorkObj + + gvr, manifestObj, err := r.decodeManifest(processingState.manifest) + // There exists a corner case where the decoding might have failed due to transient errors (e.g., the + // API discovery endpoint is not responding); in this case, if we register the error as a user-end + // decoding error and skip the manifest, and if the manifest has been applied before, the error might + // trigger the later left-over cleanup step, which would delete the previously applied manifest. It should + // be corrected in the subsequent reconciliation; however, it might trigger fluctuations. To be on the safer + // side, the work applier is set to requeue upon transient decoding errors. + // + // Note that with aggregated API discovery enabled by default in most Kubernetes clusters now, + // normally if the first decoding attempt succeeds, all subsequent decoding attempts will be resolved via + // cache. + if err != nil && kferrors.Category(err) == kferrors.ErrCategoryTransient { + wrappedErr := kferrors.Wraps(err, "failed to decode the manifest", "ordinal", pieces) + innerErr.Store(wrappedErr) + childCancel() + return + } + // Build the manifest identifier. Note that this would return an identifier even if the decoding fails. + processingState.id = buildManifestIdentifier(pieces, gvr, manifestObj) + if err != nil { + wrappedErr := kferrors.Wraps(err, "failed to decode the manifest", "ordinal", pieces) + processingState.applyErr = wrappedErr + processingState.applyRes = ApplyResTypeDecodingErred + return + } + + // Reject objects with a generate name set, regardless of whether a name is also set. + if len(manifestObj.GetGenerateName()) > 0 { + wrappedErr := kferrors.NewUserError(nil, "cannot process a manifest with a generate name", + "ordinal", pieces, "manifestObj", klog.KObj(manifestObj), "work", klog.KObj(ownerWorkObj)) + processingState.applyErr = wrappedErr + processingState.applyRes = ApplyResTypeFoundGenerateName + return + } + + // Reject objects with no name set. + if len(manifestObj.GetName()) == 0 { + wrappedErr := kferrors.NewUserError(nil, "cannot process a manifest with no name specified", + "ordinal", pieces, "manifestObj", klog.KObj(manifestObj), "work", klog.KObj(ownerWorkObj)) + processingState.applyErr = wrappedErr + processingState.applyRes = ApplyResTypeNoName + return + } + + processingState.manifestObj = manifestObj + processingState.gvr = gvr + + // Add the string representation of the manifest identifier to the bundle. + // + // Note that the string representation ignores the ordinal number in the identifier. + idStr := formatManifestIdentifierStr(processingState.id) + processingState.idStr = idStr + + klog.V(2).InfoS("Decoded a manifest", + "manifestObj", klog.KObj(manifestObj), + "GVR", *gvr, + "work", klog.KObj(ownerWorkObj)) + } + r.parallelizer.ParallelizeUntil(childCtx, len(manifestProcessingStates), preProcessOneManifest, "preProcessManifests") + + if err := innerErr.Load(); err != nil { + return kferrors.Wraps(err.(error), "failed to pre-process manifests") + } + // If the context has been cancelled, return an error immediately so that the reconciliation loop would exit now. + if ctx.Err() != nil { + return kferrors.NewTransientError(ctx.Err(), "context was canceled or timed out during manifest pre-processing") + } + return nil +} + +// Decodes the manifest JSON into a Kubernetes unstructured object. +func (r *Reconciler) decodeManifest(manifest *placementv1alpha1.Manifest) (*schema.GroupVersionResource, *unstructured.Unstructured, error) { + unstructuredObj := &unstructured.Unstructured{} + if err := unstructuredObj.UnmarshalJSON(manifest.Raw); err != nil { + return &schema.GroupVersionResource{}, nil, kferrors.NewUserError(err, "failed to unmarshal JSON") + } + + mapping, err := r.restMapper.RESTMapping(unstructuredObj.GroupVersionKind().GroupKind(), unstructuredObj.GroupVersionKind().Version) + if err != nil { + if meta.IsNoMatchError(err) { + return &schema.GroupVersionResource{}, unstructuredObj, + kferrors.NewUserError(err, "failed to find GVR from member cluster client REST mapping; the resource might not be available on the member cluster side") + } + return &schema.GroupVersionResource{}, unstructuredObj, + kferrors.NewTransientError(err, "failed to complete REST mapping for the manifest") + } + + // Verify that the REST scope matches the manifest's scope. + if mapping.Scope.Name() == meta.RESTScopeNameRoot && unstructuredObj.GetNamespace() != "" { + return &schema.GroupVersionResource{}, unstructuredObj, + kferrors.NewUserError(nil, "cluster-scoped resource should not have a namespace specified") + } + if mapping.Scope.Name() == meta.RESTScopeNameNamespace && unstructuredObj.GetNamespace() == "" { + return &schema.GroupVersionResource{}, unstructuredObj, + kferrors.NewUserError(nil, "namespace-scoped resource must have a namespace specified") + } + + return &mapping.Resource, unstructuredObj, nil +} + +func buildManifestIdentifier( + manifestIdx int, + gvr *schema.GroupVersionResource, + manifestObj *unstructured.Unstructured, +) *placementv1alpha1.ManifestIdentifier { + // The ordinal field is always set. + identifier := &placementv1alpha1.ManifestIdentifier{ + Ordinal: manifestIdx, + } + + // Set the GVK, name, namespace, and generate name information if the manifest can be decoded + // as a Kubernetes unstructured object. + // + // Note that for cluster-scoped objects, the namespace field will be empty. + if manifestObj != nil { + identifier.APIGroup = manifestObj.GroupVersionKind().Group + identifier.APIVersion = manifestObj.GroupVersionKind().Version + identifier.Kind = manifestObj.GetKind() + identifier.Name = manifestObj.GetName() + identifier.Namespace = manifestObj.GetNamespace() + } + + // Set the GVR information if the manifest object can be REST mapped. + if gvr != nil { + identifier.Resource = gvr.Resource + } + + return identifier +} + +func formatManifestIdentifierStr(id *placementv1alpha1.ManifestIdentifier) string { + // API version is omitted; KubeFleet will report an error if one attempts to place two manifests with the same + // GK but different versions. + return fmt.Sprintf("GK=%s/%s, Namespace=%s, Name=%s", + id.APIGroup, id.Kind, id.Namespace, id.Name) +} + +// markDuplicatedManifests rejects any manifest that shares an identifier with a manifest seen earlier, so that +// only its first occurrence is applied; duplicates would otherwise overwrite one another on the member cluster. +func markDuplicatedManifests(workObjProcessingStates []*workObjectProcessingState) sets.Set[string] { + // The set spans all the work objects, as linked work objects apply to the same member cluster. + seenManifestIDs := sets.New[string]() + + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + + for i := range workObjProcessingState.manifestProcessingStates { + processingState := workObjProcessingState.manifestProcessingStates[i] + if processingState.applyErr != nil { + // The manifest has already failed pre-processing; it has no usable identifier string. + klog.V(2).InfoS("Skipped a manifest in the duplicated manifest checking process as it has failed in the decoding process", + "work", klog.KObj(workObjProcessingState.work), "ordinal", i, + "applyErr", processingState.applyErr, "applyResTyp", processingState.applyRes) + continue + } + + if seenManifestIDs.Has(processingState.idStr) { + processingState.applyErr = kferrors.NewUserError(nil, "rejected a duplicated manifest", + "ordinal", processingState.id.Ordinal, "manifest", processingState.idStr, + "work", klog.KObj(processingState.fromWorkObj)) + processingState.applyRes = ApplyResTypeDuplicated + continue + } + seenManifestIDs.Insert(processingState.idStr) + } + + klog.V(2).InfoS("Completed the check for duplicated manifests", "work", klog.KObj(workObjProcessingState.work)) + } + return seenManifestIDs +} + +func (r *Reconciler) writeAheadPerWorkObjManifestProcessingAttempts(ctx context.Context, + workObjProcessingState *workObjectProcessingState, seenManifestIDs sets.Set[string]) error { + work := workObjProcessingState.work + manifestProcessingStates := workObjProcessingState.manifestProcessingStates + + // As a shortcut, if there's no spec change in the Work object and the status indicates that + // a previous apply attempt has been recorded (**successful or not**), KubeFleet will skip the write-ahead + // op. + appliedCond := meta.FindStatusCondition(work.Status.Conditions, placementv1alpha1.WorkCondTypeApplied) + if appliedCond != nil && appliedCond.ObservedGeneration == work.Generation { + klog.V(2).InfoS("Attempt to apply the current set of manifests has been made before and the results have been recorded; will skip the write-ahead process", + "work", klog.KObj(work)) + return nil + } + + // Prepare the status update (the new manifest conditions) for the write-ahead process. + perManifestStatusesToWriteAhead := make([]placementv1alpha1.PerManifestStatus, 0, len(manifestProcessingStates)) + + // Build an index of existing manifest statuses for quicker lookups. + existingManifestStatusIdx := prepareExistingManifestStatusIndex(work.Status.Manifests) + + for idx := range manifestProcessingStates { + manifestProcessingState := manifestProcessingStates[idx] + if manifestProcessingState.applyErr != nil { + // Skip a manifest if it cannot be pre-processed, e.g., it cannot be decoded. + // + // Such manifests would still be reported in the status (see the later parts of the + // reconciliation loop), it is just that they are not relevant in the write-ahead + // process. + klog.V(2).InfoS("Skipped a manifest in the write-ahead process as it has failed the decoding process or the duplicated manifest checking process", + "work", klog.KObj(work), "ordinal", idx, + "applyErr", manifestProcessingState.applyErr, "applyResTyp", manifestProcessingState.applyRes) + continue + } + + manifestStatusToWriteAhead := buildManifestStatusToWriteAhead(manifestProcessingState, work, existingManifestStatusIdx) + perManifestStatusesToWriteAhead = append(perManifestStatusesToWriteAhead, manifestStatusToWriteAhead) + + klog.V(2).InfoS("Prepared write-ahead information for a manifest", + "manifestObj", klog.KObj(manifestProcessingState.manifestObj), "manifestID", manifestProcessingState.id, "work", klog.KObj(work)) + } + + // Identify any manifests from previous runs that might have been applied and are now left + // over in the member cluster. + leftOverManifests := findLeftOverManifests(perManifestStatusesToWriteAhead, work, existingManifestStatusIdx, seenManifestIDs) + if err := r.removeLeftOverManifests(ctx, leftOverManifests, workObjProcessingState); err != nil { + return kferrors.Wraps(err, "failed to remove leftover manifests", + "work", klog.KObj(work), "leftOverManifestCount", len(leftOverManifests), "removalFailedCount", len(err.Errors())) + } + klog.V(2).InfoS("Left-over manifests are found and removed", + "leftOverManifestCount", len(leftOverManifests), "work", klog.KObj(work)) + + // Update the status. + // + // Note that the Work object might have been refreshed by controllers on the hub cluster + // before this step runs; in this case the current reconciliation loop must be abandoned. + if work.Status.Conditions == nil { + // As a sanity check, set an empty set of conditions. Currently the API definition does + // not allow nil conditions. + work.Status.Conditions = []metav1.Condition{} + } + work.Status.Manifests = perManifestStatusesToWriteAhead + if err := r.hubClient.Status().Update(ctx, work); err != nil { + return kferrors.NewAPIServerError(err, "failed to update work object status", false) + } + + klog.V(2).InfoS("Write-ahead process completed", "work", klog.KObj(work)) + return nil +} + +// prepareExistingManifestStatusIndex indexes the given manifest statuses by their identifier strings, so that +// the status of a specific manifest can be looked up without scanning the whole list. +func prepareExistingManifestStatusIndex(manifestStatuses []placementv1alpha1.PerManifestStatus) map[string]int { + existingManifestStatusIdx := make(map[string]int, len(manifestStatuses)) + for i := range manifestStatuses { + idStr := formatManifestIdentifierStr(&manifestStatuses[i].Identifier) + if _, exists := existingManifestStatusIdx[idStr]; exists { + // Skip duplicate identifiers; only the first occurrence is indexed. This is to ensure that upon + // duplicated manifests, only the first one, which is also the one that gets actually processed, + // is indexed. + continue + } + existingManifestStatusIdx[idStr] = i + } + return existingManifestStatusIdx +} + +// buildManifestStatusToWriteAhead prepares the status of a manifest for the write-ahead process. +func buildManifestStatusToWriteAhead(manifestProcessingState *manifestProcessingState, + work *placementv1alpha1.Work, + existingManifestStatusIdx map[string]int, +) placementv1alpha1.PerManifestStatus { + // If the manifest has an entry in the existing set of manifest statuses, port the previously + // recorded processing results back. + if idx, found := existingManifestStatusIdx[manifestProcessingState.idStr]; found { + return work.Status.Manifests[idx] + } + + // The manifest has not been processed before; report that KubeFleet is preparing to process it. + return placementv1alpha1.PerManifestStatus{ + Identifier: *manifestProcessingState.id, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypePreparingToProcess), + Message: "The manifest is being prepared for processing", + LastTransitionTime: metav1.Now(), + }, + }, + } +} + +// findLeftOverManifests returns the manifests that have been left over on the member cluster side. +func findLeftOverManifests( + perManifestStatusesToWriteAhead []placementv1alpha1.PerManifestStatus, + work *placementv1alpha1.Work, + existingManifestStatusIdx map[string]int, + seenManifestIDs sets.Set[string], +) []placementv1alpha1.ManifestIdentifier { + // Build an index for quicker lookup in the newly prepared write-ahead manifest conditions. + // Here the work applier uses the string representations as map keys; ordinals are omitted from any lookup. + // + // Note that before this step, the work applier has already filtered out duplicate manifests. + manifestStatusesToWriteAheadIdx := make(map[string]int, len(perManifestStatusesToWriteAhead)) + for idx := range perManifestStatusesToWriteAhead { + idStr := formatManifestIdentifierStr(&perManifestStatusesToWriteAhead[idx].Identifier) + manifestStatusesToWriteAheadIdx[idStr] = idx + } + + // For each manifest condition in the existing set of manifest conditions, check if + // there is a corresponding entry in the set of manifest conditions prepared for the write-ahead + // process. If not, the work applier will consider that the manifest has been left over on the member + // cluster side and should be removed. + + // An existing manifest status without a counterpart in the write-ahead set refers to a manifest that + // is no longer part of the work object, and that might still be present on the member cluster side. + leftOverManifests := []placementv1alpha1.ManifestIdentifier{} + for existingManifestIDStr, existingManifestStatusIdx := range existingManifestStatusIdx { + if _, found := manifestStatusesToWriteAheadIdx[existingManifestIDStr]; found { + // The current manifest condition does not have a corresponding entry in the set of manifest + // conditions prepared for the write-ahead process. + continue + } + + if seenManifestIDs.Has(existingManifestIDStr) { + // There exists a corner case where a manifest might have changed its location, i.e., it was previously + // seen on work object A but now lives on work object B. In this case the manifest should not be + // categorized as a left-over manifest. + klog.V(2).InfoS("A manifest has moved to another work object; it will not be considered a left-over manifest", + "manifestID", existingManifestIDStr, "work", klog.KObj(work)) + continue + } + + existingManifestStatus := &work.Status.Manifests[existingManifestStatusIdx] + + // Note that the manifest status might not be up-to-date, so the work applier will not check on the + // generation information. + if shouldCleanUpManifest(existingManifestStatus) { + leftOverManifests = append(leftOverManifests, existingManifestStatus.Identifier) + } + } + return leftOverManifests +} + +// removeLeftOverManifests removes applied left-over manifests from the member cluster. +func (r *Reconciler) removeLeftOverManifests( + ctx context.Context, + leftOverManifests []placementv1alpha1.ManifestIdentifier, + workObjProcessingState *workObjectProcessingState, +) utilerrors.Aggregate { + // Remove all the manifests in parallel. + // + // This is concurrency safe as each worker processes its own applied manifest and writes + // to its own error slot. + + // Cancel the child context anyway to avoid leaks. + childCtx, cancel := context.WithCancel(ctx) + defer cancel() + + // Pre-allocate the slice. + errs := make([]error, len(leftOverManifests)) + doWork := func(pieces int) { + leftOverManifest := leftOverManifests[pieces] + + if err := r.removeOneLeftOverManifest(childCtx, leftOverManifest, workObjProcessingState.appliedWorkOwnerRef); err != nil { + errs[pieces] = kferrors.Wraps(err, "failed to remove a left-over manifest", "manifestId", leftOverManifest) + } + } + r.parallelizer.ParallelizeUntil(childCtx, len(leftOverManifests), doWork, "removeLeftOverManifests") + + return utilerrors.NewAggregate(errs) +} + +// removeOneLeftOverManifest removes an applied manifest object that is left over in the member cluster. +func (r *Reconciler) removeOneLeftOverManifest( + ctx context.Context, + leftOverManifest placementv1alpha1.ManifestIdentifier, + expectedAppliedWorkOwnerRef *metav1.OwnerReference, +) error { + // Build the GVR. + gvr := schema.GroupVersionResource{ + Group: leftOverManifest.APIGroup, + Version: leftOverManifest.APIVersion, + Resource: leftOverManifest.Resource, + } + manifestNamespace := leftOverManifest.Namespace + manifestName := leftOverManifest.Name + manifestRef := klog.KRef(manifestNamespace, manifestName) + + inMemberClusterObj, err := r.spokeDynamicClient. + Resource(gvr). + Namespace(manifestNamespace). + Get(ctx, manifestName, metav1.GetOptions{}) + switch { + case err != nil && apierrors.IsNotFound(err): + // The object has been deleted from the member cluster; no further action is needed. + return nil + case err != nil: + // Failed to retrieve the object from the member cluster. + // + // The cached flag is set to false as the dynamic client is non-caching. + return kferrors.NewAPIServerError(err, "failed to retrieve the object from the member cluster", false, + "gvr", gvr, "manifestObj", manifestRef) + case inMemberClusterObj.GetDeletionTimestamp() != nil: + // The object has been marked for deletion; no further action is needed. + return nil + } + + // There are occasions, though rare, where the object has the same GVR + namespace + name + // combo but is not the applied manifest object the work applier tries to find. This could happen if the object + // has been deleted and then re-created manually without the work applier's acknowledgement. In such cases + // the work applier would ignore the object, and this is not registered as an error. + if !isInMemberClusterObjectDerivedFromManifestObj(inMemberClusterObj, expectedAppliedWorkOwnerRef) { + klog.V(2).InfoS("The object to remove is not derived from the manifest object; will not proceed with the removal", + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + return nil + } + + switch { + case len(inMemberClusterObj.GetOwnerReferences()) > 1: + // KubeFleet is not the sole owner of the object; in this case, KubeFleet will only drop the + // ownership. + klog.V(2).InfoS("The object to remove is co-owned by other sources; will drop the ownership", + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + ownerreferences.Disown(inMemberClusterObj, expectedAppliedWorkOwnerRef) + if _, err := r.spokeDynamicClient.Resource(gvr).Namespace(manifestNamespace).Update(ctx, inMemberClusterObj, metav1.UpdateOptions{}); err != nil && !apierrors.IsNotFound(err) { + return kferrors.NewAPIServerError(err, "failed to drop the ownership of the object", false, + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + } + default: + // KubeFleet is the sole owner of the object; in this case, KubeFleet will delete the object. + klog.V(2).InfoS("The object to remove is solely owned by KubeFleet; will delete the object", + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + inMemberClusterObjUID := inMemberClusterObj.GetUID() + deleteOpts := metav1.DeleteOptions{ + Preconditions: &metav1.Preconditions{ + // Add a UID pre-condition to guard against the case where the object has changed + // right before the deletion request is sent. + // + // Technically speaking resource version based concurrency control should also be + // enabled here; the work applier drops the check to avoid conflicts; this is safe as the KubeFleet + // ownership is considered to be a reserved field and other changes on the object are + // irrelevant to this step. + UID: &inMemberClusterObjUID, + }, + } + if err := r.spokeDynamicClient.Resource(gvr).Namespace(manifestNamespace).Delete(ctx, manifestName, deleteOpts); err != nil && !apierrors.IsNotFound(err) { + return kferrors.NewAPIServerError(err, "failed to delete the object", false, + "gvr", gvr, "manifestObj", manifestRef, + "inMemberClusterObj", klog.KObj(inMemberClusterObj), + "expectedAppliedWorkOwnerRef", *expectedAppliedWorkOwnerRef) + } + } + return nil +} + +// isInMemberClusterObjectDerivedFromManifestObj checks if an object in the member cluster is derived +// from a specific manifest object. +func isInMemberClusterObjectDerivedFromManifestObj(inMemberClusterObj *unstructured.Unstructured, expectedAppliedWorkOwnerRef *metav1.OwnerReference) bool { + // Do a sanity check. + if inMemberClusterObj == nil { + return false + } + + // Verify if the owner reference still stands. + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + if ownerreferences.AreEqual(&curOwners[idx], expectedAppliedWorkOwnerRef) { + return true + } + } + return false +} diff --git a/pkg/v1/controllers/workapplier/process.go b/pkg/v1/controllers/workapplier/process.go new file mode 100644 index 000000000..2a854c134 --- /dev/null +++ b/pkg/v1/controllers/workapplier/process.go @@ -0,0 +1,491 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/utils/resource" + "go.goms.io/fleet/pkg/v1/utils/ownerreferences" +) + +func (r *Reconciler) processManifests(ctx context.Context, manifestProcessingStates []*manifestProcessingState) error { + // Process all manifests in parallel. + // + // There are cases where certain groups of manifests should not be processed in parallel with + // each other (e.g., a config map must be applied after its owner namespace is applied); + // to address this situation, manifests are processed in waves: manifests in the same wave are + // processed in parallel, while different waves are processed sequentially. + + // Organize the manifests into different waves for parallel processing based on their + // GVR information. + processingWaves := organizeManifestsIntoProcessingWaves(manifestProcessingStates) + for idx := range processingWaves { + statesInWave := processingWaves[idx].states + + // TO-DO (chenyu1): evaluate if there is a need to avoid repeated closure + // assignment just for capturing variables. + doWork := func(piece int) { + state := statesInWave[piece] + if state.applyErr != nil { + // Skip a manifest if it has failed pre-processing. + // + // This added as a sanity check as the organization step normally + // would have already skipped all the manifests with processing failures. + return + } + + r.processOneManifest(ctx, state) + klog.V(2).InfoS("Processed a manifest", + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, + "work", klog.KObj(state.fromWorkObj), "primaryWork", klog.KObj(state.fromPrimaryWorkObject)) + } + + r.parallelizer.ParallelizeUntil(ctx, len(statesInWave), doWork, fmt.Sprintf("processingManifestsInWave%d", idx)) + + // Unlike some other steps in the reconciliation loop, the manifest processing step does not end + // with a contextual API call; consequently, if the context has been cancelled during this step, + // some manifest might not get processed at all, and passing such states to the next step may trigger + // unexpected behaviors. To address this, at the end of this step the work applier checks for context + // cancellation directly. + if err := ctx.Err(); err != nil { + klog.V(2).InfoS("Manifest processing has been interrupted as the main context has been cancelled") + return errors.NewTransientError(err, "manifest processing has been interrupted") + } + } + return nil +} + +func (r *Reconciler) processOneManifest(ctx context.Context, manifestProcessingState *manifestProcessingState) { + manifestObj := manifestProcessingState.manifestObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + + // Firstly, attempt to find if an object has been created in the member cluster based on the manifest object. + if shouldSkipProcessing := r.findInMemberClusterObjectFor(ctx, manifestProcessingState); shouldSkipProcessing { + return + } + + // Take over the object in the member cluster that corresponds to the manifest object + // if applicable. + // + // KubeFleet will perform the takeover if: + // a) KubeFleet can find an object in the member cluster that corresponds to the manifest object, and + // it is not currently owned by KubeFleet (specifically the expected appliedWork object); and + // b) takeover is allowed (i.e., the WhenAlreadyExists option in the sync strategy is not set to ReportError). + if shouldSkipProcessing := r.takeOverInMemberClusterObjectIfApplicable(ctx, manifestProcessingState); shouldSkipProcessing { + return + } + + // For both the ClientSideApply and ServerSideApply methods, ownership is a hard requirement. + // Skip the rest of the processing step if the resource has been created in the member cluster + // but KubeFleet is not listed as an owner of the resource in the member cluster yet (i.e., the + // takeover process does not run). This check is necessary as KubeFleet supports the + // WhenAlreadyExists option ReportError. + if !canApplyWithOwnership(manifestProcessingState) { + wrappedErr := errors.NewUserError(nil, "no ownership of the object in the member cluster; takeover is needed", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeNotTakenOver + klog.V(2).InfoS("Ownership is not established yet; skip the apply op", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return + } + + // Perform a round of drift detection before running the apply op, if the ApplyStrategy + // dictates that an apply op can only be run when there are no drifts found. + if shouldSkipProcessing := r.performPreApplyDriftDetectionIfApplicable(ctx, manifestProcessingState); shouldSkipProcessing { + return + } + + // Perform the apply op. + appliedObj, err := r.apply(ctx, manifestProcessingState) + if err != nil { + wrappedErr := errors.Wraps(err, "failed to apply the manifest", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToApply + klog.ErrorS(err, "Failed to apply the manifest", errors.Args(wrappedErr)...) + return + } + if appliedObj != nil { + // Track the newly applied object, if an apply op has been run. + manifestProcessingState.inMemberClusterObj = appliedObj + } + klog.V(2).InfoS("Apply process completed", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + + // Perform another round of drift detection after the apply op, if the ApplyStrategy dictates + // that drift detection should be done in full comparison mode. + // + // Drift detection is currently always enabled in KubeFleet. At this stage of execution, it is + // safe for us to assume that all the managed fields have been overwritten by the just + // completed apply op (or no apply op is necessary); consequently, no further drift + // detection is necessary if the partial comparison mode is used. However, for the full + // comparison mode, the apply op might not to able to resolve all the drifts, should there + // be any change made on the unmanaged fields; and KubeFleet would need to perform another + // round of drift detection. + if shouldSkipProcessing := r.performPostApplyDriftDetectionIfApplicable(ctx, manifestProcessingState); shouldSkipProcessing { + return + } + + manifestProcessingState.applyRes = ApplyResTypeApplied + klog.V(2).InfoS("Manifest processing completed", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) +} + +// findInMemberClusterObjectFor attempts to find the corresponding object in the member cluster +// for a given manifest object. +// +// Note that it is possible that the object has not been created yet in the member cluster. +func (r *Reconciler) findInMemberClusterObjectFor( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (shouldSkipProcessing bool) { + manifestObj := manifestProcessingState.manifestObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + + inMemberClusterObj, err := r.spokeDynamicClient. + Resource(*gvr). + Namespace(manifestObj.GetNamespace()). + Get(ctx, manifestObj.GetName(), metav1.GetOptions{}) + switch { + case err == nil: + // An object derived from the manifest object has been found in the member cluster. + klog.V(2).InfoS("Found the corresponding object for the manifest object in the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.inMemberClusterObj = inMemberClusterObj + return false + case apierrors.IsNotFound(err): + // The manifest object has never been applied before. + klog.V(2).InfoS("The manifest object has not been created in the member cluster yet", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + default: + // An unexpected error has occurred. + wrappedErr := errors.NewAPIServerError(err, + "failed to find the corresponding object for the manifest object in the member cluster", + false, // false as the dynamic client is non-caching. + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToFindObjInMemberCluster + klog.ErrorS(err, + "Failed to find the corresponding object for the manifest object in the member cluster", + errors.Args(wrappedErr)...) + return true + } +} + +// takeOverInMemberClusterObjectIfApplicable attempts to take over an object in the member cluster +// as needed. +func (r *Reconciler) takeOverInMemberClusterObjectIfApplicable( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (shouldSkipProcessing bool) { + manifestObj := manifestProcessingState.manifestObj + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + + if isOwnedByOlderKubeFleetAPIObjects(inMemberClusterObj) { + wrappedErr := errors.NewUserError(nil, + "the object to process is already owned by another KubeFleet placement API object", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeOwnedByOtherKubeFleetAPIObjects + klog.ErrorS(wrappedErr, "Failed to check if takeover is needed", errors.Args(wrappedErr)...) + return true + } + + if !shouldInitiateTakeOverAttempt(manifestProcessingState) { + // Takeover is not necessary; proceed with the processing. + klog.V(2).InfoS("Takeover is not needed; skip the step", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } + + // Take over the object. Note that this steps adds only the owner reference; no other + // fields are modified (on the object from the member cluster). + takenOverInMemberClusterObj, configDiffs, diffCalculatedInDegradedMode, err := r.takeOverPreExistingObject(ctx, manifestProcessingState) + switch { + case err != nil: + // An unexpected error has occurred. + wrappedErr := errors.Wraps(err, "failed to take over a pre-existing object", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToTakeOver + klog.ErrorS(err, "Failed to complete the takeover process", errors.Args(wrappedErr)...) + return true + case len(configDiffs) > 0 && diffCalculatedInDegradedMode: + // Takeover cannot be performed as configuration differences are found between the manifest + // object and the object in the member cluster in degraded mode. + // + // Note that though degraded diff calculation itself is not considered as an error, the + // presence of diffs is indeed an error. + manifestProcessingState.diffs = configDiffs + manifestProcessingState.applyErr = errors.NewUserError(nil, + "cannot take over object: configuration differences are found between the manifest object and the corresponding object in the member cluster in degraded mode (full comparison is performed instead of partial comparison, as the manifest object is considered to be invalid by the member cluster API server)", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyRes = ApplyResTypeFailedToTakeOver + klog.V(2).InfoS("Cannot take over object as configuration differences are found between the manifest object and the corresponding object in the member cluster in degraded mode", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return true + case len(configDiffs) > 0: + // Takeover cannot be performed as configuration differences are found between the manifest + // object and the object in the member cluster. + manifestProcessingState.diffs = configDiffs + manifestProcessingState.applyErr = errors.NewUserError(nil, + "cannot take over object: configuration differences are found between the manifest object and the corresponding object in the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyRes = ApplyResTypeFailedToTakeOver + klog.V(2).InfoS("Cannot take over object as configuration differences are found between the manifest object and the corresponding object in the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return true + } + + // Takeover process is completed; track the newly refreshed object from the member cluster. + manifestProcessingState.inMemberClusterObj = takenOverInMemberClusterObj + klog.V(2).InfoS("The corresponding object has been taken over", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false +} + +// canApplyWithOwnership checks if KubeFleet can perform an apply op, knowing that KubeFleet has +// acquired the ownership of the object, or that the object has not been created yet. +// +// Note that this function does not concern co-ownership; such checks are executed elsewhere. +func canApplyWithOwnership(manifestProcessingState *manifestProcessingState) bool { + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + if inMemberClusterObj == nil { + // The object has not been created yet; KubeFleet can apply the object. + return true + } + + // Verify if the object is owned by KubeFleet. + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + if ownerreferences.AreEqual(&curOwners[idx], manifestProcessingState.ownedBy) { + return true + } + } + return false +} + +// performPreApplyDriftDetectionIfApplicable checks if pre-apply drift detection is needed and +// runs the drift detection process if applicable. +func (r *Reconciler) performPreApplyDriftDetectionIfApplicable( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (shouldSkipProcessing bool) { + manifestObj := manifestProcessingState.manifestObj + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + + isPreApplyDriftDetectionNeeded, err := shouldPerformPreApplyDriftDetection(manifestProcessingState) + switch { + case err != nil: + // KubeFleet cannot determine if pre-apply drift detection is needed; this will only + // happen if the hash calculation process fails, specifically when KubeFleet cannot + // marshal the manifest object into its JSON representation. This should never happen, + // especially considering that the manifest object itself has been + // successfully decoded at this point of execution. + wrappedErr := errors.NewUnexpectedError(err, "failed to determine if pre-apply drift detection is needed", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToRunDriftDetection + klog.ErrorS(err, "Failed to determine if pre-apply drift detection is needed", errors.Args(wrappedErr)...) + return true + case !isPreApplyDriftDetectionNeeded: + // Drift detection is not needed; proceed with the processing. + klog.V(2).InfoS("Pre-apply drift detection is not needed; skip the step", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } + + // Run the drift detection process. + diffs, driftsCalculatedInDegradedMode, err := r.diffBetweenManifestAndInMemberClusterObjects(ctx, + gvr, manifestObj, inMemberClusterObj, + manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy.ComparisonOption) + switch { + case err != nil: + // An unexpected error has occurred. + wrappedErr := errors.Wraps(err, "failed to calculate pre-apply drifts between the manifest and the object from the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeFailedToRunDriftDetection + klog.ErrorS(err, "Failed to complete pre-apply drift detection", errors.Args(wrappedErr)...) + return true + case len(diffs) > 0 && driftsCalculatedInDegradedMode: + // Configuration drifts are found in degraded mode. + // + // Note that though degraded drift calculation itself is not considered as an error, the + // presence of drifts in the pre-apply drift detection step is indeed an error. + manifestProcessingState.diffs = diffs + manifestProcessingState.applyErr = errors.NewUserError(nil, + "cannot apply manifest: drifts are found between the manifest and the object from the member cluster in degraded mode (full comparison is performed instead of partial comparison, as the manifest object is considered to be invalid by the member cluster API server)", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyRes = ApplyResTypeFoundDriftsInDegradedMode + klog.V(2).InfoS("Cannot apply manifest: drifts are found between the manifest and the object from the member cluster in degraded mode", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return true + case len(diffs) > 0: + // Drifts are found in the pre-apply drift detection process. + manifestProcessingState.diffs = diffs + manifestProcessingState.applyErr = errors.NewUserError(nil, + "cannot apply manifest: drifts are found between the manifest and the object from the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyRes = ApplyResTypeFoundDrifts + klog.V(2).InfoS("Cannot apply manifest: drifts are found between the manifest and the object from the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return true + default: + // No drifts are found in the pre-apply drift detection process; carry on with the apply op. + klog.V(2).InfoS("Pre-apply drift detection completed; no drifts are found", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } +} + +// shouldPerformPreApplyDriftDetection checks if pre-apply drift detection should be performed. +func shouldPerformPreApplyDriftDetection(manifestProcessingState *manifestProcessingState) (bool, error) { + // Drift detection is performed before the apply op if (and only if): + // * KubeFleet reports that the manifest has been applied before (i.e., inMemberClusterObj exists); and + // * The sync strategy dictates that a drift should be reported rather than overwritten; and + // * The hash of the manifest object is consistent with the last applied manifest object hash + // annotation on the corresponding resource in the member cluster (i.e., the same manifest + // object has been applied before). + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + if syncStrategy.WhenDrifted != placementv1alpha1.WhenDriftedOptionReportError || inMemberClusterObj == nil { + // A shortcut to save some overhead. + return false, nil + } + + cleanedManifestObj := discardFieldsIrrelevantInComparisonFrom(manifestProcessingState.manifestObj) + manifestObjHash, err := resource.HashOf(cleanedManifestObj.Object) + if err != nil { + return false, err + } + + inMemberClusterObjLastAppliedManifestObjHash := inMemberClusterObj.GetAnnotations()[placementv1alpha1.LastAppliedManifestHashAnnotationKey] + return manifestObjHash == inMemberClusterObjLastAppliedManifestObjHash, nil +} + +// performPostApplyDriftDetectionIfApplicable checks if post-apply drift detection is needed and +// runs the drift detection process if applicable. +func (r *Reconciler) performPostApplyDriftDetectionIfApplicable( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (shouldSkipProcessing bool) { + manifestObj := manifestProcessingState.manifestObj + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + ownerWorkObj := manifestProcessingState.fromWorkObj + gvr := manifestProcessingState.gvr + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + + if !shouldPerformPostApplyDriftDetection(syncStrategy) { + // Post-apply drift detection is not needed; proceed with the processing. + klog.V(2).InfoS("Post-apply drift detection is not needed; skip the step", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } + + diffs, driftsCalculatedInDegradedMode, err := r.diffBetweenManifestAndInMemberClusterObjects(ctx, + gvr, manifestObj, inMemberClusterObj, syncStrategy.ComparisonOption) + switch { + case err != nil: + // An unexpected error has occurred. + // + // This case counts as a partial error; the apply op has been completed, but KubeFleet + // cannot determine if there are any drifts. + wrappedErr := errors.Wraps(err, "failed to calculate post-apply drifts between the manifest object and the object from the member cluster", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + manifestProcessingState.applyErr = wrappedErr + manifestProcessingState.applyRes = ApplyResTypeAppliedWithFailedDriftDetection + klog.ErrorS(err, + "Failed to complete post-apply drift detection", + errors.Args(wrappedErr)...) + return true + case len(diffs) > 0 && driftsCalculatedInDegradedMode: + // Configuration drifts are found, but they were calculated in degraded mode. + // + // Normally this should never happen, as post-apply drift detection will only run if the full + // comparison mode is enabled, but degraded mode only applies to partial comparison mode. + manifestProcessingState.diffs = diffs + klog.V(2).InfoS("Post-apply drift detection completed; drifts are found in degraded mode", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + // The presence of such drifts is not considered as an error. + return false + case len(diffs) > 0: + // Drifts are found in the post-apply drift detection process. + manifestProcessingState.diffs = diffs + klog.V(2).InfoS("Post-apply drift detection completed; drifts are found", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + // The presence of such drifts is not considered as an error. + return false + default: + // No drifts are found in the post-apply drift detection process. + klog.V(2).InfoS("Post-apply drift detection completed; no drifts are found", + "manifestObj", klog.KObj(manifestObj), "GVR", *gvr, + "work", klog.KObj(ownerWorkObj), "primaryWork", klog.KObj(manifestProcessingState.fromPrimaryWorkObject)) + return false + } +} + +// shouldPerformPostApplyDriftDetection checks if post-apply drift detection should be performed. +func shouldPerformPostApplyDriftDetection(syncStrategy *placementv1alpha1.SyncStrategy) bool { + // Post-apply drift detection is performed if (and only if): + // * The sync strategy dictates that drift detection should run in full comparison mode. + return syncStrategy.ComparisonOption == placementv1alpha1.ComparisonOptionFullComparison +} diff --git a/pkg/v1/controllers/workapplier/requeue_integration_test.go b/pkg/v1/controllers/workapplier/requeue_integration_test.go new file mode 100644 index 000000000..e6f59b5e4 --- /dev/null +++ b/pkg/v1/controllers/workapplier/requeue_integration_test.go @@ -0,0 +1,613 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +// The test cases below use the work applier dedicated to the requeue and status update test cases +// (workApplierInstrumented), whose clients record the start of each reconciliation and the status updates on +// work/appliedWork objects, and whose requeue rate limiter uses short delays (see instrumentedRequeueRateLimiter). +// +// Note that the test cases must not let the work applier take over objects, as the takeover process might retrieve +// work objects, which would be mistaken for the start of reconciliations. +var _ = Describe("requeues and status updates", func() { + Context("status updates skipped when nothing changes", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workKey := types.NamespacedName{Namespace: memberReservedNSNameInstrumented, Name: workName} + appliedWorkKey := types.NamespacedName{Name: workName} + + BeforeAll(func() { + // Prepare the manifests. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + createWorkObject(workName, memberReservedNSNameInstrumented, nil, regularNSJSON, regularConfigMapJSON) + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSNameInstrumented, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedNS := &corev1.Namespace{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Name: nsName}, appliedNS)).To(Succeed(), "Failed to retrieve the namespace") + appliedConfigMap := &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, appliedConfigMap)).To(Succeed(), "Failed to retrieve the config map") + + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: appliedNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: appliedConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("should have updated the work object status exactly twice and the appliedWork object status exactly once", func() { + // The work object status is updated once to write ahead the manifests to process, and once to report the + // results; the appliedWork object status is updated once to report the applied resources. + Expect(hubClientRecorderInstrumented.statusUpdateCount(workKey)).To(Equal(2), "The work object status is not updated exactly twice") + Expect(memberClientRecorderInstrumented.statusUpdateCount(appliedWorkKey)).To(Equal(1), "The appliedWork object status is not updated exactly once") + }) + + It("should skip the status updates in later reconciliations", func() { + reconcileCount := len(hubClientRecorderInstrumented.reconcileStartTimestamps(workKey)) + Eventually(func() int { + return len(hubClientRecorderInstrumented.reconcileStartTimestamps(workKey)) + }, 2*eventuallyDuration, eventuallyInterval).Should(BeNumerically(">=", reconcileCount+2), "The work object has not been reconciled at least twice more") + + Expect(hubClientRecorderInstrumented.statusUpdateCount(workKey)).To(Equal(2), "The work object status has been updated again with no change") + Expect(memberClientRecorderInstrumented.statusUpdateCount(appliedWorkKey)).To(Equal(1), "The appliedWork object status has been updated again with no change") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSNameInstrumented) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSNameInstrumented, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("status updates skipped when nothing changes (linked work objects)", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and the + // same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + primaryConfigMapName := "configmap-primary" + secondaryConfigMapName := "configmap-secondary" + workNames := []string{primaryWorkName, secondaryWorkName} + configMapNames := map[string]string{ + primaryWorkName: primaryConfigMapName, + secondaryWorkName: secondaryConfigMapName, + } + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work objects do not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + configMapJSONs := map[string][]byte{} + for _, workName := range workNames { + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapNames[workName] + configMapJSONs[workName] = marshalK8sObjJSON(regularConfigMap) + } + + // Create the secondary work object first, so that the work applier can find all the linked work + // objects as soon as it starts processing the primary work object. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSNameInstrumented, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + configMapJSONs[secondaryWorkName], + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSNameInstrumented, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + configMapJSONs[primaryWorkName], + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + for _, workName := range workNames { + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapNames[workName], + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSNameInstrumented, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of work %s", workName) + } + }) + + It("should update the appliedWork object statuses", func() { + // Each appliedWork object reports the resources applied from its own work object. + for _, workName := range workNames { + appliedConfigMap := &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapNames[workName]}, appliedConfigMap)).To(Succeed(), "Failed to retrieve the config map %s", configMapNames[workName]) + + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapNames[workName], + Namespace: nsName, + }, + UID: appliedConfigMap.UID, + }, + } + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the appliedWork object %s", workName) + } + }) + + It("should have updated each work object status exactly twice and each appliedWork object status exactly once", func() { + for _, workName := range workNames { + Expect(hubClientRecorderInstrumented.statusUpdateCount(types.NamespacedName{Namespace: memberReservedNSNameInstrumented, Name: workName})).To(Equal(2), "The status of work %s is not updated exactly twice", workName) + Expect(memberClientRecorderInstrumented.statusUpdateCount(types.NamespacedName{Name: workName})).To(Equal(1), "The status of the appliedWork object %s is not updated exactly once", workName) + } + }) + + It("should skip the status updates in later reconciliations", func() { + // Only the primary work object is reconciled; the secondary work object is processed along with it. + primaryWorkKey := types.NamespacedName{Namespace: memberReservedNSNameInstrumented, Name: primaryWorkName} + reconcileCount := len(hubClientRecorderInstrumented.reconcileStartTimestamps(primaryWorkKey)) + Eventually(func() int { + return len(hubClientRecorderInstrumented.reconcileStartTimestamps(primaryWorkKey)) + }, 2*eventuallyDuration, eventuallyInterval).Should(BeNumerically(">=", reconcileCount+2), "The primary work object has not been reconciled at least twice more") + + for _, workName := range workNames { + Expect(hubClientRecorderInstrumented.statusUpdateCount(types.NamespacedName{Namespace: memberReservedNSNameInstrumented, Name: workName})).To(Equal(2), "The status of work %s has been updated again with no change", workName) + Expect(memberClientRecorderInstrumented.statusUpdateCount(types.NamespacedName{Name: workName})).To(Equal(1), "The status of the appliedWork object %s has been updated again with no change", workName) + } + }) + + AfterAll(func() { + // There is no GC controller in the hub cluster test environment to delete the secondary work object along + // with the primary one; delete both of them. + for _, workName := range workNames { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSNameInstrumented, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete work object %s", workName) + } + + // Remove the config maps manually, as there is no GC controller in the test environment. + for _, workName := range workNames { + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapNames[workName]) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map %s", configMapNames[workName]) + } + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork objects. + for _, workName := range workNames { + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object %s", workName) + + workRemovedActual := workRemovedActual(memberReservedNSNameInstrumented, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object %s", workName) + } + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("requeue backoff (manifests not yet available), and reset upon processing result changes", Ordered, Label("slow"), func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workKey := types.NamespacedName{Namespace: memberReservedNSNameInstrumented, Name: workName} + + // The time when the deployment is marked as available. + var markedAsAvailableAt time.Time + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + createWorkObject(workName, memberReservedNSNameInstrumented, nil, regularDeployJSON) + }) + + It("should apply the manifest", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSNameInstrumented, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef := prepareAppliedWorkOwnerRef(memberClient1, workName) + + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the deployment object") + }) + + It("should back off slowly, then back off fast", func() { + // The deployment never becomes available in the test environment unless marked so; the processing result + // stays the same, and the requeue delays grow as configured in instrumentedRequeueRateLimiter (2s, 2s, 3s, + // 6s, 8s, 8s, ...). + // + // The gaps are counted from the second reconciliation of the work object, i.e., the fixed delay is not + // checked here (it is checked after the processing result changes). The gap after the first reconciliation + // also covers the reconciliation itself, which creates the appliedWork object and applies the manifest; it + // might take seconds when the test environment has just started. + wantGaps := []time.Duration{ + time.Second * 2, + time.Second * 3, + time.Second * 6, + time.Second * 8, + time.Second * 8, + } + requeueGapsActual := requeueGapsActual(hubClientRecorderInstrumented, workKey, time.Time{}, 1, wantGaps) + Eventually(requeueGapsActual, 6*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to requeue the work object with backoff") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + markedAsAvailableAt = time.Now() + }) + + It("should requeue with the fixed delay again, as the processing result changes", func() { + // The change in the member cluster triggers no event for the work object; the next reconciliation (up to + // 8 seconds later) finds the deployment available, which resets the requeue counter. Without the reset, + // the work object would have been requeued after 8 seconds again. + requeueGapsActual := requeueGapsActual(hubClientRecorderInstrumented, workKey, markedAsAvailableAt, 0, []time.Duration{time.Second * 2}) + Eventually(requeueGapsActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to reset the requeue counter") + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSNameInstrumented, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSNameInstrumented) + + // Remove the deployment manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSNameInstrumented, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("requeue backoff (manifests available), and reset upon work object changes", Ordered, Label("slow"), func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workKey := types.NamespacedName{Namespace: memberReservedNSNameInstrumented, Name: workName} + + // The time when the work object is updated. + var nudgedAt time.Time + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + createWorkObject(workName, memberReservedNSNameInstrumented, nil, regularConfigMapJSON) + }) + + It("should apply the manifest", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSNameInstrumented, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + appliedWorkOwnerRef := prepareAppliedWorkOwnerRef(memberClient1, workName) + + regularConfigMapObjectAppliedActual := regularConfigMapObjectAppliedActual(memberClient1, nsName, configMapName, appliedWorkOwnerRef) + Eventually(regularConfigMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map object") + }) + + It("should skip to fast backoff", func() { + // The config map is available right away; as all the applied manifests are available, the requeue rate + // limiter skips the slow backoff stage (2s, 2s, 4s, 8s, 8s, ...). + // + // The gaps are counted from the second reconciliation of the work object, for the same reason as with + // the not yet available manifests. + wantGaps := []time.Duration{ + time.Second * 2, + time.Second * 4, + time.Second * 8, + time.Second * 8, + } + requeueGapsActual := requeueGapsActual(hubClientRecorderInstrumented, workKey, time.Time{}, 1, wantGaps) + Eventually(requeueGapsActual, 6*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to requeue the work object with backoff") + }) + + It("can update the work object", func() { + nudgedAt = time.Now() + nudgeWork(workName, memberReservedNSNameInstrumented) + }) + + It("should requeue after 2 seconds again, as the work object changes", func() { + // The update triggers a reconciliation right away, which finds the work object changed and resets the + // requeue counter; the work object is then requeued with the fixed delay (2s), and then with the initial + // slow backoff delay (also 2s). Without the reset, the work object would have been requeued after 8 + // seconds again. + // + // A periodic requeue might start right after nudgedAt is recorded, yet still read the previous generation + // of the work object from the cache; the reconciliation triggered by the update would then follow it + // immediately. To stay robust, the gap is checked after skipping the first reconciliation at or after + // nudgedAt; it is 2 seconds either way. + requeueGapsActual := requeueGapsActual(hubClientRecorderInstrumented, workKey, nudgedAt, 1, []time.Duration{time.Second * 2}) + Eventually(requeueGapsActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to reset the requeue counter") + }) + + It("should update the work object status", func() { + // The work object conditions observe the latest generation of the work object. + workProcessedAtLatestGenerationActual := workProcessedAtLatestGenerationActual(memberReservedNSNameInstrumented, workName) + Eventually(workProcessedAtLatestGenerationActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to process the latest generation of the work object") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSNameInstrumented) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSNameInstrumented, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/retrieval.go b/pkg/v1/controllers/workapplier/retrieval.go new file mode 100644 index 000000000..7b0a3a0b9 --- /dev/null +++ b/pkg/v1/controllers/workapplier/retrieval.go @@ -0,0 +1,213 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + "slices" + "strconv" + "strings" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +// retrieveLinkedAndLeftOverWorks retrieves two sets of work objects: +// +// a) linked work objects: these work objects share the same owner placement binding as the primary work object, and +// +// are all linked to the same primary placement resource snapshot as the primary work object. In other words, +// these work objects together form a consistent state of manifests that the work applier should process. +// +// b) leftover work objects: these work objects are also owned by the same placement binding as the primary work +// +// object, but are no longer linked to the current primary placement resource snapshot and are considered stale. +// All manifests in these work objects should be cleaned up, unless they are also referenced by another linked work +// object. +// +// This method will return an error if it observed any incomplete/inconsistent state among the two sets. Specifically, +// it will return a transient error when: +// +// a) the number of linked work objects does not match the linked work count recorded on the primary work object, or +// +// any of the linked work objects is not linked to the same primary placement resource snapshot as the primary +// work object. +// +// b) any of the leftover work objects hasn't been marked for deletion. +// +// Note that the first work object in the array of linked work objects is always the primary work object. +func (r *Reconciler) retrieveLinkedAndLeftOverWorks(ctx context.Context, + primaryWork *placementv1alpha1.Work) ([]*placementv1alpha1.Work, []*placementv1alpha1.Work, error) { + ownedBy := primaryWork.GetLabels()[placementv1alpha1.WorkOwnedByPlacementBindingLabelKey] + if ownedBy == "" { + return nil, nil, errors.NewUnexpectedError(nil, "the primary work is missing the owner placement binding label") + } + // An empty owner namespace signals a cluster-scoped placement binding. + ownerNS := primaryWork.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + + wantLinkedWorkCountVal := primaryWork.GetAnnotations()[placementv1alpha1.LinkedWorkCountAnnotationKey] + wantLinkedWorkCount, err := strconv.Atoi(wantLinkedWorkCountVal) + if err != nil { + return nil, nil, errors.NewUnexpectedError(err, "failed to parse the linked work count annotation on the primary work", + "linkedWorkCount", wantLinkedWorkCountVal) + } + + wantPrimarySnapshotName := primaryWork.GetAnnotations()[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey] + if wantPrimarySnapshotName == "" { + return nil, nil, errors.NewUnexpectedError(nil, "the primary work is missing the primary placement resource snapshot annotation") + } + + // List all work objects owned by the same placement binding as the primary work. + ownedByFieldVal := fmt.Sprintf(fieldindexers.WorkOwnedByPlacementBindingCustomFieldValFmt, ownerNS, ownedBy) + workList := &placementv1alpha1.WorkList{} + if err := r.hubClient.List(ctx, workList, + client.InNamespace(primaryWork.Namespace), + client.MatchingFields{fieldindexers.WorkOwnedByPlacementBindingCustomFieldName: ownedByFieldVal}, + ); err != nil { + return nil, nil, errors.NewAPIServerError(err, "failed to list the work objects owned by the placement binding", true, + "ownerPlacementBinding", ownedByFieldVal) + } + + var linkedWorks []*placementv1alpha1.Work + var leftOverWorks []*placementv1alpha1.Work + + linkedWorks = append(linkedWorks, primaryWork) + for idx := range workList.Items { + linkedWork := &workList.Items[idx] + primarySnapshotName := linkedWork.GetAnnotations()[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey] + if primarySnapshotName != wantPrimarySnapshotName { + // This work is not linked to the same primary placement resource snapshot as the primary work; add it + // to the set of leftover works. + leftOverWorks = append(leftOverWorks, linkedWork) + continue + } + + if linkedWork.Name != primaryWork.Name { + // This work is linked to the same primary placement resource snapshot as the primary work; add it + // to the set of linked works. + linkedWorks = append(linkedWorks, linkedWork) + } + } + + // Verify that the number of linked work objects matches the count recorded on the primary work. + if len(linkedWorks) != wantLinkedWorkCount { + return nil, nil, errors.NewTransientError(nil, "the number of linked work objects found is inconsistent with the count recorded on the primary work", + "ownerPlacementBinding", ownedByFieldVal, + "observedLinkedWorkCount", len(linkedWorks), "wantLinkedWorkCount", wantLinkedWorkCount) + } + + // Sort the linked work objects by their names to ensure deterministic processing order. + // + // The primary work object is excluded from sorting as it must always be the first item in the array. + slices.SortFunc(linkedWorks[1:], func(a, b *placementv1alpha1.Work) int { + return strings.Compare(a.Name, b.Name) + }) + + // Verify that all leftover work objects have been marked for deletion. + for idx := range leftOverWorks { + leftOverWork := leftOverWorks[idx] + if leftOverWork.DeletionTimestamp == nil { + return nil, nil, errors.NewTransientError(nil, "a leftover work object has not been marked for deletion as expected", + "leftOverWork", klog.KObj(leftOverWork)) + } + } + + return linkedWorks, leftOverWorks, nil +} + +// addCleanupFinalizerTo adds the cleanup finalizer to the given work objects, so that the applied resources +// on the member cluster side can be cleaned up before the work objects are removed. +func (r *Reconciler) addCleanupFinalizerTo(ctx context.Context, linkedWorks []*placementv1alpha1.Work) error { + for idx := range linkedWorks { + linkedWork := linkedWorks[idx] + if controllerutil.ContainsFinalizer(linkedWork, workApplierCleanupFinalizer) { + continue + } + + controllerutil.AddFinalizer(linkedWork, workApplierCleanupFinalizer) + if err := r.hubClient.Update(ctx, linkedWork); err != nil { + // Reset the finalizer list to its previous state. + controllerutil.RemoveFinalizer(linkedWork, workApplierCleanupFinalizer) + return errors.NewAPIServerError(err, "failed to add the cleanup finalizer to the work", false, + "work", klog.KObj(linkedWork)) + } + klog.V(2).InfoS("Added the cleanup finalizer to the work", "work", klog.KObj(linkedWork)) + } + return nil +} + +func (r *Reconciler) ensureAppliedWorks(ctx context.Context, works []*placementv1alpha1.Work) ([]*placementv1alpha1.AppliedWork, error) { + appliedWorks := make([]*placementv1alpha1.AppliedWork, 0, len(works)) + for idx := range works { + work := works[idx] + + // Check if an appliedWork object already exists for the primary work object. + // + // Since we only create an appliedWork object after adding the finalizer to the primary work object, + // it might appear that if the finalizer is absent, the appliedWork object should not exist. This + // is not the case with the work applier though, as the controller features a + // Leave method that will strip all work objects off their finalizers, which is called when the + // member cluster leaves the fleet. If the member cluster chooses to later re-join the fleet, the controller + // might see a work object with no finalizer but with an appliedWork object. Because of this, here we always + // check for the existence of the appliedWork object, with or without the finalizer. + appliedWork := &placementv1alpha1.AppliedWork{} + err := r.spokeClient.Get(ctx, types.NamespacedName{Name: work.Name}, appliedWork) + switch { + case err == nil && !appliedWork.GetDeletionTimestamp().IsZero(): + // The appliedWork object is marked for deletion; wait for the cleanup to complete. + return nil, errors.NewTransientError(nil, "found an appliedWork object that has been marked for deletion", "appliedWork", klog.KObj(appliedWork)) + case err == nil: + // The AppliedWork already exists; no further action is needed. + klog.V(2).InfoS("Found an appliedWork object for the work object", "appliedWork", klog.KObj(appliedWork)) + appliedWorks = append(appliedWorks, appliedWork) + continue + case !apierrors.IsNotFound(err): + return nil, errors.NewAPIServerError(err, "failed to retrieve the appliedWork object", true) + } + + // The appliedWork object does not exist; create one. + appliedWork = &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{ + Name: work.Name, + }, + Spec: placementv1alpha1.AppliedWorkSpec{ + WorkName: work.Name, + WorkNamespace: work.Namespace, + }, + } + if err := r.spokeClient.Create(ctx, appliedWork); err != nil { + // Note that the controller must retry on AppliedWork AlreadyExists errors; otherwise the + // controller will run the reconciliation loop with an AppliedWork that has no UID, + // which might lead to takeover failures in later steps. + return nil, errors.NewAPIServerError(err, "failed to create an appliedWork object for the work object", false, + "appliedWork", klog.KObj(appliedWork)) + } + klog.V(2).InfoS("Created an appliedWork object for the work object", "appliedWork", klog.KObj(appliedWork)) + appliedWorks = append(appliedWorks, appliedWork) + } + + return appliedWorks, nil +} diff --git a/pkg/v1/controllers/workapplier/status.go b/pkg/v1/controllers/workapplier/status.go new file mode 100644 index 000000000..b8f74b963 --- /dev/null +++ b/pkg/v1/controllers/workapplier/status.go @@ -0,0 +1,541 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + + "k8s.io/apimachinery/pkg/api/equality" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +// refreshWorkStatus refreshes the status of a Work object based on the processing results of its manifests. +func (r *Reconciler) refreshWorkStatus( + ctx context.Context, + workObjProcessingStates []*workObjectProcessingState, +) error { + // Note (chenyu1): it is possible to run this method in parallel; however, for simplicity reasons, + // considering that in most of the time the count of work objects would be low, currently + // KubeFleet still does the status refresh sequentially. + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + if err := r.refreshOneWorkStatus(ctx, workObjProcessingState); err != nil { + wrappedErr := errors.Wraps(err, "", "work", klog.KObj(workObjProcessingState.work)) + return wrappedErr + } + } + return nil +} + +func (r *Reconciler) refreshOneWorkStatus( + ctx context.Context, + workObjProcessingState *workObjectProcessingState, +) error { + work := workObjProcessingState.work + statusCopy := work.Status.DeepCopy() + + // Note (chenyu1): it is possible to run this method in parallel; however, for simplicity reasons, + // considering that in most of the time the count of manifests would be low, currently + // KubeFleet still does the status refresh sequentially. + + // Set up the counters. + manifestCount := len(workObjProcessingState.manifestProcessingStates) + appliedManifestsCount := 0 + availableAppliedObjectsCount := 0 + untrackableAppliedObjectsCount := 0 + + // Use the now timestamp as the observation time. + now := metav1.Now() + + // Build an empty list of per-manifest statuses and port back existing per-manifest statuses to the list + // as applicable. + // + // This step is necessary at the moment primarily because the manifest status features `metav1.Condition`, + // the LastTransitionTime field of which requires that KubeFleet track the last known condition. + // + // Note that the refreshed manifest status array is pre-allocated and is of the same length as the number of + // manifest processing states. + refreshedManifestStatuses, refreshedManifestStatusesIndex := prepareRefreshedManifestStatusWithIndex(workObjProcessingState.manifestProcessingStates) + portBackExistingManifestStatuses(statusCopy.Manifests, refreshedManifestStatuses, refreshedManifestStatusesIndex) + + // Refresh per-manifest statuses. + for idx := range workObjProcessingState.manifestProcessingStates { + manifestProcessingState := workObjProcessingState.manifestProcessingStates[idx] + refreshOneManifestStatus(manifestProcessingState, &refreshedManifestStatuses[idx], now) + + // Tally the stats. + if isManifestObjectApplied(manifestProcessingState.applyRes) { + appliedManifestsCount++ + } + if isAppliedObjectAvailable(manifestProcessingState.availabilityCheckRes) { + availableAppliedObjectsCount++ + } + if manifestProcessingState.availabilityCheckRes == AvailabilityResultTypeNotTrackable { + untrackableAppliedObjectsCount++ + } + } + statusCopy.Manifests = refreshedManifestStatuses + + // Set work object status conditions. + + // Do a sanity check. + if appliedManifestsCount > manifestCount || + availableAppliedObjectsCount > manifestCount || + untrackableAppliedObjectsCount > manifestCount { + // Normally this should never happen. + return errors.NewUnexpectedError(nil, + "the number of applied manifests, available applied objects, or untrackable applied objects exceeds the total number of manifests", + "appliedManifestsCount", appliedManifestsCount, + "availableAppliedObjectsCount", availableAppliedObjectsCount, + "untrackableAppliedObjectsCount", untrackableAppliedObjectsCount, + "manifestCount", manifestCount) + } + if statusCopy.Conditions == nil { + statusCopy.Conditions = []metav1.Condition{} + } + setWorkAppliedCondition(statusCopy, manifestCount, appliedManifestsCount, work.Generation) + setWorkAvailableCondition(statusCopy, manifestCount, availableAppliedObjectsCount, untrackableAppliedObjectsCount, work.Generation) + + // TO-DO (chenyu1): if needed, trim the reported status so that the work object will not grow over Kubernetes' + // object size limit. + if shouldSkipStatusUpdate(&work.Status, statusCopy) { + // No status change found; skip the update. + klog.V(2).InfoS("No status change found for Work object; skip the status update", "work", klog.KObj(work)) + } else { + klog.V(2).InfoS("Refreshing work object status", "work", klog.KObj(work)) + + work.Status = *statusCopy + if err := r.hubClient.Status().Update(ctx, work); err != nil { + return errors.NewAPIServerError(err, "failed to update work object status", false, "work", klog.KObj(work)) + } + } + + return nil +} + +func prepareRefreshedManifestStatusWithIndex(manifestProcessingStates []*manifestProcessingState) ( + []placementv1alpha1.PerManifestStatus, map[string][]int) { + // Pre-allocate the status slice. + refreshedManifestStatuses := make([]placementv1alpha1.PerManifestStatus, len(manifestProcessingStates)) + + // Prepare the index. Note that there might be multiple manifests with the same identifier (duplicated + // manifests), so the index returns a slice of indices. It is guaranteed in normal ops that all indices + // except for the first point to duplicated manifests. + refreshedManifestStatusesIndex := make(map[string][]int, len(manifestProcessingStates)) + for idx := range manifestProcessingStates { + manifestProcessingState := manifestProcessingStates[idx] + idStr := manifestProcessingState.idStr + + if len(manifestProcessingState.idStr) == 0 { + // There might be manifests without a valid identifier in the bundle set + // (e.g., a decoding error has occurred when processing a bundle). + // KubeFleet will skip these bundles, as there is no need to port back + // information for such manifests any way for obvious reasons (manifest itself is not + // identifiable). This is not considered as an error. + continue + } + refreshedManifestStatusesIndex[idStr] = append(refreshedManifestStatusesIndex[idStr], idx) + } + return refreshedManifestStatuses, refreshedManifestStatusesIndex +} + +func portBackExistingManifestStatuses( + existingManifestStatuses []placementv1alpha1.PerManifestStatus, + refreshedManifestStatuses []placementv1alpha1.PerManifestStatus, + refreshedManifestStatusesIndex map[string][]int, +) { + for idx := range existingManifestStatuses { + existingManifestStatus := existingManifestStatuses[idx] + existingManifestId := &existingManifestStatus.Identifier + if existingManifestId.Kind == "" || + existingManifestId.Resource == "" || + existingManifestId.Name == "" { + // It is OK for an existing manifest status to not have a valid identifier; this + // happens when the manifest status was previously associated with a manifest + // that cannot be decoded. For obvious reasons KubeFleet does not need to port back + // such manifest conditions any way. + continue + } + existingManifestIdStr := formatManifestIdentifierStr(existingManifestId) + + if idxes, found := refreshedManifestStatusesIndex[existingManifestIdStr]; found { + // A match can be found via the index; port back the existing manifest status. + // + // Normally there should only be one match. However, in case of duplicated manifests, the index + // lookup might yield multiple indices. The system ensures that when there are duplicate manifests, + // the one that actually gets processed always appears first in the manifest list (if applicable), + // and the portback here would match that order as well. This helps avoid fluctuations when + // refreshing work object status. + if len(idxes) > 0 { + curIdx := idxes[0] + refreshedManifestStatuses[curIdx] = *existingManifestStatus.DeepCopy() + refreshedManifestStatusesIndex[existingManifestIdStr] = idxes[1:] + } + } + } +} + +func refreshOneManifestStatus(manifestProcessingState *manifestProcessingState, + manifestStatusToRefresh *placementv1alpha1.PerManifestStatus, + now metav1.Time, +) { + // Populate the identifier in the manifest status. + // + // If the status has been ported back, the identifier written here is guaranteed to match the existing one. + manifestStatusToRefresh.Identifier = *manifestProcessingState.id.DeepCopy() + + // Prepare the list of conditions in the manifest status. + if manifestStatusToRefresh.Conditions == nil { + manifestStatusToRefresh.Conditions = []metav1.Condition{} + } + + // Set the conditions. + // + // Note that per API definition, the observed generation of a manifest condition is that + // of the applied resource, not that of the Work object. + var inMemberClusterObjGeneration *int64 + if manifestProcessingState.inMemberClusterObj != nil { + inMemberClusterObjGeneration = ptr.To(manifestProcessingState.inMemberClusterObj.GetGeneration()) + } + setManifestStatusAppliedCondition(manifestStatusToRefresh, manifestProcessingState.applyRes, manifestProcessingState.applyErr, inMemberClusterObjGeneration) + setManifestStatusAvailableCondition(manifestStatusToRefresh, manifestProcessingState.availabilityCheckRes, manifestProcessingState.availabilityCheckErr, inMemberClusterObjGeneration) + setManifestStatusDiffDetails(manifestStatusToRefresh, manifestProcessingState.diffs, inMemberClusterObjGeneration, now) +} + +// setManifestStatusAppliedCondition sets the Applied condition on a manifest status. +func setManifestStatusAppliedCondition( + manifestStatus *placementv1alpha1.PerManifestStatus, + applyRes ApplyResultType, + applyErr error, + inMemberClusterObjGeneration *int64, +) { + // If the in-cluster object generation is not available (i.e., there is no in-cluster object), use the default + // value 0. + observedGeneration := int64(0) + if inMemberClusterObjGeneration != nil { + observedGeneration = *inMemberClusterObjGeneration + } + + var appliedCond metav1.Condition + switch { + case applyRes == ApplyResTypeApplied: + // The manifest has been successfully applied. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + Message: ApplyResTypeAppliedDescription, + ObservedGeneration: observedGeneration, + } + case applyRes == ApplyResTypeAppliedWithFailedDriftDetection: + // The manifest has been successfully applied, but drift detection has failed. + // + // At this moment KubeFleet does not prepare a dedicated condition for drift detection outcomes. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeAppliedWithFailedDriftDetection), + Message: ApplyResTypeAppliedWithFailedDriftDetectionDescription, + ObservedGeneration: observedGeneration, + } + case !allApplyResTypes.Has(applyRes): + // Do a sanity check; verify if the returned result type is a valid one. + // Normally this branch should never run. + wrappedErr := errors.NewUnexpectedError(applyErr, "found an unexpected apply result type", + "manifestID", manifestStatus.Identifier, "applyRes", applyRes) + klog.ErrorS(wrappedErr, "Failed to set the Applied condition", errors.Args(wrappedErr)...) + // The work applier will consider this to be an apply failure. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToApply), + Message: fmt.Sprintf("An unexpected apply result is returned (%s) with error (%s)", applyRes, applyErr.Error()), + ObservedGeneration: observedGeneration, + } + default: + // The apply op fails. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(applyRes), + Message: fmt.Sprintf(ApplyResTypeFailedToApplyDescription, applyErr), + ObservedGeneration: observedGeneration, + } + } + + meta.SetStatusCondition(&manifestStatus.Conditions, appliedCond) +} + +// setManifestStatusAvailableCondition sets the Available condition on a manifest status. +func setManifestStatusAvailableCondition( + manifestStatus *placementv1alpha1.PerManifestStatus, + availabilityCheckRes AvailabilityCheckResultType, + availabilityCheckErr error, + inMemberClusterObjGeneration *int64, +) { + // If the in-cluster object generation is not available (i.e., there is no in-cluster object), use the default + // value 0. + observedGeneration := int64(0) + if inMemberClusterObjGeneration != nil { + observedGeneration = *inMemberClusterObjGeneration + } + + var availableCond *metav1.Condition + switch availabilityCheckRes { + case AvailabilityResultTypeSkipped: + // The availability check has been skipped as the manifest has not been applied yet; in this + // case no Available condition is set. + case AvailabilityResultTypeFailed: + // The availability check has failed. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeFailed), + Message: fmt.Sprintf(AvailabilityResultTypeFailedDescription, availabilityCheckErr), + ObservedGeneration: observedGeneration, + } + case AvailabilityResultTypeNotYetAvailable: + // The manifest is not yet available. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeNotYetAvailable), + Message: AvailabilityResultTypeNotYetAvailableDescription, + ObservedGeneration: observedGeneration, + } + case AvailabilityResultTypeNotTrackable: + // KubeFleet cannot track the availability of the manifest. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeNotTrackable), + Message: AvailabilityResultTypeNotTrackableDescription, + ObservedGeneration: observedGeneration, + } + default: + // The manifest is available. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + Message: AvailabilityResultTypeAvailableDescription, + ObservedGeneration: observedGeneration, + } + } + + if availableCond == nil { + // The conditions might be ported back from the previous run; a stale Available condition must be + // removed when no new one is set. + meta.RemoveStatusCondition(&manifestStatus.Conditions, placementv1alpha1.ManifestCondTypeAvailable) + return + } + meta.SetStatusCondition(&manifestStatus.Conditions, *availableCond) +} + +// setManifestStatusDiffDetails sets the diff details on a manifest status. +func setManifestStatusDiffDetails( + manifestStatus *placementv1alpha1.PerManifestStatus, + diffs []placementv1alpha1.PatchDetail, + observedInMemberClusterGeneration *int64, + now metav1.Time, +) { + // Check if a first diffed timestamp has been set; if not, set it to the current time. + firstDiffedTimestamp := now + if manifestStatus.DiffDetails != nil && !manifestStatus.DiffDetails.FirstDiffedObservedTimestamp.IsZero() { + firstDiffedTimestamp = manifestStatus.DiffDetails.FirstDiffedObservedTimestamp + } + + // Reset the diff details (such details need no port-back). + manifestStatus.DiffDetails = nil + if len(diffs) == 0 { + return + } + + // Populate the diff details. + manifestStatus.DiffDetails = &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: observedInMemberClusterGeneration, + FirstDiffedObservedTimestamp: firstDiffedTimestamp, + ObservedDiffs: diffs, + } +} + +// isAppliedObjectAvailable returns whether an availability check result type indicates that an +// applied manifest object is available. +func isAppliedObjectAvailable(availabilityCheckRes AvailabilityCheckResultType) bool { + return availabilityCheckRes == AvailabilityResultTypeAvailable || availabilityCheckRes == AvailabilityResultTypeNotTrackable +} + +// setWorkAppliedCondition sets the Applied condition on a work object status. +// +// A work object is considered to be applied if all of its manifests have been successfully applied. +func setWorkAppliedCondition( + workStatus *placementv1alpha1.WorkStatus, + manifestCount, appliedManifestCount int, + workGeneration int64, +) { + var appliedCond metav1.Condition + switch { + case appliedManifestCount == manifestCount: + // All manifests have been successfully applied. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + // Here KubeFleet reuses the same reason for individual manifests. + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + Message: fmt.Sprintf("All %d manifest(s) have been successfully applied", manifestCount), + ObservedGeneration: workGeneration, + } + default: + // Not all manifests have been successfully applied. + appliedCond = metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + Message: fmt.Sprintf("%d out of %d manifest(s) have not yet been applied", manifestCount-appliedManifestCount, manifestCount), + ObservedGeneration: workGeneration, + } + } + + meta.SetStatusCondition(&workStatus.Conditions, appliedCond) + klog.V(2).InfoS("Applied condition set on the work object status", + "appliedManifestCount", appliedManifestCount, "manifestCount", manifestCount) +} + +// setWorkAvailableCondition sets the Available condition on a work object status. +// +// A work object is considered to be available if all of its applied manifests are available. +func setWorkAvailableCondition( + workStatus *placementv1alpha1.WorkStatus, + manifestCount, availableManifestCount, untrackableAppliedObjectsCount int, + workGeneration int64, +) { + var availableCond *metav1.Condition + switch { + case availableManifestCount == manifestCount && untrackableAppliedObjectsCount == 0: + // All manifests are available. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + Message: fmt.Sprintf("All %d manifest(s) are available", manifestCount), + ObservedGeneration: workGeneration, + } + case availableManifestCount == manifestCount: + // Some manifests are not trackable. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsTrackableReason, + Message: fmt.Sprintf("All %d manifest(s) are available (with %d manifests untrackable)", manifestCount, untrackableAppliedObjectsCount), + ObservedGeneration: workGeneration, + } + default: + // Not all manifests are available. + availableCond = &metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + Message: fmt.Sprintf("%d out of %d manifest(s) are not yet available", manifestCount-availableManifestCount, manifestCount), + ObservedGeneration: workGeneration, + } + } + + meta.SetStatusCondition(&workStatus.Conditions, *availableCond) + klog.V(2).InfoS("Available condition set on the work object status", + "availableManifestCount", availableManifestCount, + "untrackableAppliedObjectsCount", untrackableAppliedObjectsCount, "manifestCount", manifestCount) +} + +func shouldSkipStatusUpdate(oldStatus, newStatus *placementv1alpha1.WorkStatus) bool { + // Skip status update if there is no change in the status. + return equality.Semantic.DeepEqual(oldStatus, newStatus) +} + +// refreshAppliedWorkStatus refreshes the status of appliedWork objects. +func (r *Reconciler) refreshAppliedWorkStatus( + ctx context.Context, + workObjProcessingStates []*workObjectProcessingState, +) error { + // Note (chenyu1): it is possible to run this method in parallel; however, for simplicity reasons, + // considering that in most of the time the count of work objects would be low, currently + // KubeFleet still does the status refresh sequentially. + for idx := range workObjProcessingStates { + workObjProcessingState := workObjProcessingStates[idx] + if err := r.refreshOneAppliedWorkStatus(ctx, workObjProcessingState); err != nil { + wrappedErr := errors.Wraps(err, "", "appliedWork", klog.KObj(workObjProcessingState.appliedWork)) + return wrappedErr + } + } + return nil +} + +func (r *Reconciler) refreshOneAppliedWorkStatus( + ctx context.Context, + workObjProcessingState *workObjectProcessingState, +) error { + appliedWork := workObjProcessingState.appliedWork + appliedWorkStatusCopy := appliedWork.Status.DeepCopy() + + manifestProcessingStates := workObjProcessingState.manifestProcessingStates + + // Note (chenyu1): It is possible to run this method in parallel; however, for simplicity reasons, + // considering that in most of the time the count of manifests would be low, currently + // KubeFleet still does the status refresh sequentially. + + // Pre-allocate the slice. + // + // Manifests that failed to get applied are not included in this list, hence + // empty length. + appliedResources := make([]placementv1alpha1.AppliedResource, 0, len(manifestProcessingStates)) + + // Build the list of applied resources. + for idx := range manifestProcessingStates { + manifestProcessingState := manifestProcessingStates[idx] + + if isManifestObjectApplied(manifestProcessingState.applyRes) { + appliedResources = append(appliedResources, placementv1alpha1.AppliedResource{ + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier(*manifestProcessingState.id), + UID: manifestProcessingState.inMemberClusterObj.GetUID(), + }) + } + } + + // Update the AppliedWork object status. + appliedWorkStatusCopy.AppliedResources = appliedResources + + // Skip the status update if no change found. + if equality.Semantic.DeepEqual(&appliedWork.Status, appliedWorkStatusCopy) { + klog.V(2).InfoS("No status change found for appliedWork object; skip the status update", "appliedWork", klog.KObj(appliedWork)) + } else { + klog.V(2).InfoS("Refreshing appliedWork object status", "appliedWork", klog.KObj(appliedWork)) + + appliedWork.Status = *appliedWorkStatusCopy + if err := r.spokeClient.Status().Update(ctx, appliedWork); err != nil { + klog.ErrorS(err, "Failed to update appliedWork status", + "appliedWork", klog.KObj(appliedWork)) + return errors.NewAPIServerError(err, "failed to update appliedWork object status", false, "appliedWork", klog.KObj(appliedWork)) + } + } + return nil +} diff --git a/pkg/v1/controllers/workapplier/suite_test.go b/pkg/v1/controllers/workapplier/suite_test.go new file mode 100644 index 000000000..c1b50c93c --- /dev/null +++ b/pkg/v1/controllers/workapplier/suite_test.go @@ -0,0 +1,367 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "flag" + "log" + "os" + "path/filepath" + "sync" + "testing" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + "k8s.io/utils/ptr" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/cache" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/config" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/log/zap" + "sigs.k8s.io/controller-runtime/pkg/manager" + "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/parallelizer" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" + testv1alpha1 "go.goms.io/fleet/test/apis/v1alpha1" +) + +// These tests use Ginkgo (BDD-style Go testing framework). Refer to +// http://onsi.github.io/ginkgo/ to learn more about Ginkgo. +var ( + hubCfg *rest.Config + hubEnv *envtest.Environment + hubClient client.Client + + memberCfg1 *rest.Config + memberEnv1 *envtest.Environment + hubMgr1 manager.Manager + memberClient1 client.Client + memberDynamicClient1 dynamic.Interface + workApplier1 *Reconciler + + // A work applier dedicated to the test cases that change its lifecycle (i.e., leave and re-join the fleet), so + // that such changes never affect other test cases. It watches its own reserved namespace in the hub cluster, and + // shares the member cluster (memberEnv1) with workApplier1. + hubMgrLifecycle manager.Manager + workApplierLifecycle *Reconciler + + // A work applier dedicated to the wave test cases. Its parallelizer pauses for a fixed delay after processing + // each wave of manifests, so that the order of the waves shows in the creation timestamps of the applied objects; + // its hub cluster client records the status updates on work objects. It watches its own reserved namespace in the + // hub cluster, and shares the member cluster (memberEnv1) with workApplier1. + hubMgrWaves manager.Manager + workApplierWaves *Reconciler + hubClientRecorderWaves *recordingClient + + // A work applier dedicated to the requeue and status update test cases. Its hub/member cluster clients record the + // start of each reconciliation and the status updates on work/appliedWork objects, and it uses a requeue rate + // limiter with short delays (see instrumentedRequeueRateLimiter). It watches its own reserved namespace in the hub + // cluster, and shares the member cluster (memberEnv1) with workApplier1. + hubMgrInstrumented manager.Manager + workApplierInstrumented *Reconciler + hubClientRecorderInstrumented *recordingClient + memberClientRecorderInstrumented *recordingClient + + ctx context.Context + cancel context.CancelFunc + wg sync.WaitGroup +) + +const ( + // The number of max. concurrent reconciliations for the work applier controller. + maxConcurrentReconciles = 5 + // The count of workers for the work applier controller. + workerCount = 4 + + // The max. time the work applier waits for an appliedWork object to be deleted before it unblocks + // the deletion manually. + // + // Note that envtest runs no GC controller; foreground deletions of appliedWork objects never complete + // in the test environments, and the work applier will always wait for 2 * cleanupWaitTime before + // it removes the cleanup finalizers. A short wait time is used here to keep the tests fast. + cleanupWaitTime = time.Second * 5 + + memberReservedNSName1 = "fleet-member-experimental-1" + // The reserved namespace for the work applier dedicated to the lifecycle test cases. + memberReservedNSNameLifecycle = "fleet-member-experimental-lifecycle" + // The reserved namespace for the work applier dedicated to the wave test cases. + memberReservedNSNameWaves = "fleet-member-experimental-waves" + // The reserved namespace for the work applier dedicated to the requeue and status update test cases. + memberReservedNSNameInstrumented = "fleet-member-experimental-instrumented" + + // The delay that the parallelizer of the work applier for the wave test cases adds after processing each wave of + // manifests. Creation timestamps are of the second granularity; the delay must be at least two seconds for + // consecutive waves to be told apart reliably. + parallelizerFixedDelay = time.Second * 3 +) + +// instrumentedRequeueRateLimiter returns the requeue rate limiter for the work applier dedicated to the requeue test +// cases. It requeues after 2 seconds once, then backs off slowly (by a factor of 1.5, from 2 seconds up to 4 seconds), +// then backs off fast (by a factor of 2, up to 8 seconds); it skips to the fast backoff stage when all the applied +// manifests are available. In other words, the requeue delays are 2s, 2s, 3s, 6s, 8s, 8s, ... if some manifests are +// not available, and 2s, 2s, 4s, 8s, 8s, ... if all the manifests are available. +func instrumentedRequeueRateLimiter() *MultiStageRequeueWithExponentialBackoffRateLimiter { + rateLimiter, err := NewMultiStageRequeueWithExponentialBackoffRateLimiter(1, 2, 1.5, 2, 4, 2, 8, true) + Expect(err).ToNot(HaveOccurred(), "Failed to create the requeue rate limiter for the instrumented work applier") + return rateLimiter +} + +const ( + // The environment variable that the CI pipeline sets to tell which runner invokes the tests. + testRunnerNameEnvVarName = "KUBEFLEET_CI_TEST_RUNNER_NAME" + // The runner (`go test`, via `make test`) with which the tests are skipped in the CI pipeline; the pipeline runs + // this package in a separate step with the Ginkgo CLI (in parallel) instead. + runnerNameToSkipTestsInCI = "default" +) + +func TestMain(m *testing.M) { + // Skip the tests if in the CI environment the tests are invoked with `go test` instead of the Ginkgo CLI. + // This has no effect outside the CI environment. + if v := os.Getenv(testRunnerNameEnvVarName); v == runnerNameToSkipTestsInCI { + log.Println("Skipping the tests in CI as they are not run with the expected runner") + os.Exit(0) + } + + os.Exit(m.Run()) +} + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Work Applier Integration Test Suite") +} + +func setupResources() { + for _, nsName := range []string{memberReservedNSName1, memberReservedNSNameLifecycle, memberReservedNSNameWaves, memberReservedNSNameInstrumented} { + ns := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: nsName, + }, + } + Expect(hubClient.Create(ctx, ns)).To(Succeed(), "Failed to create the reserved namespace %s", nsName) + } +} + +// Note: each Ginkgo process must do the same setup; unlike our E2E tests, the integration +// tests uses in-memory testing environments, and as a result cannot be shared across processes. +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + logger := zap.New(zap.WriteTo(GinkgoWriter), zap.UseDevMode(true)) + klog.SetLogger(logger) + ctrl.SetLogger(logger) + + By("Bootstrapping test environments") + hubEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{ + filepath.Join("../../../../", "config", "crd", "bases"), + filepath.Join("../../../../", "test", "manifests"), + }, + ErrorIfCRDPathMissing: true, + } + // memberEnv1 is the test environment for verifying most work applier behaviors. + memberEnv1 = &envtest.Environment{ + CRDDirectoryPaths: []string{ + filepath.Join("../../../../", "config", "crd", "bases"), + filepath.Join("../../../../", "test", "manifests"), + }, + ErrorIfCRDPathMissing: true, + } + + var err error + hubCfg, err = hubEnv.Start() + Expect(err).ToNot(HaveOccurred()) + Expect(hubCfg).ToNot(BeNil()) + + memberCfg1, err = memberEnv1.Start() + Expect(err).ToNot(HaveOccurred()) + Expect(memberCfg1).ToNot(BeNil()) + + Expect(placementv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed()) + Expect(testv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed()) + + By("Building the K8s clients") + hubClient, err = client.New(hubCfg, client.Options{Scheme: scheme.Scheme}) + Expect(err).ToNot(HaveOccurred()) + Expect(hubClient).ToNot(BeNil()) + + memberClient1, err = client.New(memberCfg1, client.Options{Scheme: scheme.Scheme}) + Expect(err).ToNot(HaveOccurred()) + Expect(memberClient1).ToNot(BeNil()) + + memberDynamicClient1, err = dynamic.NewForConfig(memberCfg1) + Expect(err).ToNot(HaveOccurred()) + + By("Setting up the resources") + setupResources() + + By("Setting up the controller and the controller manager for member cluster 1") + hubMgr1 = newHubManager(memberReservedNSName1, false) + workApplier1 = New( + memberReservedNSName1, + // Use the cached client from the manager, as the work applier relies on the field indexes. + hubMgr1.GetClient(), + hubMgr1.GetAPIReader(), + memberClient1, + memberDynamicClient1, + memberClient1.RESTMapper(), + parallelizer.NewParallelizer(workerCount), + maxConcurrentReconciles, + workqueue.NewTypedItemExponentialFailureRateLimiter[types.NamespacedName](DefaultCleanupBaseDelay, DefaultCleanupMaxDelay), + defaultRequeueRateLimiter, + cleanupWaitTime, + ) + Expect(workApplier1.SetupWithManager(hubMgr1)).To(Succeed()) + + // The work appliers below all share the member cluster with workApplier1, and watch their own reserved + // namespaces in the hub cluster. The requeue rate limiter tracks work objects by their namespaces and names; it is + // safe to share the default one, as the work appliers watch different namespaces. + + By("Setting up the controller and the controller manager for the lifecycle test cases") + hubMgrLifecycle = newHubManager(memberReservedNSNameLifecycle, true) + workApplierLifecycle = New( + memberReservedNSNameLifecycle, + hubMgrLifecycle.GetClient(), + hubMgrLifecycle.GetAPIReader(), + memberClient1, + memberDynamicClient1, + memberClient1.RESTMapper(), + parallelizer.NewParallelizer(workerCount), + maxConcurrentReconciles, + workqueue.NewTypedItemExponentialFailureRateLimiter[types.NamespacedName](DefaultCleanupBaseDelay, DefaultCleanupMaxDelay), + defaultRequeueRateLimiter, + cleanupWaitTime, + ) + Expect(workApplierLifecycle.SetupWithManager(hubMgrLifecycle)).To(Succeed()) + + By("Setting up the controller and the controller manager for the wave test cases") + hubMgrWaves = newHubManager(memberReservedNSNameWaves, true) + hubClientRecorderWaves = newRecordingClient(hubMgrWaves.GetClient()) + workApplierWaves = New( + memberReservedNSNameWaves, + hubClientRecorderWaves, + hubMgrWaves.GetAPIReader(), + memberClient1, + memberDynamicClient1, + memberClient1.RESTMapper(), + ¶llelizerWithFixedDelay{ + Parallelizer: parallelizer.NewParallelizer(workerCount), + delay: parallelizerFixedDelay, + }, + maxConcurrentReconciles, + workqueue.NewTypedItemExponentialFailureRateLimiter[types.NamespacedName](DefaultCleanupBaseDelay, DefaultCleanupMaxDelay), + defaultRequeueRateLimiter, + cleanupWaitTime, + ) + Expect(workApplierWaves.SetupWithManager(hubMgrWaves)).To(Succeed()) + + By("Setting up the controller and the controller manager for the requeue and status update test cases") + hubMgrInstrumented = newHubManager(memberReservedNSNameInstrumented, true) + hubClientRecorderInstrumented = newRecordingClient(hubMgrInstrumented.GetClient()) + memberClientRecorderInstrumented = newRecordingClient(memberClient1) + workApplierInstrumented = New( + memberReservedNSNameInstrumented, + hubClientRecorderInstrumented, + hubMgrInstrumented.GetAPIReader(), + memberClientRecorderInstrumented, + memberDynamicClient1, + memberClient1.RESTMapper(), + parallelizer.NewParallelizer(workerCount), + maxConcurrentReconciles, + workqueue.NewTypedItemExponentialFailureRateLimiter[types.NamespacedName](DefaultCleanupBaseDelay, DefaultCleanupMaxDelay), + instrumentedRequeueRateLimiter(), + cleanupWaitTime, + ) + Expect(workApplierInstrumented.SetupWithManager(hubMgrInstrumented)).To(Succeed()) + + wg = sync.WaitGroup{} + for _, pair := range []struct { + mgr manager.Manager + workApplier *Reconciler + }{ + {hubMgr1, workApplier1}, + {hubMgrLifecycle, workApplierLifecycle}, + {hubMgrWaves, workApplierWaves}, + {hubMgrInstrumented, workApplierInstrumented}, + } { + wg.Add(1) + go func() { + defer GinkgoRecover() + defer wg.Done() + Expect(pair.workApplier.Join(ctx)).To(Succeed()) + Expect(pair.mgr.Start(ctx)).To(Succeed()) + }() + } +}) + +// newHubManager creates a controller manager for the hub cluster, whose cache covers only the given reserved +// namespace, and sets up the field indexes that the work applier needs. +// +// The work applier always registers its controller under the same name; for all but the first work applier in the +// test environment, the controller name uniqueness check must be skipped. +func newHubManager(reservedNSName string, skipControllerNameValidation bool) manager.Manager { + mgr, err := ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: server.Options{ + BindAddress: "0", + }, + Cache: cache.Options{ + DefaultNamespaces: map[string]cache.Config{ + reservedNSName: {}, + }, + }, + Controller: config.Controller{ + SkipNameValidation: ptr.To(skipControllerNameValidation), + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).ToNot(HaveOccurred(), "Failed to create the controller manager for reserved namespace %s", reservedNSName) + + // The work applier lists work objects via custom field indexes; the indexes must be set up + // (with the manager's cache) before the manager starts. + Expect(fieldindexers.SetupWithMemberAgentControllerManager(ctx, mgr)).To(Succeed(), "Failed to set up the field indexes for reserved namespace %s", reservedNSName) + return mgr +} + +var _ = AfterSuite(func() { + defer klog.Flush() + + cancel() + wg.Wait() + By("Tearing down the test environment") + Expect(hubEnv.Stop()).To(Succeed()) + Expect(memberEnv1.Stop()).To(Succeed()) +}) diff --git a/pkg/v1/controllers/workapplier/takeover_drift_detection.go b/pkg/v1/controllers/workapplier/takeover_drift_detection.go new file mode 100644 index 000000000..aaa4d9c8a --- /dev/null +++ b/pkg/v1/controllers/workapplier/takeover_drift_detection.go @@ -0,0 +1,583 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "fmt" + "strings" + + "github.com/qri-io/jsonpointer" + "github.com/wI2L/jsondiff" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfioplacementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/ownerreferences" +) + +const ( + patchDetailPerObjLimit = 100 +) + +const ( + k8sReservedLabelAnnotationFullDomain = "kubernetes.io/" + k8sReservedLabelAnnotationAbbrDomain = "k8s.io/" + kubefleetReservedLabelAnnotationDomain = "kubefleet.dev/" +) + +// isOwnedByOlderKubeFleetAPIObjects checks if the object in the member cluster is owned by older KubeFleet API +// objects, specifically the appliedWork objects in the `placement.kubernetes-fleet.io` API group. +// +// This is added as a safeguard to avoid scenarios where a manifest is applied via both the newer PlacementPolicy +// APIs and the older ResourcePlacement APIs. And the safeguard always runs regardless of the sync strategy in use. +// +// Owner references are matched by API group and kind only, as owner references are not migrated when an API +// version changes; an object might carry an owner reference of any version that the API group has ever served +// (e.g., the `placement.kubernetes-fleet.io` API group serves AppliedWork objects in both v1 and v1beta1). +func isOwnedByOlderKubeFleetAPIObjects(inMemberClusterObj *unstructured.Unstructured) bool { + if inMemberClusterObj == nil { + return false + } + + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + owner := &curOwners[idx] + gv, err := schema.ParseGroupVersion(owner.APIVersion) + if err != nil { + // The API server rejects owner references with unparsable API versions; normally this should never + // happen. + continue + } + if gv.Group == kfioplacementv1beta1.GroupVersion.Group && owner.Kind == kfioplacementv1beta1.AppliedWorkKind { + return true + } + } + return false +} + +// shouldInitiateTakeOverAttempt checks if KubeFleet should initiate the takeover process for an object. +// +// A takeover process is initiated when: +// +// - An object that matches with the given manifest has been created; but +// - The object is not owned by KubeFleet (more specifically, the object is not owned by the +// expected AppliedWork object). +func shouldInitiateTakeOverAttempt(manifestProcessingState *manifestProcessingState) bool { + inMemberClusterObj := manifestProcessingState.inMemberClusterObj + if inMemberClusterObj == nil { + // Obviously, if the corresponding live object is not found, no takeover is + // needed. + return false + } + + // Skip the takeover process if the sync strategy forbids so. + if manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy.WhenAlreadyExists == placementv1alpha1.WhenAlreadyExistsOptionReportError { + return false + } + + // Check if the live object is owned by KubeFleet. + curOwners := inMemberClusterObj.GetOwnerReferences() + for idx := range curOwners { + if ownerreferences.AreEqual(&curOwners[idx], manifestProcessingState.ownedBy) { + // The live object is owned by KubeFleet; no takeover is needed. + return false + } + } + return true +} + +// takeOverPreExistingObject takes over a pre-existing object in the member cluster. +func (r *Reconciler) takeOverPreExistingObject( + ctx context.Context, + manifestProcessingState *manifestProcessingState, +) (*unstructured.Unstructured, []placementv1alpha1.PatchDetail, bool, error) { + gvr := manifestProcessingState.gvr + syncStrategy := manifestProcessingState.fromPrimaryWorkObject.Spec.SyncStrategy + expectedAppliedWorkOwnerRef := manifestProcessingState.ownedBy + + inMemberClusterObjCopy := manifestProcessingState.inMemberClusterObj.DeepCopy() + existingOwnerRefs := inMemberClusterObjCopy.GetOwnerReferences() + + // At this moment KubeFleet is set to leave applied resources in the member cluster if the cluster + // decides to leave its fleet; when this happens, the resources will be owned by an AppliedWork + // object that might not have a corresponding Work object anymore as the cluster might have + // been de-selected. If the cluster decides to re-join the fleet, and the same set of manifests + // are being applied again, one may encounter unexpected errors due to the presence of the + // left behind AppliedWork object. To address this issue, KubeFleet here will perform a cleanup, + // removing any owner reference that points to an orphaned AppliedWork object. + existingOwnerRefs, err := r.removeLeftBehindAppliedWorkOwnerRefs(ctx, existingOwnerRefs) + if err != nil { + return nil, nil, false, errors.Wraps(err, "failed to remove left-behind AppliedWork owner references") + } + + // Check this object is already owned by another object (or controller); if so, KubeFleet will only + // add itself as an additional owner if co-ownership is allowed. + if len(existingOwnerRefs) >= 1 && syncStrategy.WhenOwnedByOthers != placementv1alpha1.WhenOwnedByOthersOptionShareOwnership { + // The object is already owned by another object, and co-ownership is forbidden. + // No takeover will be performed. + // + // Note that this will be registered as an (apply) error. + return nil, nil, false, errors.NewUserError(nil, + "the object is already owned by some other source(s) and co-ownership is disallowed", + "existingOwnerRefs", existingOwnerRefs) + } + + // Check if the object is already owned by KubeFleet, but the owner is a different AppliedWork + // object, i.e., the object has been placed in duplicate. + // + // Originally KubeFleet does allow placing the same object multiple times; each placement + // attempt might have its own object spec (envelopes might be used and in each envelope + // the object looks different). This would lead to the object being constantly overwritten, + // yet no error would be raised on the user-end. With the drift detection feature now added to KubeFleet, however, + // this scenario would repeatedly trigger drift-related alarms, which could lead to user confusion. + // To address this corner case, KubeFleet would now refuse to place the same object twice. + if isPlacedByKubeFleetInDuplicate(existingOwnerRefs, expectedAppliedWorkOwnerRef) { + return nil, nil, false, errors.NewUserError(nil, "the object is already owned by another KubeFleet AppliedWork object") + } + + // Check if the takeover action requires additional steps (configuration difference inspection). + if syncStrategy.WhenAlreadyExists == placementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff { + configDiffs, diffCalculatedInDegradedMode, err := r.diffBetweenManifestAndInMemberClusterObjects(ctx, + gvr, manifestProcessingState.manifestObj, inMemberClusterObjCopy, syncStrategy.ComparisonOption) + switch { + case err != nil: + return nil, nil, false, errors.Wraps(err, "failed to calculate configuration diffs between the manifest object and the object from the member cluster") + case len(configDiffs) > 0: + return nil, configDiffs, diffCalculatedInDegradedMode, nil + } + } + + // Take over the object. + updatedOwnerRefs := append(existingOwnerRefs, *expectedAppliedWorkOwnerRef) + inMemberClusterObjCopy.SetOwnerReferences(updatedOwnerRefs) + takenOverInMemberClusterObj, err := r.spokeDynamicClient. + Resource(*gvr).Namespace(inMemberClusterObjCopy.GetNamespace()). + Update(ctx, inMemberClusterObjCopy, metav1.UpdateOptions{}) + if err != nil { + // false as the dynamic client is non-caching. + return nil, nil, false, errors.NewAPIServerError(err, "failed to take over the object", false) + } + + return takenOverInMemberClusterObj, nil, false, nil +} + +// removeLeftBehindAppliedWorkOwnerRefs removes owner references that point to orphaned AppliedWork objects. +func (r *Reconciler) removeLeftBehindAppliedWorkOwnerRefs(ctx context.Context, ownerRefs []metav1.OwnerReference) ([]metav1.OwnerReference, error) { + updatedOwnerRefs := make([]metav1.OwnerReference, 0, len(ownerRefs)) + for idx := range ownerRefs { + ownerRef := ownerRefs[idx] + if ownerRef.APIVersion != placementv1alpha1.GroupVersion.String() || ownerRef.Kind != placementv1alpha1.AppliedWorkKind { + // Skip non-appliedWork owner references. + updatedOwnerRefs = append(updatedOwnerRefs, ownerRef) + continue + } + + // Check if the appliedWork object has a corresponding work object. + workObj := &placementv1alpha1.Work{} + err := r.hubClient.Get(ctx, types.NamespacedName{Namespace: r.workNSName, Name: ownerRef.Name}, workObj) + switch { + case err != nil && !apierrors.IsNotFound(err): + // An unexpected error occurred. + return nil, errors.NewAPIServerError(err, "failed to retrieve the work object", true, + "workNamespace", r.workNSName, "workName", ownerRef.Name) + case err == nil: + // The appliedWork owner reference is valid; no need for removal. + // + // Note that no UID check is performed here; KubeFleet can (and will) reuse the same appliedWork + // as long as it has the same name as a work object, even if the appliedWork object is not + // originally derived from it. This is safe as the appliedWork object is in essence a delegate + // and does not keep any additional information. + updatedOwnerRefs = append(updatedOwnerRefs, ownerRef) + continue + default: + // The appliedWork owner reference is invalid; the work object does not exist. + // + // Remove the owner reference. + klog.V(2).InfoS("Found an owner reference that points to an orphaned AppliedWork object", "ownerRef", ownerRef) + continue + } + } + + return updatedOwnerRefs, nil +} + +// isPlacedByKubeFleetInDuplicate checks if the object has already been placed by KubeFleet via another +// placement policy. +func isPlacedByKubeFleetInDuplicate(ownerRefs []metav1.OwnerReference, expectedAppliedWorkOwnerRef *metav1.OwnerReference) bool { + for idx := range ownerRefs { + ownerRef := ownerRefs[idx] + if ownerRef.APIVersion == placementv1alpha1.GroupVersion.String() && + ownerRef.Kind == placementv1alpha1.AppliedWorkKind && + ownerRef.UID != expectedAppliedWorkOwnerRef.UID { + return true + } + } + return false +} + +// diffBetweenManifestAndInMemberClusterObjects calculates the differences between the manifest object +// and its corresponding object in the member cluster. +func (r *Reconciler) diffBetweenManifestAndInMemberClusterObjects( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, + cmpOption placementv1alpha1.ComparisonOption, +) ([]placementv1alpha1.PatchDetail, bool, error) { + switch cmpOption { + case placementv1alpha1.ComparisonOptionPartialComparison: + return r.partialDiffBetweenManifestAndInMemberClusterObjects(ctx, gvr, manifestObj, inMemberClusterObj) + case placementv1alpha1.ComparisonOptionFullComparison: + // For the full comparison, KubeFleet compares directly the JSON representations of the + // manifest object and the object in the member cluster. + patchDetails, err := preparePatchDetails(manifestObj, inMemberClusterObj) + return patchDetails, false, err + default: + return nil, false, errors.NewUnexpectedError(nil, "an invalid comparison option is specified", "comparisonOption", cmpOption) + } +} + +// partialDiffBetweenManifestAndInMemberClusterObjects calculates the differences between the +// manifest object and its corresponding object in the member cluster by performing a dry-run +// apply op; this would ignore differences in the unmanaged fields and report only those +// in the managed fields. +func (r *Reconciler) partialDiffBetweenManifestAndInMemberClusterObjects( + ctx context.Context, + gvr *schema.GroupVersionResource, + manifestObj, inMemberClusterObj *unstructured.Unstructured, +) ([]placementv1alpha1.PatchDetail, bool, error) { + // KubeFleet calculates the partial diff between two objects by running apply ops in the dry-run + // mode. + appliedObj, err := r.applyInDryRunMode(ctx, gvr, manifestObj, inMemberClusterObj) + + // After the dry-run apply op, all the managed fields should have been overwritten using the + // values from the manifest object, while leaving all the unmanaged fields untouched. This + // would allow KubeFleet to compare the object returned by the dry-run apply op with the object + // that is currently in the member cluster; if all the fields are consistent, it is safe + // for us to assume that there are no drifts, otherwise, any fields that are different + // imply that running an actual apply op would lead to unexpected changes, which signifies + // the presence of partial drifts (drifts in managed fields). + + switch { + case err == nil: + // The dry-run apply op has succeeded. All managed fields should have been overwritten using the + // values from the manifest object, while leaving all the unmanaged fields untouched. This + // would allow KubeFleet to compare the object returned by the dry-run apply op with the object + // that is currently in the member cluster; if all the fields are consistent, it is safe + // for us to assume that there are no drifts, otherwise, any fields that are different + // imply that running an actual apply op would lead to unexpected changes, which signifies + // the presence of partial drifts (drifts in managed fields). + patchDetails, err := preparePatchDetails(appliedObj, inMemberClusterObj) + return patchDetails, false, err + case apierrors.IsInvalid(err): + // The dry-run apply op has failed as the manifest object provided is not valid. This could + // happen when the apply op involves fields that are immutable or the apply op attempts to + // set invalid values. This error implies that the in-cluster object has already been modified, + // and any change that the user supplies right now cannot be accepted. + // + // In this case, fall back to full comparison. Report that the diff is being calculated in a + // degraded manner. + // + // This is not considered as a diff calculation error. + klog.V(2).InfoS("Calculate diffs in degraded mode as the manifest object cannot be server-side applied in dry-run mode", + "gvr", gvr, "manifestObj", klog.KObj(manifestObj), "serverErr", err) + patchDetails, err := preparePatchDetails(manifestObj, inMemberClusterObj) + return patchDetails, true, err + default: + // An unexpected error has occurred. + return nil, false, errors.Wraps(err, "failed to apply the manifest in dry-run mode") + } +} + +// preparePatchDetails calculates the differences between two objects in the form +// of KubeFleet patch details. +func preparePatchDetails(srcObj, destObj *unstructured.Unstructured) ([]placementv1alpha1.PatchDetail, error) { + // Discard certain fields from both objects before comparison. + srcObjCopy := discardFieldsIrrelevantInComparisonFrom(srcObj) + destObjCopy := discardFieldsIrrelevantInComparisonFrom(destObj) + + // Marshal the objects into JSON. + srcObjJSONBytes, err := srcObjCopy.MarshalJSON() + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to marshal the source object into JSON") + } + + destObjJSONBytes, err := destObjCopy.MarshalJSON() + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to marshal the destination object into JSON") + } + + // Compare the JSON representations. + patch, err := jsondiff.CompareJSON(srcObjJSONBytes, destObjJSONBytes) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to compare the JSON representations of the source and destination objects") + } + + // Prepare KubeFleet patch details from the JSON patches. + details, err := organizeJSONPatchIntoPatchDetails(patch, srcObjCopy.Object) + if err != nil { + return nil, errors.Wraps(err, "failed to organize JSON patch operations into patch details") + } + + // Obscure sensitive fields in the patch details. + // + // This currently only concerns Secret objects (core API group); all the drift/diff outputs regarding + // a Secret object's data (`.data` or `.stringData` fields) are obscured. + details = obscureSensitiveFieldsInPatchDetails(srcObj, details) + return details, nil +} + +// discardFieldsIrrelevantInComparisonFrom discards fields that are irrelevant when comparing +// the manifest and live objects (or two manifest objects). +// +// Note that this method will return an object copy; the original object will be left untouched. +func discardFieldsIrrelevantInComparisonFrom(obj *unstructured.Unstructured) *unstructured.Unstructured { + // Create a deep copy of the object. + objCopy := obj.DeepCopy() + + // Remove object meta fields that are irrelevant in comparison. + + // Clear out the object's name/namespace. For regular objects, the names/namespaces will + // always be same between the manifest and the live objects, as guaranteed by the object + // retrieval step earlier, so a comparison on these fields is unnecessary anyway. + objCopy.SetName("") + objCopy.SetNamespace("") + + // Clear out the object's generate name. This is a field that is irrelevant in comparison. + objCopy.SetGenerateName("") + + // Remove certain labels and annotations. + // + // KubeFleet will remove labels/annotations that are reserved for KubeFleet own use cases, plus + // well-known Kubernetes labels and annotations, as these cannot (should not) be set by users + // directly. + annotations := objCopy.GetAnnotations() + cleanedAnnotations := map[string]string{} + for k, v := range annotations { + if strings.Contains(k, k8sReservedLabelAnnotationFullDomain) { + // Skip Kubernetes reserved annotations. + continue + } + + if strings.Contains(k, k8sReservedLabelAnnotationAbbrDomain) { + // Skip Kubernetes reserved annotations. + continue + } + + if strings.Contains(k, kubefleetReservedLabelAnnotationDomain) { + // Skip KubeFleet reserved annotations. + continue + } + cleanedAnnotations[k] = v + } + objCopy.SetAnnotations(cleanedAnnotations) + + labels := objCopy.GetLabels() + cleanedLabels := map[string]string{} + for k, v := range labels { + if strings.Contains(k, k8sReservedLabelAnnotationFullDomain) { + // Skip Kubernetes reserved labels. + continue + } + + if strings.Contains(k, k8sReservedLabelAnnotationAbbrDomain) { + // Skip Kubernetes reserved labels. + continue + } + + if strings.Contains(k, kubefleetReservedLabelAnnotationDomain) { + // Skip KubeFleet reserved labels. + continue + } + cleanedLabels[k] = v + } + objCopy.SetLabels(cleanedLabels) + + // Fields below are system-reserved fields in object meta. Technically speaking they can be + // set in the manifests, but this is a very uncommon practice, and currently KubeFleet will clear + // these fields (except for the finalizers) before applying the manifests. + // As a result, for now KubeFleet will ignore them in the comparison process as well. + // + // TO-DO (chenyu1): evaluate if this is a correct assumption for most (if not all) KubeFleet + // users. + objCopy.SetFinalizers([]string{}) + objCopy.SetManagedFields([]metav1.ManagedFieldsEntry{}) + objCopy.SetOwnerReferences([]metav1.OwnerReference{}) + + // Fields below are read-only fields in object meta. KubeFleet will ignore them in the comparison + // process. + // Deleted objects are handled separately in the apply process; for comparison purposes, + // KubeFleet will ignore the deletion timestamp and grace period seconds. + objCopy.SetDeletionTimestamp(nil) + objCopy.SetDeletionGracePeriodSeconds(nil) + objCopy.SetGeneration(0) + objCopy.SetResourceVersion("") + objCopy.SetSelfLink("") + objCopy.SetUID("") + + // Remove the status field. + unstructured.RemoveNestedField(objCopy.Object, "status") + + // Remove all creationTimestamp fields. + unstructured.RemoveNestedField(objCopy.Object, "metadata", "creationTimestamp") + // Resources that have .spec.template.metadata include: Deployment, Job, StatefulSet, + // DaemonSet, ReplicaSet, and CronJob. + unstructured.RemoveNestedField(objCopy.Object, "spec", "template", "metadata", "creationTimestamp") + // Also handle CronJob's .spec.jobTemplate.spec.template.metadata + unstructured.RemoveNestedField(objCopy.Object, "spec", "jobTemplate", "spec", "template", "metadata", "creationTimestamp") + + return objCopy +} + +// organizeJSONPatchIntoPatchDetails organizes the JSON patch operations into KubeFleet patch details. +func organizeJSONPatchIntoPatchDetails(patch jsondiff.Patch, manifestObjMap map[string]interface{}) ([]placementv1alpha1.PatchDetail, error) { + // Pre-allocate the slice for the patch details. The organization procedure typically will yield + // the same number of PatchDetail items as the JSON patch operations. + details := make([]placementv1alpha1.PatchDetail, 0, len(patch)) + + // A side note: here KubeFleet takes an expedient approach processing null JSON paths, by treating + // null paths as empty strings. + + // Process only the first 100 ops. + // TO-DO (chenyu1): Impose additional size limits. + for idx := 0; idx < len(patch) && idx < patchDetailPerObjLimit; idx++ { + op := patch[idx] + pathPtr, err := jsonpointer.Parse(op.Path) + if err != nil { + // An invalid path is found; normally this should not happen. + return nil, errors.NewUnexpectedError(err, "failed to parse the JSON path", "path", op.Path) + } + fromPtr, err := jsonpointer.Parse(op.From) + if err != nil { + // An invalid path is found; normally this should not happen. + return nil, errors.NewUnexpectedError(err, "failed to parse the JSON path", "path", op.From) + } + + switch op.Type { + case jsondiff.OperationAdd: + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInMember: fmt.Sprint(op.Value), + }) + case jsondiff.OperationRemove: + // KubeFleet here skips validation as the JSON data is just marshalled. + hubValue, err := pathPtr.Eval(manifestObjMap) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to evaluate the JSON path in the manifest object", "path", op.Path) + } + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInHub: fmt.Sprintf("%v", hubValue), + }) + case jsondiff.OperationReplace: + // KubeFleet here skips validation as the JSON data is just marshalled. + hubValue, err := pathPtr.Eval(manifestObjMap) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to evaluate the JSON path in the manifest object", "path", op.Path) + } + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInMember: fmt.Sprint(op.Value), + ValueInHub: fmt.Sprintf("%v", hubValue), + }) + case jsondiff.OperationMove: + // Normally the Move operation will not be returned as factorization is disabled + // for the JSON patch calculation process; however, KubeFleet here still processes them + // just in case. + // + // Each Move operation will be parsed into two separate operations. + hubValue, err := fromPtr.Eval(manifestObjMap) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to evaluate the JSON path in the manifest object", "path", op.From) + } + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.From, + ValueInHub: fmt.Sprintf("%v", hubValue), + }) + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInMember: fmt.Sprintf("%v", hubValue), + }) + case jsondiff.OperationCopy: + // Normally the Copy operation will not be returned as factorization is disabled + // for the JSON patch calculation process; however, KubeFleet here still processes them + // just in case. + // + // Each Copy operation will be parsed into an Add operation. + hubValue, err := fromPtr.Eval(manifestObjMap) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to evaluate the JSON path in the manifest object", "path", op.From) + } + details = append(details, placementv1alpha1.PatchDetail{ + Path: op.Path, + ValueInMember: fmt.Sprintf("%v", hubValue), + }) + case jsondiff.OperationTest: + // The Test op is a no-op in KubeFleet's use case. Normally it will not be returned, either. + default: + // An unexpected op is returned. + return nil, errors.NewUnexpectedError(nil, "an unexpected JSON patch operation is returned", "op", op) + } + } + + return details, nil +} + +// obscureSensitiveFieldsInPatchDetails obscures sensitive fields from the patch details so that +// such information will not be included in the drift/diff outputs. +// +// At this moment fields in the following API objects are discarded: +// +// * `.data` and `.stringData` field (and their children) in all Secret objects (`core` API group). +// +// Note (chenyu1): there are other Kubernetes API objects that also feature sensitive data, +// such as TokenRequest and CertificateSigningRequest; these objects are not included in the list +// as they have the sensitive information in the status, which are not accounted for in the +// drift/diff calculation in the very beginning. +func obscureSensitiveFieldsInPatchDetails( + srcObj *unstructured.Unstructured, details []placementv1alpha1.PatchDetail, +) (sanitizedPatchDetails []placementv1alpha1.PatchDetail) { + // Verify if the object is a Secret. + if srcObj.GetAPIVersion() != "v1" || srcObj.GetKind() != "Secret" { + return details + } + + for idx := range details { + pd := &details[idx] + // Note (chenyu1): the string data field in the Secret object is provided by Kubernetes + // as a write-only field for convenience reasons; all entries shall be merged into the + // data field. Here the code still processes the field just for completeness reasons; + // in practice it will never be included as part of the patch details. + if strings.HasPrefix(pd.Path, "/data") || strings.HasPrefix(pd.Path, "/stringData") { + // Obscure all patch details that concerns the Secret object's data. + + if len(pd.ValueInHub) > 0 { + pd.ValueInHub = "(redacted for security reasons)" + } + if len(pd.ValueInMember) > 0 { + pd.ValueInMember = "(redacted for security reasons)" + } + } + } + return details +} diff --git a/pkg/v1/controllers/workapplier/takeover_drift_detection_test.go b/pkg/v1/controllers/workapplier/takeover_drift_detection_test.go new file mode 100644 index 000000000..e7efb5e15 --- /dev/null +++ b/pkg/v1/controllers/workapplier/takeover_drift_detection_test.go @@ -0,0 +1,142 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "testing" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/utils/ptr" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfioplacementv1 "go.goms.io/fleet/apis/placement/v1" + kfioplacementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" +) + +// TestIsOwnedByOlderKubeFleetAPIObjects tests the isOwnedByOlderKubeFleetAPIObjects function. +func TestIsOwnedByOlderKubeFleetAPIObjects(t *testing.T) { + olderAPIAppliedWorkRef := metav1.OwnerReference{ + APIVersion: kfioplacementv1beta1.GroupVersion.String(), + Kind: kfioplacementv1beta1.AppliedWorkKind, + Name: "applied-work-old", + UID: "uid-applied-work-old", + BlockOwnerDeletion: ptr.To(true), + } + // The older KubeFleet API group serves AppliedWork objects in both v1 and v1beta1. + olderAPIV1AppliedWorkRef := metav1.OwnerReference{ + APIVersion: kfioplacementv1.GroupVersion.String(), + Kind: kfioplacementv1beta1.AppliedWorkKind, + Name: "applied-work-old-v1", + UID: "uid-applied-work-old-v1", + BlockOwnerDeletion: ptr.To(true), + } + olderAPIWorkRef := metav1.OwnerReference{ + APIVersion: kfioplacementv1beta1.GroupVersion.String(), + Kind: "Work", + Name: "work-old", + UID: "uid-work-old", + } + newerAPIAppliedWorkRef := metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.AppliedWorkKind, + Name: "applied-work-new", + UID: "uid-applied-work-new", + BlockOwnerDeletion: ptr.To(true), + } + nonKubeFleetRef := metav1.OwnerReference{ + APIVersion: "example.com/v1", + Kind: "AppliedWork", + Name: "non-kubefleet-owner", + UID: "uid-non-kubefleet-owner", + } + // An owner reference of the AppliedWork kind and of the same version as the older KubeFleet APIs, but + // in an API group that merely shares a prefix with the older KubeFleet API group. + lookalikeGroupRef := metav1.OwnerReference{ + APIVersion: kfioplacementv1beta1.GroupVersion.Group + ".example.com/" + kfioplacementv1beta1.GroupVersion.Version, + Kind: kfioplacementv1beta1.AppliedWorkKind, + Name: "applied-work-lookalike", + UID: "uid-applied-work-lookalike", + } + + configMapWithOwners := func(ownerRefs ...metav1.OwnerReference) *unstructured.Unstructured { + obj := &unstructured.Unstructured{} + obj.SetAPIVersion("v1") + obj.SetKind("ConfigMap") + obj.SetNamespace("ns-1") + obj.SetName("configmap-1") + obj.SetOwnerReferences(ownerRefs) + return obj + } + + testCases := []struct { + name string + inMemberClusterObj *unstructured.Unstructured + want bool + }{ + { + name: "no object", + }, + { + name: "no owners", + inMemberClusterObj: configMapWithOwners(), + }, + { + name: "owned by a non-KubeFleet owner of the AppliedWork kind", + inMemberClusterObj: configMapWithOwners(nonKubeFleetRef), + }, + { + name: "owned by an AppliedWork object from the newer KubeFleet APIs", + inMemberClusterObj: configMapWithOwners(newerAPIAppliedWorkRef), + }, + { + name: "owned by a non-AppliedWork object from the older KubeFleet APIs", + inMemberClusterObj: configMapWithOwners(olderAPIWorkRef), + }, + { + name: "owned by an AppliedWork object from an API group that shares a prefix with the older KubeFleet API group", + inMemberClusterObj: configMapWithOwners(lookalikeGroupRef), + }, + { + name: "owned by an AppliedWork object from the older KubeFleet APIs", + inMemberClusterObj: configMapWithOwners(olderAPIAppliedWorkRef), + want: true, + }, + { + name: "owned by an AppliedWork object from the older KubeFleet APIs (v1)", + inMemberClusterObj: configMapWithOwners(olderAPIV1AppliedWorkRef), + want: true, + }, + { + name: "co-owned by AppliedWork objects from both the newer and the older KubeFleet APIs", + inMemberClusterObj: configMapWithOwners(newerAPIAppliedWorkRef, nonKubeFleetRef, olderAPIAppliedWorkRef), + want: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + var ownerRefs []metav1.OwnerReference + if tc.inMemberClusterObj != nil { + ownerRefs = tc.inMemberClusterObj.GetOwnerReferences() + } + if got := isOwnedByOlderKubeFleetAPIObjects(tc.inMemberClusterObj); got != tc.want { + t.Errorf("isOwnedByOlderKubeFleetAPIObjects(object with owners %v) = %t, want %t", ownerRefs, got, tc.want) + } + }) + } +} diff --git a/pkg/v1/controllers/workapplier/takeover_integration_test.go b/pkg/v1/controllers/workapplier/takeover_integration_test.go new file mode 100644 index 000000000..5e6288980 --- /dev/null +++ b/pkg/v1/controllers/workapplier/takeover_integration_test.go @@ -0,0 +1,2001 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("takeover", func() { + Context("pre-existing objects with the default sync strategy (report error, no takeover)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + // The objects created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + var preExistingDeploy *appsv1.Deployment + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create the objects in the member cluster before the work object, as if a user or another controller + // has created them. The config map differs from its manifest, while the deployment is identical to its + // manifest; the default sync strategy forbids takeovers regardless of diffs. + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + preExistingDeploy = regularDeploy.DeepCopy() + Expect(memberClient1.Create(ctx, preExistingDeploy)).To(Succeed(), "Failed to create the pre-existing deployment") + + // Create a new work object with no sync strategy; the work applier falls back to the default one. + createWorkObject(workName, memberReservedNSName1, nil, regularConfigMapJSON, regularDeployJSON) + }) + + It("should add cleanup finalizer to the work object", func() { + finalizerAddedActual := workFinalizerAddedActual(memberReservedNSName1, workName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the work object") + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + }) + + It("should update the work object status", func() { + // Prepare the status information. + // + // The manifests are not applied and have no Available condition; no diff is calculated either, as the + // work applier never attempts to take over the objects. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeNotTakenOver), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeNotTakenOver), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should report the missing ownership as the cause of the failures", func() { + for _, ordinal := range []int{0, 1} { + manifestAppliedCondMessageActual := manifestAppliedCondMessageActual(memberReservedNSName1, workName, ordinal, notTakenOverErrMsg) + Eventually(manifestAppliedCondMessageActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the missing ownership for the manifest with ordinal %d", ordinal) + } + }) + + It("should not report any applied resource in the appliedWork object status", func() { + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, nil) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("should not take over or modify the pre-existing objects", func() { + // The check spans at least one periodic requeue of the work object. + configMapUntouchedActual := objectUntouchedActual(memberClient1, preExistingConfigMap) + deployUntouchedActual := objectUntouchedActual(memberClient1, preExistingDeploy) + Consistently(func() error { + if err := configMapUntouchedActual(); err != nil { + return err + } + return deployUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The pre-existing objects have been modified") + }) + + It("can delete the work object", func() { + deleteWorkObject(workName, memberReservedNSName1) + }) + + It("should start deleting the appliedWork object in the foreground", func() { + appliedWorkBeingDeletedActual := appliedWorkBeingDeletedInForegroundActual(memberClient1, workName) + Eventually(appliedWorkBeingDeletedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start deleting the appliedWork object in the foreground") + }) + + It("should remove the work object", func() { + // As there is no GC controller in the test environment, the appliedWork object will be stuck in the + // pending deletion state; the test case does not simulate the GC controller here, so that the work + // applier goes through the forced cleanup path, which visits all the manifests that have an Applied + // condition (including those that have not been taken over). The work applier proceeds to remove the + // cleanup finalizer after 2 * cleanupWaitTime. + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, 2*cleanupWaitTime+eventuallyDuration, eventuallyInterval*2).Should(Succeed(), "Failed to remove the work object") + }) + + It("should leave the pre-existing objects untouched", func() { + Expect(objectUntouchedActual(memberClient1, preExistingConfigMap)()).To(Succeed(), "The pre-existing config map has been modified") + Expect(objectUntouchedActual(memberClient1, preExistingDeploy)()).To(Succeed(), "The pre-existing deployment has been modified") + }) + + AfterAll(func() { + // The work object might still exist if the test case fails before the work object is deleted. + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete the work object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // Remove the pre-existing objects; the work applier never manages them. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("always take over", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + // The object created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the config map in the member cluster before the work object, as if a user or another + // controller has created it. The pre-existing config map differs from its manifest in a field that the + // manifest specifies (the data), and features a label that the manifest does not specify. + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + preExistingConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + // Create a new work object that always takes over pre-existing objects; the API server fills in the + // defaults for the other fields in the sync strategy. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should take over the pre-existing config map and apply the manifest", func() { + // The data is overwritten with the value from the manifest, while the label that the manifest does not + // specify is left alone. + wantConfigMap := regularConfigMap.DeepCopy() + wantConfigMap.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the config map") + + Expect(objectNotRecreatedActual(memberClient1, preExistingConfigMap)()).To(Succeed(), "The config map has been re-created instead of being taken over") + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: preExistingConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("take over if no diff (no diff)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + // The objects created in the member cluster before the work object. + var preExistingNS *corev1.Namespace + var preExistingDeploy *appsv1.Deployment + + BeforeAll(func() { + // Prepare the manifests. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create the objects in the member cluster before the work object, as if a user or another controller + // has created them; the objects are identical to their manifests. + preExistingNS = regularNS.DeepCopy() + Expect(memberClient1.Create(ctx, preExistingNS)).To(Succeed(), "Failed to create the pre-existing namespace") + + preExistingDeploy = regularDeploy.DeepCopy() + Expect(memberClient1.Create(ctx, preExistingDeploy)).To(Succeed(), "Failed to create the pre-existing deployment") + + // Create a new work object that takes over pre-existing objects only if they have no diff; the API + // server fills in the defaults for the other fields in the sync strategy (partial comparison). + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularNSJSON, regularDeployJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should take over the pre-existing objects", func() { + regularNSObjectAppliedActual := regularNSObjectAppliedActual(memberClient1, nsName, appliedWorkOwnerRef) + Eventually(regularNSObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the namespace object") + + regularDeploymentObjectAppliedActual := regularDeploymentObjectAppliedActual(memberClient1, nsName, deployName, appliedWorkOwnerRef) + Eventually(regularDeploymentObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the deployment object") + + Expect(objectNotRecreatedActual(memberClient1, preExistingNS)()).To(Succeed(), "The namespace has been re-created instead of being taken over") + Expect(objectNotRecreatedActual(memberClient1, preExistingDeploy)()).To(Succeed(), "The deployment has been re-created instead of being taken over") + }) + + It("should update the work object status (deployment not yet available)", func() { + // Verify that the manifests have been applied before marking the deployment as available; the takeover + // and the apply op are separate writes, and the status update would otherwise race with the apply op. + // + // Note that the API server bumps the generation of a deployment upon annotation changes (in addition to + // spec changes); the apply op adds the KubeFleet annotations to the taken-over deployment, which brings + // its generation to 2. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 2, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: string(AvailabilityResultTypeNotYetAvailable), + ObservedGeneration: 2, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("can mark the deployment as available", func() { + markDeploymentAsAvailable(memberClient1, nsName, deployName) + }) + + It("should update the work object status (all available)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 2, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 2, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + UID: preExistingNS.UID, + }, + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + UID: preExistingDeploy.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the deployment manually, as there is no GC controller in the test environment. + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("take over if no diff (diffs in managed fields, partial comparison)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + // The objects created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + var preExistingDeploy *appsv1.Deployment + // The timestamp when the diffs on the deployment were first observed. + var deployFirstDiffedTimestamp metav1.Time + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create the objects in the member cluster before the work object, as if a user or another controller + // has created them; each object differs from its manifest in a field that the manifest specifies. + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + preExistingDeploy = regularDeploy.DeepCopy() + preExistingDeploy.Spec.Replicas = ptr.To(int32(2)) + Expect(memberClient1.Create(ctx, preExistingDeploy)).To(Succeed(), "Failed to create the pre-existing deployment") + + // Create a new work object that takes over pre-existing objects only if they have no diff. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff, + ComparisonOption: placementv1alpha1.ComparisonOptionPartialComparison, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON, regularDeployJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should update the work object status", func() { + // Prepare the status information. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToTakeOver), + ObservedGeneration: 0, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: fmt.Sprintf("/data/%s", configMapDataKey), + ValueInMember: "baz", + ValueInHub: "bar", + }, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToTakeOver), + ObservedGeneration: 1, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(1)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "2", + ValueInHub: "1", + }, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + + deployFirstDiffedTimestamp = retrieveManifestFirstDiffedTimestamp(memberReservedNSName1, workName, 1) + }) + + It("should not take over or modify the pre-existing objects", func() { + // The check spans at least one periodic requeue of the work object. + configMapUntouchedActual := objectUntouchedActual(memberClient1, preExistingConfigMap) + deployUntouchedActual := objectUntouchedActual(memberClient1, preExistingDeploy) + Consistently(func() error { + if err := configMapUntouchedActual(); err != nil { + return err + } + return deployUntouchedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The pre-existing objects have been modified") + }) + + It("can resolve the diffs on the config map in the member cluster", func() { + gotConfigMap := &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, gotConfigMap)).To(Succeed(), "Failed to retrieve the config map") + gotConfigMap.Data = regularConfigMap.Data + Expect(memberClient1.Update(ctx, gotConfigMap)).To(Succeed(), "Failed to resolve the diffs on the config map") + }) + + It("should take over the config map", func() { + // The change in the member cluster triggers no event for the work object; the config map is taken over + // at the next requeue of the work object. The work object stays in the slow backoff stage at this + // moment, where requeues are at most 15 seconds apart; allow more time for the next attempt to happen. + wantConfigMap := regularConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the config map") + + Expect(objectNotRecreatedActual(memberClient1, preExistingConfigMap)()).To(Succeed(), "The config map has been re-created instead of being taken over") + }) + + It("should update the work object status (diffs resolved on the config map only)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Resource: "deployments", + Name: deployName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToTakeOver), + ObservedGeneration: 1, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(1)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "2", + ValueInHub: "1", + }, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should keep the timestamp when the diffs on the deployment were first observed", func() { + // The work object status has been rewritten since the timestamp was captured (the config map has been + // taken over). + gotTimestamp := retrieveManifestFirstDiffedTimestamp(memberReservedNSName1, workName, 1) + Expect(gotTimestamp.Equal(&deployFirstDiffedTimestamp)).To(BeTrue(), "first diffed timestamp = %v, want %v", gotTimestamp, deployFirstDiffedTimestamp) + }) + + It("should not take over or modify the pre-existing deployment", func() { + deployUntouchedActual := objectUntouchedActual(memberClient1, preExistingDeploy) + Consistently(deployUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The pre-existing deployment has been modified") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: preExistingConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the objects manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + regularDeployRemovedActual := regularDeployRemovedActual(memberClient1, nsName, deployName) + Eventually(regularDeployRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the deployment object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("take over if no diff (diffs in unmanaged fields only, partial comparison)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + // The object created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the config map in the member cluster before the work object, as if a user or another + // controller has created it. The pre-existing config map is identical to its manifest, except for a + // label that the manifest does not specify. + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + // Create a new work object that takes over pre-existing objects only if they have no diff; partial + // comparison considers only the fields that the manifest specifies. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff, + ComparisonOption: placementv1alpha1.ComparisonOptionPartialComparison, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should take over the pre-existing config map", func() { + // The label that the manifest does not specify is left alone. + wantConfigMap := regularConfigMap.DeepCopy() + wantConfigMap.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + wantConfigMap.OwnerReferences = []metav1.OwnerReference{*appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the config map") + + Expect(objectNotRecreatedActual(memberClient1, preExistingConfigMap)()).To(Succeed(), "The config map has been re-created instead of being taken over") + }) + + It("should update the work object status", func() { + // No diff is reported. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: preExistingConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("take over if no diff (diffs in unmanaged fields only, full comparison)", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + // The object created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create the config map in the member cluster before the work object, as if a user or another + // controller has created it. The pre-existing config map is identical to its manifest, except for a + // label that the manifest does not specify. + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.Labels = map[string]string{ + memberOnlyLabelKey: memberOnlyLabelValue, + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + // Create a new work object that takes over pre-existing objects only if they have no diff; full + // comparison considers all the fields, including those that the manifest does not specify. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff, + ComparisonOption: placementv1alpha1.ComparisonOptionFullComparison, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + }) + + It("should update the work object status", func() { + // The label is reported as a diff, which exists only in the member cluster. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToTakeOver), + ObservedGeneration: 0, + }, + }, + DiffDetails: &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: fmt.Sprintf("/metadata/labels/%s", memberOnlyLabelKey), + ValueInMember: memberOnlyLabelValue, + }, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should not take over or modify the pre-existing config map", func() { + // The check spans at least one periodic requeue of the work object. + configMapUntouchedActual := objectUntouchedActual(memberClient1, preExistingConfigMap) + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The pre-existing config map has been modified") + }) + + It("should not report any applied resource in the appliedWork object status", func() { + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, nil) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // Remove the pre-existing config map; the work applier never manages it. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("co-ownership", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRef *metav1.OwnerReference + var regularConfigMap *corev1.ConfigMap + var regularConfigMapJSON []byte + // The object created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + // The config map as it is after KubeFleet has taken it over as a co-owner. + var coOwnedConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap = configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON = marshalK8sObjJSON(regularConfigMap) + + // Create the config map in the member cluster before the work object, as if another controller has + // created (and owns) it. The pre-existing config map differs from its manifest in a field that the + // manifest specifies (the data). + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.OwnerReferences = []metav1.OwnerReference{externalOwnerRef} + preExistingConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + // Create a new work object that always takes over pre-existing objects, but forbids co-ownership. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: placementv1alpha1.WhenOwnedByOthersOptionReportError, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + + appliedWorkOwnerRef = prepareAppliedWorkOwnerRef(memberClient1, workName) + }) + + It("should update the work object status (co-ownership disallowed)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToTakeOver), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should report the existing owner as the cause of the failure", func() { + manifestAppliedCondMessageActual := manifestAppliedCondMessageActual(memberReservedNSName1, workName, 0, ownedByOthersErrMsg) + Eventually(manifestAppliedCondMessageActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the existing owner as the cause of the failure") + }) + + It("should not take over or modify the pre-existing config map", func() { + // The check spans at least one periodic requeue of the work object. + configMapUntouchedActual := objectUntouchedActual(memberClient1, preExistingConfigMap) + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The pre-existing config map has been modified") + }) + + It("can update the work object to allow co-ownership", func() { + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: placementv1alpha1.WhenOwnedByOthersOptionShareOwnership, + } + updateWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should take over the config map as a co-owner and apply the manifest", func() { + // The existing owner is kept. + wantConfigMap := regularConfigMap.DeepCopy() + wantConfigMap.OwnerReferences = []metav1.OwnerReference{externalOwnerRef, *appliedWorkOwnerRef} + configMapObjectActual := configMapObjectActual(memberClient1, wantConfigMap) + Eventually(configMapObjectActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the config map as a co-owner") + + Expect(objectNotRecreatedActual(memberClient1, preExistingConfigMap)()).To(Succeed(), "The config map has been re-created instead of being taken over") + + coOwnedConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, coOwnedConfigMap)).To(Succeed(), "Failed to retrieve the config map") + }) + + It("should update the work object status (co-owned)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should update the appliedWork object status (co-owned)", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: preExistingConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("can update the work object to disallow co-ownership again", func() { + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: placementv1alpha1.WhenOwnedByOthersOptionReportError, + } + updateWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should update the work object status (co-ownership disallowed after the takeover)", func() { + // KubeFleet still co-owns the config map, but may no longer apply the manifest to it. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToApply), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should report the co-ownership as the cause of the failure", func() { + manifestAppliedCondMessageActual := manifestAppliedCondMessageActual(memberReservedNSName1, workName, 0, coOwnershipDisallowedErrMsg) + Eventually(manifestAppliedCondMessageActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the co-ownership as the cause of the failure") + }) + + It("should leave the co-owned config map as it is", func() { + // The owner references stay as they are; the check spans at least one periodic requeue of the work + // object. + configMapUntouchedActual := objectUntouchedActual(memberClient1, coOwnedConfigMap) + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The co-owned config map has been modified") + }) + + It("should not report any applied resource in the appliedWork object status", func() { + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, nil) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSName1) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("objects owned by the older KubeFleet APIs", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var regularConfigMapJSON []byte + // The object created in the member cluster before the work object. + var preExistingConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work object does not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifest. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON = marshalK8sObjJSON(regularConfigMap) + + // Create the config map in the member cluster before the work object, as if the older work applier has + // placed it via the older KubeFleet APIs. The pre-existing config map differs from its manifest in a + // field that the manifest specifies (the data). + preExistingConfigMap = regularConfigMap.DeepCopy() + preExistingConfigMap.OwnerReferences = []metav1.OwnerReference{olderAPIAppliedWorkOwnerRef} + preExistingConfigMap.Data = map[string]string{ + configMapDataKey: "baz", + } + Expect(memberClient1.Create(ctx, preExistingConfigMap)).To(Succeed(), "Failed to create the pre-existing config map") + + // Create a new work object with the most permissive sync strategy: it always takes over pre-existing + // objects and allows co-ownership. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: placementv1alpha1.WhenOwnedByOthersOptionShareOwnership, + } + createWorkObject(workName, memberReservedNSName1, syncStrategy, regularConfigMapJSON) + }) + + It("should prepare an appliedWork object", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workName) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object") + }) + + It("should update the work object status (takeover and co-ownership allowed)", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeOwnedByOtherKubeFleetAPIObjects), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should report the ownership by the older KubeFleet APIs as the cause of the failure", func() { + manifestAppliedCondMessageActual := manifestAppliedCondMessageActual(memberReservedNSName1, workName, 0, ownedByOlderKubeFleetAPIObjectsErrMsg) + Eventually(manifestAppliedCondMessageActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the ownership by the older KubeFleet APIs as the cause of the failure") + }) + + It("should not report any applied resource in the appliedWork object status", func() { + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workName, nil) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update appliedWork status") + }) + + It("should not take over or modify the pre-existing config map (takeover and co-ownership allowed)", func() { + // The check spans at least one periodic requeue of the work object. + configMapUntouchedActual := objectUntouchedActual(memberClient1, preExistingConfigMap) + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The pre-existing config map has been modified") + }) + + It("can update the work object to use the default sync strategy", func() { + updateWorkObject(workName, memberReservedNSName1, nil, regularConfigMapJSON) + }) + + It("should update the work object status (default sync strategy)", func() { + // The safeguard runs regardless of the sync strategy in use; without it, the default sync strategy would + // have the work applier report the config map as not taken over instead. + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeOwnedByOtherKubeFleetAPIObjects), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should not take over or modify the pre-existing config map (default sync strategy)", func() { + // The check spans at least one periodic requeue of the work object. + configMapUntouchedActual := objectUntouchedActual(memberClient1, preExistingConfigMap) + Consistently(configMapUntouchedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The pre-existing config map has been modified") + }) + + It("can delete the work object", func() { + deleteWorkObject(workName, memberReservedNSName1) + }) + + It("should start deleting the appliedWork object in the foreground", func() { + appliedWorkBeingDeletedActual := appliedWorkBeingDeletedInForegroundActual(memberClient1, workName) + Eventually(appliedWorkBeingDeletedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start deleting the appliedWork object in the foreground") + }) + + It("should remove the work object", func() { + // As there is no GC controller in the test environment, the appliedWork object will be stuck in the + // pending deletion state; the test case does not simulate the GC controller here, so that the work + // applier goes through the forced cleanup path, which visits all the manifests that have an Applied + // condition (including the config map, which features an AppliedWork owner reference that blocks owner + // deletion, but from the older KubeFleet APIs). The work applier proceeds to remove the cleanup + // finalizer after 2 * cleanupWaitTime. + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, 2*cleanupWaitTime+eventuallyDuration, eventuallyInterval*2).Should(Succeed(), "Failed to remove the work object") + }) + + It("should leave the pre-existing config map untouched", func() { + Expect(objectUntouchedActual(memberClient1, preExistingConfigMap)()).To(Succeed(), "The pre-existing config map has been modified") + }) + + AfterAll(func() { + // The work object might still exist if the test case fails before the work object is deleted. + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete the work object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // Remove the pre-existing config map; the work applier never manages it. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("objects placed by another work object (duplicate placement)", Ordered, func() { + // The two work objects are not linked: each is the primary (and only) work object of its own placement + // binding. Both place the same config map. + workNameA := fmt.Sprintf(workNameTemplate, utils.RandStr()) + workNameB := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + var appliedWorkOwnerRefA *metav1.OwnerReference + var appliedWorkOwnerRefB *metav1.OwnerReference + // The config map as placed by each work object; the data differs. + var configMapFromA *corev1.ConfigMap + var configMapFromB *corev1.ConfigMap + var configMapFromBJSON []byte + // The config map as created in the member cluster via work A. + var placedConfigMap *corev1.ConfigMap + + BeforeAll(func() { + // Create the namespace directly in the member cluster, as the work objects do not feature the + // namespace manifest. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + Expect(memberClient1.Create(ctx, regularNS)).To(Succeed(), "Failed to create the namespace") + + // Prepare the manifests. + configMapFromA = configMap.DeepCopy() + configMapFromA.Namespace = nsName + configMapFromAJSON := marshalK8sObjJSON(configMapFromA) + + configMapFromB = configMapFromA.DeepCopy() + configMapFromB.Data = map[string]string{ + configMapDataKey: "baz", + } + configMapFromBJSON = marshalK8sObjJSON(configMapFromB) + + // Create work A with the default sync strategy. + createWorkObject(workNameA, memberReservedNSName1, nil, configMapFromAJSON) + }) + + It("should apply the config map via work A", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workNameA) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for work A") + appliedWorkOwnerRefA = prepareAppliedWorkOwnerRef(memberClient1, workNameA) + + configMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, configMapFromA, appliedWorkOwnerRefA) + Eventually(configMapObjectAppliedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to apply the config map via work A") + + placedConfigMap = &corev1.ConfigMap{} + Expect(memberClient1.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, placedConfigMap)).To(Succeed(), "Failed to retrieve the config map") + }) + + It("can create work B to place the same config map (default sync strategy)", func() { + createWorkObject(workNameB, memberReservedNSName1, nil, configMapFromBJSON) + }) + + It("should prepare an appliedWork object for work B", func() { + appliedWorkCreatedActual := appliedWorkCreatedActual(memberClient1, memberReservedNSName1, workNameB) + Eventually(appliedWorkCreatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to prepare an appliedWork object for work B") + appliedWorkOwnerRefB = prepareAppliedWorkOwnerRef(memberClient1, workNameB) + }) + + It("should report that work B has not taken over the config map", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeNotTakenOver), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workNameB, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of work B") + }) + + It("can update work B to always take over pre-existing objects", func() { + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + } + updateWorkObject(workNameB, memberReservedNSName1, syncStrategy, configMapFromBJSON) + }) + + It("should report that work B cannot take over the config map as it is owned by others", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToTakeOver), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workNameB, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of work B") + + manifestAppliedCondMessageActual := manifestAppliedCondMessageActual(memberReservedNSName1, workNameB, 0, ownedByOthersErrMsg) + Eventually(manifestAppliedCondMessageActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the existing owner as the cause of the failure") + }) + + It("can update work B to allow co-ownership", func() { + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenAlreadyExists: placementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: placementv1alpha1.WhenOwnedByOthersOptionShareOwnership, + } + updateWorkObject(workNameB, memberReservedNSName1, syncStrategy, configMapFromBJSON) + }) + + It("should report that work B cannot take over the config map as it has been placed by another work object", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAppliedCondNotAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.WorkAvailableCondNotAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + Reason: string(ApplyResTypeFailedToTakeOver), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workNameB, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of work B") + + // The reason stays the same as before; the message tells the two failures apart. + manifestAppliedCondMessageActual := manifestAppliedCondMessageActual(memberReservedNSName1, workNameB, 0, placedInDuplicateErrMsg) + Eventually(manifestAppliedCondMessageActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the duplicate placement as the cause of the failure") + }) + + It("should keep the config map as placed by work A", func() { + // The check spans at least one periodic requeue of work B. + configMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, configMapFromA, appliedWorkOwnerRefA) + Consistently(configMapObjectAppliedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The config map placed by work A has been changed") + }) + + It("can delete work A", func() { + deleteWorkObject(workNameA, memberReservedNSName1) + }) + + It("should remove work A", func() { + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object of + // work A. As there is no GC controller in the test environment, the config map is not deleted, and it keeps + // a (now dangling) owner reference to that appliedWork object; this mimics a GC controller that has not + // caught up yet. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workNameA) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object of work A") + + workRemovedActual := workRemovedActual(memberReservedNSName1, workNameA) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work A") + }) + + It("should take over the config map via work B", func() { + // Work B drops the owner reference left behind by the appliedWork object of work A, as work A no longer + // exists, and then takes over the config map. Deleting work A triggers no event for work B; the takeover + // happens at the next requeue of work B, which might be up to 15 seconds away in the slow backoff stage. + configMapObjectAppliedActual := configMapObjectAppliedActual(memberClient1, configMapFromB, appliedWorkOwnerRefB) + Eventually(configMapObjectAppliedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take over the config map via work B") + + Expect(objectNotRecreatedActual(memberClient1, placedConfigMap)()).To(Succeed(), "The config map has been re-created instead of being taken over") + }) + + It("should update the status of work B", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSName1, workNameB, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of work B") + }) + + It("should update the appliedWork object status of work B", func() { + appliedResources := []placementv1alpha1.AppliedResource{ + { + AppliedResourceIdentifier: placementv1alpha1.AppliedResourceIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + UID: placedConfigMap.UID, + }, + } + + appliedWorkStatusUpdatedActual := appliedWorkStatusUpdated(memberClient1, workNameB, appliedResources) + Eventually(appliedWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the appliedWork status of work B") + }) + + AfterAll(func() { + // Work A might still exist if the test case fails before work A is deleted, and work B might not exist yet + // if the test case fails before work B is created. + for _, workName := range []string{workNameA, workNameB} { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSName1, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete work %s", workName) + } + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork objects. + for _, workName := range []string{workNameA, workNameB} { + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object of work %s", workName) + + workRemovedActual := workRemovedActual(memberReservedNSName1, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove work %s", workName) + } + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workapplier/utils_integration_test.go b/pkg/v1/controllers/workapplier/utils_integration_test.go new file mode 100644 index 000000000..35b1d975a --- /dev/null +++ b/pkg/v1/controllers/workapplier/utils_integration_test.go @@ -0,0 +1,1567 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "context" + "crypto/sha256" + "encoding/json" + "fmt" + "slices" + "strings" + "sync" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + autoscalingv2 "k8s.io/api/autoscaling/v2" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/util/workqueue" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfioplacementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/utils/parallelizer" + testutilsresource "go.goms.io/fleet/test/utils/resource" +) + +const ( + workNameTemplate = "work-%s" + nsNameTemplate = "ns-%s" + placementPolicyNameTemplate = "policy-%s" + placementBindingNameTemplate = "binding-%s" + primarySnapshotNameTemplate = "snapshot-%s-0" + deployName = "deploy-1" + configMapName = "configmap-1" + configMapDataKey = "foo" + singleWorkLinkedWorkCountAnno = "1" + // A test-only label that nudgeWork sets on manifests to bump the generations of Work objects. + nudgeLabelKey = "nudge" + // The UID used in the owner references that link secondary work objects to their primary work objects. The API + // server only requires the UID to be non-empty, and there is no GC controller in the test environment to check + // it against the actual primary work object. + dummyPrimaryWorkUID types.UID = "dummy-primary-work-uid" + // The message the work applier reports when it finds an object that it does not own and may not take over. + notTakenOverErrMsg = "no ownership of the object in the member cluster; takeover is needed" + // The message the work applier reports when it may not take over an object as the object is owned by other + // sources and co-ownership is disallowed. + ownedByOthersErrMsg = "the object is already owned by some other source(s) and co-ownership is disallowed" + // The message the work applier reports when it may not take over an object as the object has been placed by + // another KubeFleet AppliedWork object. + placedInDuplicateErrMsg = "the object is already owned by another KubeFleet AppliedWork object" + // The message the work applier reports when it may not apply to an object it co-owns, as co-ownership has been + // disallowed since the takeover. + coOwnershipDisallowedErrMsg = "object is co-owned by multiple objects but co-ownership has been disallowed" + // The message the work applier reports when it may not process an object as the object has been placed via the + // older KubeFleet APIs (i.e., it is owned by an AppliedWork object in the placement.kubernetes-fleet.io API group). + ownedByOlderKubeFleetAPIObjectsErrMsg = "the object to process is already owned by another KubeFleet placement API object" + + // A label that the test cases add to objects in the member cluster, to mimic changes made by users or other + // controllers on fields that KubeFleet does not manage. + memberOnlyLabelKey = "member-only" + memberOnlyLabelValue = "true" + + // The field manager that the test cases use when they write to objects in the member cluster, to mimic + // changes made by users or other controllers. + thirdPartyFieldManagerName = "integration-test-manager" +) + +const ( + eventuallyDuration = time.Second * 10 + eventuallyInterval = time.Second * 1 + consistentlyDuration = time.Second * 5 + consistentlyInterval = time.Millisecond * 500 +) + +var ( + ignoreFieldObjectMetaAutoGenFields = cmpopts.IgnoreFields(metav1.ObjectMeta{}, "CreationTimestamp", "Generation", "ResourceVersion", "SelfLink", "UID", "ManagedFields") + ignoreFieldAppliedWorkStatus = cmpopts.IgnoreFields(placementv1alpha1.AppliedWork{}, "Status") + ignoreFieldConditionLTTMsg = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message") + ignoreFieldDiffDetailsObsTime = cmpopts.IgnoreFields(placementv1alpha1.DiffDetails{}, "FirstDiffedObservedTimestamp") +) + +var ( + ns = &corev1.Namespace{ + TypeMeta: metav1.TypeMeta{ + Kind: "Namespace", + APIVersion: "v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: "ns-1", + }, + } + + configMap = &corev1.ConfigMap{ + TypeMeta: metav1.TypeMeta{ + Kind: "ConfigMap", + APIVersion: "v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: "ns-1", + }, + Data: map[string]string{ + configMapDataKey: "bar", + }, + } + + deploy = &appsv1.Deployment{ + TypeMeta: metav1.TypeMeta{ + Kind: "Deployment", + APIVersion: "apps/v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: deployName, + Namespace: "ns-1", + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(int32(1)), + Selector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "app": "nginx", + }, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{ + "app": "nginx", + }, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "nginx", + Image: "nginx", + Ports: []corev1.ContainerPort{ + { + ContainerPort: 80, + }, + }, + }, + }, + }, + }, + }, + } + + // An owner reference that points to an object managed by a non-KubeFleet controller. The API server does not + // verify that owners exist, and there is no GC controller in the test environment to act on the reference. + externalOwnerRef = metav1.OwnerReference{ + APIVersion: "example.com/v1", + Kind: "ExternalOwner", + Name: "external-owner", + UID: "external-owner-uid", + } + + // An owner reference that points to an AppliedWork object of the older KubeFleet APIs (the + // placement.kubernetes-fleet.io API group), in the same form as the older work applier sets it on the objects + // it places. The API server does not verify that owners exist, and there is no GC controller in the test + // environment to act on the reference. + olderAPIAppliedWorkOwnerRef = metav1.OwnerReference{ + APIVersion: kfioplacementv1beta1.GroupVersion.String(), + Kind: kfioplacementv1beta1.AppliedWorkKind, + Name: "older-api-applied-work", + UID: "older-api-applied-work-uid", + BlockOwnerDeletion: ptr.To(true), + } +) + +// buildWorkObject builds a work object with the given work name/namespace, owner placement policy/binding, +// primary placement resource snapshot, sync strategy, and raw manifest JSONs. +// +// Note that the returned work object does not have the linked work count annotation set; the annotation +// is only present on primary work objects. +func buildWorkObject( + workName, memberClusterReservedNSName string, + placementPolicyName, placementBindingName, primarySnapshotName string, + syncStrategy *placementv1alpha1.SyncStrategy, + rawManifestJSON ...[]byte, +) *placementv1alpha1.Work { + manifests := make([]placementv1alpha1.Manifest, len(rawManifestJSON)) + for idx := range rawManifestJSON { + manifests[idx] = placementv1alpha1.Manifest{ + RawExtension: runtime.RawExtension{ + Raw: rawManifestJSON[idx], + }, + } + } + + return &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberClusterReservedNSName, + Labels: map[string]string{ + // An empty owner namespace signals cluster-scoped owners. + placementv1alpha1.WorkOwnerNamespaceLabelKey: "", + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: placementPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: placementBindingName, + }, + Annotations: map[string]string{ + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: placementPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: placementBindingName, + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: primarySnapshotName, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: manifests, + SyncStrategy: syncStrategy, + }, + } +} + +// createWorkObject creates a new (primary) work object with the given work name/namespace, sync strategy, +// and raw manifest JSONs. +// +// The work object is set up as the only work object linked to a dedicated (cluster-scoped) placement policy and +// placement binding, so that work objects created in different test cases never get linked with each other. +func createWorkObject(workName, memberClusterReservedNSName string, syncStrategy *placementv1alpha1.SyncStrategy, rawManifestJSON ...[]byte) { + work := buildWorkObject( + workName, memberClusterReservedNSName, + fmt.Sprintf(placementPolicyNameTemplate, workName), + fmt.Sprintf(placementBindingNameTemplate, workName), + fmt.Sprintf(primarySnapshotNameTemplate, workName), + syncStrategy, + rawManifestJSON..., + ) + work.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = singleWorkLinkedWorkCountAnno + Expect(hubClient.Create(ctx, work)).To(Succeed(), "Failed to create the Work object") +} + +// setPrimaryWorkAsOwner sets the primary work object as the controller owner of a secondary work object, which +// mimics how KubeFleet links secondary work objects to their primary work objects; the work applier relies on +// this owner reference to map events on secondary work objects to their primary work objects. +func setPrimaryWorkAsOwner(secondaryWork *placementv1alpha1.Work, primaryWorkName string) { + secondaryWork.OwnerReferences = []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.WorkKind, + Name: primaryWorkName, + UID: dummyPrimaryWorkUID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } +} + +func marshalK8sObjJSON(obj runtime.Object) []byte { + json, err := testutilsresource.MarshalRuntimeObjToJSONForTest(obj) + Expect(err).To(BeNil(), "Failed to marshal the k8s object to JSON") + return json +} + +func workFinalizerAddedActual(workNS, workName string) func() error { + return func() error { + // Retrieve the Work object. + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + // Check that the cleanup finalizer has been added. + if !controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + return fmt.Errorf("cleanup finalizer has not been added") + } + return nil + } +} + +func appliedWorkCreatedActual(memberClient client.Client, workNS, workName string) func() error { + return func() error { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + + wantAppliedWork := &placementv1alpha1.AppliedWork{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + }, + Spec: placementv1alpha1.AppliedWorkSpec{ + WorkName: workName, + WorkNamespace: workNS, + }, + } + if diff := cmp.Diff( + appliedWork, wantAppliedWork, + ignoreFieldObjectMetaAutoGenFields, + ignoreFieldAppliedWorkStatus, + ); diff != "" { + return fmt.Errorf("appliedWork diff (-got +want):\n%s", diff) + } + return nil + } +} + +func prepareAppliedWorkOwnerRef(memberClient client.Client, workName string) *metav1.OwnerReference { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + Expect(memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork)).To(Succeed(), "Failed to retrieve the AppliedWork object") + + // Prepare the expected OwnerReference. + return &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "AppliedWork", + Name: appliedWork.Name, + UID: appliedWork.GetUID(), + BlockOwnerDeletion: ptr.To(true), + } +} + +func regularNSObjectAppliedActual(memberClient client.Client, nsName string, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + return func() error { + // Retrieve the NS object. + gotNS := &corev1.Namespace{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: nsName}, gotNS); err != nil { + return fmt.Errorf("failed to retrieve the NS object: %w", err) + } + + // Check that the NS object has been created as expected. + + // To ignore default values automatically, here the test suite rebuilds the objects. + wantNS := ns.DeepCopy() + wantNS.TypeMeta = metav1.TypeMeta{} + wantNS.Name = nsName + wantNS.OwnerReferences = []metav1.OwnerReference{ + *appliedWorkOwnerRef, + } + + rebuiltGotNS := &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{ + Name: gotNS.Name, + OwnerReferences: gotNS.OwnerReferences, + }, + } + + if diff := cmp.Diff(rebuiltGotNS, wantNS); diff != "" { + return fmt.Errorf("namespace diff (-got +want):\n%s", diff) + } + return nil + } +} + +func regularDeploymentObjectAppliedActual(memberClient client.Client, nsName, deployName string, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + wantDeploy := deploy.DeepCopy() + wantDeploy.Namespace = nsName + wantDeploy.Name = deployName + wantDeploy.OwnerReferences = []metav1.OwnerReference{ + *appliedWorkOwnerRef, + } + return deploymentObjectActual(memberClient, wantDeploy) +} + +// deploymentObjectActual checks that the Deployment object in the member cluster has the namespace, name, owner +// references, replicas, selector, and pod template (labels, container names, images, and ports) of the given +// Deployment object, which is expected to feature exactly one container with exactly one port. +func deploymentObjectActual(memberClient client.Client, wantDeploy *appsv1.Deployment) func() error { + return func() error { + // Retrieve the Deployment object. + gotDeploy := &appsv1.Deployment{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: wantDeploy.Namespace, Name: wantDeploy.Name}, gotDeploy); err != nil { + return fmt.Errorf("failed to retrieve the Deployment object: %w", err) + } + + // Check that the Deployment object has been created as expected. + + // To ignore default values automatically, here the test suite rebuilds the objects. + rebuiltWantDeploy := &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: wantDeploy.Namespace, + Name: wantDeploy.Name, + OwnerReferences: wantDeploy.OwnerReferences, + }, + Spec: appsv1.DeploymentSpec{ + Replicas: wantDeploy.Spec.Replicas, + Selector: wantDeploy.Spec.Selector, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: wantDeploy.Spec.Template.Labels, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: wantDeploy.Spec.Template.Spec.Containers[0].Name, + Image: wantDeploy.Spec.Template.Spec.Containers[0].Image, + Ports: []corev1.ContainerPort{ + { + ContainerPort: wantDeploy.Spec.Template.Spec.Containers[0].Ports[0].ContainerPort, + }, + }, + }, + }, + }, + }, + }, + } + + if len(gotDeploy.Spec.Template.Spec.Containers) != 1 { + return fmt.Errorf("number of containers in the Deployment object, got %d, want %d", len(gotDeploy.Spec.Template.Spec.Containers), 1) + } + if len(gotDeploy.Spec.Template.Spec.Containers[0].Ports) != 1 { + return fmt.Errorf("number of ports in the first container, got %d, want %d", len(gotDeploy.Spec.Template.Spec.Containers[0].Ports), 1) + } + rebuiltGotDeploy := &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: gotDeploy.Namespace, + Name: gotDeploy.Name, + OwnerReferences: gotDeploy.OwnerReferences, + }, + Spec: appsv1.DeploymentSpec{ + Replicas: gotDeploy.Spec.Replicas, + Selector: gotDeploy.Spec.Selector, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{ + "app": gotDeploy.Spec.Template.Labels["app"], + }, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: gotDeploy.Spec.Template.Spec.Containers[0].Name, + Image: gotDeploy.Spec.Template.Spec.Containers[0].Image, + Ports: []corev1.ContainerPort{ + { + ContainerPort: gotDeploy.Spec.Template.Spec.Containers[0].Ports[0].ContainerPort, + }, + }, + }, + }, + }, + }, + }, + } + if diff := cmp.Diff(rebuiltGotDeploy, rebuiltWantDeploy); diff != "" { + return fmt.Errorf("deployment diff (-got +want):\n%s", diff) + } + return nil + } +} + +// updateDeploymentInMemberCluster updates a Deployment object in the member cluster with the given mutation, as if +// a user or another controller has made the change, and returns the updated object. +func updateDeploymentInMemberCluster(memberClient client.Client, nsName, deployName string, mutate func(deploy *appsv1.Deployment)) *appsv1.Deployment { + updatedDeploy := &appsv1.Deployment{} + Eventually(func() error { + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, updatedDeploy); err != nil { + return fmt.Errorf("failed to retrieve the Deployment object: %w", err) + } + mutate(updatedDeploy) + // The work applier might update the object at the same time; retry on conflicts. + return memberClient.Update(ctx, updatedDeploy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the Deployment object in the member cluster") + return updatedDeploy +} + +// updateConfigMapInMemberCluster updates a ConfigMap object in the member cluster with the given mutation, as if a +// user or another controller has made the change, and returns the updated object. +func updateConfigMapInMemberCluster(memberClient client.Client, nsName, configMapName string, mutate func(configMap *corev1.ConfigMap)) *corev1.ConfigMap { + updatedConfigMap := &corev1.ConfigMap{} + Eventually(func() error { + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, updatedConfigMap); err != nil { + return fmt.Errorf("failed to retrieve the ConfigMap object: %w", err) + } + mutate(updatedConfigMap) + // The work applier might update the object at the same time; retry on conflicts. + return memberClient.Update(ctx, updatedConfigMap) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the ConfigMap object in the member cluster") + return updatedConfigMap +} + +// objectLabelsActual checks that an object in the member cluster has exactly the given labels. +func objectLabelsActual(memberClient client.Client, obj client.Object, wantLabels map[string]string) func() error { + return func() error { + gotObj, ok := obj.DeepCopyObject().(client.Object) + if !ok { + return fmt.Errorf("failed to copy the object %s", client.ObjectKeyFromObject(obj)) + } + if err := memberClient.Get(ctx, client.ObjectKeyFromObject(obj), gotObj); err != nil { + return fmt.Errorf("failed to retrieve the object %s: %w", client.ObjectKeyFromObject(obj), err) + } + + if diff := cmp.Diff(gotObj.GetLabels(), wantLabels); diff != "" { + return fmt.Errorf("object %s labels diff (-got +want):\n%s", client.ObjectKeyFromObject(obj), diff) + } + return nil + } +} + +func regularConfigMapObjectAppliedActual(memberClient client.Client, nsName, configMapName string, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + wantConfigMap := configMap.DeepCopy() + wantConfigMap.Namespace = nsName + wantConfigMap.Name = configMapName + return configMapObjectAppliedActual(memberClient, wantConfigMap, appliedWorkOwnerRef) +} + +// configMapObjectAppliedActual checks that the ConfigMap object has been applied as expected, i.e., it has the +// same namespace, name, and data as the given ConfigMap object, and is owned by the AppliedWork object. +func configMapObjectAppliedActual(memberClient client.Client, configMapToApply *corev1.ConfigMap, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + return func() error { + // Retrieve the ConfigMap object. + gotConfigMap := &corev1.ConfigMap{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: configMapToApply.Namespace, Name: configMapToApply.Name}, gotConfigMap); err != nil { + return fmt.Errorf("failed to retrieve the ConfigMap object: %w", err) + } + + // Check that the ConfigMap object has been created as expected. + + // To ignore default values automatically, here the test suite rebuilds the objects. + wantConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: configMapToApply.Namespace, + Name: configMapToApply.Name, + OwnerReferences: []metav1.OwnerReference{ + *appliedWorkOwnerRef, + }, + }, + Data: configMapToApply.Data, + } + + rebuiltGotConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: gotConfigMap.Namespace, + Name: gotConfigMap.Name, + OwnerReferences: gotConfigMap.OwnerReferences, + }, + Data: gotConfigMap.Data, + } + if diff := cmp.Diff(rebuiltGotConfigMap, wantConfigMap); diff != "" { + return fmt.Errorf("configMap diff (-got +want):\n%s", diff) + } + return nil + } +} + +// configMapObjectActual checks that the ConfigMap object in the member cluster has exactly the namespace, name, +// labels, annotations (sans those reserved for KubeFleet), owner references, and data of the given ConfigMap object. +// +// Unlike configMapObjectAppliedActual, this function compares labels and annotations, and allows any set of owners +// (e.g., when KubeFleet co-owns the object with other sources). +func configMapObjectActual(memberClient client.Client, wantConfigMap *corev1.ConfigMap) func() error { + return func() error { + gotConfigMap := &corev1.ConfigMap{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: wantConfigMap.Namespace, Name: wantConfigMap.Name}, gotConfigMap); err != nil { + return fmt.Errorf("failed to retrieve the ConfigMap object: %w", err) + } + + // To ignore default values and fields managed by KubeFleet (e.g., the KubeFleet annotations) automatically, + // here the test suite rebuilds the objects. + rebuiltWantConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: wantConfigMap.Namespace, + Name: wantConfigMap.Name, + Labels: wantConfigMap.Labels, + Annotations: wantConfigMap.Annotations, + OwnerReferences: wantConfigMap.OwnerReferences, + }, + Data: wantConfigMap.Data, + } + rebuiltGotConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: gotConfigMap.Namespace, + Name: gotConfigMap.Name, + Labels: gotConfigMap.Labels, + Annotations: nonKubeFleetAnnotations(gotConfigMap.Annotations), + OwnerReferences: gotConfigMap.OwnerReferences, + }, + Data: gotConfigMap.Data, + } + if diff := cmp.Diff(rebuiltGotConfigMap, rebuiltWantConfigMap); diff != "" { + return fmt.Errorf("configMap diff (-got +want):\n%s", diff) + } + return nil + } +} + +// nonKubeFleetAnnotations returns the annotations that are not reserved for KubeFleet, or nil if there are none. +func nonKubeFleetAnnotations(annotations map[string]string) map[string]string { + var filtered map[string]string + for key, val := range annotations { + if strings.Contains(key, placementv1alpha1.KubeFleetPrefix) { + continue + } + if filtered == nil { + filtered = map[string]string{} + } + filtered[key] = val + } + return filtered +} + +// configMapObjectWithDataDigestActual checks that the ConfigMap object in the member cluster has the namespace, +// name, owner references, and data of the given ConfigMap object. +// +// Unlike configMapObjectActual, this function compares the lengths and digests of the data values instead of the +// values themselves, so that the outputs stay readable when the data is large. +func configMapObjectWithDataDigestActual(memberClient client.Client, wantConfigMap *corev1.ConfigMap) func() error { + digestsOf := func(data map[string]string) map[string]string { + digests := make(map[string]string, len(data)) + for key, val := range data { + digests[key] = fmt.Sprintf("len=%d, sha256=%x", len(val), sha256.Sum256([]byte(val))) + } + return digests + } + + return func() error { + gotConfigMap := &corev1.ConfigMap{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: wantConfigMap.Namespace, Name: wantConfigMap.Name}, gotConfigMap); err != nil { + return fmt.Errorf("failed to retrieve the ConfigMap object: %w", err) + } + + if diff := cmp.Diff(gotConfigMap.OwnerReferences, wantConfigMap.OwnerReferences); diff != "" { + return fmt.Errorf("configMap owner references diff (-got +want):\n%s", diff) + } + if diff := cmp.Diff(digestsOf(gotConfigMap.Data), digestsOf(wantConfigMap.Data)); diff != "" { + return fmt.Errorf("configMap data digests diff (-got +want):\n%s", diff) + } + return nil + } +} + +// objectAnnotationActual checks that an object in the member cluster has the given annotation with the given value +// (which might be empty). +func objectAnnotationActual(memberClient client.Client, obj client.Object, key, wantValue string) func() error { + return func() error { + gotObj, ok := obj.DeepCopyObject().(client.Object) + if !ok { + return fmt.Errorf("failed to copy the object %s", client.ObjectKeyFromObject(obj)) + } + if err := memberClient.Get(ctx, client.ObjectKeyFromObject(obj), gotObj); err != nil { + return fmt.Errorf("failed to retrieve the object %s: %w", client.ObjectKeyFromObject(obj), err) + } + + gotValue, found := gotObj.GetAnnotations()[key] + if !found { + return fmt.Errorf("object %s does not have the annotation %s", client.ObjectKeyFromObject(obj), key) + } + if gotValue != wantValue { + return fmt.Errorf("object %s annotation %s = %q, want %q", client.ObjectKeyFromObject(obj), key, gotValue, wantValue) + } + return nil + } +} + +// fieldManagersActual checks that a field of an object in the member cluster is managed exactly by the given field +// managers, each formatted as `[MANAGER]/[OPERATION]` (e.g., `kubefleet-member-agent/Apply`). +// +// The field is specified as a path in the managed fields format, e.g., `f:data`, `f:foo` for the `foo` key in the +// data of a ConfigMap object. +func fieldManagersActual(memberClient client.Client, obj client.Object, fieldPath []string, wantManagers []string) func() error { + return func() error { + gotObj, ok := obj.DeepCopyObject().(client.Object) + if !ok { + return fmt.Errorf("failed to copy the object %s", client.ObjectKeyFromObject(obj)) + } + if err := memberClient.Get(ctx, client.ObjectKeyFromObject(obj), gotObj); err != nil { + return fmt.Errorf("failed to retrieve the object %s: %w", client.ObjectKeyFromObject(obj), err) + } + + gotManagers := []string{} + for _, managedFieldsEntry := range gotObj.GetManagedFields() { + if managedFieldsEntry.FieldsV1 == nil { + continue + } + fields := map[string]interface{}{} + if err := json.Unmarshal(managedFieldsEntry.FieldsV1.Raw, &fields); err != nil { + return fmt.Errorf("failed to unmarshal the managed fields of manager %s: %w", managedFieldsEntry.Manager, err) + } + + cur := fields + found := true + for _, segment := range fieldPath { + next, ok := cur[segment].(map[string]interface{}) + if !ok { + found = false + break + } + cur = next + } + if found { + gotManagers = append(gotManagers, fmt.Sprintf("%s/%s", managedFieldsEntry.Manager, managedFieldsEntry.Operation)) + } + } + + if diff := cmp.Diff(gotManagers, wantManagers, cmpopts.SortSlices(func(a, b string) bool { return a < b })); diff != "" { + return fmt.Errorf("object %s field %v managers diff (-got +want):\n%s", client.ObjectKeyFromObject(obj), fieldPath, diff) + } + return nil + } +} + +func markDeploymentAsAvailable(memberClient client.Client, nsName, deployName string) { + // Retrieve the Deployment object. + gotDeploy := &appsv1.Deployment{} + Expect(memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, gotDeploy)).To(Succeed(), "Failed to retrieve the Deployment object") + + // Mark the Deployment object as available. + now := metav1.Now() + requiredReplicas := int32(1) + if gotDeploy.Spec.Replicas != nil { + requiredReplicas = *gotDeploy.Spec.Replicas + } + gotDeploy.Status = appsv1.DeploymentStatus{ + ObservedGeneration: gotDeploy.Generation, + Replicas: requiredReplicas, + UpdatedReplicas: requiredReplicas, + ReadyReplicas: requiredReplicas, + AvailableReplicas: requiredReplicas, + UnavailableReplicas: 0, + Conditions: []appsv1.DeploymentCondition{ + { + Type: appsv1.DeploymentAvailable, + Status: corev1.ConditionTrue, + Reason: "MarkedAsAvailable", + Message: "Deployment has been marked as available", + LastUpdateTime: now, + LastTransitionTime: now, + }, + }, + } + Expect(memberClient.Status().Update(ctx, gotDeploy)).To(Succeed(), "Failed to mark the Deployment object as available") +} + +func workStatusUpdated( + workNS, workName string, + workConds []metav1.Condition, + manifestStatuses []placementv1alpha1.PerManifestStatus, +) func() error { + return func() error { + // Retrieve the Work object. + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + // Prepare the expected Work object status. + + // Update the conditions with the observed generation. + // + // Note that the observed generation of a manifest condition is that of an applied + // resource, not that of the Work object. + for idx := range workConds { + workConds[idx].ObservedGeneration = work.Generation + } + wantWorkStatus := placementv1alpha1.WorkStatus{ + Conditions: workConds, + Manifests: manifestStatuses, + } + + // Check that the Work object status has been updated as expected. + if diff := cmp.Diff( + work.Status, wantWorkStatus, + ignoreFieldConditionLTTMsg, + ignoreFieldDiffDetailsObsTime, + ); diff != "" { + return fmt.Errorf("work status diff (-got, +want):\n%s", diff) + } + return nil + } +} + +// workStatusStaleActual checks that the status of a Work object has not been refreshed since the Work object was last +// processed at the given (now outdated) generation, i.e., the Work object has been updated since then, and all of its +// conditions have become stale. +func workStatusStaleActual( + workNS, workName string, + staleWorkGeneration int64, + workConds []metav1.Condition, + manifestStatuses []placementv1alpha1.PerManifestStatus, +) func() error { + return func() error { + // Retrieve the Work object. + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + if work.Generation <= staleWorkGeneration { + return fmt.Errorf("work generation = %d, want a generation newer than %d", work.Generation, staleWorkGeneration) + } + + // Prepare the expected Work object status, with all the conditions observed at the outdated generation. + // + // Note that the observed generation of a manifest condition is that of an applied + // resource, not that of the Work object. + wantWorkConds := make([]metav1.Condition, len(workConds)) + for idx := range workConds { + wantWorkConds[idx] = workConds[idx] + wantWorkConds[idx].ObservedGeneration = staleWorkGeneration + } + wantWorkStatus := placementv1alpha1.WorkStatus{ + Conditions: wantWorkConds, + Manifests: manifestStatuses, + } + + // Check that the Work object status has stayed the same. + if diff := cmp.Diff( + work.Status, wantWorkStatus, + ignoreFieldConditionLTTMsg, + ignoreFieldDiffDetailsObsTime, + ); diff != "" { + return fmt.Errorf("work status diff (-got, +want):\n%s", diff) + } + return nil + } +} + +func appliedWorkStatusUpdated(memberClient client.Client, workName string, appliedResources []placementv1alpha1.AppliedResource) func() error { + return func() error { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + + // Prepare the expected AppliedWork object status. + wantAppliedWorkStatus := placementv1alpha1.AppliedWorkStatus{ + AppliedResources: appliedResources, + } + if diff := cmp.Diff(appliedWork.Status, wantAppliedWorkStatus); diff != "" { + return fmt.Errorf("appliedWork status diff (-got, +want):\n%s", diff) + } + return nil + } +} + +func deleteWorkObject(workName, memberClusterReservedNSName string) { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberClusterReservedNSName, + }, + } + Expect(hubClient.Delete(ctx, work)).To(Succeed(), "Failed to delete the Work object") +} + +// updateWorkObject replaces the manifests and the sync strategy of an existing work object. +func updateWorkObject(workName, memberClusterReservedNSName string, syncStrategy *placementv1alpha1.SyncStrategy, rawManifestJSON ...[]byte) { + manifests := make([]placementv1alpha1.Manifest, len(rawManifestJSON)) + for idx := range rawManifestJSON { + manifests[idx] = placementv1alpha1.Manifest{ + RawExtension: runtime.RawExtension{ + Raw: rawManifestJSON[idx], + }, + } + } + + Eventually(func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: memberClusterReservedNSName}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + work.Spec.Manifests = manifests + work.Spec.SyncStrategy = syncStrategy + // The work applier might update the work object (e.g., its status) at the same time; retry on conflicts. + return hubClient.Update(ctx, work) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the Work object") +} + +// updateLinkedWorkObject replaces the manifests and the sync strategy of an existing work object, and links it +// to the given primary placement resource snapshot; this mimics how KubeFleet refreshes work objects during a +// rollout. +func updateLinkedWorkObject(workName, memberClusterReservedNSName, primarySnapshotName string, syncStrategy *placementv1alpha1.SyncStrategy, rawManifestJSON ...[]byte) { + manifests := make([]placementv1alpha1.Manifest, len(rawManifestJSON)) + for idx := range rawManifestJSON { + manifests[idx] = placementv1alpha1.Manifest{ + RawExtension: runtime.RawExtension{ + Raw: rawManifestJSON[idx], + }, + } + } + + Eventually(func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: memberClusterReservedNSName}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + work.Annotations[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey] = primarySnapshotName + work.Spec.Manifests = manifests + work.Spec.SyncStrategy = syncStrategy + // The work applier might update the work object (e.g., its status) at the same time; retry on conflicts. + return hubClient.Update(ctx, work) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the Work object") +} + +// updatePrimaryLinkedWorkObject replaces the manifests and the sync strategy of an existing primary work object, +// links it to the given primary placement resource snapshot, and sets its linked work count; this mimics how +// KubeFleet refreshes primary work objects during a rollout. +func updatePrimaryLinkedWorkObject(workName, memberClusterReservedNSName, primarySnapshotName, linkedWorkCount string, syncStrategy *placementv1alpha1.SyncStrategy, rawManifestJSON ...[]byte) { + manifests := make([]placementv1alpha1.Manifest, len(rawManifestJSON)) + for idx := range rawManifestJSON { + manifests[idx] = placementv1alpha1.Manifest{ + RawExtension: runtime.RawExtension{ + Raw: rawManifestJSON[idx], + }, + } + } + + Eventually(func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: memberClusterReservedNSName}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + work.Annotations[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey] = primarySnapshotName + work.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = linkedWorkCount + work.Spec.Manifests = manifests + work.Spec.SyncStrategy = syncStrategy + // The work applier might update the work object (e.g., its status) at the same time; retry on conflicts. + return hubClient.Update(ctx, work) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the primary Work object") +} + +func appliedWorkBeingDeletedInForegroundActual(memberClient client.Client, workName string) func() error { + return func() error { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + + // Check that the AppliedWork object has been marked for deletion with the foreground propagation policy. + // + // Note that as there are no real built-in controllers (the GC controller) in this test environment, the + // AppliedWork object will stay in the pending deletion state with the foreground deletion finalizer. + if appliedWork.DeletionTimestamp.IsZero() { + return fmt.Errorf("appliedWork object has not been marked for deletion") + } + if !controllerutil.ContainsFinalizer(appliedWork, metav1.FinalizerDeleteDependents) { + return fmt.Errorf("appliedWork object is not being deleted in the foreground (finalizers: %v)", appliedWork.Finalizers) + } + return nil + } +} + +// appliedWorkOwnerRefUnblockedActual checks that the given object is still owned by the AppliedWork object, +// but the owner reference no longer blocks the deletion of the AppliedWork object. +func appliedWorkOwnerRefUnblockedActual(memberClient client.Client, obj client.Object, appliedWorkOwnerRef *metav1.OwnerReference) func() error { + return func() error { + if err := memberClient.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil { + return fmt.Errorf("failed to retrieve the object: %w", err) + } + + wantOwnerRef := appliedWorkOwnerRef.DeepCopy() + wantOwnerRef.BlockOwnerDeletion = ptr.To(false) + wantOwnerRefs := []metav1.OwnerReference{*wantOwnerRef} + if diff := cmp.Diff(obj.GetOwnerReferences(), wantOwnerRefs); diff != "" { + return fmt.Errorf("owner references diff (-got +want):\n%s", diff) + } + return nil + } +} + +// workFinalizerRemovedActual checks that the work object still exists, but no longer has the cleanup finalizer. +func workFinalizerRemovedActual(workNS, workName string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + if controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + return fmt.Errorf("cleanup finalizer has not been removed") + } + return nil + } +} + +// workNotProcessedActual checks that the work applier has never processed a work object, i.e., the work object exists +// with no cleanup finalizer and no status, and no appliedWork object has been created for it. +func workNotProcessedActual(memberClient client.Client, workNS, workName string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + if controllerutil.ContainsFinalizer(work, workApplierCleanupFinalizer) { + return fmt.Errorf("work object %s has the cleanup finalizer", workName) + } + if diff := cmp.Diff(work.Status, placementv1alpha1.WorkStatus{}, cmpopts.EquateEmpty()); diff != "" { + return fmt.Errorf("work object %s status diff (-got +want):\n%s", workName, diff) + } + + appliedWork := &placementv1alpha1.AppliedWork{} + err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork) + switch { + case err == nil: + return fmt.Errorf("an appliedWork object has been created for work object %s", workName) + case !apierrors.IsNotFound(err): + return fmt.Errorf("failed to retrieve the appliedWork object: %w", err) + } + return nil + } +} + +// appliedWorkAbsentActual checks that the AppliedWork object does not exist in the member cluster. +// +// Unlike appliedWorkRemovedActual, this function does not attempt to remove any finalizer from the object. +func appliedWorkAbsentActual(memberClient client.Client, workName string) func() error { + return func() error { + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); !apierrors.IsNotFound(err) { + return fmt.Errorf("appliedWork object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func appliedWorkRemovedActual(memberClient client.Client, workName string) func() error { + return func() error { + // Retrieve the AppliedWork object. + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + if apierrors.IsNotFound(err) { + // The AppliedWork object has been deleted, which is expected. + return nil + } + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + if !appliedWork.DeletionTimestamp.IsZero() && + (controllerutil.ContainsFinalizer(appliedWork, metav1.FinalizerDeleteDependents) || controllerutil.ContainsFinalizer(appliedWork, metav1.FinalizerOrphanDependents)) { + // The AppliedWork object is being deleted (in the foreground, or with its dependents orphaned), but the + // finalizer is still present. Remove the finalizer as there are no real built-in controllers in this + // test environment to handle garbage collection. + controllerutil.RemoveFinalizer(appliedWork, metav1.FinalizerDeleteDependents) + controllerutil.RemoveFinalizer(appliedWork, metav1.FinalizerOrphanDependents) + Expect(memberClient.Update(ctx, appliedWork)).To(Succeed(), "Failed to remove the finalizer from the AppliedWork object") + } + return fmt.Errorf("appliedWork object still exists") + } +} + +// appliedWorkBeingDeletedWithOrphanActual checks that the AppliedWork object has been marked for deletion with the +// orphan propagation policy, i.e., its dependents are to be left behind. +func appliedWorkBeingDeletedWithOrphanActual(memberClient client.Client, workName string) func() error { + return func() error { + appliedWork := &placementv1alpha1.AppliedWork{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: workName}, appliedWork); err != nil { + return fmt.Errorf("failed to retrieve the AppliedWork object: %w", err) + } + + // Note that as there are no real built-in controllers (the GC controller) in this test environment, the + // AppliedWork object will stay in the pending deletion state with the orphan finalizer. + if appliedWork.DeletionTimestamp.IsZero() { + return fmt.Errorf("appliedWork object has not been marked for deletion") + } + if !controllerutil.ContainsFinalizer(appliedWork, metav1.FinalizerOrphanDependents) { + return fmt.Errorf("appliedWork object is not being deleted with the orphan propagation policy (finalizers: %v)", appliedWork.Finalizers) + } + return nil + } +} + +func regularDeployRemovedActual(memberClient client.Client, nsName, deployName string) func() error { + return func() error { + // Delete the Deployment object manually, as there are no real built-in controllers in this + // test environment to handle garbage collection. + deploy := &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: deployName, + }, + } + if err := memberClient.Delete(ctx, deploy); err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("failed to delete the Deployment object: %w", err) + } + + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, deploy); !apierrors.IsNotFound(err) { + return fmt.Errorf("deployment object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func regularConfigMapRemovedActual(memberClient client.Client, nsName, configMapName string) func() error { + return func() error { + // Delete the ConfigMap object manually, as there are no real built-in controllers in this + // test environment to handle garbage collection. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: configMapName, + }, + } + if err := memberClient.Delete(ctx, configMap); err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("failed to delete the ConfigMap object: %w", err) + } + + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, configMap); !apierrors.IsNotFound(err) { + return fmt.Errorf("configMap object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +// regularConfigMapAbsentActual checks that the ConfigMap object does not exist in the member cluster. +// +// Unlike regularConfigMapRemovedActual, this function does not attempt to delete the object by itself. +func regularConfigMapAbsentActual(memberClient client.Client, nsName, configMapName string) func() error { + return func() error { + configMap := &corev1.ConfigMap{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, configMap); !apierrors.IsNotFound(err) { + return fmt.Errorf("configMap object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +// regularDeployAbsentActual checks that the Deployment object does not exist in the member cluster. +// +// Unlike regularDeployRemovedActual, this function does not attempt to delete the object by itself. +func regularDeployAbsentActual(memberClient client.Client, nsName, deployName string) func() error { + return func() error { + deploy := &appsv1.Deployment{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: deployName}, deploy); !apierrors.IsNotFound(err) { + return fmt.Errorf("deployment object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func workRemovedActual(workNS, workName string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + if apierrors.IsNotFound(err) { + return nil + } + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + return fmt.Errorf("work object still exists (finalizers: %v)", work.Finalizers) + } +} + +func regularHPARemovedActual(memberClient client.Client, nsName, hpaName string) func() error { + return func() error { + // Delete the HorizontalPodAutoscaler object manually, as there are no real built-in controllers in this + // test environment to handle garbage collection. + hpa := &autoscalingv2.HorizontalPodAutoscaler{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: hpaName, + }, + } + if err := memberClient.Delete(ctx, hpa); err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("failed to delete the HorizontalPodAutoscaler object: %w", err) + } + + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: hpaName}, hpa); !apierrors.IsNotFound(err) { + return fmt.Errorf("horizontalPodAutoscaler object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +// configMapStableActual checks that the ConfigMap object exists with the given UID, i.e., it has never been deleted +// or re-created, and that it has the given data. If wantData is nil, the data is not checked. +func configMapStableActual(memberClient client.Client, nsName, configMapName string, wantUID types.UID, wantData map[string]string) func() error { + return func() error { + gotConfigMap := &corev1.ConfigMap{} + if err := memberClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: configMapName}, gotConfigMap); err != nil { + return fmt.Errorf("failed to retrieve the ConfigMap object: %w", err) + } + + if gotConfigMap.UID != wantUID { + return fmt.Errorf("configMap UID = %s, want %s (the object has been re-created)", gotConfigMap.UID, wantUID) + } + if wantData == nil { + return nil + } + if diff := cmp.Diff(gotConfigMap.Data, wantData); diff != "" { + return fmt.Errorf("configMap data diff (-got +want):\n%s", diff) + } + return nil + } +} + +// objectUntouchedActual checks that an object in the member cluster still exists and has not been written to since +// it was last retrieved, i.e., it keeps the resource version of the given object. Any write to the object, including +// changes to its owner references, bumps the resource version. +func objectUntouchedActual(memberClient client.Client, wantObj client.Object) func() error { + return func() error { + gotObj, ok := wantObj.DeepCopyObject().(client.Object) + if !ok { + return fmt.Errorf("failed to copy the object %s", client.ObjectKeyFromObject(wantObj)) + } + if err := memberClient.Get(ctx, client.ObjectKeyFromObject(wantObj), gotObj); err != nil { + return fmt.Errorf("failed to retrieve the object %s: %w", client.ObjectKeyFromObject(wantObj), err) + } + + if gotObj.GetResourceVersion() != wantObj.GetResourceVersion() { + return fmt.Errorf("object %s resource version = %s, want %s (the object has been modified; owner references: %v)", + client.ObjectKeyFromObject(wantObj), gotObj.GetResourceVersion(), wantObj.GetResourceVersion(), gotObj.GetOwnerReferences()) + } + return nil + } +} + +// objectNotRecreatedActual checks that an object in the member cluster still exists with the UID of the given +// object, i.e., it has never been deleted or re-created since it was last retrieved. +func objectNotRecreatedActual(memberClient client.Client, wantObj client.Object) func() error { + return func() error { + gotObj, ok := wantObj.DeepCopyObject().(client.Object) + if !ok { + return fmt.Errorf("failed to copy the object %s", client.ObjectKeyFromObject(wantObj)) + } + if err := memberClient.Get(ctx, client.ObjectKeyFromObject(wantObj), gotObj); err != nil { + return fmt.Errorf("failed to retrieve the object %s: %w", client.ObjectKeyFromObject(wantObj), err) + } + + if gotObj.GetUID() != wantObj.GetUID() { + return fmt.Errorf("object %s UID = %s, want %s (the object has been re-created)", + client.ObjectKeyFromObject(wantObj), gotObj.GetUID(), wantObj.GetUID()) + } + return nil + } +} + +// configMapsWithPrefixAbsentActual checks that no ConfigMap object in the given namespace has a name that starts +// with the given prefix. +func configMapsWithPrefixAbsentActual(memberClient client.Client, nsName, prefix string) func() error { + return func() error { + configMapList := &corev1.ConfigMapList{} + if err := memberClient.List(ctx, configMapList, client.InNamespace(nsName)); err != nil { + return fmt.Errorf("failed to list ConfigMap objects: %w", err) + } + + for idx := range configMapList.Items { + if strings.HasPrefix(configMapList.Items[idx].Name, prefix) { + return fmt.Errorf("found a ConfigMap object (%s) with the name prefix %s", configMapList.Items[idx].Name, prefix) + } + } + return nil + } +} + +// namespaceAbsentActual checks that the namespace does not exist in the member cluster. +func namespaceAbsentActual(memberClient client.Client, nsName string) func() error { + return func() error { + gotNS := &corev1.Namespace{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: nsName}, gotNS); !apierrors.IsNotFound(err) { + return fmt.Errorf("namespace object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +// namespaceNotManagedActual checks that the namespace exists in the member cluster and is not being deleted, but has +// not been adopted by KubeFleet, i.e., it has no owner references and no KubeFleet annotations. +func namespaceNotManagedActual(memberClient client.Client, nsName string) func() error { + return func() error { + gotNS := &corev1.Namespace{} + if err := memberClient.Get(ctx, client.ObjectKey{Name: nsName}, gotNS); err != nil { + return fmt.Errorf("failed to retrieve the namespace object: %w", err) + } + + if !gotNS.DeletionTimestamp.IsZero() { + return fmt.Errorf("namespace is being deleted (deletion timestamp: %v)", gotNS.DeletionTimestamp) + } + if len(gotNS.OwnerReferences) > 0 { + return fmt.Errorf("namespace has owner references: %v", gotNS.OwnerReferences) + } + for key := range gotNS.Annotations { + if strings.Contains(key, placementv1alpha1.KubeFleetPrefix) { + return fmt.Errorf("namespace has a KubeFleet annotation: %s", key) + } + } + return nil + } +} + +// findManifestStatusByOrdinal returns the status of the manifest with the given ordinal in a Work object. +func findManifestStatusByOrdinal(work *placementv1alpha1.Work, ordinal int) *placementv1alpha1.PerManifestStatus { + for idx := range work.Status.Manifests { + if work.Status.Manifests[idx].Identifier.Ordinal == ordinal { + return &work.Status.Manifests[idx] + } + } + return nil +} + +// manifestAppliedCondMessageActual checks that the Applied condition of the manifest with the given ordinal in a +// Work object has a message that contains the given substring. +func manifestAppliedCondMessageActual(workNS, workName string, ordinal int, wantSubstr string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + manifestStatus := findManifestStatusByOrdinal(work, ordinal) + if manifestStatus == nil { + return fmt.Errorf("no status found for the manifest with ordinal %d", ordinal) + } + appliedCond := meta.FindStatusCondition(manifestStatus.Conditions, placementv1alpha1.ManifestCondTypeApplied) + if appliedCond == nil { + return fmt.Errorf("no Applied condition found for the manifest with ordinal %d", ordinal) + } + if !strings.Contains(appliedCond.Message, wantSubstr) { + return fmt.Errorf("applied condition message = %q, want a message that contains %q", appliedCond.Message, wantSubstr) + } + return nil + } +} + +// retrieveManifestCondLTT returns the last transition time of a condition of the manifest with the given ordinal +// in a Work object. +func retrieveManifestCondLTT(workNS, workName string, ordinal int, condType string) metav1.Time { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work)).To(Succeed(), "Failed to retrieve the Work object") + + manifestStatus := findManifestStatusByOrdinal(work, ordinal) + Expect(manifestStatus).NotTo(BeNil(), "No status found for the manifest with ordinal %d", ordinal) + cond := meta.FindStatusCondition(manifestStatus.Conditions, condType) + Expect(cond).NotTo(BeNil(), "No %s condition found for the manifest with ordinal %d", condType, ordinal) + return cond.LastTransitionTime +} + +// retrieveManifestFirstDiffedTimestamp returns the timestamp when diffs were first observed on the manifest with the +// given ordinal in a Work object. +func retrieveManifestFirstDiffedTimestamp(workNS, workName string, ordinal int) metav1.Time { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work)).To(Succeed(), "Failed to retrieve the Work object") + + manifestStatus := findManifestStatusByOrdinal(work, ordinal) + Expect(manifestStatus).NotTo(BeNil(), "No status found for the manifest with ordinal %d", ordinal) + Expect(manifestStatus.DiffDetails).NotTo(BeNil(), "No diff details found for the manifest with ordinal %d", ordinal) + return manifestStatus.DiffDetails.FirstDiffedObservedTimestamp +} + +// manifestCondLTTActual checks that a condition of the manifest with the given ordinal in a Work object has the +// given last transition time, i.e., the condition has not transitioned since the time was captured. +func manifestCondLTTActual(workNS, workName string, ordinal int, condType string, wantLTT metav1.Time) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + manifestStatus := findManifestStatusByOrdinal(work, ordinal) + if manifestStatus == nil { + return fmt.Errorf("no status found for the manifest with ordinal %d", ordinal) + } + cond := meta.FindStatusCondition(manifestStatus.Conditions, condType) + if cond == nil { + return fmt.Errorf("no %s condition found for the manifest with ordinal %d", condType, ordinal) + } + if !cond.LastTransitionTime.Equal(&wantLTT) { + return fmt.Errorf("%s condition last transition time = %v, want %v", condType, cond.LastTransitionTime, wantLTT) + } + return nil + } +} + +// nudgeWork bumps the generation of a Work object by setting a random value on a test-only label of its first +// manifest. For a primary Work object, this triggers an immediate reconciliation. +func nudgeWork(workName, memberClusterReservedNSName string) { + Eventually(func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: memberClusterReservedNSName}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + if len(work.Spec.Manifests) == 0 { + return fmt.Errorf("the Work object has no manifest to nudge") + } + + manifestObj := &unstructured.Unstructured{} + if err := manifestObj.UnmarshalJSON(work.Spec.Manifests[0].Raw); err != nil { + return fmt.Errorf("failed to unmarshal the first manifest: %w", err) + } + labels := manifestObj.GetLabels() + if labels == nil { + labels = map[string]string{} + } + labels[nudgeLabelKey] = utils.RandStr() + manifestObj.SetLabels(labels) + nudgedManifestJSON, err := manifestObj.MarshalJSON() + if err != nil { + return fmt.Errorf("failed to marshal the first manifest: %w", err) + } + + work.Spec.Manifests[0].Raw = nudgedManifestJSON + // The work applier might update the work object (e.g., its status) at the same time; retry on conflicts. + return hubClient.Update(ctx, work) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to nudge the Work object") +} + +// workProcessedAtLatestGenerationActual checks that the work applier has processed the latest generation of a +// Work object, i.e., the Applied condition of the Work object reports the current generation. +func workProcessedAtLatestGenerationActual(workNS, workName string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: workName, Namespace: workNS}, work); err != nil { + return fmt.Errorf("failed to retrieve the Work object: %w", err) + } + + appliedCond := meta.FindStatusCondition(work.Status.Conditions, placementv1alpha1.WorkCondTypeApplied) + if appliedCond == nil { + return fmt.Errorf("no Applied condition found on the Work object") + } + if appliedCond.ObservedGeneration != work.Generation { + return fmt.Errorf("applied condition observed generation = %d, want %d", appliedCond.ObservedGeneration, work.Generation) + } + return nil + } +} + +const ( + // The tolerances for comparing the observed gap between two consecutive reconciliations with the requeue delay + // that the requeue rate limiter returns. The observed gap also includes the duration of the reconciliation itself + // (and the scheduling latency), which might take a while when the test suite runs with the race detector and in + // parallel. + requeueGapLowerTolerance = time.Millisecond * 250 + requeueGapUpperTolerance = time.Millisecond * 1500 +) + +// parallelizerWithFixedDelay is a parallelizer that pauses for a fixed delay after the work applier processes each +// wave of manifests, so that the order in which the waves are processed shows in the creation timestamps (which are +// of the second granularity) of the applied objects. +type parallelizerWithFixedDelay struct { + parallelizer.Parallelizer + delay time.Duration +} + +// ParallelizeUntil runs the tasks in parallel, and pauses for the fixed delay afterwards if the tasks process a wave +// of manifests. +func (p *parallelizerWithFixedDelay) ParallelizeUntil(ctx context.Context, pieces int, doWork workqueue.DoWorkPieceFunc, operation string) { + p.Parallelizer.ParallelizeUntil(ctx, pieces, doWork, operation) + if !strings.HasPrefix(operation, "processingManifestsInWave") { + return + } + + select { + case <-ctx.Done(): + case <-time.After(p.delay): + } +} + +// recordingClient is a client that records the timestamps of the Get calls on work objects (the work applier starts +// each reconciliation of a primary work object with such a call) and counts the successful status updates on each +// object. +type recordingClient struct { + client.Client + + mu sync.Mutex + workGetTimestamps map[types.NamespacedName][]time.Time + statusUpdateCounts map[types.NamespacedName]int +} + +// newRecordingClient wraps the given client with a recordingClient. +func newRecordingClient(c client.Client) *recordingClient { + return &recordingClient{ + Client: c, + workGetTimestamps: make(map[types.NamespacedName][]time.Time), + statusUpdateCounts: make(map[types.NamespacedName]int), + } +} + +// Get retrieves an object, and records the timestamp if the object is a work object. +func (c *recordingClient) Get(ctx context.Context, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, isWork := obj.(*placementv1alpha1.Work); isWork { + c.mu.Lock() + c.workGetTimestamps[key] = append(c.workGetTimestamps[key], time.Now()) + c.mu.Unlock() + } + return c.Client.Get(ctx, key, obj, opts...) +} + +// Status returns a status writer that counts the successful status updates. +func (c *recordingClient) Status() client.SubResourceWriter { + return &recordingStatusWriter{ + SubResourceWriter: c.Client.Status(), + recorder: c, + } +} + +// reconcileStartTimestamps returns the timestamps of the Get calls on the given work object, i.e., the timestamps +// when the reconciliations of the (primary) work object started. +func (c *recordingClient) reconcileStartTimestamps(key types.NamespacedName) []time.Time { + c.mu.Lock() + defer c.mu.Unlock() + return slices.Clone(c.workGetTimestamps[key]) +} + +// statusUpdateCount returns the count of the successful status updates on the given object. +func (c *recordingClient) statusUpdateCount(key types.NamespacedName) int { + c.mu.Lock() + defer c.mu.Unlock() + return c.statusUpdateCounts[key] +} + +// recordingStatusWriter is a status writer that counts the successful status updates on each object. +type recordingStatusWriter struct { + client.SubResourceWriter + recorder *recordingClient +} + +// Update updates the status of an object, and counts the update if it succeeds. +func (w *recordingStatusWriter) Update(ctx context.Context, obj client.Object, opts ...client.SubResourceUpdateOption) error { + if err := w.SubResourceWriter.Update(ctx, obj, opts...); err != nil { + return err + } + w.recorder.mu.Lock() + w.recorder.statusUpdateCounts[client.ObjectKeyFromObject(obj)]++ + w.recorder.mu.Unlock() + return nil +} + +// requeueGapsActual checks that the gaps between consecutive reconciliations of a work object, as recorded by the +// given recording client, match the given requeue delays (within tolerance). The gaps are counted from the first +// reconciliation at or after the given time, after skipping the given number of reconciliations. +func requeueGapsActual(recorder *recordingClient, workKey types.NamespacedName, since time.Time, skip int, wantGaps []time.Duration) func() error { + return func() error { + timestamps := recorder.reconcileStartTimestamps(workKey) + startIdx := slices.IndexFunc(timestamps, func(t time.Time) bool { return !t.Before(since) }) + if startIdx != -1 { + startIdx += skip + } + if startIdx == -1 || len(timestamps)-startIdx < len(wantGaps)+1 { + return fmt.Errorf("not enough reconciliations have been observed since %v (observed: %v)", since, timestamps) + } + + gotGaps := make([]time.Duration, 0, len(wantGaps)) + for idx := startIdx + 1; idx <= startIdx+len(wantGaps); idx++ { + gotGaps = append(gotGaps, timestamps[idx].Sub(timestamps[idx-1])) + } + for idx := range wantGaps { + if gotGaps[idx] < wantGaps[idx]-requeueGapLowerTolerance || gotGaps[idx] > wantGaps[idx]+requeueGapUpperTolerance { + return fmt.Errorf("requeue gaps = %v, want %v (with tolerance -%v/+%v)", gotGaps, wantGaps, requeueGapLowerTolerance, requeueGapUpperTolerance) + } + } + return nil + } +} + +// retrieveCreationTimestamp returns the creation timestamp of an object in the member cluster. +func retrieveCreationTimestamp(memberClient client.Client, obj client.Object) time.Time { + gotObj, ok := obj.DeepCopyObject().(client.Object) + Expect(ok).To(BeTrue(), "Failed to copy the object %s", client.ObjectKeyFromObject(obj)) + Expect(memberClient.Get(ctx, client.ObjectKeyFromObject(obj), gotObj)).To(Succeed(), "Failed to retrieve the object %s", client.ObjectKeyFromObject(obj)) + return gotObj.GetCreationTimestamp().Time +} diff --git a/pkg/v1/controllers/workapplier/waves.go b/pkg/v1/controllers/workapplier/waves.go new file mode 100644 index 000000000..16074cbf4 --- /dev/null +++ b/pkg/v1/controllers/workapplier/waves.go @@ -0,0 +1,180 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "slices" + + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/klog/v2" +) + +type waveNumber int + +const ( + lastWave waveNumber = 999 +) + +var ( + // The default wave number for all known Kubernetes resource type. + // + // Note (chenyu1): the waves below are based on the Helm resource installation + // order (see also the Helm source code). Similar objects are grouped together + // to achieve best performance. + defaultWaveNumberByResourceType = map[string]waveNumber{ + // Apply namespaces and priority classes first. + "namespaces": 0, + "priorityclasses": 0, + // Apply policies, configuration data, and other static resources second. + "networkpolicies": 1, + "resourcequotas": 1, + "limitranges": 1, + "podsecuritypolicies": 1, + "poddisruptionbudgets": 1, + "serviceaccounts": 1, + "secrets": 1, + "configmaps": 1, + "storageclasses": 1, + "persistentvolumes": 1, + "persistentvolumeclaims": 1, + "customresourcedefinitions": 1, + "ingressclasses": 1, + // Apply RBAC resources (cluster roles and roles). + "clusterroles": 2, + "roles": 2, + // Apply RBAC resources (cluster role bindings and role bindings). + "clusterrolebindings": 3, + "rolebindings": 3, + // Apply workloads and services. + "services": 4, + "daemonsets": 4, + "pods": 4, + "replicationcontrollers": 4, + "replicasets": 4, + "deployments": 4, + "horizontalpodautoscalers": 4, + "statefulsets": 4, + "jobs": 4, + "cronjobs": 4, + "ingresses": 4, + // Apply API services and webhooks. + "apiservices": 5, + "validatingwebhookconfigurations": 5, + "mutatingwebhookconfigurations": 5, + } + + // The API groups for all known Kubernetes resource types. + knownAPIGroups = sets.New( + // The core API group. + "", + // The networking API group (`networking.k8s.io`). + "networking.k8s.io", + // The scheduling API group (`scheduling.k8s.io`). + "scheduling.k8s.io", + // The policy API group (`policy.k8s.io`). + "policy", + // The storage API group (`storage.k8s.io`). + "storage.k8s.io", + // The API extensions API group (`apiextensions.k8s.io`). + "apiextensions.k8s.io", + // The RBAC authorization API group (`rbac.authorization.k8s.io`). + "rbac.authorization.k8s.io", + // The apps API group (`apps`). + "apps", + // The autoscaling API group (`autoscaling`). + "autoscaling", + // The API registration API group (`apiregistration.k8s.io`). + "apiregistration.k8s.io", + // The admission registration API group (`admissionregistration.k8s.io`). + "admissionregistration.k8s.io", + // The batch API group (`batch`). + "batch", + ) +) + +// manifestProcessingWave is a wave of manifests that can be processed in parallel. +type manifestProcessingWave struct { + num waveNumber + states []*manifestProcessingState +} + +// organizeManifestsIntoProcessingWaves organizes the list of manifest processing states into +// different waves for parallel processing based on their GVR information. +func organizeManifestsIntoProcessingWaves(manifestProcessingStates []*manifestProcessingState) []*manifestProcessingWave { + // Pre-allocate the map; 7 is the total count of default wave numbers, though + // not all wave numbers might be used. + waveByNum := make(map[waveNumber]*manifestProcessingWave, 7) + + getOrAddWave := func(num waveNumber) *manifestProcessingWave { + wave, ok := waveByNum[num] + if !ok { + wave = &manifestProcessingWave{ + num: num, + // Pre-allocate with a reasonable size. + states: make([]*manifestProcessingState, 0, 5), + } + waveByNum[num] = wave + } + return wave + } + + // For simplicity reasons, the organization itself runs in sequential order. + // Considering that the categorization itself is quick and the total number of manifests + // should be limited in most cases, this should not introduce significant overhead. + for idx := range manifestProcessingStates { + state := manifestProcessingStates[idx] + if state.gvr == nil { + // For manifest data that cannot be decoded, there might not be any available GVR + // information. Skip such manifests; this is not considered as an error. + klog.V(2).InfoS("Skipping a manifest with no GVR; no wave is assigned", + "ordinal", idx, "work", klog.KObj(state.fromWorkObj)) + continue + } + + if state.applyErr != nil { + // An error has occurred before this step; such manifests need no further processing, + // skip them. This is not considered as an error. + klog.V(2).InfoS("Skipping a manifest with prior processing error; no wave is assigned", + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, "work", klog.KObj(state.fromWorkObj)) + continue + } + + waveNum := lastWave + defaultWaveNum, foundInDefaultWaveNumber := defaultWaveNumberByResourceType[state.gvr.Resource] + if foundInDefaultWaveNumber && knownAPIGroups.Has(state.gvr.Group) { + // The resource is a known one; assign the manifest to its default wave. + waveNum = defaultWaveNum + } + + wave := getOrAddWave(waveNum) + wave.states = append(wave.states, state) + klog.V(2).InfoS("Assigned manifest to a wave", + "waveNumber", waveNum, + "manifestObj", klog.KObj(state.manifestObj), "GVR", *state.gvr, "work", klog.KObj(state.fromWorkObj)) + } + + // Retrieve all the waves and sort them by their wave number. + waves := make([]*manifestProcessingWave, 0, len(waveByNum)) + for _, w := range waveByNum { + waves = append(waves, w) + } + // Sort the waves in ascending order. + slices.SortFunc(waves, func(a, b *manifestProcessingWave) int { + return int(a.num) - int(b.num) + }) + return waves +} diff --git a/pkg/v1/controllers/workapplier/waves_integration_test.go b/pkg/v1/controllers/workapplier/waves_integration_test.go new file mode 100644 index 000000000..934910a81 --- /dev/null +++ b/pkg/v1/controllers/workapplier/waves_integration_test.go @@ -0,0 +1,588 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workapplier + +import ( + "fmt" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + schedulingv1 "k8s.io/api/scheduling/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +// The test cases below use the work applier dedicated to the wave test cases (workApplierWaves), which pauses for +// parallelizerFixedDelay after processing each wave of manifests. Objects in the same wave are created at about the +// same time; objects in consecutive waves are created at least parallelizerFixedDelay apart. Had an object not been +// applied in the expected wave, it would have been applied in a later reconciliation, at least one requeue delay +// (and several wave delays) later; the gaps are hence also checked against an upper bound. +var _ = Describe("waves", func() { + Context("single wave", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + priorityClassName := fmt.Sprintf("priority-class-%s", utils.RandStr()) + + BeforeAll(func() { + // Prepare the manifests; namespaces and priority classes are both in the first wave. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularPriorityClass := &schedulingv1.PriorityClass{ + TypeMeta: metav1.TypeMeta{ + APIVersion: "scheduling.k8s.io/v1", + Kind: "PriorityClass", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: priorityClassName, + }, + Value: 1000, + } + regularPriorityClassJSON := marshalK8sObjJSON(regularPriorityClass) + + createWorkObject(workName, memberReservedNSNameWaves, nil, regularNSJSON, regularPriorityClassJSON) + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "scheduling.k8s.io", + APIVersion: "v1", + Kind: "PriorityClass", + Resource: "priorityclasses", + Name: priorityClassName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 1, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 1, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSNameWaves, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should create the objects in the same wave", func() { + nsCreationTimestamp := retrieveCreationTimestamp(memberClient1, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}}) + priorityClassCreationTimestamp := retrieveCreationTimestamp(memberClient1, &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: priorityClassName}}) + gap := priorityClassCreationTimestamp.Sub(nsCreationTimestamp) + Expect(gap).To(BeNumerically("~", 0, time.Second), "Objects in the same wave are not created at about the same time (gap: %v)", gap) + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSNameWaves) + + // Remove the priority class manually, as there is no GC controller in the test environment. + priorityClass := &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: priorityClassName}} + Expect(client.IgnoreNotFound(memberClient1.Delete(ctx, priorityClass))).To(Succeed(), "Failed to delete the priority class") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSNameWaves, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("two consecutive waves", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + + BeforeAll(func() { + // Prepare the manifests; namespaces are in the first wave (wave 0), and config maps are in the second + // wave (wave 1). + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // List the config map first; the manifest order does not matter. + createWorkObject(workName, memberReservedNSNameWaves, nil, regularConfigMapJSON, regularNSJSON) + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSNameWaves, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should create the objects in waves", func() { + nsCreationTimestamp := retrieveCreationTimestamp(memberClient1, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}}) + configMapCreationTimestamp := retrieveCreationTimestamp(memberClient1, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: configMapName}}) + gap := configMapCreationTimestamp.Sub(nsCreationTimestamp) + Expect(gap).To(BeNumerically(">=", parallelizerFixedDelay), "The config map is not created one wave after the namespace (gap: %v)", gap) + Expect(gap).To(BeNumerically("<", 2*parallelizerFixedDelay), "The config map is not created in the wave right after the namespace (gap: %v)", gap) + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSNameWaves) + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSNameWaves, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("two non-consecutive waves", Ordered, func() { + workName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + // The environment prepared by the envtest package does not support namespace + // deletion; each test case would use a new namespace. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + roleName := "role-1" + + BeforeAll(func() { + // Prepare the manifests; namespaces are in the first wave (wave 0), and roles are in the third wave (wave + // 2). No manifest is in the second wave (wave 1). + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularRole := &rbacv1.Role{ + TypeMeta: metav1.TypeMeta{ + APIVersion: "rbac.authorization.k8s.io/v1", + Kind: "Role", + }, + ObjectMeta: metav1.ObjectMeta{ + Namespace: nsName, + Name: roleName, + }, + Rules: []rbacv1.PolicyRule{ + { + APIGroups: []string{""}, + Resources: []string{"configmaps"}, + Verbs: []string{"get"}, + }, + }, + } + regularRoleJSON := marshalK8sObjJSON(regularRole) + + createWorkObject(workName, memberReservedNSNameWaves, nil, regularNSJSON, regularRoleJSON) + }) + + It("should update the work object status", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + manifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 1, + APIGroup: "rbac.authorization.k8s.io", + APIVersion: "v1", + Kind: "Role", + Resource: "roles", + Name: roleName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + workStatusUpdatedActual := workStatusUpdated(memberReservedNSNameWaves, workName, workConds, manifestStatuses) + Eventually(workStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update work status") + }) + + It("should create the objects in waves, skipping the empty wave", func() { + // No delay is added for the empty wave. + nsCreationTimestamp := retrieveCreationTimestamp(memberClient1, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}}) + roleCreationTimestamp := retrieveCreationTimestamp(memberClient1, &rbacv1.Role{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: roleName}}) + gap := roleCreationTimestamp.Sub(nsCreationTimestamp) + Expect(gap).To(BeNumerically(">=", parallelizerFixedDelay), "The role is not created one wave after the namespace (gap: %v)", gap) + Expect(gap).To(BeNumerically("<", 2*parallelizerFixedDelay), "The role is not created in the wave right after the namespace (gap: %v)", gap) + }) + + AfterAll(func() { + // Delete the work object and related resources. + deleteWorkObject(workName, memberReservedNSNameWaves) + + // Remove the role manually, as there is no GC controller in the test environment. + role := &rbacv1.Role{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: roleName}} + Expect(client.IgnoreNotFound(memberClient1.Delete(ctx, role))).To(Succeed(), "Failed to delete the role") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork object. + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object") + + workRemovedActual := workRemovedActual(memberReservedNSNameWaves, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) + + Context("waves across linked work objects", Ordered, func() { + // The two work objects are linked to each other: they share the same owner placement policy/binding and the + // same primary placement resource snapshot; the primary work object features the linked work count + // annotation. + primaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + secondaryWorkName := fmt.Sprintf(workNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, primaryWorkName) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, primaryWorkName) + primarySnapshotName := fmt.Sprintf(primarySnapshotNameTemplate, primaryWorkName) + // The namespace is created from the namespace manifest in the secondary work object. + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + primaryWorkKey := types.NamespacedName{Namespace: memberReservedNSNameWaves, Name: primaryWorkName} + secondaryWorkKey := types.NamespacedName{Namespace: memberReservedNSNameWaves, Name: secondaryWorkName} + + BeforeAll(func() { + // Prepare the manifests. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // The secondary work object features the namespace manifest. + secondaryWork := buildWorkObject( + secondaryWorkName, memberReservedNSNameWaves, + placementPolicyName, placementBindingName, primarySnapshotName, + nil, + regularNSJSON, + ) + setPrimaryWorkAsOwner(secondaryWork, primaryWorkName) + Expect(hubClient.Create(ctx, secondaryWork)).To(Succeed(), "Failed to create the secondary work object") + + // The primary work object features the config map manifest, and forbids the work applier from creating + // namespaces on its own; the config map can be applied only after the namespace manifest from the + // secondary work object has been applied. + syncStrategy := &placementv1alpha1.SyncStrategy{ + WhenNamespaceDoesNotExist: placementv1alpha1.WhenNamespaceDoesNotExistOptionReportError, + } + primaryWork := buildWorkObject( + primaryWorkName, memberReservedNSNameWaves, + placementPolicyName, placementBindingName, primarySnapshotName, + syncStrategy, + regularConfigMapJSON, + ) + primaryWork.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = "2" + Expect(hubClient.Create(ctx, primaryWork)).To(Succeed(), "Failed to create the primary work object") + }) + + It("should update the work object statuses", func() { + workConds := []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAppliedCondAllManifestsAppliedReason, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.WorkAvailableCondAllManifestsAvailableReason, + }, + } + primaryManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "ConfigMap", + Resource: "configmaps", + Name: configMapName, + Namespace: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + secondaryManifestStatuses := []placementv1alpha1.PerManifestStatus{ + { + Identifier: placementv1alpha1.ManifestIdentifier{ + Ordinal: 0, + APIGroup: "", + APIVersion: "v1", + Kind: "Namespace", + Resource: "namespaces", + Name: nsName, + }, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + Reason: string(ApplyResTypeApplied), + ObservedGeneration: 0, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: string(AvailabilityResultTypeAvailable), + ObservedGeneration: 0, + }, + }, + }, + } + + primaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSNameWaves, primaryWorkName, workConds, primaryManifestStatuses) + Eventually(primaryWorkStatusUpdatedActual, 2*eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the primary work object") + + secondaryWorkStatusUpdatedActual := workStatusUpdated(memberReservedNSNameWaves, secondaryWorkName, workConds, secondaryManifestStatuses) + Eventually(secondaryWorkStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the status of the secondary work object") + }) + + It("should apply the manifests in waves across the linked work objects", func() { + nsCreationTimestamp := retrieveCreationTimestamp(memberClient1, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}}) + configMapCreationTimestamp := retrieveCreationTimestamp(memberClient1, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Namespace: nsName, Name: configMapName}}) + gap := configMapCreationTimestamp.Sub(nsCreationTimestamp) + Expect(gap).To(BeNumerically(">=", parallelizerFixedDelay), "The config map is not created one wave after the namespace (gap: %v)", gap) + Expect(gap).To(BeNumerically("<", 2*parallelizerFixedDelay), "The config map is not created in the wave right after the namespace (gap: %v)", gap) + }) + + It("should apply all the manifests in one pass", func() { + // Each work object status is updated exactly twice in the first (and only effective) reconciliation: + // once to write ahead the manifests to process, and once to report the results. Had the config map + // failed to apply in the first reconciliation, the primary work object status would have been updated + // again once the config map got applied. + Expect(hubClientRecorderWaves.statusUpdateCount(primaryWorkKey)).To(Equal(2), "The status of the primary work object is not updated exactly twice") + Expect(hubClientRecorderWaves.statusUpdateCount(secondaryWorkKey)).To(Equal(2), "The status of the secondary work object is not updated exactly twice") + }) + + AfterAll(func() { + // There is no GC controller in the hub cluster test environment to delete the secondary work object along + // with the primary one; delete both of them. + for _, workName := range []string{primaryWorkName, secondaryWorkName} { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: workName, + Namespace: memberReservedNSNameWaves, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete work object %s", workName) + } + + // Remove the config map manually, as there is no GC controller in the test environment. + regularConfigMapRemovedActual := regularConfigMapRemovedActual(memberClient1, nsName, configMapName) + Eventually(regularConfigMapRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the config map object") + + // Simulate the GC controller by removing the foreground deletion finalizer from the appliedWork objects. + for _, workName := range []string{primaryWorkName, secondaryWorkName} { + appliedWorkRemovedActual := appliedWorkRemovedActual(memberClient1, workName) + Eventually(appliedWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the appliedWork object %s", workName) + + workRemovedActual := workRemovedActual(memberReservedNSNameWaves, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object %s", workName) + } + + // The namespace is left behind, as the environment prepared by the envtest package does not support + // namespace deletion. + }) + }) +}) diff --git a/pkg/v1/controllers/workgenerator/controller.go b/pkg/v1/controllers/workgenerator/controller.go index be37106e2..480d63187 100644 --- a/pkg/v1/controllers/workgenerator/controller.go +++ b/pkg/v1/controllers/workgenerator/controller.go @@ -51,14 +51,20 @@ type Reconciler struct { hubClient client.Client parallelizer parallelizerutil.Parallelizer + + // rateLimiter is the rate limiter for requeues of failed reconciliations; nil means the controller-runtime default. + rateLimiter workqueue.TypedRateLimiter[reconcile.Request] } -func New(hubClient client.Client, workerCnt int) *Reconciler { +// New returns a new work generator reconciler. If rateLimiter is nil, the default rate limiter of controller-runtime +// is used for requeues of failed reconciliations. +func New(hubClient client.Client, workerCnt int, rateLimiter workqueue.TypedRateLimiter[reconcile.Request]) *Reconciler { parallelizer := parallelizerutil.NewParallelizer(workerCnt) return &Reconciler{ hubClient: hubClient, parallelizer: parallelizer, + rateLimiter: rateLimiter, } } @@ -88,8 +94,7 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu // suspended. if placementBinding.GetDeletionTimestamp() != nil || placementBindingSpec.Suspended { if err := r.cleanupWorks(ctx, placementBinding); err != nil { - wrappedErr := errors.Wraps(err, "failed to clean up work objects for placement binding", - "placementBinding", klog.KObj(placementBinding), "controller", controllerName) + wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), "controller", controllerName) klog.ErrorS(wrappedErr, "failed to clean up work objects for placement binding", errors.Args(wrappedErr)...) return ctrl.Result{}, wrappedErr @@ -98,8 +103,7 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu } // Add the cleanup finalizer if it is not already present. if err := r.addPlacementBindingCleanupFinalizer(ctx, placementBinding); err != nil { - wrappedErr := errors.Wraps(err, "failed to add cleanup finalizer to placement binding", - "placementBinding", klog.KObj(placementBinding), "controller", controllerName) + wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), "controller", controllerName) klog.ErrorS(wrappedErr, "failed to add cleanup finalizer to placement binding", errors.Args(wrappedErr)...) return ctrl.Result{}, wrappedErr } @@ -112,7 +116,7 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{}, wrappedErr } - // Retrieve the Work objects owned by the placement binding. + // Retrieve the work objects owned by the placement binding. works, err := r.listWorksByOwnerBinding(ctx, placementBindingSpec.ClusterName, placementBinding.GetNamespace(), placementBinding.GetName()) if err != nil { wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), @@ -121,65 +125,110 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu return ctrl.Result{}, wrappedErr } - // Check if the Work objects are consistent with the assigned primary and secondary placement resource snapshots. - // If so, no need to update the spec of the Work objects; just sync the status back to the placement binding + // Check if the work objects are consistent with the assigned primary and secondary placement resource snapshots. + // If so, no need to update the spec of the work objects; just sync the status back to the placement binding // instead. // // Note (chenyu1): this check is intended as a shortcut to avoid constant re-generation and validation of // work objects (which can be expensive when there are a large number of manifests to place); once the controller // signals that it has completed processing a placement binding given a specific configuration (a specific // set of placement resource snapshots) and generated all the needed work objects, the control loop will skip - // to status reporting. In general we do not try to guard against byzantine faults here, especially - // considering that work objects are KubeFleet internal API objects that reside in reserved namespaces; if a - // non-KubeFleet agent decides to tamper with work objects, the system is not guaranteed to auto-recover. - // The changes, however, will be overwritten upon rollouts. - upToDate, err := areWorksUpToDate(placementBinding, works) // codespell:ignore + // to status reporting. The check only covers coarse-grained consistency (the last processed snapshot, the + // snapshot links, the work count and the sync strategy), so missing or partially missing work objects are + // regenerated; it does not, however, verify the content of the work objects. In general we do not try to guard + // against byzantine faults here, especially considering that work objects are KubeFleet internal API objects that + // reside in reserved namespaces; if a non-KubeFleet agent tampers with the content of work objects, the system is + // not guaranteed to auto-recover. Any inadvertent changes, however, will be overwritten upon rollouts. + workSyncRes, err := areWorksInSync(placementBinding, works) if err != nil { - wrappedErr := errors.Wraps(err, "failed to check if work objects are up-to-date", + wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, "controller", controllerName) - klog.ErrorS(wrappedErr, "failed to check if work objects are up-to-date", errors.Args(wrappedErr)...) + klog.ErrorS(wrappedErr, "failed to check if work objects are in sync", errors.Args(wrappedErr)...) return ctrl.Result{}, wrappedErr } - if upToDate { // codespell:ignore + switch workSyncRes { + case workSyncStatusInSync: + klog.V(2).InfoS("Work objects are fully in sync with the placement binding; refreshing placement binding status", + "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, + "controller", controllerName) if err := r.refreshPlacementBindingStatus(ctx, placementBinding, works); err != nil { - wrappedErr := errors.Wraps(err, "failed to refresh placement binding status", + wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, "controller", controllerName) klog.ErrorS(wrappedErr, "failed to refresh placement binding status", errors.Args(wrappedErr)...) return ctrl.Result{}, wrappedErr } return ctrl.Result{}, nil + case workSyncStatusStrategyOutOfSync: + // In this situation the work objects always have a spec change, so that the work generator can just + // wait for the status change events on the work objects to arrive to have the binding status refreshed. + // + // Note that the controller does not reset the binding status in this situation, unlike in the scenario below + // (resource changes to apply); it is considered OK as the set of selected resources remains consistent. + klog.V(2).InfoS("There are sync strategy changes not yet applied to the work objects; reconciling", + "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, + "controller", controllerName) + if err := r.refreshSyncStrategyOnlyForWorks(ctx, placementBinding, works); err != nil { + wrappedErr := errors.Wraps(err, "", + "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, + "controller", controllerName) + klog.ErrorS(wrappedErr, "failed to refresh sync strategy for work objects", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + return ctrl.Result{}, nil } + klog.V(2).InfoS("Found resource changes not in sync with the work objects; reconciling", + "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, + "workSyncStatus", workSyncRes, "controller", controllerName) - // The Work objects are absent or not up-to-date. Retrieve the placement resource snapshots and create/update - // the Work objects accordingly. + // The work objects are not in sync (the last processed snapshot differs from the primary snapshot in the spec, or + // some work objects are absent or inconsistent). Retrieve the placement resource snapshots and create/update + // the work objects accordingly. // Retrieve the assigned primary and secondary placement resource snapshots referenced by the placement binding. placementResourceSnapshots, err := r.retrievePrimaryAndSecondaryPlacementResourceSnapshots(ctx, placementBinding) if err != nil { - wrappedErr := errors.Wraps(err, "failed to retrieve placement resource snapshots", + if apierrors.IsNotFound(err) || errors.Category(err) == errors.ErrCategoryUser { + // There exists a corner case where, due to stale caches, a binding updated with a new placement resource + // snapshot fails its reconciliation at this step due to the snapshot not being found in the cache. + // The situation should self-recover upon retries; for simplicity and performance reasons + // no additional checks are made in the control loop. + if statusErr := r.markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots(ctx, placementBinding, works, err.Error()); statusErr != nil { + wrappedErr := errors.Wraps(statusErr, + "failed to mark placement binding as failed to synchronize due to invalid snapshots", + "originalErr", err, + "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, + "controller", controllerName) + klog.ErrorS(wrappedErr, "failed to mark placement binding as failed to synchronize due to invalid snapshots", errors.Args(wrappedErr)...) + return ctrl.Result{}, wrappedErr + } + } + wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, "controller", controllerName) - klog.ErrorS(wrappedErr, "failed to retrieve placement resource snapshots referenced by binding", + klog.ErrorS(wrappedErr, "failed to retrieve placement resource snapshots", errors.Args(wrappedErr)...) + // Keep retrying as the work generator does not watch for snapshot changes. return ctrl.Result{}, wrappedErr } // Create or update the work objects. - createdOrUpdatedWorks, writtenToStorage, err := r.refreshWorks(ctx, placementBinding, placementResourceSnapshots, works) + // + // Note that if any of the work objects we have checked are stale, the refreshing would fail with an error. + createdOrUpdatedWorks, workSpecsChanged, err := r.refreshWorks(ctx, placementBinding, placementResourceSnapshots, works) if err != nil { - wrappedErr := errors.Wraps(err, "failed to refresh work objects", + wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, "controller", controllerName) - klog.ErrorS(wrappedErr, "failed to refresh work objects for placement binding", + klog.ErrorS(wrappedErr, "failed to refresh work objects", errors.Args(wrappedErr)...) return ctrl.Result{}, wrappedErr } // Report the processing progress via placement binding status. if err := r.reportPlacementBindingProcessingProgress(ctx, placementBinding, placementResourceSnapshots[0], createdOrUpdatedWorks); err != nil { - wrappedErr := errors.Wraps(err, "failed to report placement binding processing progress", + wrappedErr := errors.Wraps(err, "", "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBindingSpec.ClusterName, "controller", controllerName) klog.ErrorS(wrappedErr, "failed to report placement binding processing progress", @@ -189,16 +238,18 @@ func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Resu // The work objects have been refreshed. Normally the controller needs only to wait for the work objects // to be processed by the KubeFleet member agent, then refresh the placement binding status upon receiving - // create/update events from the work objects, and there is no need to requeue manually. However, there exists - // a corner case in which a rollout attempt does not involve any change in the work objects; in this case there - // will not be any change events from the work objects and the work generator needs to requeue manually to - // have the placement binding status refreshed. - if !writtenToStorage { - // The work objects have not been created or updated; requeue manually. + // create/status update events from the work objects, and there is no need to requeue manually. However, there + // exists a corner case in which a rollout attempt does not involve any spec change in the work objects (e.g., + // the new placement resource snapshot has the same content as the old one, and the work objects are only + // re-linked via metadata changes); in this case the member agent will not re-process the work objects, there + // will not be any status change events, and the work generator needs to requeue manually to have the placement + // binding status refreshed. + if !workSpecsChanged { + // No work object has been created or has had its spec changed; requeue manually. return ctrl.Result{RequeueAfter: 1 * time.Second}, nil } - // The work objects have been created or updated; wait for change events from the work objects to refresh - // the placement binding status. + // Some work objects have been created or have had their specs changed; wait for change events from the work + // objects to refresh the placement binding status. return ctrl.Result{}, nil } @@ -260,7 +311,7 @@ func (r *Reconciler) SetupWithManager(mgr ctrl.Manager, maxConcurrentReconciles return ctrl.NewControllerManagedBy(mgr). Named(controllerName). - WithOptions(controller.Options{MaxConcurrentReconciles: maxConcurrentReconciles}). + WithOptions(controller.Options{MaxConcurrentReconciles: maxConcurrentReconciles, RateLimiter: r.rateLimiter}). // The controller watches placement binding objects (both namespace-scoped and cluster-scoped) for spec // changes (generation predicate). Watches(&placementv1alpha1.PlacementBinding{}, &handler.EnqueueRequestForObject{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). diff --git a/pkg/v1/controllers/workgenerator/core_ops_integration_test.go b/pkg/v1/controllers/workgenerator/core_ops_integration_test.go new file mode 100644 index 000000000..b066bc347 --- /dev/null +++ b/pkg/v1/controllers/workgenerator/core_ops_integration_test.go @@ -0,0 +1,4287 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package workgenerator contains the controller logic for reconciling placement binding objects. +package workgenerator + +import ( + "fmt" + "strconv" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("reconciling placement bindings (single placement resource snapshot)", func() { + Context("generate works", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a placement resource snapshot with the NS and the Deployment objects. + snapshot = createPlacementResourceSnapshot(appNamespaceName, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + }) + }) + + Context("update works", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + oldSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + newSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var oldSnapshot, newSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + nsIdentifier := placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + } + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + configMapIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + } + + oldManifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + newManifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + oldManifestIdentifiers[0], + { + Ordinal: 1, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a placement resource snapshot with the NS and the Deployment objects. + oldSnapshot = createPlacementResourceSnapshot(appNamespaceName, oldSnapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: nsIdentifier, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, oldSnapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, oldManifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out a new placement resource snapshot", func() { + // Prepare a NS object (unchanged). + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a ConfigMap object (new); the Deployment object is dropped. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a new placement resource snapshot with the NS and the new ConfigMap objects. + newSnapshot = createPlacementResourceSnapshot(appNamespaceName, newSnapshotName, placementPolicyName, 1, + placementv1alpha1.SnapshottedResource{ + Identifier: nsIdentifier, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: configMapIdentifier, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + ) + + // Point the placement binding to the new snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, newSnapshotName) + }) + + It("should update the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, newSnapshot), + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, newManifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + oldSnapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, oldSnapshotName) + Eventually(oldSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the old placement resource snapshot") + + newSnapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, newSnapshotName) + Eventually(newSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new placement resource snapshot") + }) + }) + + Context("refresh binding status upon work status changes", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a placement resource snapshot with the NS and the Deployment objects. + snapshot = createPlacementResourceSnapshot(appNamespaceName, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the deployment (and consequently the work object) as unavailable", func() { + markWorkManifestAsUnavailable(workNS, workName, 1) + }) + + It("should refresh the placement binding status", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(2)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: deployIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + }) + }) + + Context("update works (no spec changes)", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + oldSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + newSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var oldSnapshot, newSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + var resources []placementv1alpha1.SnapshottedResource + var oldBindingGeneration int64 + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a placement resource snapshot with the NS and the Deployment objects. + resources = []placementv1alpha1.SnapshottedResource{ + { + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + } + oldSnapshot = createPlacementResourceSnapshot(appNamespaceName, oldSnapshotName, placementPolicyName, 0, resources...) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, oldSnapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out a new placement resource snapshot with the same content", func() { + // Record the current generation of the placement binding. + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + oldBindingGeneration = binding.GetGeneration() + + // Create a new placement resource snapshot with exactly the same resources. + newSnapshot = createPlacementResourceSnapshot(appNamespaceName, newSnapshotName, placementPolicyName, 1, resources...) + + // Point the placement binding to the new snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, newSnapshotName) + }) + + It("should update the work objects", func() { + // The manifests stay the same; only the link to the primary placement resource snapshot changes. + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, newSnapshot), + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the placement binding status without waiting for the work objects to be re-processed", func() { + // As the work objects have no spec changes, their existing status still applies; the placement binding + // should be reported as synchronized and available, with the conditions observing the new generation. + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + Expect(binding.GetGeneration()).To(BeNumerically(">", oldBindingGeneration), "The placement binding generation has not been bumped") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + oldSnapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, oldSnapshotName) + Eventually(oldSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the old placement resource snapshot") + + newSnapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, newSnapshotName) + Eventually(newSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new placement resource snapshot") + }) + }) + + Context("backport status for failed manifests", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + configMapIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + } + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + { + Ordinal: 2, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + + // The diff details reported for the ConfigMap object, as if the member agent has failed to take over an + // existing ConfigMap object with different data in the member cluster. + // + // Note that the timestamp is truncated to the second, as this is the precision kept by the API server. + diffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/data/" + configMapDataKey, + ValueInMember: altConfigMapDataValue1, + ValueInHub: configMapDataValue, + }, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a placement resource snapshot with the NS, the Deployment, and the ConfigMap objects. + snapshot = createPlacementResourceSnapshot(appNamespaceName, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: configMapIdentifier, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, appNamespaceName, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(3, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the deployment as unavailable, and the config map as failed to be applied (with diff details)", func() { + markWorkManifestsAsFailed(workNS, workName, manifestIdentifiers, + map[int]*placementv1alpha1.DiffDetails{2: diffDetails}, + sets.New(1), + ) + }) + + It("should refresh the placement binding status with the failure info", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: deployIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + { + ObjectRef: configMapIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: diffDetails, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + }) + }) +}) + +var _ = Describe("reconciling placement bindings (multiple placement resource snapshots)", func() { + Context("generate works", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // The primary snapshot (sub-index 0) and two secondary snapshots (sub-indices 1 and 2) of the same index. + snapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 2), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects (and the snapshots they are derived from), ordered by snapshot sub-index. + workNames := make([]string, len(snapshotNames)) + snapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNames)) + + // Each snapshot (and consequently each work object) features its own set of differently named NS, + // Deployment, and ConfigMap objects. + nsNames := make([]string, len(snapshotNames)) + deployNames := make([]string, len(snapshotNames)) + configMapNames := make([]string, len(snapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNames)) + for subIdx := range snapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsNames[subIdx], + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsNames[subIdx], + Name: deployNames[subIdx], + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + { + Ordinal: 2, + Namespace: nsNames[subIdx], + Name: configMapNames[subIdx], + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + } + + BeforeAll(func() { + resourcesPerSnapshot := make([][]placementv1alpha1.SnapshottedResource, len(snapshotNames)) + for subIdx := range snapshotNames { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsNames[subIdx] + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsNames[subIdx] + regularDeploy.Name = deployNames[subIdx] + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsNames[subIdx] + regularConfigMap.Name = configMapNames[subIdx] + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + resourcesPerSnapshot[subIdx] = []placementv1alpha1.SnapshottedResource{ + { + Identifier: placementv1alpha1.ObjectReference{ + Name: nsNames[subIdx], + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsNames[subIdx], + Name: deployNames[subIdx], + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsNames[subIdx], + Name: configMapNames[subIdx], + APIVersion: "v1", + Kind: configMapKind, + }, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + } + } + + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(snapshotNames) - 1; subIdx >= 0; subIdx-- { + snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, snapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(snapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Create a placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotNames[0]) + for subIdx := range snapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work objects should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(snapshotNames)) + for subIdx := range snapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + snapshotNames[0], subIdx, len(snapshotNames), ownerRef, snapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the primary work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workNames[0], manifestIdentifiersPerWork[0]) + }) + + It("should not refresh the placement binding status until all work objects have been processed", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Consistently(statusUpdatedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status has been refreshed prematurely") + }) + + It("can mark the secondary work objects as applied and available", func() { + for subIdx := 1; subIdx < len(snapshotNames); subIdx++ { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually. + for subIdx := 1; subIdx < len(snapshotNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + for subIdx := range snapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotNames[subIdx]) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) + + Context("update works", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // Before the rollout: the primary snapshot (sub-index 0) and two secondary snapshots (sub-indices 1 and 2) of + // index 0. + oldSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 2), + } + // After the first rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of + // index 1. + newSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + // After the second rollout: the primary snapshot (sub-index 0) and three secondary snapshots (sub-indices 1, + // 2, and 3) of index 2. + latestSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 3), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from; work objects derived + // from snapshots of the same sub-index share the same name across snapshot indices. + workNames := make([]string, len(latestSnapshotNames)) + oldSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(oldSnapshotNames)) + newSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(newSnapshotNames)) + latestSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(latestSnapshotNames)) + + // The UIDs of the work objects that should be kept (updated rather than re-created) across the rollout. + var primaryWorkUID, secondaryWorkUID types.UID + var ownerRef *metav1.OwnerReference + + // The differently named NS, Deployment, and ConfigMap objects to place, one set per sub-index. + nsNames := make([]string, len(latestSnapshotNames)) + deployNames := make([]string, len(latestSnapshotNames)) + configMapNames := make([]string, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + } + + // Before the rollouts, each snapshot features its own set of NS, Deployment, and ConfigMap objects. + oldManifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + oldManifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // After the first rollout, the snapshot of sub-index 2 is gone; its Deployment and ConfigMap objects are moved to the + // snapshots of sub-index 0 and 1 respectively, and its NS object is dropped. + newManifestIdentifiersPerWork := [][]placementv1alpha1.ManifestIdentifier{ + { + nsManifestIdentifier(0, nsNames[0]), + deployManifestIdentifier(1, nsNames[0], deployNames[0]), + configMapManifestIdentifier(2, nsNames[0], configMapNames[0]), + deployManifestIdentifier(3, nsNames[2], deployNames[2]), + }, + { + nsManifestIdentifier(0, nsNames[1]), + deployManifestIdentifier(1, nsNames[1], deployNames[1]), + configMapManifestIdentifier(2, nsNames[1], configMapNames[1]), + configMapManifestIdentifier(3, nsNames[2], configMapNames[2]), + }, + } + + // After the second rollout, each snapshot again features its own set of NS, Deployment, and ConfigMap + // objects. + latestManifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + latestManifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + BeforeAll(func() { + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(oldSnapshotNames) - 1; subIdx >= 0; subIdx-- { + oldSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, oldSnapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(oldSnapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Create a placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotNames[0]) + for subIdx := range latestSnapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work objects should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + primaryWorkUID = primaryWork.UID + ownerRef = &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + oldSnapshotNames[0], subIdx, len(oldSnapshotNames), ownerRef, oldSnapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UID of the secondary work object that should be kept across the rollout. + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + secondaryWorkUID = secondaryWork.UID + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range oldSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], oldManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out a new set of placement resource snapshots", func() { + // Create the new placement resource snapshots; the Deployment and ConfigMap objects from the old snapshot + // of sub-index 2 are moved to the new snapshots of sub-index 0 and 1 respectively. + newSnapshots[1] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, newSnapshotNames[1], placementPolicyName, + 1, 1, len(newSnapshotNames), + nsSnapshottedResource(nsNames[1]), + deploySnapshottedResource(nsNames[1], deployNames[1]), + configMapSnapshottedResource(nsNames[1], configMapNames[1]), + configMapSnapshottedResource(nsNames[2], configMapNames[2]), + ) + newSnapshots[0] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, newSnapshotNames[0], placementPolicyName, + 1, 0, len(newSnapshotNames), + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResource(nsNames[0], configMapNames[0]), + deploySnapshottedResource(nsNames[2], deployNames[2]), + ) + + // Point the placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, newSnapshotNames[0]) + }) + + It("should update the work objects", func() { + // The work object derived from the snapshot of sub-index 2 should be gone; the other two should be + // updated in place, with the secondary one still owned by the (same) primary work object. + wantWorks := make([]placementv1alpha1.Work, len(newSnapshotNames)) + for subIdx := range newSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + newSnapshotNames[0], subIdx, len(newSnapshotNames), ownerRef, newSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should delete the work object that is no longer needed", func() { + workRemovedActual := workRemovedActual(workNS, workNames[2]) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to delete the work object that is no longer needed") + }) + + It("should update the other work objects in place (not re-create them)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + Expect(secondaryWork.UID).To(Equal(secondaryWorkUID), "The secondary work object has been re-created") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(8, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range newSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], newManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(8, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out another new set of placement resource snapshots", func() { + // Create the latest placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(latestSnapshotNames) - 1; subIdx >= 0; subIdx-- { + latestSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, latestSnapshotNames[subIdx], placementPolicyName, + 2, subIdx, len(latestSnapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Point the placement binding to the latest primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, latestSnapshotNames[0]) + }) + + It("should update and create the work objects", func() { + // The work objects derived from the snapshots of sub-index 0 and 1 should be updated; the ones derived from + // the snapshots of sub-index 2 and 3 should be created, and owned by the (same) primary work object. + wantWorks := make([]placementv1alpha1.Work, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + latestSnapshotNames[0], subIdx, len(latestSnapshotNames), ownerRef, latestSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should update the existing work objects in place (not re-create them)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + Expect(secondaryWork.UID).To(Equal(secondaryWorkUID), "The secondary work object has been re-created") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(12, latestSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range latestSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], latestManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(12, latestSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually (if they still exist). + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append(append([]string{}, oldSnapshotNames...), newSnapshotNames...), latestSnapshotNames...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) + + Context("handle inconsistent resource snapshots", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // Index 0: a single primary snapshot. + snapshotNameIdx0 := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + // Index 1: a primary snapshot that expects one secondary snapshot. + snapshotNamesIdx1 := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + // Index 2: a primary snapshot that expects two secondary snapshots. + snapshotNamesIdx2 := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 2), + } + contentsHashIdx1 := "hash-1" + mismatchedContentsHashIdx1 := "hash-1-mismatched" + contentsHashIdx2 := "hash-2" + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from. + workNames := make([]string, len(snapshotNamesIdx2)) + var snapshotIdx0 placementv1alpha1.PlacementResourceSnapshotAccessor + snapshotsIdx2 := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNamesIdx2)) + + var primaryWorkUID types.UID + // The expected state of the work objects before any rollout attempt; it should not change as long as the + // snapshots in use are inconsistent. + var wantWorksBeforeRollouts []placementv1alpha1.Work + // The expected placement binding status (with the observed generations set) once the work generator has + // marked the placement binding as failed to synchronize after a rollout attempt; it should not change until + // the next rollout attempt. + var wantStatusAfterRollout *placementv1alpha1.PlacementBindingStatus + var statusBeforeRollouts *placementv1alpha1.PlacementBindingStatus + + // The differently named NS, Deployment, and ConfigMap objects to place, one set per sub-index. + nsNames := make([]string, len(snapshotNamesIdx2)) + deployNames := make([]string, len(snapshotNamesIdx2)) + configMapNames := make([]string, len(snapshotNamesIdx2)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNamesIdx2)) + for subIdx := range snapshotNamesIdx2 { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // After a rollout to the given primary snapshot, the placement binding should be marked as failed to synchronize + // due to the missing secondary snapshots, with the resource counters tallied from the work object of index 0 + // (the last processed snapshot). The spec is checked first, so that the observed generations are not verified + // against a stale (pre-rollout) generation from the cache; the status before and after the second rollout + // differs only in the Synchronized condition's observed generation. + markedAsFailedToSynchronizeActual := func(wantSnapshotName string) func() error { + return func() error { + binding, err := getPlacementBinding(appNamespaceName, bindingName) + if err != nil { + return err + } + if got := binding.GetSpec().ResourceSnapshotName; got != wantSnapshotName { + return fmt.Errorf("the placement binding points to snapshot %s, want %s", got, wantSnapshotName) + } + + wantStatus := wantPlacementBindingStatusInvalidSnapshots(3, snapshotNameIdx0, statusBeforeRollouts) + wantStatus.Conditions[0].ObservedGeneration = binding.GetGeneration() + if err := placementBindingStatusMatchesActual(appNamespaceName, bindingName, wantStatus)(); err != nil { + return err + } + return placementBindingSyncCondMessageActual(appNamespaceName, bindingName, fewerSnapshotsErrMsg)() + } + } + + recordStatusAfterRollout := func() { + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + wantStatusAfterRollout = wantPlacementBindingStatusInvalidSnapshots(3, snapshotNameIdx0, statusBeforeRollouts) + wantStatusAfterRollout.Conditions[0].ObservedGeneration = binding.GetGeneration() + } + + bindingAndWorksUnchangedActual := func() error { + if err := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorksBeforeRollouts)(); err != nil { + return err + } + if err := placementBindingStatusMatchesActual(appNamespaceName, bindingName, wantStatusAfterRollout)(); err != nil { + return err + } + // The status comparison ignores condition messages; the message should still report the missing snapshots. + return placementBindingSyncCondMessageActual(appNamespaceName, bindingName, fewerSnapshotsErrMsg)() + } + + BeforeAll(func() { + // Create a single placement resource snapshot of index 0. + snapshotIdx0 = createPlacementResourceSnapshot(appNamespaceName, snapshotNameIdx0, placementPolicyName, 0, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResource(nsNames[0], configMapNames[0]), + ) + + // Create a placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotNameIdx0) + for subIdx := range snapshotNamesIdx2 { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + wantWorksBeforeRollouts = []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workNames[0], appNamespaceName, bindingName, placementPolicyName, snapshotIdx0), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorksBeforeRollouts) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UID of the primary work object, which should be kept across rollouts. + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + primaryWorkUID = primaryWork.UID + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(3, snapshotNameIdx0) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workNames[0], manifestIdentifiersPerWork[0]) + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(3, snapshotNameIdx0) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + statusBeforeRollouts = binding.GetStatus().DeepCopy() + }) + + It("can roll out a new primary snapshot (index 1) without its secondary snapshot", func() { + // The new primary snapshot features new ConfigMap data, and expects one secondary snapshot, which is + // absent for now. + createSubIndexedPlacementResourceSnapshotWithHash(appNamespaceName, snapshotNamesIdx1[0], placementPolicyName, + 1, 0, len(snapshotNamesIdx1), contentsHashIdx1, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResourceWithData(nsNames[0], configMapNames[0], map[string]string{configMapDataKey: altConfigMapDataValue1}), + ) + + // Point the placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, snapshotNamesIdx1[0]) + }) + + It("should mark the placement binding as failed to synchronize due to the missing secondary snapshot", func() { + Eventually(markedAsFailedToSynchronizeActual(snapshotNamesIdx1[0]), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + recordStatusAfterRollout() + }) + + It("should not update the placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status or the work objects have been updated unexpectedly") + }) + + It("can add a secondary snapshot (index 1) with a mismatched contents hash", func() { + createSubIndexedPlacementResourceSnapshotWithHash(appNamespaceName, snapshotNamesIdx1[1], placementPolicyName, + 1, 1, len(snapshotNamesIdx1), mismatchedContentsHashIdx1, + nsSnapshottedResource(nsNames[1]), + deploySnapshottedResource(nsNames[1], deployNames[1]), + configMapSnapshottedResource(nsNames[1], configMapNames[1]), + ) + }) + + // A contents hash mismatch is not reported on the placement binding; the status, including the message about + // the missing snapshot, should stay as it is. + It("should not update the placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status or the work objects have been updated unexpectedly") + }) + + It("can roll out a new primary snapshot (index 2) without its secondary snapshots", func() { + // The new primary snapshot features new ConfigMap data, and expects two secondary snapshots, which are + // absent for now. + snapshotsIdx2[0] = createSubIndexedPlacementResourceSnapshotWithHash(appNamespaceName, snapshotNamesIdx2[0], placementPolicyName, + 2, 0, len(snapshotNamesIdx2), contentsHashIdx2, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResourceWithData(nsNames[0], configMapNames[0], map[string]string{configMapDataKey: altConfigMapDataValue2}), + ) + + // Point the placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, snapshotNamesIdx2[0]) + }) + + It("should mark the placement binding as failed to synchronize due to the missing secondary snapshots", func() { + Eventually(markedAsFailedToSynchronizeActual(snapshotNamesIdx2[0]), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + recordStatusAfterRollout() + }) + + It("should not update the placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status or the work objects have been updated unexpectedly") + }) + + It("can add the missing secondary snapshots (index 2) with the expected contents hash", func() { + for subIdx := len(snapshotNamesIdx2) - 1; subIdx >= 1; subIdx-- { + snapshotsIdx2[subIdx] = createSubIndexedPlacementResourceSnapshotWithHash(appNamespaceName, snapshotNamesIdx2[subIdx], placementPolicyName, + 2, subIdx, len(snapshotNamesIdx2), contentsHashIdx2, + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + }) + + It("should update and create the work objects", func() { + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: workNames[0], + UID: primaryWorkUID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + wantWorks := make([]placementv1alpha1.Work, len(snapshotNamesIdx2)) + for subIdx := range snapshotNamesIdx2 { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + snapshotNamesIdx2[0], subIdx, len(snapshotNamesIdx2), ownerRef, snapshotsIdx2[subIdx]) + } + + // The work generator does not watch placement resource snapshots; it picks up the newly added secondary + // snapshots only when it retries the failed reconciliation, whose backoff is capped at a second in this suite (see requeueMaxDelay). + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should update the primary work object in place (not re-create it)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNamesIdx2[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range snapshotNamesIdx2 { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, snapshotNamesIdx2[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually (if they still exist). + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append([]string{snapshotNameIdx0}, snapshotNamesIdx1...), snapshotNamesIdx2...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) + + Context("update works (no spec changes)", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // Before the rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 0. + oldSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + } + // After the rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 1, + // with exactly the same content as their counterparts of index 0. + newSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from. + workNames := make([]string, len(oldSnapshotNames)) + workUIDs := make([]types.UID, len(oldSnapshotNames)) + oldSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(oldSnapshotNames)) + newSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(newSnapshotNames)) + resourcesPerSnapshot := make([][]placementv1alpha1.SnapshottedResource, len(oldSnapshotNames)) + var ownerRef *metav1.OwnerReference + var oldBindingGeneration int64 + + // Each snapshot features its own set of differently named NS, Deployment, and ConfigMap objects. + nsNames := make([]string, len(oldSnapshotNames)) + deployNames := make([]string, len(oldSnapshotNames)) + configMapNames := make([]string, len(oldSnapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + BeforeAll(func() { + for subIdx := range oldSnapshotNames { + resourcesPerSnapshot[subIdx] = []placementv1alpha1.SnapshottedResource{ + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + } + } + + // Create the placement resource snapshots of index 0. The secondary snapshot is created first, as the + // placement resource snapshot manager would do. + for subIdx := len(oldSnapshotNames) - 1; subIdx >= 0; subIdx-- { + oldSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, oldSnapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(oldSnapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Create a placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotNames[0]) + for subIdx := range oldSnapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work object should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef = &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + oldSnapshotNames[0], subIdx, len(oldSnapshotNames), ownerRef, oldSnapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UIDs of the work objects, which should be kept across the rollout. + for subIdx := range oldSnapshotNames { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[subIdx]}, work)).To(Succeed(), "Failed to retrieve the work object") + workUIDs[subIdx] = work.UID + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(6, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range oldSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(6, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can roll out a new set of placement resource snapshots with the same content", func() { + // Record the current generation of the placement binding. + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + oldBindingGeneration = binding.GetGeneration() + + // Create the new placement resource snapshots with exactly the same resources as their counterparts. + for subIdx := len(newSnapshotNames) - 1; subIdx >= 0; subIdx-- { + newSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, newSnapshotNames[subIdx], placementPolicyName, + 1, subIdx, len(newSnapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Point the placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(appNamespaceName, bindingName, newSnapshotNames[0]) + }) + + It("should update the work objects", func() { + // The manifests stay the same; only the link to the primary placement resource snapshot changes. + wantWorks := make([]placementv1alpha1.Work, len(newSnapshotNames)) + for subIdx := range newSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + newSnapshotNames[0], subIdx, len(newSnapshotNames), ownerRef, newSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should update the work objects in place (not re-create them)", func() { + for subIdx := range newSnapshotNames { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[subIdx]}, work)).To(Succeed(), "Failed to retrieve the work object") + Expect(work.UID).To(Equal(workUIDs[subIdx]), "The work object has been re-created") + } + }) + + It("should refresh the placement binding status without waiting for the work objects to be re-processed", func() { + // As the work objects have no spec changes, their existing status still applies; the placement binding + // should be reported as synchronized and available, with the conditions observing the new generation. + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(6, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + + binding, err := getPlacementBinding(appNamespaceName, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + Expect(binding.GetGeneration()).To(BeNumerically(">", oldBindingGeneration), "The placement binding generation has not been bumped") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work object is owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes it manually. + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append([]string{}, oldSnapshotNames...), newSnapshotNames...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) + + Context("backport status for failed manifests", Ordered, func() { + bindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + // The primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 0. + snapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects (and the snapshots they are derived from), ordered by snapshot sub-index. + workNames := make([]string, len(snapshotNames)) + snapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNames)) + + // Each snapshot features its own set of differently named NS, Deployment, and ConfigMap objects. + nsNames := make([]string, len(snapshotNames)) + deployNames := make([]string, len(snapshotNames)) + configMapNames := make([]string, len(snapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNames)) + for subIdx := range snapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // The diff details reported for the ConfigMap object in the primary work object, as if the member agent has + // failed to take over an existing ConfigMap object with different data in the member cluster. + // + // Note that the timestamps are truncated to the second, as this is the precision kept by the API server. + configMapDiffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/data/" + configMapDataKey, + ValueInMember: altConfigMapDataValue1, + ValueInHub: configMapDataValue, + }, + }, + } + // The diff details reported for the Deployment object in the secondary work object, as if the member agent + // has failed to take over an existing Deployment object with a different replica count in the member cluster. + deployDiffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(2)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "2", + ValueInHub: "1", + }, + }, + } + + BeforeAll(func() { + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshot is created first, as the placement resource snapshot manager would do. + for subIdx := len(snapshotNames) - 1; subIdx >= 0; subIdx-- { + snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(appNamespaceName, snapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(snapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Create a placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(appNamespaceName, bindingName, placementPolicyName, memberCluster1Name, snapshotNames[0]) + for subIdx := range snapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(appNamespaceName, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work object should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(snapshotNames)) + for subIdx := range snapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], appNamespaceName, bindingName, placementPolicyName, + snapshotNames[0], subIdx, len(snapshotNames), ownerRef, snapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, appNamespaceName, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(6, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark some manifests in the primary work object as failed", func() { + // The Deployment object fails to become available; the ConfigMap object fails to be applied (with diff + // details). + markWorkManifestsAsFailed(workNS, workNames[0], manifestIdentifiersPerWork[0], + map[int]*placementv1alpha1.DiffDetails{2: configMapDiffDetails}, + sets.New(1), + ) + }) + + It("can mark some manifests in the secondary work object as failed", func() { + // The Deployment object fails to be applied (with diff details); the ConfigMap object fails to become + // available. + markWorkManifestsAsFailed(workNS, workNames[1], manifestIdentifiersPerWork[1], + map[int]*placementv1alpha1.DiffDetails{1: deployDiffDetails}, + sets.New(2), + ) + }) + + It("should refresh the placement binding status with the failure info from all work objects", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + // In each work object, the NS object is applied and available; the Deployment and ConfigMap objects + // have failed in different ways. + SelectedResources: ptr.To(int32(6)), + SynchronizedResources: ptr.To(int32(4)), + AvailableResources: ptr.To(int32(2)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[0][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[0][2]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: configMapDiffDetails, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[1][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: deployDiffDetails, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[1][2]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotNames[0]), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(appNamespaceName, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + // Delete the placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(appNamespaceName, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(appNamespaceName, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The secondary work object is owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes it manually. + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + for _, snapshotName := range snapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(appNamespaceName, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + }) + }) +}) + +var _ = Describe("reconciling cluster placement bindings (single placement resource snapshot)", func() { + Context("generate works", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a cluster placement resource snapshot with the NS and the Deployment objects. + snapshot = createPlacementResourceSnapshot(clusterScopedNS, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + }) + }) + + Context("update works", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + oldSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + newSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var oldSnapshot, newSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + nsIdentifier := placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + } + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + configMapIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + } + + oldManifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + newManifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + oldManifestIdentifiers[0], + { + Ordinal: 1, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a cluster placement resource snapshot with the NS and the Deployment objects. + oldSnapshot = createPlacementResourceSnapshot(clusterScopedNS, oldSnapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: nsIdentifier, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, oldSnapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, oldManifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out a new cluster placement resource snapshot", func() { + // Prepare a NS object (unchanged). + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a ConfigMap object (new); the Deployment object is dropped. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a new cluster placement resource snapshot with the NS and the new ConfigMap objects. + newSnapshot = createPlacementResourceSnapshot(clusterScopedNS, newSnapshotName, placementPolicyName, 1, + placementv1alpha1.SnapshottedResource{ + Identifier: nsIdentifier, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: configMapIdentifier, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + ) + + // Point the cluster placement binding to the new snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, newSnapshotName) + }) + + It("should update the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, newSnapshot), + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, newManifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + oldSnapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, oldSnapshotName) + Eventually(oldSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the old cluster placement resource snapshot") + + newSnapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, newSnapshotName) + Eventually(newSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new cluster placement resource snapshot") + }) + }) + + Context("refresh binding status upon work status changes", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a cluster placement resource snapshot with the NS and the Deployment objects. + snapshot = createPlacementResourceSnapshot(clusterScopedNS, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the deployment (and consequently the work object) as unavailable", func() { + markWorkManifestAsUnavailable(workNS, workName, 1) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(2)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: deployIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + }) + }) + + Context("update works (no spec changes)", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + oldSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + newSnapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var oldSnapshot, newSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + var resources []placementv1alpha1.SnapshottedResource + var oldBindingGeneration int64 + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Create a cluster placement resource snapshot with the NS and the Deployment objects. + resources = []placementv1alpha1.SnapshottedResource{ + { + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + } + oldSnapshot = createPlacementResourceSnapshot(clusterScopedNS, oldSnapshotName, placementPolicyName, 0, resources...) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, oldSnapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workName, manifestIdentifiers) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, oldSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out a new cluster placement resource snapshot with the same content", func() { + // Record the current generation of the cluster placement binding. + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + oldBindingGeneration = binding.GetGeneration() + + // Create a new cluster placement resource snapshot with exactly the same resources. + newSnapshot = createPlacementResourceSnapshot(clusterScopedNS, newSnapshotName, placementPolicyName, 1, resources...) + + // Point the cluster placement binding to the new snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, newSnapshotName) + }) + + It("should update the work objects", func() { + // The manifests stay the same; only the link to the primary cluster placement resource snapshot changes. + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, newSnapshot), + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the cluster placement binding status without waiting for the work objects to be re-processed", func() { + // As the work objects have no spec changes, their existing status still applies; the cluster placement binding + // should be reported as synchronized and available, with the conditions observing the new generation. + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(2, newSnapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + Expect(binding.GetGeneration()).To(BeNumerically(">", oldBindingGeneration), "The cluster placement binding generation has not been bumped") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + oldSnapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, oldSnapshotName) + Eventually(oldSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the old cluster placement resource snapshot") + + newSnapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, newSnapshotName) + Eventually(newSnapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the new cluster placement resource snapshot") + }) + }) + + Context("backport status for failed manifests", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + snapshotName := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + var workName string + var snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + + deployIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + } + configMapIdentifier := placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + } + + manifestIdentifiers := []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + { + Ordinal: 2, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + + // The diff details reported for the ConfigMap object, as if the member agent has failed to take over an + // existing ConfigMap object with different data in the member cluster. + // + // Note that the timestamp is truncated to the second, as this is the precision kept by the API server. + diffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/data/" + configMapDataKey, + ValueInMember: altConfigMapDataValue1, + ValueInHub: configMapDataValue, + }, + }, + } + + BeforeAll(func() { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsName + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + // Create a cluster placement resource snapshot with the NS, the Deployment, and the ConfigMap objects. + snapshot = createPlacementResourceSnapshot(clusterScopedNS, snapshotName, placementPolicyName, 0, + placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: deployIdentifier, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + placementv1alpha1.SnapshottedResource{ + Identifier: configMapIdentifier, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotName) + workName = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}) + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorks := []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workName, clusterScopedNS, bindingName, placementPolicyName, snapshot), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(3, snapshotName) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the deployment as unavailable, and the config map as failed to be applied (with diff details)", func() { + markWorkManifestsAsFailed(workNS, workName, manifestIdentifiers, + map[int]*placementv1alpha1.DiffDetails{2: diffDetails}, + sets.New(1), + ) + }) + + It("should refresh the cluster placement binding status with the failure info", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: deployIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + { + ObjectRef: configMapIdentifier, + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: diffDetails, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the work objects and then + // remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + workRemovedActual := workRemovedActual(workNS, workName) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + }) + }) +}) + +var _ = Describe("reconciling cluster placement bindings (multiple placement resource snapshots)", func() { + Context("generate works", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // The primary snapshot (sub-index 0) and two secondary snapshots (sub-indices 1 and 2) of the same index. + snapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 2), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects (and the snapshots they are derived from), ordered by snapshot sub-index. + workNames := make([]string, len(snapshotNames)) + snapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNames)) + + // Each snapshot (and consequently each work object) features its own set of differently named NS, + // Deployment, and ConfigMap objects. + nsNames := make([]string, len(snapshotNames)) + deployNames := make([]string, len(snapshotNames)) + configMapNames := make([]string, len(snapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNames)) + for subIdx := range snapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + { + Ordinal: 0, + Name: nsNames[subIdx], + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + }, + { + Ordinal: 1, + Namespace: nsNames[subIdx], + Name: deployNames[subIdx], + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + }, + { + Ordinal: 2, + Namespace: nsNames[subIdx], + Name: configMapNames[subIdx], + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + }, + } + } + + BeforeAll(func() { + resourcesPerSnapshot := make([][]placementv1alpha1.SnapshottedResource, len(snapshotNames)) + for subIdx := range snapshotNames { + // Prepare a NS object. + regularNS := ns.DeepCopy() + regularNS.Name = nsNames[subIdx] + regularNSJSON := marshalK8sObjJSON(regularNS) + + // Prepare a Deployment object. + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsNames[subIdx] + regularDeploy.Name = deployNames[subIdx] + regularDeployJSON := marshalK8sObjJSON(regularDeploy) + + // Prepare a ConfigMap object. + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsNames[subIdx] + regularConfigMap.Name = configMapNames[subIdx] + regularConfigMapJSON := marshalK8sObjJSON(regularConfigMap) + + resourcesPerSnapshot[subIdx] = []placementv1alpha1.SnapshottedResource{ + { + Identifier: placementv1alpha1.ObjectReference{ + Name: nsNames[subIdx], + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: regularNSJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsNames[subIdx], + Name: deployNames[subIdx], + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: regularDeployJSON}, + }, + { + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsNames[subIdx], + Name: configMapNames[subIdx], + APIVersion: "v1", + Kind: configMapKind, + }, + Manifest: runtime.RawExtension{Raw: regularConfigMapJSON}, + }, + } + } + + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(snapshotNames) - 1; subIdx >= 0; subIdx-- { + snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, snapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(snapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Create a cluster placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotNames[0]) + for subIdx := range snapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work objects should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(snapshotNames)) + for subIdx := range snapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + snapshotNames[0], subIdx, len(snapshotNames), ownerRef, snapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the primary work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workNames[0], manifestIdentifiersPerWork[0]) + }) + + It("should not refresh the cluster placement binding status until all work objects have been processed", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Consistently(statusUpdatedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster placement binding status has been refreshed prematurely") + }) + + It("can mark the secondary work objects as applied and available", func() { + for subIdx := 1; subIdx < len(snapshotNames); subIdx++ { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually. + for subIdx := 1; subIdx < len(snapshotNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + for subIdx := range snapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotNames[subIdx]) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) + + Context("update works", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // Before the rollout: the primary snapshot (sub-index 0) and two secondary snapshots (sub-indices 1 and 2) of + // index 0. + oldSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 2), + } + // After the first rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of + // index 1. + newSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + // After the second rollout: the primary snapshot (sub-index 0) and three secondary snapshots (sub-indices 1, + // 2, and 3) of index 2. + latestSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 3), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from; work objects derived + // from snapshots of the same sub-index share the same name across snapshot indices. + workNames := make([]string, len(latestSnapshotNames)) + oldSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(oldSnapshotNames)) + newSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(newSnapshotNames)) + latestSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(latestSnapshotNames)) + + // The UIDs of the work objects that should be kept (updated rather than re-created) across the rollout. + var primaryWorkUID, secondaryWorkUID types.UID + var ownerRef *metav1.OwnerReference + + // The differently named NS, Deployment, and ConfigMap objects to place, one set per sub-index. + nsNames := make([]string, len(latestSnapshotNames)) + deployNames := make([]string, len(latestSnapshotNames)) + configMapNames := make([]string, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + } + + // Before the rollouts, each snapshot features its own set of NS, Deployment, and ConfigMap objects. + oldManifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + oldManifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // After the first rollout, the snapshot of sub-index 2 is gone; its Deployment and ConfigMap objects are moved to the + // snapshots of sub-index 0 and 1 respectively, and its NS object is dropped. + newManifestIdentifiersPerWork := [][]placementv1alpha1.ManifestIdentifier{ + { + nsManifestIdentifier(0, nsNames[0]), + deployManifestIdentifier(1, nsNames[0], deployNames[0]), + configMapManifestIdentifier(2, nsNames[0], configMapNames[0]), + deployManifestIdentifier(3, nsNames[2], deployNames[2]), + }, + { + nsManifestIdentifier(0, nsNames[1]), + deployManifestIdentifier(1, nsNames[1], deployNames[1]), + configMapManifestIdentifier(2, nsNames[1], configMapNames[1]), + configMapManifestIdentifier(3, nsNames[2], configMapNames[2]), + }, + } + + // After the second rollout, each snapshot again features its own set of NS, Deployment, and ConfigMap + // objects. + latestManifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + latestManifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + BeforeAll(func() { + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(oldSnapshotNames) - 1; subIdx >= 0; subIdx-- { + oldSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, oldSnapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(oldSnapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Create a cluster placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotNames[0]) + for subIdx := range latestSnapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work objects should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + primaryWorkUID = primaryWork.UID + ownerRef = &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + oldSnapshotNames[0], subIdx, len(oldSnapshotNames), ownerRef, oldSnapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UID of the secondary work object that should be kept across the rollout. + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + secondaryWorkUID = secondaryWork.UID + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range oldSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], oldManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out a new set of cluster placement resource snapshots", func() { + // Create the new placement resource snapshots; the Deployment and ConfigMap objects from the old snapshot + // of sub-index 2 are moved to the new snapshots of sub-index 0 and 1 respectively. + newSnapshots[1] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, newSnapshotNames[1], placementPolicyName, + 1, 1, len(newSnapshotNames), + nsSnapshottedResource(nsNames[1]), + deploySnapshottedResource(nsNames[1], deployNames[1]), + configMapSnapshottedResource(nsNames[1], configMapNames[1]), + configMapSnapshottedResource(nsNames[2], configMapNames[2]), + ) + newSnapshots[0] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, newSnapshotNames[0], placementPolicyName, + 1, 0, len(newSnapshotNames), + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResource(nsNames[0], configMapNames[0]), + deploySnapshottedResource(nsNames[2], deployNames[2]), + ) + + // Point the cluster placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, newSnapshotNames[0]) + }) + + It("should update the work objects", func() { + // The work object derived from the snapshot of sub-index 2 should be gone; the other two should be + // updated in place, with the secondary one still owned by the (same) primary work object. + wantWorks := make([]placementv1alpha1.Work, len(newSnapshotNames)) + for subIdx := range newSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + newSnapshotNames[0], subIdx, len(newSnapshotNames), ownerRef, newSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should delete the work object that is no longer needed", func() { + workRemovedActual := workRemovedActual(workNS, workNames[2]) + Eventually(workRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to delete the work object that is no longer needed") + }) + + It("should update the other work objects in place (not re-create them)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + Expect(secondaryWork.UID).To(Equal(secondaryWorkUID), "The secondary work object has been re-created") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(8, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range newSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], newManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(8, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out another new set of cluster placement resource snapshots", func() { + // Create the latest placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshots are created first, as the placement resource snapshot manager would do. + for subIdx := len(latestSnapshotNames) - 1; subIdx >= 0; subIdx-- { + latestSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, latestSnapshotNames[subIdx], placementPolicyName, + 2, subIdx, len(latestSnapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Point the cluster placement binding to the latest primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, latestSnapshotNames[0]) + }) + + It("should update and create the work objects", func() { + // The work objects derived from the snapshots of sub-index 0 and 1 should be updated; the ones derived from + // the snapshots of sub-index 2 and 3 should be created, and owned by the (same) primary work object. + wantWorks := make([]placementv1alpha1.Work, len(latestSnapshotNames)) + for subIdx := range latestSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + latestSnapshotNames[0], subIdx, len(latestSnapshotNames), ownerRef, latestSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should update the existing work objects in place (not re-create them)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + + secondaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[1]}, secondaryWork)).To(Succeed(), "Failed to retrieve the secondary work object") + Expect(secondaryWork.UID).To(Equal(secondaryWorkUID), "The secondary work object has been re-created") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(12, latestSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range latestSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], latestManifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(12, latestSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually (if they still exist). + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append(append([]string{}, oldSnapshotNames...), newSnapshotNames...), latestSnapshotNames...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) + + Context("handle inconsistent resource snapshots", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // Index 0: a single primary snapshot. + snapshotNameIdx0 := fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0) + // Index 1: a primary snapshot that expects one secondary snapshot. + snapshotNamesIdx1 := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + // Index 2: a primary snapshot that expects two secondary snapshots. + snapshotNamesIdx2 := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 2), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 2, 2), + } + contentsHashIdx1 := "hash-1" + mismatchedContentsHashIdx1 := "hash-1-mismatched" + contentsHashIdx2 := "hash-2" + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from. + workNames := make([]string, len(snapshotNamesIdx2)) + var snapshotIdx0 placementv1alpha1.PlacementResourceSnapshotAccessor + snapshotsIdx2 := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNamesIdx2)) + + var primaryWorkUID types.UID + // The expected state of the work objects before any rollout attempt; it should not change as long as the + // snapshots in use are inconsistent. + var wantWorksBeforeRollouts []placementv1alpha1.Work + // The expected cluster placement binding status (with the observed generations set) once the work generator + // has marked the cluster placement binding as failed to synchronize after a rollout attempt; it should not + // change until the next rollout attempt. + var wantStatusAfterRollout *placementv1alpha1.PlacementBindingStatus + var statusBeforeRollouts *placementv1alpha1.PlacementBindingStatus + + // The differently named NS, Deployment, and ConfigMap objects to place, one set per sub-index. + nsNames := make([]string, len(snapshotNamesIdx2)) + deployNames := make([]string, len(snapshotNamesIdx2)) + configMapNames := make([]string, len(snapshotNamesIdx2)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNamesIdx2)) + for subIdx := range snapshotNamesIdx2 { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // After a rollout to the given primary snapshot, the cluster placement binding should be marked as failed to + // synchronize due to the missing secondary snapshots, with the resource counters tallied from the work object of + // index 0 (the last processed snapshot). The spec is checked first, so that the observed generations are not + // verified against a stale (pre-rollout) generation from the cache; the status before and after the second + // rollout differs only in the Synchronized condition's observed generation. + markedAsFailedToSynchronizeActual := func(wantSnapshotName string) func() error { + return func() error { + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + if err != nil { + return err + } + if got := binding.GetSpec().ResourceSnapshotName; got != wantSnapshotName { + return fmt.Errorf("the cluster placement binding points to snapshot %s, want %s", got, wantSnapshotName) + } + + wantStatus := wantPlacementBindingStatusInvalidSnapshots(3, snapshotNameIdx0, statusBeforeRollouts) + wantStatus.Conditions[0].ObservedGeneration = binding.GetGeneration() + if err := placementBindingStatusMatchesActual(clusterScopedNS, bindingName, wantStatus)(); err != nil { + return err + } + return placementBindingSyncCondMessageActual(clusterScopedNS, bindingName, fewerSnapshotsErrMsg)() + } + } + + recordStatusAfterRollout := func() { + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + wantStatusAfterRollout = wantPlacementBindingStatusInvalidSnapshots(3, snapshotNameIdx0, statusBeforeRollouts) + wantStatusAfterRollout.Conditions[0].ObservedGeneration = binding.GetGeneration() + } + + bindingAndWorksUnchangedActual := func() error { + if err := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorksBeforeRollouts)(); err != nil { + return err + } + if err := placementBindingStatusMatchesActual(clusterScopedNS, bindingName, wantStatusAfterRollout)(); err != nil { + return err + } + // The status comparison ignores condition messages; the message should still report the missing snapshots. + return placementBindingSyncCondMessageActual(clusterScopedNS, bindingName, fewerSnapshotsErrMsg)() + } + + BeforeAll(func() { + // Create a single placement resource snapshot of index 0. + snapshotIdx0 = createPlacementResourceSnapshot(clusterScopedNS, snapshotNameIdx0, placementPolicyName, 0, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResource(nsNames[0], configMapNames[0]), + ) + + // Create a cluster placement binding that binds the snapshot to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotNameIdx0) + for subIdx := range snapshotNamesIdx2 { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + wantWorksBeforeRollouts = []placementv1alpha1.Work{ + wantWorkForPrimaryPlacementResourceSnapshot(workNS, workNames[0], clusterScopedNS, bindingName, placementPolicyName, snapshotIdx0), + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorksBeforeRollouts) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UID of the primary work object, which should be kept across rollouts. + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + primaryWorkUID = primaryWork.UID + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(3, snapshotNameIdx0) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work object as applied and available", func() { + markWorkAsAppliedAndAvailable(workNS, workNames[0], manifestIdentifiersPerWork[0]) + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(3, snapshotNameIdx0) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + statusBeforeRollouts = binding.GetStatus().DeepCopy() + }) + + It("can roll out a new primary snapshot (index 1) without its secondary snapshot", func() { + // The new primary snapshot features new ConfigMap data, and expects one secondary snapshot, which is + // absent for now. + createSubIndexedPlacementResourceSnapshotWithHash(clusterScopedNS, snapshotNamesIdx1[0], placementPolicyName, + 1, 0, len(snapshotNamesIdx1), contentsHashIdx1, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResourceWithData(nsNames[0], configMapNames[0], map[string]string{configMapDataKey: altConfigMapDataValue1}), + ) + + // Point the cluster placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, snapshotNamesIdx1[0]) + }) + + It("should mark the cluster placement binding as failed to synchronize due to the missing secondary snapshot", func() { + Eventually(markedAsFailedToSynchronizeActual(snapshotNamesIdx1[0]), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the cluster placement binding as failed to synchronize") + recordStatusAfterRollout() + }) + + It("should not update the cluster placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster placement binding status or the work objects have been updated unexpectedly") + }) + + It("can add a secondary snapshot (index 1) with a mismatched contents hash", func() { + createSubIndexedPlacementResourceSnapshotWithHash(clusterScopedNS, snapshotNamesIdx1[1], placementPolicyName, + 1, 1, len(snapshotNamesIdx1), mismatchedContentsHashIdx1, + nsSnapshottedResource(nsNames[1]), + deploySnapshottedResource(nsNames[1], deployNames[1]), + configMapSnapshottedResource(nsNames[1], configMapNames[1]), + ) + }) + + // A contents hash mismatch is not reported on the cluster placement binding; the status, including the + // message about the missing snapshot, should stay as it is. + It("should not update the cluster placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster placement binding status or the work objects have been updated unexpectedly") + }) + + It("can roll out a new primary snapshot (index 2) without its secondary snapshots", func() { + // The new primary snapshot features new ConfigMap data, and expects two secondary snapshots, which are + // absent for now. + snapshotsIdx2[0] = createSubIndexedPlacementResourceSnapshotWithHash(clusterScopedNS, snapshotNamesIdx2[0], placementPolicyName, + 2, 0, len(snapshotNamesIdx2), contentsHashIdx2, + nsSnapshottedResource(nsNames[0]), + deploySnapshottedResource(nsNames[0], deployNames[0]), + configMapSnapshottedResourceWithData(nsNames[0], configMapNames[0], map[string]string{configMapDataKey: altConfigMapDataValue2}), + ) + + // Point the cluster placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, snapshotNamesIdx2[0]) + }) + + It("should mark the cluster placement binding as failed to synchronize due to the missing secondary snapshots", func() { + Eventually(markedAsFailedToSynchronizeActual(snapshotNamesIdx2[0]), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the cluster placement binding as failed to synchronize") + recordStatusAfterRollout() + }) + + It("should not update the cluster placement binding status or the work objects", func() { + Consistently(bindingAndWorksUnchangedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster placement binding status or the work objects have been updated unexpectedly") + }) + + It("can add the missing secondary snapshots (index 2) with the expected contents hash", func() { + for subIdx := len(snapshotNamesIdx2) - 1; subIdx >= 1; subIdx-- { + snapshotsIdx2[subIdx] = createSubIndexedPlacementResourceSnapshotWithHash(clusterScopedNS, snapshotNamesIdx2[subIdx], placementPolicyName, + 2, subIdx, len(snapshotNamesIdx2), contentsHashIdx2, + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + }) + + It("should update and create the work objects", func() { + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: workNames[0], + UID: primaryWorkUID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + wantWorks := make([]placementv1alpha1.Work, len(snapshotNamesIdx2)) + for subIdx := range snapshotNamesIdx2 { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + snapshotNamesIdx2[0], subIdx, len(snapshotNamesIdx2), ownerRef, snapshotsIdx2[subIdx]) + } + + // The work generator does not watch placement resource snapshots; it picks up the newly added secondary + // snapshots only when it retries the failed reconciliation, whose backoff is capped at a second in this suite (see requeueMaxDelay). + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should update the primary work object in place (not re-create it)", func() { + primaryWork := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork)).To(Succeed(), "Failed to retrieve the primary work object") + Expect(primaryWork.UID).To(Equal(primaryWorkUID), "The primary work object has been re-created") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(9, snapshotNamesIdx2[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range snapshotNamesIdx2 { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(9, snapshotNamesIdx2[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work objects are owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes them manually (if they still exist). + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append([]string{snapshotNameIdx0}, snapshotNamesIdx1...), snapshotNamesIdx2...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) + + Context("update works (no spec changes)", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // Before the rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 0. + oldSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + } + // After the rollout: the primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 1, + // with exactly the same content as their counterparts of index 0. + newSnapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 1), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 1, 1), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects, ordered by the sub-index of the snapshots they are derived from. + workNames := make([]string, len(oldSnapshotNames)) + workUIDs := make([]types.UID, len(oldSnapshotNames)) + oldSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(oldSnapshotNames)) + newSnapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(newSnapshotNames)) + resourcesPerSnapshot := make([][]placementv1alpha1.SnapshottedResource, len(oldSnapshotNames)) + var ownerRef *metav1.OwnerReference + var oldBindingGeneration int64 + + // Each snapshot features its own set of differently named NS, Deployment, and ConfigMap objects. + nsNames := make([]string, len(oldSnapshotNames)) + deployNames := make([]string, len(oldSnapshotNames)) + configMapNames := make([]string, len(oldSnapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + BeforeAll(func() { + for subIdx := range oldSnapshotNames { + resourcesPerSnapshot[subIdx] = []placementv1alpha1.SnapshottedResource{ + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + } + } + + // Create the placement resource snapshots of index 0. The secondary snapshot is created first, as the + // placement resource snapshot manager would do. + for subIdx := len(oldSnapshotNames) - 1; subIdx >= 0; subIdx-- { + oldSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, oldSnapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(oldSnapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Create a cluster placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, oldSnapshotNames[0]) + for subIdx := range oldSnapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work object should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef = &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(oldSnapshotNames)) + for subIdx := range oldSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + oldSnapshotNames[0], subIdx, len(oldSnapshotNames), ownerRef, oldSnapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + + // Record the UIDs of the work objects, which should be kept across the rollout. + for subIdx := range oldSnapshotNames { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[subIdx]}, work)).To(Succeed(), "Failed to retrieve the work object") + workUIDs[subIdx] = work.UID + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(6, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + for subIdx := range oldSnapshotNames { + markWorkAsAppliedAndAvailable(workNS, workNames[subIdx], manifestIdentifiersPerWork[subIdx]) + } + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(6, oldSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can roll out a new set of cluster placement resource snapshots with the same content", func() { + // Record the current generation of the cluster placement binding. + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + oldBindingGeneration = binding.GetGeneration() + + // Create the new placement resource snapshots with exactly the same resources as their counterparts. + for subIdx := len(newSnapshotNames) - 1; subIdx >= 0; subIdx-- { + newSnapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, newSnapshotNames[subIdx], placementPolicyName, + 1, subIdx, len(newSnapshotNames), resourcesPerSnapshot[subIdx]...) + } + + // Point the cluster placement binding to the new primary snapshot. + updatePlacementBindingResourceSnapshot(clusterScopedNS, bindingName, newSnapshotNames[0]) + }) + + It("should update the work objects", func() { + // The manifests stay the same; only the link to the primary placement resource snapshot changes. + wantWorks := make([]placementv1alpha1.Work, len(newSnapshotNames)) + for subIdx := range newSnapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + newSnapshotNames[0], subIdx, len(newSnapshotNames), ownerRef, newSnapshots[subIdx]) + } + + worksUpdatedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should update the work objects in place (not re-create them)", func() { + for subIdx := range newSnapshotNames { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[subIdx]}, work)).To(Succeed(), "Failed to retrieve the work object") + Expect(work.UID).To(Equal(workUIDs[subIdx]), "The work object has been re-created") + } + }) + + It("should refresh the cluster placement binding status without waiting for the work objects to be re-processed", func() { + // As the work objects have no spec changes, their existing status still applies; the cluster placement + // binding should be reported as synchronized and available, with the conditions observing the new + // generation. + wantStatus := wantPlacementBindingStatusSyncedAndAvailable(6, newSnapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + + binding, err := getPlacementBinding(clusterScopedNS, bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the cluster placement binding") + Expect(binding.GetGeneration()).To(BeNumerically(">", oldBindingGeneration), "The cluster placement binding generation has not been bumped") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work object is owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes it manually. + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + allSnapshotNames := append(append([]string{}, oldSnapshotNames...), newSnapshotNames...) + for _, snapshotName := range allSnapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) + + Context("backport status for failed manifests", Ordered, func() { + bindingName := fmt.Sprintf(clusterPlacementBindingNameTemplate, utils.RandStr()) + placementPolicyName := fmt.Sprintf(clusterPlacementPolicyNameTemplate, utils.RandStr()) + // The primary snapshot (sub-index 0) and one secondary snapshot (sub-index 1) of index 0. + snapshotNames := []string{ + fmt.Sprintf(placementResourceSnapshotNameTemplate, placementPolicyName, 0), + fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, placementPolicyName, 0, 1), + } + workNS := fmt.Sprintf(utils.NamespaceNameFormat, memberCluster1Name) + + // The work objects (and the snapshots they are derived from), ordered by snapshot sub-index. + workNames := make([]string, len(snapshotNames)) + snapshots := make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotNames)) + + // Each snapshot features its own set of differently named NS, Deployment, and ConfigMap objects. + nsNames := make([]string, len(snapshotNames)) + deployNames := make([]string, len(snapshotNames)) + configMapNames := make([]string, len(snapshotNames)) + manifestIdentifiersPerWork := make([][]placementv1alpha1.ManifestIdentifier, len(snapshotNames)) + for subIdx := range snapshotNames { + nsNames[subIdx] = fmt.Sprintf(indexedNSNameTemplate, subIdx) + deployNames[subIdx] = fmt.Sprintf(deployNameTemplate, subIdx) + configMapNames[subIdx] = fmt.Sprintf(configMapNameTemplate, subIdx) + manifestIdentifiersPerWork[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsNames[subIdx]), + deployManifestIdentifier(1, nsNames[subIdx], deployNames[subIdx]), + configMapManifestIdentifier(2, nsNames[subIdx], configMapNames[subIdx]), + } + } + + // The diff details reported for the ConfigMap object in the primary work object, as if the member agent has + // failed to take over an existing ConfigMap object with different data in the member cluster. + // + // Note that the timestamps are truncated to the second, as this is the precision kept by the API server. + configMapDiffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/data/" + configMapDataKey, + ValueInMember: altConfigMapDataValue1, + ValueInHub: configMapDataValue, + }, + }, + } + // The diff details reported for the Deployment object in the secondary work object, as if the member agent + // has failed to take over an existing Deployment object with a different replica count in the member cluster. + deployDiffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(2)), + FirstDiffedObservedTimestamp: metav1.NewTime(time.Now().Truncate(time.Second)), + ObservedDiffs: []placementv1alpha1.PatchDetail{ + { + Path: "/spec/replicas", + ValueInMember: "2", + ValueInHub: "1", + }, + }, + } + + BeforeAll(func() { + // Create the placement resource snapshots, with a different set of resources in each snapshot. The + // secondary snapshot is created first, as the placement resource snapshot manager would do. + for subIdx := len(snapshotNames) - 1; subIdx >= 0; subIdx-- { + snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(clusterScopedNS, snapshotNames[subIdx], placementPolicyName, + 0, subIdx, len(snapshotNames), + nsSnapshottedResource(nsNames[subIdx]), + deploySnapshottedResource(nsNames[subIdx], deployNames[subIdx]), + configMapSnapshottedResource(nsNames[subIdx], configMapNames[subIdx]), + ) + } + + // Create a cluster placement binding that binds the snapshots to a member cluster. + binding := createPlacementBinding(clusterScopedNS, bindingName, placementPolicyName, memberCluster1Name, snapshotNames[0]) + for subIdx := range snapshotNames { + workNames[subIdx] = uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) + } + }) + + It("should add cleanup finalizer to the cluster placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(clusterScopedNS, bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the cluster placement binding") + }) + + It("should generate the work objects", func() { + // Retrieve the primary work object; the secondary work object should be owned by it. + primaryWork := &placementv1alpha1.Work{} + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workNames[0]}, primaryWork) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the primary work object") + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(snapshotNames)) + for subIdx := range snapshotNames { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(workNS, workNames[subIdx], clusterScopedNS, bindingName, placementPolicyName, + snapshotNames[0], subIdx, len(snapshotNames), ownerRef, snapshots[subIdx]) + } + + worksGeneratedActual := worksGeneratedActual(memberCluster1Name, clusterScopedNS, bindingName, wantWorks) + Eventually(worksGeneratedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the cluster placement binding status", func() { + wantStatus := wantPlacementBindingStatusWaitingForSync(6, snapshotNames[0]) + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + It("can mark some manifests in the primary work object as failed", func() { + // The Deployment object fails to become available; the ConfigMap object fails to be applied (with diff + // details). + markWorkManifestsAsFailed(workNS, workNames[0], manifestIdentifiersPerWork[0], + map[int]*placementv1alpha1.DiffDetails{2: configMapDiffDetails}, + sets.New(1), + ) + }) + + It("can mark some manifests in the secondary work object as failed", func() { + // The Deployment object fails to be applied (with diff details); the ConfigMap object fails to become + // available. + markWorkManifestsAsFailed(workNS, workNames[1], manifestIdentifiersPerWork[1], + map[int]*placementv1alpha1.DiffDetails{1: deployDiffDetails}, + sets.New(2), + ) + }) + + It("should refresh the cluster placement binding status with the failure info from all work objects", func() { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + // In each work object, the NS object is applied and available; the Deployment and ConfigMap objects + // have failed in different ways. + SelectedResources: ptr.To(int32(6)), + SynchronizedResources: ptr.To(int32(4)), + AvailableResources: ptr.To(int32(2)), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[0][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[0][2]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: configMapDiffDetails, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[1][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + DiffDetails: deployDiffDetails, + }, + { + ObjectRef: objectRefFromManifestIdentifier(manifestIdentifiersPerWork[1][2]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(snapshotNames[0]), + } + + statusUpdatedActual := placementBindingStatusUpdatedActual(clusterScopedNS, bindingName, wantStatus) + Eventually(statusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the cluster placement binding status") + }) + + AfterAll(func() { + // Delete the cluster placement binding; the work generator should clean up the primary work object and + // then remove the cleanup finalizer. + removePlacementBinding(clusterScopedNS, bindingName) + + primaryWorkRemovedActual := workRemovedActual(workNS, workNames[0]) + Eventually(primaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + + bindingRemovedActual := placementBindingRemovedActual(clusterScopedNS, bindingName) + Eventually(bindingRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement binding") + + // The secondary work object is owned by the primary work object and normally would be garbage collected + // by Kubernetes; however, the environment prepared by the envtest package does not run the built-in + // garbage collector, so the test suite deletes it manually. + for subIdx := 1; subIdx < len(workNames); subIdx++ { + secondaryWork := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workNames[subIdx], + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, secondaryWork))).To(Succeed(), "Failed to delete the secondary work object") + + secondaryWorkRemovedActual := workRemovedActual(workNS, workNames[subIdx]) + Eventually(secondaryWorkRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the secondary work object") + } + + for _, snapshotName := range snapshotNames { + snapshotRemovedActual := placementResourceSnapshotRemovedActual(clusterScopedNS, snapshotName) + Eventually(snapshotRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cluster placement resource snapshot") + } + }) + }) +}) diff --git a/pkg/v1/controllers/workgenerator/invalid_snapshots_integration_test.go b/pkg/v1/controllers/workgenerator/invalid_snapshots_integration_test.go new file mode 100644 index 000000000..1b13d074a --- /dev/null +++ b/pkg/v1/controllers/workgenerator/invalid_snapshots_integration_test.go @@ -0,0 +1,696 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// The work generator does not watch placement resource snapshots; it picks up snapshot changes only when it retries +// a failed reconciliation, whose backoff is capped at a second in this suite (see requeueMaxDelay). +var _ = Describe("handling invalid placement resource snapshots", func() { + serverSideApplyStrategy := &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodServerSideApply, + ServerSideApplyOptions: &placementv1alpha1.ServerSideApplyOptions{ + ForceConflicts: true, + }, + } + + for _, sc := range scopes { + Context(fmt.Sprintf("roll out to a missing snapshot (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + var set1 *snapshotSet + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs of the work objects", func() { + f.recordWorkUIDs() + }) + + It("can roll out a snapshot that does not exist yet", func() { + set1 = f.newSnapshotSet(1, 1) + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set1.primaryName()) + }) + + It("should mark the placement binding as failed to synchronize due to invalid snapshots", func() { + set0 := f.sets[0] + wantStatus := wantPlacementBindingStatusInvalidSnapshots(set0.resourceCount(), set0.primaryName(), f.status) + Eventually(f.invalidSnapshotsStatusActual(wantStatus, fmt.Sprintf(snapshotNotFoundErrMsgFmt, set1.primaryName())), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + f.recordStatus() + }) + + It("should not write to the placement binding or update the work objects while retrying", func() { + Consistently(func() error { + if err := f.worksMatchActual(0, nil)(); err != nil { + return err + } + return f.resourceVersionUnchangedActual()() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding or the work objects have been updated unexpectedly") + }) + + It("can mark the deployment in the placed work object as unavailable", func() { + markWorkManifestAsUnavailable(f.workNS, f.workName(0), 1) + }) + + It("should recount the resources from the placed work objects", func() { + set0 := f.sets[0] + wantStatus := wantPlacementBindingStatusInvalidSnapshots(set0.resourceCount(), set0.primaryName(), f.status) + wantStatus.AvailableResources = ptr.To(set0.resourceCount() - 1) + wantStatus.FailedResources = []placementv1alpha1.FailedResource{ + { + ObjectRef: objectRefFromManifestIdentifier(set0.identifiers[0][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + } + Eventually(f.invalidSnapshotsStatusActual(wantStatus, fmt.Sprintf(snapshotNotFoundErrMsgFmt, set1.primaryName())), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to recount the resources") + }) + + It("can create the missing snapshot", func() { + f.createSnapshotsInSet(set1) + }) + + It("should update the work object in place", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work object") + f.expectWorksNotRecreated() + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + for _, tc := range []struct { + name string + strayHash string + }{ + {name: "with the same contents hash", strayHash: dummyContentsHash}, + {name: "with a different contents hash", strayHash: "stray-contents-hash"}, + } { + Context(fmt.Sprintf("stray snapshot duplicating the last counted sub-index, %s (%s)", tc.name, sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + var set1 *snapshotSet + var strayName string + + BeforeAll(func() { + f.createSnapshots(0, 2) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs of the work objects", func() { + f.recordWorkUIDs() + }) + + It("can roll out a snapshot set that does not exist yet", func() { + set1 = f.newSnapshotSet(1, 2) + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set1.primaryName()) + }) + + It("should mark the placement binding as failed to synchronize as the snapshot is not found", func() { + set0 := f.sets[0] + wantStatus := wantPlacementBindingStatusInvalidSnapshots(set0.resourceCount(), set0.primaryName(), f.status) + Eventually(f.invalidSnapshotsStatusActual(wantStatus, fmt.Sprintf(snapshotNotFoundErrMsgFmt, set1.primaryName())), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + }) + + It("can create the snapshot set along with a stray snapshot duplicating its last sub-index", func() { + // The stray snapshot is created first, so that the work generator never sees the snapshot set + // without it. + strayName = f.createStraySnapshot(1, 1, 2, tc.strayHash) + f.createSnapshotsInSet(set1) + }) + + It("should report the duplicate sub-indices", func() { + set0 := f.sets[0] + wantStatus := wantPlacementBindingStatusInvalidSnapshots(set0.resourceCount(), set0.primaryName(), f.status) + Eventually(f.invalidSnapshotsStatusActual(wantStatus, duplicateSubIdxErrMsg), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the duplicate sub-indices") + f.recordStatus() + }) + + It("should not write to the placement binding or update the work objects while retrying", func() { + Consistently(func() error { + if err := f.worksMatchActual(0, nil)(); err != nil { + return err + } + return f.resourceVersionUnchangedActual()() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding or the work objects have been updated unexpectedly") + }) + + It("can delete the stray snapshot", func() { + f.deleteStraySnapshot(strayName) + }) + + It("should update the work objects in place", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + f.expectWorksNotRecreated() + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } + + Context(fmt.Sprintf("stray snapshots further down the listing (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + var set1 *snapshotSet + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can create a snapshot set along with two stray snapshots beyond its count", func() { + // Both stray snapshots have the sub-index right after the counted ones; only the first of them in the + // sorted listing is checked (and found not to be a duplicate of the last counted snapshot), and the + // second, a duplicate of the first, is ignored. + set1 = f.newSnapshotSet(1, 2) + f.createStraySnapshot(1, 2, 2, dummyContentsHash) + f.createStraySnapshot(1, 2, 2, dummyContentsHash) + f.createSnapshotsInSet(set1) + }) + + It("can roll out the snapshot set", func() { + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set1.primaryName()) + }) + + It("should update the work objects (and not create any for the stray snapshots)", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("stray primary snapshot at the same index (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + var set1 *snapshotSet + var strayName string + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can create a snapshot set along with a stray primary snapshot at the same index", func() { + set1 = f.newSnapshotSet(1, 2) + strayName = f.createStraySnapshot(1, 0, 2, dummyContentsHash) + f.createSnapshotsInSet(set1) + }) + + It("can roll out the snapshot set", func() { + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set1.primaryName()) + }) + + It("should report the duplicate sub-indices", func() { + set0 := f.sets[0] + wantStatus := wantPlacementBindingStatusInvalidSnapshots(set0.resourceCount(), set0.primaryName(), f.status) + Eventually(f.invalidSnapshotsStatusActual(wantStatus, duplicateSubIdxErrMsg), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the duplicate sub-indices") + }) + + It("should not update the work objects", func() { + Consistently(f.worksMatchActual(0, nil), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been updated unexpectedly") + }) + + It("can delete the stray snapshot", func() { + f.deleteStraySnapshot(strayName) + }) + + It("should update the work objects", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("gap in the counted sub-indices (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + var set1 *snapshotSet + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can create a snapshot set with a missing sub-index and an extra snapshot beyond its count", func() { + // Sub-indices 0 and 2 (of 3), plus a stray snapshot of sub-index 3. + set1 = f.newSnapshotSet(1, 3) + f.createStraySnapshot(1, 3, 3, dummyContentsHash) + f.createSnapshotsInSet(set1, 2, 0) + }) + + It("can roll out the snapshot set", func() { + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set1.primaryName()) + }) + + It("should report the inconsistent sub-indices", func() { + set0 := f.sets[0] + wantStatus := wantPlacementBindingStatusInvalidSnapshots(set0.resourceCount(), set0.primaryName(), f.status) + Eventually(f.invalidSnapshotsStatusActual(wantStatus, inconsistentSubIdxErrMsg), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the inconsistent sub-indices") + }) + + It("should not update the work objects", func() { + Consistently(f.worksMatchActual(0, nil), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been updated unexpectedly") + }) + + It("can create the missing snapshot", func() { + f.createSnapshotsInSet(set1, 1) + }) + + It("should update and create the work objects (and not create any for the extra snapshot)", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update and create the work objects") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("new binding with a missing snapshot (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + var set0 *snapshotSet + + BeforeAll(func() { + set0 = f.newSnapshotSet(0, 1) + f.createBinding(0) + }) + + It("should add cleanup finalizer to the placement binding", func() { + Eventually(placementBindingFinalizerAddedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should mark the placement binding as failed to synchronize due to invalid snapshots", func() { + // Nothing has been processed (or placed) yet. + wantStatus := wantPlacementBindingStatusInvalidSnapshots(0, "", nil) + wantStatus.LastProcessedResourceSnapshotName = nil + Eventually(f.invalidSnapshotsStatusActual(wantStatus, fmt.Sprintf(snapshotNotFoundErrMsgFmt, set0.primaryName())), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + }) + + It("should not generate any work object", func() { + Consistently(f.noWorksActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Work objects have been generated unexpectedly") + }) + + It("can create the missing snapshot", func() { + f.createSnapshotsInSet(set0) + }) + + It("should generate the work objects", func() { + Eventually(f.worksMatchActual(0, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(0) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("sync strategy change while the processed snapshot is missing (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 2) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs of the work objects and the placement binding status", func() { + f.recordWorkUIDs() + f.recordStatus() + }) + + It("can delete the primary snapshot in use", func() { + f.deleteSnapshot(0, 0) + }) + + It("can set a sync strategy on the placement binding", func() { + updatePlacementBindingSyncStrategy(sc.bindingNS, f.bindingName, serverSideApplyStrategy) + }) + + It("should update the sync strategy of the work objects in place, without the snapshots", func() { + Eventually(f.worksMatchBindingStrategyActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + f.expectWorksNotRecreated() + }) + + // A sync strategy change alone is applied to the work objects without resetting the placement binding + // status (and without reporting the missing snapshot). + It("should not refresh the placement binding status before the work objects are processed", func() { + Consistently(f.statusUnchangedActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status has been refreshed unexpectedly") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(0) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("sync strategy change along with a rollout to a missing snapshot (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + var set1 *snapshotSet + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs of the work objects", func() { + f.recordWorkUIDs() + }) + + It("can set a sync strategy and roll out a snapshot that does not exist yet in one update", func() { + set1 = f.newSnapshotSet(1, 1) + f.updateSpec(func(spec *placementv1alpha1.PlacementBindingSpec) { + spec.SyncStrategy = serverSideApplyStrategy.DeepCopy() + spec.ResourceSnapshotName = set1.primaryName() + }) + }) + + It("should mark the placement binding as failed to synchronize due to invalid snapshots", func() { + set0 := f.sets[0] + wantStatus := wantPlacementBindingStatusInvalidSnapshots(set0.resourceCount(), set0.primaryName(), f.status) + Eventually(f.invalidSnapshotsStatusActual(wantStatus, fmt.Sprintf(snapshotNotFoundErrMsgFmt, set1.primaryName())), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + }) + + It("should not update the work objects (including their sync strategy)", func() { + Consistently(f.worksMatchActual(0, nil), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been updated unexpectedly") + }) + + It("can create the missing snapshot", func() { + f.createSnapshotsInSet(set1) + }) + + It("should update the work object in place, with the new snapshot and the sync strategy", func() { + Eventually(f.worksMatchBindingStrategyActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work object") + f.expectWorksNotRecreated() + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("work object deleted out of band while the processed snapshot is missing (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 2) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs of the work objects", func() { + f.recordWorkUIDs() + }) + + It("can delete the primary snapshot in use", func() { + f.deleteSnapshot(0, 0) + }) + + It("can delete the secondary work object", func() { + deleteWork(f.workNS, f.workName(1)) + }) + + It("should mark the placement binding as failed to synchronize, at the same generation", func() { + // The placement binding still points to the processed snapshot, so availability is invalidated; + // the resource counters are tallied from the remaining (primary) work object. + set0 := f.sets[0] + placedResources := int32(resourcesPerSnapshot) + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonInvalidSnapshots, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonIncompleteSetOfResources, + }, + }, + SelectedResources: ptr.To(placedResources), + SynchronizedResources: ptr.To(placedResources), + AvailableResources: ptr.To(placedResources), + LastProcessedResourceSnapshotName: ptr.To(set0.primaryName()), + } + Eventually(f.invalidSnapshotsStatusActual(wantStatus, fmt.Sprintf(snapshotNotFoundErrMsgFmt, set0.primaryName())), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + }) + + It("should not re-create the secondary work object", func() { + Consistently(func() error { + uids, err := f.listWorkUIDs() + if err != nil { + return err + } + if _, found := uids[f.workName(1)]; found { + return fmt.Errorf("the secondary work object has been re-created") + } + if uids[f.workName(0)] != f.workUIDs[f.workName(0)] { + return fmt.Errorf("the primary work object has been re-created or removed") + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have been changed unexpectedly") + }) + + It("can re-create the primary snapshot", func() { + f.createSnapshotsInSet(f.sets[0], 0) + }) + + It("should re-create the secondary work object", func() { + Eventually(f.worksMatchActual(0, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the secondary work object") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(0) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("suspend, resume, and delete a binding with a missing snapshot (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + var set1 *snapshotSet + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can roll out a snapshot that does not exist yet", func() { + set1 = f.newSnapshotSet(1, 1) + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set1.primaryName()) + }) + + It("should mark the placement binding as failed to synchronize due to invalid snapshots", func() { + set0 := f.sets[0] + wantStatus := wantPlacementBindingStatusInvalidSnapshots(set0.resourceCount(), set0.primaryName(), f.status) + Eventually(f.invalidSnapshotsStatusActual(wantStatus, fmt.Sprintf(snapshotNotFoundErrMsgFmt, set1.primaryName())), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + f.recordStatus() + }) + + It("can suspend the placement binding", func() { + updatePlacementBindingSuspended(sc.bindingNS, f.bindingName, true) + }) + + It("should remove the work object", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + }) + + It("should unset the last processed snapshot and remove the cleanup finalizer, leaving the rest of the status as it is", func() { + wantStatus := f.status.DeepCopy() + wantStatus.LastProcessedResourceSnapshotName = nil + Eventually(placementBindingStatusMatchesActual(sc.bindingNS, f.bindingName, wantStatus), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unset the last processed snapshot") + Eventually(placementBindingFinalizerRemovedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("can resume the placement binding", func() { + updatePlacementBindingSuspended(sc.bindingNS, f.bindingName, false) + }) + + It("should add cleanup finalizer to the placement binding", func() { + Eventually(placementBindingFinalizerAddedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should mark the placement binding as failed to synchronize again, at the new generation", func() { + // No snapshot has been processed (or resource placed) since the suspension. + wantStatus := wantPlacementBindingStatusInvalidSnapshots(0, "", nil) + wantStatus.LastProcessedResourceSnapshotName = nil + Eventually(f.invalidSnapshotsStatusActual(wantStatus, fmt.Sprintf(snapshotNotFoundErrMsgFmt, set1.primaryName())), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the placement binding as failed to synchronize") + }) + + It("should not generate any work object", func() { + Consistently(f.noWorksActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Work objects have been generated unexpectedly") + }) + + It("can delete the placement binding", func() { + removePlacementBinding(sc.bindingNS, f.bindingName) + }) + + It("should remove the placement binding", func() { + Eventually(placementBindingRemovedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } +}) diff --git a/pkg/v1/controllers/workgenerator/lifecycle_integration_test.go b/pkg/v1/controllers/workgenerator/lifecycle_integration_test.go new file mode 100644 index 000000000..29ecc578b --- /dev/null +++ b/pkg/v1/controllers/workgenerator/lifecycle_integration_test.go @@ -0,0 +1,815 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "fmt" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/types" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("deleting placement bindings", func() { + for _, sc := range scopes { + Context(fmt.Sprintf("delete a binding with a single placement resource snapshot (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + // A binding on another member cluster, which should not be affected. + other := newBindingFixture(sc, memberCluster2Name) + + BeforeAll(func() { + for _, fixture := range []*bindingFixture{f, other} { + fixture.createSnapshots(0, 1) + fixture.createBinding(0) + } + }) + + f.registerBaselineSpecs(0) + + It("should generate the work objects and sync the status of the other placement binding", func() { + other.syncWorks(0) + }) + + It("can delete the placement binding", func() { + removePlacementBinding(sc.bindingNS, f.bindingName) + }) + + It("should remove the work object and the placement binding", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + Eventually(placementBindingRemovedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + }) + + It("should not affect the other placement binding", func() { + Consistently(other.worksMatchActual(0, nil), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects of the other placement binding have changed") + Consistently(other.syncedStatusActual(0), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The status of the other placement binding has changed") + }) + + AfterAll(func() { + f.teardown() + other.teardown() + }) + }) + + Context(fmt.Sprintf("delete a binding with multiple placement resource snapshots (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 2) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can delete the placement binding", func() { + removePlacementBinding(sc.bindingNS, f.bindingName) + }) + + // The secondary work object is owned by the primary one, and is left for K8s' garbage collection. + It("should remove the primary work object and the placement binding", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + Eventually(placementBindingRemovedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } +}) + +var _ = Describe("suspending placement bindings", func() { + for _, sc := range scopes { + Context(fmt.Sprintf("suspend and resume a binding (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can suspend the placement binding", func() { + updatePlacementBindingSuspended(sc.bindingNS, f.bindingName, true) + }) + + It("should remove the work object", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + }) + + It("should unset the last processed placement resource snapshot name", func() { + unsetActual := placementBindingLastProcessedSnapshotUnsetActual(sc.bindingNS, f.bindingName) + Eventually(unsetActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unset the last processed placement resource snapshot name") + }) + + It("should remove the cleanup finalizer from the placement binding", func() { + finalizerRemovedActual := placementBindingFinalizerRemovedActual(sc.bindingNS, f.bindingName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("can roll out a new placement resource snapshot while the placement binding is suspended", func() { + set := f.createSnapshots(1, 1) + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set.primaryName()) + }) + + It("should not generate any work object while the placement binding is suspended", func() { + Consistently(f.noWorksActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Work objects have been generated for a suspended placement binding") + finalizerRemovedActual := placementBindingFinalizerRemovedActual(sc.bindingNS, f.bindingName) + Consistently(finalizerRemovedActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cleanup finalizer has been added back to a suspended placement binding") + }) + + It("can resume the placement binding", func() { + updatePlacementBindingSuspended(sc.bindingNS, f.bindingName, false) + }) + + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(sc.bindingNS, f.bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work object from the new placement resource snapshot", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work object") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("suspend and resume a binding whose work objects have not been processed (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + It("should generate the work object", func() { + Eventually(f.worksMatchActual(0, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work object") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can suspend the placement binding", func() { + updatePlacementBindingSuspended(sc.bindingNS, f.bindingName, true) + }) + + It("should remove the work object and unset the last processed placement resource snapshot name", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + unsetActual := placementBindingLastProcessedSnapshotUnsetActual(sc.bindingNS, f.bindingName) + Eventually(unsetActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unset the last processed placement resource snapshot name") + }) + + It("can resume the placement binding", func() { + updatePlacementBindingSuspended(sc.bindingNS, f.bindingName, false) + }) + + It("should re-generate the work object", func() { + Eventually(f.worksMatchActual(0, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-generate the work object") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + f.markWorks(0) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("delete a suspended binding (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can suspend the placement binding", func() { + updatePlacementBindingSuspended(sc.bindingNS, f.bindingName, true) + }) + + It("should remove the work object and the cleanup finalizer", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + finalizerRemovedActual := placementBindingFinalizerRemovedActual(sc.bindingNS, f.bindingName) + Eventually(finalizerRemovedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the cleanup finalizer") + }) + + It("can delete the placement binding", func() { + removePlacementBinding(sc.bindingNS, f.bindingName) + }) + + It("should remove the placement binding without generating any work object", func() { + Eventually(placementBindingRemovedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + Consistently(f.noWorksActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Work objects have been generated for a deleted placement binding") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } +}) + +var _ = Describe("changing sync strategies of placement bindings", func() { + serverSideApplyStrategy := &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodServerSideApply, + ServerSideApplyOptions: &placementv1alpha1.ServerSideApplyOptions{ + ForceConflicts: true, + }, + } + + for _, sc := range scopes { + for _, snapshotCount := range []int{1, 2} { + Context(fmt.Sprintf("update the sync strategy with %d placement resource snapshot(s) (%s)", snapshotCount, sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, snapshotCount) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs of the work objects", func() { + f.recordWorkUIDs() + Expect(f.workUIDs).To(HaveLen(snapshotCount), "Unexpected number of work objects") + }) + + It("can record the placement binding status", func() { + f.recordStatus() + }) + + It("can set a sync strategy on the placement binding", func() { + updatePlacementBindingSyncStrategy(sc.bindingNS, f.bindingName, serverSideApplyStrategy) + }) + + It("should update the work objects in place", func() { + // The sync strategy on the work objects should match the one on the placement binding as written + // to the API server (which might carry defaulted fields). + Eventually(func() error { + binding, err := getPlacementBinding(sc.bindingNS, f.bindingName) + if err != nil { + return err + } + syncStrategy := binding.GetSpec().SyncStrategy + if syncStrategy == nil { + return fmt.Errorf("the sync strategy has not been set on the placement binding") + } + return f.worksMatchActual(0, syncStrategy)() + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + + uids, err := f.listWorkUIDs() + Expect(err).To(Succeed(), "Failed to list the work objects") + if diff := cmp.Diff(uids, f.workUIDs); diff != "" { + Fail(fmt.Sprintf("work objects have been re-created (-got +want):\n%s", diff)) + } + }) + + // A sync strategy change alone is applied to the work objects without resetting the placement binding + // status; the status is refreshed only after the updated work objects have been processed. + It("should not refresh the placement binding status before the work objects are processed", func() { + Consistently(f.statusUnchangedActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status has been refreshed unexpectedly") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(0) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can record the placement binding status again", func() { + f.recordStatus() + }) + + It("can unset the sync strategy on the placement binding", func() { + updatePlacementBindingSyncStrategy(sc.bindingNS, f.bindingName, nil) + }) + + It("should update the work objects in place", func() { + Eventually(f.worksMatchActual(0, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + + uids, err := f.listWorkUIDs() + Expect(err).To(Succeed(), "Failed to list the work objects") + if diff := cmp.Diff(uids, f.workUIDs); diff != "" { + Fail(fmt.Sprintf("work objects have been re-created (-got +want):\n%s", diff)) + } + }) + + It("should not refresh the placement binding status before the work objects are processed", func() { + Consistently(f.statusUnchangedActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status has been refreshed unexpectedly") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(0) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } + } +}) + +var _ = Describe("handling work object events", func() { + for _, sc := range scopes { + Context(fmt.Sprintf("work objects that the work generator does not own (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + foreignWorkNames := []string{ + fmt.Sprintf("foreign-work-%s", utils.RandStr()), + fmt.Sprintf("foreign-work-%s", utils.RandStr()), + fmt.Sprintf("foreign-work-%s", utils.RandStr()), + } + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can create work objects that are not owned by the placement binding", func() { + ownerNSLabel := map[string]string{placementv1alpha1.WorkOwnerNamespaceLabelKey: sc.bindingNS} + + // A work object without any KubeFleet owner metadata. + createForeignWork(f.workNS, foreignWorkNames[0], nil, nil) + // A work object owned by a placement binding that does not exist. + createForeignWork(f.workNS, foreignWorkNames[1], ownerNSLabel, map[string]string{ + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: fmt.Sprintf("non-existent-binding-%s", utils.RandStr()), + }) + // A work object that points to the placement binding, but is not labeled as being owned by it. + createForeignWork(f.workNS, foreignWorkNames[2], ownerNSLabel, map[string]string{ + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: f.bindingName, + }) + }) + + It("can mark the work objects as applied and available", func() { + for _, workName := range foreignWorkNames { + markWorkAsAppliedAndAvailable(f.workNS, workName, nil) + } + }) + + It("should not affect the placement binding", func() { + Consistently(f.worksMatchActual(0, nil), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have changed") + Consistently(f.syncedStatusActual(0), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status has changed") + }) + + It("can delete the work objects", func() { + for _, workName := range foreignWorkNames { + deleteWork(f.workNS, workName) + Eventually(workRemovedActual(f.workNS, workName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + } + }) + + It("should not affect the placement binding", func() { + Consistently(f.worksMatchActual(0, nil), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The work objects have changed") + Consistently(f.syncedStatusActual(0), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status has changed") + }) + + It("can roll out a new placement resource snapshot", func() { + set := f.createSnapshots(1, 1) + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set.primaryName()) + }) + + It("should update the work object", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work object") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work object as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + // The work generator re-creates work objects that have been deleted out of band, without waiting for a rollout. + for _, tc := range []struct { + name string + snapshotCount int + // The sub-indices of the work objects to delete. + deleteSubIdxs []int + }{ + {name: "the secondary work object is deleted", snapshotCount: 2, deleteSubIdxs: []int{1}}, + {name: "the primary work object is deleted", snapshotCount: 2, deleteSubIdxs: []int{0}}, + {name: "the only work object is deleted", snapshotCount: 1, deleteSubIdxs: []int{0}}, + } { + Context(fmt.Sprintf("work objects deleted out of band: %s (%s)", tc.name, sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, tc.snapshotCount) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs of the work objects", func() { + f.recordWorkUIDs() + Expect(f.workUIDs).To(HaveLen(tc.snapshotCount), "Unexpected number of work objects") + }) + + It("can delete the work objects", func() { + for _, subIdx := range tc.deleteSubIdxs { + deleteWork(f.workNS, f.workName(subIdx)) + } + }) + + It("should re-create the deleted work objects", func() { + Eventually(func() error { + uids, err := f.listWorkUIDs() + if err != nil { + return err + } + // The cache might still serve the deleted work objects. + for _, subIdx := range tc.deleteSubIdxs { + workName := f.workName(subIdx) + if uid, found := uids[workName]; found && uid == f.workUIDs[workName] { + return fmt.Errorf("work object %s has not been re-created", workName) + } + } + return f.worksMatchActual(0, nil)() + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the deleted work objects") + }) + + It("should leave the other work objects untouched", func() { + uids, err := f.listWorkUIDs() + Expect(err).To(Succeed(), "Failed to list the work objects") + for subIdx := 0; subIdx < tc.snapshotCount; subIdx++ { + deleted := false + for _, deletedSubIdx := range tc.deleteSubIdxs { + deleted = deleted || deletedSubIdx == subIdx + } + workName := f.workName(subIdx) + if !deleted && uids[workName] != f.workUIDs[workName] { + Fail(fmt.Sprintf("work object %s has been re-created unexpectedly", workName)) + } + } + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the re-created work objects as applied and available", func() { + f.markWorks(0, tc.deleteSubIdxs...) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } + } +}) + +var _ = Describe("binding a placement policy to multiple clusters", func() { + for _, sc := range scopes { + Context(fmt.Sprintf("two bindings of the same placement policy (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + // Another binding of the same placement policy (sharing the same snapshots), on another member cluster. + other := f.newFixtureOfSamePolicy(memberCluster2Name) + + BeforeAll(func() { + f.createSnapshots(0, 2) + f.createBinding(0) + other.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("should generate the work objects of the other placement binding, with the same names", func() { + Eventually(other.worksMatchActual(0, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects of the other placement binding") + for subIdx := 0; subIdx < 2; subIdx++ { + Expect(other.workName(subIdx)).To(Equal(f.workName(subIdx)), "The work objects of the two placement bindings should have the same names") + } + }) + + It("should keep the other placement binding waiting, as its work objects have not been processed", func() { + Eventually(other.waitingStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the status of the other placement binding") + Consistently(other.waitingStatusActual(0), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The status of the other placement binding has changed unexpectedly") + }) + + It("can record the UIDs of the work objects and the status of the other placement binding", func() { + other.recordWorkUIDs() + other.recordStatus() + }) + + It("can roll out a new snapshot set to the first placement binding only", func() { + set := f.createSnapshots(1, 2) + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set.primaryName()) + }) + + It("should update the work objects of the first placement binding", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + }) + + It("should refresh the status of the first placement binding", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects of the first placement binding as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the status of the first placement binding", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("should leave the other placement binding untouched", func() { + Consistently(func() error { + if err := other.worksMatchActual(0, nil)(); err != nil { + return err + } + return other.resourceVersionUnchangedActual()() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The other placement binding or its work objects have changed") + other.expectWorksNotRecreated() + }) + + It("can delete the first placement binding", func() { + removePlacementBinding(sc.bindingNS, f.bindingName) + }) + + It("should remove the primary work object of the first placement binding and the placement binding itself", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + Eventually(placementBindingRemovedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + }) + + It("should leave the other placement binding untouched", func() { + Consistently(func() error { + if err := other.worksMatchActual(0, nil)(); err != nil { + return err + } + return other.resourceVersionUnchangedActual()() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The other placement binding or its work objects have changed") + other.expectWorksNotRecreated() + }) + + AfterAll(func() { + f.teardown() + other.teardown() + }) + }) + } +}) + +var _ = Describe("placement bindings with long and dotted names", func() { + for _, sc := range scopes { + Context(fmt.Sprintf("names over the label value length limit, with dots (%s)", sc.name), Ordered, func() { + // The placement policy name is long enough for the work names to be truncated, while keeping the + // snapshot names (`[POLICY-NAME]-snapshot-[INDEX]-[SUB-INDEX]`) within the 253-character limit; both names + // are too long (and dotted) to be used as label values as they are. + f := newBindingFixtureWithNames(sc, memberCluster1Name, longDottedName("binding", 100), longDottedName("policy", 240)) + + BeforeAll(func() { + f.createSnapshots(0, 2) + f.createBinding(0) + }) + + // The work objects are listed through the owner binding field index (keyed on the hashed owner label + // value), and their events are mapped to the placement binding through the owner annotation (which + // carries the full name); the baseline specs exercise both. + f.registerBaselineSpecs(0) + + It("should use names long enough to exercise label value hashing and work name truncation", func() { + Expect(workOwnerLabelValue(f.bindingName)).NotTo(Equal(f.bindingName), "The owner label value of the placement binding should be hashed") + Expect(workOwnerLabelValue(f.policyName)).NotTo(Equal(f.policyName), "The owner label value of the placement policy should be hashed") + Expect(f.workName(0)).To(HaveLen(nameLenLimit), "The name of the primary work object should be truncated") + }) + + It("can record the UIDs of the work objects", func() { + f.recordWorkUIDs() + }) + + It("can roll out a new snapshot set", func() { + set := f.createSnapshots(1, 2) + updatePlacementBindingResourceSnapshot(sc.bindingNS, f.bindingName, set.primaryName()) + }) + + It("should update the work objects in place", func() { + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + f.expectWorksNotRecreated() + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can delete the placement binding", func() { + removePlacementBinding(sc.bindingNS, f.bindingName) + }) + + It("should remove the primary work object and the placement binding", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + Eventually(placementBindingRemovedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } +}) + +var _ = Describe("re-creating placement bindings", func() { + for _, sc := range scopes { + Context(fmt.Sprintf("re-create a binding while the secondary work objects of the previous one linger (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 3) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs of the work objects", func() { + f.recordWorkUIDs() + }) + + It("can delete the placement binding", func() { + removePlacementBinding(sc.bindingNS, f.bindingName) + }) + + It("should remove the primary work object and the placement binding", func() { + Eventually(workRemovedActual(f.workNS, f.workName(0)), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the primary work object") + Eventually(placementBindingRemovedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + }) + + // The secondary work objects are owned by the primary work object and are left to K8s' garbage collection, + // which does not run in the test environment; this simulates a garbage collector that has yet to catch up. + It("should leave the secondary work objects behind", func() { + uids, err := f.listWorkUIDs() + Expect(err).To(Succeed(), "Failed to list the work objects") + wantUIDs := map[string]types.UID{ + f.workName(1): f.workUIDs[f.workName(1)], + f.workName(2): f.workUIDs[f.workName(2)], + } + Expect(cmp.Diff(uids, wantUIDs)).To(BeEmpty(), "Unexpected work objects (-got +want)") + }) + + It("can re-create the placement binding with a new snapshot set of fewer snapshots", func() { + f.createSnapshots(1, 2) + f.createBinding(1) + }) + + It("should add cleanup finalizer to the placement binding", func() { + Eventually(placementBindingFinalizerAddedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should adopt the lingering secondary work object still needed, and delete the other one", func() { + // The work objects should match the new snapshot set exactly, i.e., the secondary work object of + // sub-index 1 now points to the new primary work object, and the one of sub-index 2 is gone. + Eventually(f.worksMatchActual(1, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the work objects") + + uids, err := f.listWorkUIDs() + Expect(err).To(Succeed(), "Failed to list the work objects") + Expect(uids[f.workName(1)]).To(Equal(f.workUIDs[f.workName(1)]), "The lingering secondary work object has been re-created") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(1) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(1), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } +}) + +var _ = Describe("missing reserved namespaces", func() { + // The handling of reserved namespaces does not depend on the scope of the placement binding; the namespace-scoped + // one is used. + sc := scopes[0] + + Context("the reserved namespace of the target cluster does not exist yet", Ordered, func() { + // A member cluster whose reserved namespace has not been created (namespaces cannot be deleted in the test + // environment, so a new one is used each time). + clusterName := fmt.Sprintf("member-%s", utils.RandStr()) + f := newBindingFixture(sc, clusterName) + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + It("should add cleanup finalizer to the placement binding", func() { + Eventually(placementBindingFinalizerAddedActual(sc.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should not generate any work object or report any status", func() { + Consistently(func() error { + if err := f.noWorksActual()(); err != nil { + return err + } + return placementBindingStatusMatchesActual(sc.bindingNS, f.bindingName, &placementv1alpha1.PlacementBindingStatus{})() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Work objects have been generated or the status has been reported unexpectedly") + }) + + It("can create the reserved namespace", func() { + createReservedNamespace(clusterName) + }) + + // The work generator does not watch namespaces; it picks up the new namespace when it retries the failed + // reconciliation, whose backoff is capped at a second in this suite (see requeueMaxDelay). + It("should generate the work objects", func() { + Eventually(f.worksMatchActual(0, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(0) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) +}) diff --git a/pkg/v1/controllers/workgenerator/retrieval.go b/pkg/v1/controllers/workgenerator/retrieval.go index cb0a10b57..c04b0e350 100644 --- a/pkg/v1/controllers/workgenerator/retrieval.go +++ b/pkg/v1/controllers/workgenerator/retrieval.go @@ -29,6 +29,7 @@ import ( placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" "go.goms.io/fleet/pkg/utils" "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" ) func (r *Reconciler) retrievePlacementBinding(ctx context.Context, namespacedName types.NamespacedName) (placementv1alpha1.PlacementBindingAccessor, error) { @@ -55,9 +56,12 @@ func (r *Reconciler) listWorksByOwnerBinding(ctx context.Context, clusterName, o workList := &placementv1alpha1.WorkList{} listOptions := []client.ListOption{ client.InNamespace(memberClusterNamespace), - client.MatchingLabels{ - placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: workOwnerLabelValue(ownerBindingName), - placementv1alpha1.WorkOwnerNamespaceLabelKey: ownerBindingNSName, + client.MatchingFields{ + fieldindexers.WorkOwnedByBindingCustomFieldName: fmt.Sprintf( + fieldindexers.WorkOwnedByBindingCustomFieldValFmt, + ownerBindingNSName, + workOwnerLabelValue(ownerBindingName), + ), }, } if err := r.hubClient.List(ctx, workList, listOptions...); err != nil { @@ -103,21 +107,24 @@ func (r *Reconciler) retrievePrimaryAndSecondaryPlacementResourceSnapshots( } // There are secondary snapshots; list all snapshots that share the same owner and index. - ownedBy := primarySnapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey] + ownedBy := primarySnapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] index := primarySnapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotIndexLabelKey] if ownedBy == "" || index == "" { - return nil, errors.NewUnexpectedError(nil, "the primary placement resource snapshot is missing required labels", + return nil, errors.NewUnexpectedError(nil, "the primary placement resource snapshot is missing required labels or annotations", "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot)) } - labelMatchers := client.MatchingLabels{ - placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: ownedBy, - placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: index, + fieldMatchers := client.MatchingFields{ + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, + ownedBy, + index, + ), } var snapshots []placementv1alpha1.PlacementResourceSnapshotAccessor if namespace == "" { snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} - if err := r.hubClient.List(ctx, snapshotList, labelMatchers); err != nil { + if err := r.hubClient.List(ctx, snapshotList, fieldMatchers); err != nil { return nil, errors.NewAPIServerError(err, "failed to list cluster placement resource snapshots", true) } snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotList.Items)) @@ -126,7 +133,7 @@ func (r *Reconciler) retrievePrimaryAndSecondaryPlacementResourceSnapshots( } } else { snapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} - if err := r.hubClient.List(ctx, snapshotList, client.InNamespace(namespace), labelMatchers); err != nil { + if err := r.hubClient.List(ctx, snapshotList, client.InNamespace(namespace), fieldMatchers); err != nil { return nil, errors.NewAPIServerError(err, "failed to list placement resource snapshots", true) } snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotList.Items)) @@ -154,16 +161,53 @@ func (r *Reconciler) retrievePrimaryAndSecondaryPlacementResourceSnapshots( return nil, errors.NewUnexpectedError(nil, "failed to sort placement resource snapshots by sub-index", "errs", sortErrs) } - // Do some sanity checks; verify that all snapshots dictated by the count label are present and they have - // the same snapshotted resource hash. + // Do some sanity checks; verify that all snapshots dictated by the count label are present with consecutive + // sub-indices starting from 0 (no duplicates or gaps), and that they have the same snapshotted resource hash. + + // First, verify the sub-indices. After sorting, the sub-index of each snapshot dictated by the count label should + // match its position in the listing. The snapshot right after them (if any) is also checked for being a duplicate + // of the last one, as the sorting is not stable: such a duplicate might otherwise take the place of the last + // snapshot in the result at random. Snapshots further down the listing are orphans from an overwritten resource + // change; they are not returned, and no checks are needed. + prevSubIdx := -1 + for i := 0; i < len(snapshots) && i <= count; i++ { + subIdxStr := snapshots[i].GetLabels()[placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey] + subIdx, err := strconv.Atoi(subIdxStr) + if err != nil { + return nil, errors.NewUnexpectedError(err, "failed to convert sub-index label to integer", + "placementResourceSnapshot", klog.KObj(snapshots[i]), "observedSubIndex", subIdxStr) + } + + switch { + case i > 0 && subIdx == prevSubIdx: + // A duplicate sub-index signals a stray snapshot (e.g., a leftover from a previous placement policy of + // the same name) at the same index. After sorting, duplicates (if any) are adjacent to each other. + return nil, errors.NewUserError(nil, "found placement resource snapshots with duplicate sub-indices; some snapshots might have been orphaned from previous placements, and a rollout might be needed to fix the situation", + "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot), + "placementResourceSnapshot", klog.KObj(snapshots[i]), + "duplicatePlacementResourceSnapshot", klog.KObj(snapshots[i-1]), + "duplicateSubIndex", subIdx) + case i < count && subIdx != i: + // A gap signals a missing snapshot. + return nil, errors.NewUserError(nil, "found placement resource snapshots with inconsistent sub-indices; some snapshots might have been orphaned from previous placements, and a rollout might be needed to fix the situation", + "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot), + "placementResourceSnapshot", klog.KObj(snapshots[i]), + "expectedSubIndex", i, "observedSubIndex", subIdx) + } + prevSubIdx = subIdx + } + // Next, verify that all snapshots dictated by the count label are present. + // + // The placement resource snapshot manager creates secondary snapshots first, then the primary snapshot with the + // count label, and all snapshots are observed via the same informer; the secondary snapshots are always visible + // once the primary one is. A shortfall here (like a gap above) signals that some snapshots have been deleted. if len(snapshots) < count { - // Normally this branch will never run, as the placement resource snapshot manager creates secondary - // snapshots first, then the primary snapshot with the count label. - return nil, errors.NewUnexpectedError(nil, "there are fewer placement resource snapshots than the count label indicates", + return nil, errors.NewUserError(nil, "there are fewer placement resource snapshots than the count label indicates; some snapshots might have been deleted, and a rollout might be needed to fix the situation", "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot), "expectedCount", count, "actualCount", len(snapshots)) } + // Finally, verify that all snapshots dictated by the count label have the same snapshotted resource hash. primarySnapshottedResHash := primarySnapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] for i := range snapshots[:count] { snapshottedResHash := snapshots[i].GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] diff --git a/pkg/v1/controllers/workgenerator/status.go b/pkg/v1/controllers/workgenerator/status.go index dc5b89147..b6a6ed8bd 100644 --- a/pkg/v1/controllers/workgenerator/status.go +++ b/pkg/v1/controllers/workgenerator/status.go @@ -17,7 +17,10 @@ limitations under the License. package workgenerator import ( + "cmp" "context" + "fmt" + "slices" "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/meta" @@ -30,6 +33,10 @@ import ( "go.goms.io/fleet/pkg/utils/errors" ) +const ( + maxFailedResourcesToReport = 50 +) + func (r *Reconciler) refreshPlacementBindingStatus( ctx context.Context, placementBinding placementv1alpha1.PlacementBindingAccessor, @@ -50,10 +57,11 @@ func (r *Reconciler) refreshPlacementBindingStatus( placementBinding.GetStatus().SelectedResources = ptr.To(total) placementBinding.GetStatus().SynchronizedResources = ptr.To(synced) placementBinding.GetStatus().AvailableResources = ptr.To(available) - if len(failed) > 50 { - klog.V(2).InfoS("Too many failed resources to report in placement binding status; truncating the list to 50", - "placementBinding", klog.KObj(placementBinding), "totalFailedResources", len(failed)) - failed = failed[:50] + if len(failed) > maxFailedResourcesToReport { + klog.V(2).InfoS("Too many failed resources to report in placement binding status; truncating the list", + "placementBinding", klog.KObj(placementBinding), "totalFailedResources", len(failed), + "maxFailedResourcesToReport", maxFailedResourcesToReport) + failed = failed[:maxFailedResourcesToReport] } placementBinding.GetStatus().FailedResources = failed @@ -76,6 +84,80 @@ func (r *Reconciler) refreshPlacementBindingStatus( return nil } +// markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots marks the placement binding as failed to +// synchronize due to invalid placement resource snapshots. The resource counters are tallied only from the work +// objects linked to the last processed placement resource snapshot, i.e., the resources that have been placed. +func (r *Reconciler) markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots( + ctx context.Context, + placementBinding placementv1alpha1.PlacementBindingAccessor, + works []placementv1alpha1.Work, + errDetails string, +) error { + placementBindingStatus := placementBinding.GetStatus() + oldStatus := placementBindingStatus.DeepCopy() + + // Set a false Synchronized condition with the InvalidSnapshots reason. + meta.SetStatusCondition(&placementBindingStatus.Conditions, metav1.Condition{ + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + ObservedGeneration: placementBinding.GetGeneration(), + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonInvalidSnapshots, + Message: fmt.Sprintf("Failed to synchronize resources as the given placement resource snapshots are invalid: %s", errDetails), + }) + // Set a false Available condition with the IncompleteSetOfResources reason, if the last processed placement + // resource snapshot is the same as the primary snapshot in the spec; otherwise the existing condition + // is left stale. + lastProcessedSnapshotName := placementBindingStatus.LastProcessedResourceSnapshotName + if lastProcessedSnapshotName == nil || *lastProcessedSnapshotName == placementBinding.GetSpec().ResourceSnapshotName { + meta.SetStatusCondition(&placementBindingStatus.Conditions, metav1.Condition{ + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: placementBinding.GetGeneration(), + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonIncompleteSetOfResources, + Message: "Cannot check the availability of resources as the given placement resource snapshots are invalid; some resources might be missing from the target cluster", + }) + } + + // Tally the counters using only the work objects linked to the last processed placement resource snapshot. + var lastProcessedWorks []placementv1alpha1.Work + if lastProcessedSnapshotName != nil { + lastProcessedWorks = make([]placementv1alpha1.Work, 0, len(works)) + for idx := range works { + linked := works[idx].GetAnnotations()[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey] + if linked == *lastProcessedSnapshotName { + lastProcessedWorks = append(lastProcessedWorks, works[idx]) + } + } + } + total, synced, available, failed := countResourcesInWorksByProcessingResults(lastProcessedWorks) + placementBindingStatus.SelectedResources = ptr.To(total) + placementBindingStatus.SynchronizedResources = ptr.To(synced) + placementBindingStatus.AvailableResources = ptr.To(available) + if len(failed) > maxFailedResourcesToReport { + klog.V(2).InfoS("Too many failed resources to report in placement binding status; truncating the list", + "placementBinding", klog.KObj(placementBinding), "totalFailedResources", len(failed), + "maxFailedResourcesToReport", maxFailedResourcesToReport) + failed = failed[:maxFailedResourcesToReport] + } + placementBindingStatus.FailedResources = failed + + // Skip the update if the status has not changed. + if equality.Semantic.DeepEqual(oldStatus, placementBindingStatus) { + klog.V(2).InfoS("No need to update placement binding status as it has not changed", + "placementBinding", klog.KObj(placementBinding), "errDetails", errDetails) + return nil + } + + if err := r.hubClient.Status().Update(ctx, placementBinding); err != nil { + return errors.NewAPIServerError(err, "failed to update placement binding status", true) + } + klog.V(2).InfoS("Marked placement binding as failed to synchronize due to invalid placement resource snapshots", + "placementBinding", klog.KObj(placementBinding), "errDetails", errDetails, + "selectedResources", total, "synchronizedResources", synced, "availableResources", available, + "failedResources", len(failed)) + return nil +} + func (r *Reconciler) reportPlacementBindingProcessingProgress( ctx context.Context, placementBinding placementv1alpha1.PlacementBindingAccessor, @@ -246,6 +328,17 @@ func countResourcesInWorksByProcessingResults(works []placementv1alpha1.Work) ( } } } + + // Sort the failed resources to ensure deterministic status reporting. + slices.SortFunc(failed, func(a, b placementv1alpha1.FailedResource) int { + return cmp.Or( + cmp.Compare(a.ObjectRef.APIGroup, b.ObjectRef.APIGroup), + cmp.Compare(a.ObjectRef.APIVersion, b.ObjectRef.APIVersion), + cmp.Compare(a.ObjectRef.Kind, b.ObjectRef.Kind), + cmp.Compare(a.ObjectRef.Namespace, b.ObjectRef.Namespace), + cmp.Compare(a.ObjectRef.Name, b.ObjectRef.Name), + ) + }) return total, synced, available, failed } diff --git a/pkg/v1/controllers/workgenerator/status_integration_test.go b/pkg/v1/controllers/workgenerator/status_integration_test.go new file mode 100644 index 000000000..acf9400ce --- /dev/null +++ b/pkg/v1/controllers/workgenerator/status_integration_test.go @@ -0,0 +1,270 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/utils/ptr" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +var _ = Describe("refreshing placement binding status", func() { + for _, sc := range scopes { + Context(fmt.Sprintf("work objects applied, but not yet checked for availability (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 2) + f.createBinding(0) + }) + + f.registerWorkGenerationSpecs(0) + + It("can record the placement binding status", func() { + f.recordStatus() + }) + + It("can mark the work objects as applied", func() { + f.markWorksAsApplied(0) + }) + + It("should not refresh the placement binding status until the availability checks complete", func() { + Consistently(f.statusUnchangedActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding status has been refreshed unexpectedly") + }) + + It("can mark the work objects as applied and available", func() { + f.markWorks(0) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(0), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("work status changes that do not change the processing results (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the placement binding status", func() { + f.recordStatus() + }) + + It("can update the status of the work object without changing its processing results", func() { + touchWorkStatus(f.workNS, f.workName(0)) + }) + + It("should not write to the placement binding", func() { + Consistently(f.resourceVersionUnchangedActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding has been written to unexpectedly") + }) + + It("can mark the deployment (and consequently the work object) as unavailable", func() { + markWorkManifestAsUnavailable(f.workNS, f.workName(0), 1) + }) + + It("should refresh the placement binding status", func() { + set := f.sets[0] + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(set.resourceCount()), + SynchronizedResources: ptr.To(set.resourceCount()), + AvailableResources: ptr.To(set.resourceCount() - 1), + FailedResources: []placementv1alpha1.FailedResource{ + { + ObjectRef: objectRefFromManifestIdentifier(set.identifiers[0][1]), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + }, + }, + }, + }, + LastProcessedResourceSnapshotName: ptr.To(set.primaryName()), + } + Eventually(placementBindingStatusUpdatedActual(sc.bindingNS, f.bindingName, wantStatus), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + + Context(fmt.Sprintf("work object metadata tampered with (%s)", sc.name), Ordered, func() { + f := newBindingFixture(sc, memberCluster1Name) + tamperedSnapshotName := fmt.Sprintf("tampered-snapshot-%s", utils.RandStr()) + var primaryWorkGeneration int64 + + BeforeAll(func() { + f.createSnapshots(0, 1) + f.createBinding(0) + }) + + f.registerBaselineSpecs(0) + + It("can record the UIDs and generation of the work objects, and the placement binding status", func() { + f.recordWorkUIDs() + primaryWorkGeneration = f.workGeneration(0) + f.recordStatus() + }) + + It("can link the work object to another snapshot", func() { + setWorkAnnotation(f.workNS, f.workName(0), placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey, tamperedSnapshotName) + }) + + // Metadata-only changes to work objects are not events that the work generator reacts to. + It("should not reconcile the placement binding", func() { + tamperedActual := workAnnotationActual(f.workNS, f.workName(0), placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey, tamperedSnapshotName) + Eventually(tamperedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the tampered work object") + Consistently(func() error { + if err := tamperedActual(); err != nil { + return err + } + return f.statusUnchangedActual()() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement binding has been reconciled unexpectedly") + }) + + It("can update the status of the work object", func() { + touchWorkStatus(f.workNS, f.workName(0)) + }) + + It("should re-link the work object in place, without changing its spec", func() { + Eventually(f.worksMatchActual(0, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-link the work object") + f.expectWorksNotRecreated() + Expect(f.workGeneration(0)).To(Equal(primaryWorkGeneration), "The spec of the work object has been changed") + }) + + // The work generator resets the placement binding status upon re-linking, then refreshes it after a + // requeue, as the work object does not need to be re-processed. + It("should refresh the placement binding status", func() { + Eventually(func() error { + if err := f.syncedStatusActual(0)(); err != nil { + return err + } + return f.syncCondTransitionedSinceRecordActual()() + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) + + AfterAll(func() { + f.teardown() + }) + }) + } + + Context("more failed resources than the reporting limit", Ordered, func() { + // The handling of failed resources does not depend on the scope of the placement binding; the namespace-scoped + // one is used. + sc := scopes[0] + f := newBindingFixture(sc, memberCluster1Name) + configMapCount := maxFailedResourcesToReport + 1 + nsName := fmt.Sprintf(nsNameTemplate, utils.RandStr()) + identifiers := make([]placementv1alpha1.ManifestIdentifier, configMapCount) + + BeforeAll(func() { + // The ConfigMaps are placed in descending order of their names, so that the failed resources reported + // (sorted, then truncated) differ from the first ones in the work object. + resources := make([]placementv1alpha1.SnapshottedResource, configMapCount) + for idx := 0; idx < configMapCount; idx++ { + configMapName := fmt.Sprintf(paddedConfigMapNameTemplate, configMapCount-1-idx) + resources[idx] = configMapSnapshottedResource(nsName, configMapName) + identifiers[idx] = configMapManifestIdentifier(idx, nsName, configMapName) + } + f.createSnapshotWithResources(0, resources, identifiers) + f.createBinding(0) + }) + + f.registerWorkGenerationSpecs(0) + + It("can mark all the manifests as failed to be applied", func() { + failedToApplyOrdinals := make(map[int]*placementv1alpha1.DiffDetails, configMapCount) + for idx := range identifiers { + failedToApplyOrdinals[identifiers[idx].Ordinal] = nil + } + markWorkManifestsAsFailed(f.workNS, f.workName(0), identifiers, failedToApplyOrdinals, nil) + }) + + It("should report the failed resources in order, up to the limit", func() { + wantFailedResources := make([]placementv1alpha1.FailedResource, 0, maxFailedResourcesToReport) + for num := 0; num < maxFailedResourcesToReport; num++ { + configMapName := fmt.Sprintf(paddedConfigMapNameTemplate, num) + wantFailedResources = append(wantFailedResources, placementv1alpha1.FailedResource{ + ObjectRef: objectRefFromManifestIdentifier(configMapManifestIdentifier(0, nsName, configMapName)), + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + }, + }, + }) + } + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + }, + }, + SelectedResources: ptr.To(int32(configMapCount)), + SynchronizedResources: ptr.To(int32(0)), + AvailableResources: ptr.To(int32(0)), + FailedResources: wantFailedResources, + LastProcessedResourceSnapshotName: ptr.To(f.sets[0].primaryName()), + } + Eventually(placementBindingStatusInOrderActual(sc.bindingNS, f.bindingName, wantStatus), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the failed resources") + }) + + AfterAll(func() { + f.teardown() + }) + }) +}) diff --git a/pkg/v1/controllers/workgenerator/status_test.go b/pkg/v1/controllers/workgenerator/status_test.go new file mode 100644 index 000000000..0d53183d5 --- /dev/null +++ b/pkg/v1/controllers/workgenerator/status_test.go @@ -0,0 +1,894 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "context" + "fmt" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" +) + +const ( + unitTestBindingName = "binding" + unitTestPolicyName = "policy" + unitTestResourceNS = "resource-ns" + unitTestSnapshotName0 = "policy-snapshot-0" + unitTestSnapshotName1 = "policy-snapshot-1" +) + +var ( + // A fixed last transition time for the conditions that the tests build, so that conditions copied as they are by + // the work generator (e.g., into failed resources) compare as equal. + unitTestLTT = metav1.NewTime(time.Date(2026, time.January, 1, 0, 0, 0, 0, time.UTC)) + + // The work generator sets the last transition times of placement binding conditions to the current time. + ignoreConditionLTT = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime") + + // Common work-level and manifest-level conditions. + manifestAppliedCond = newCondition(placementv1alpha1.ManifestCondTypeApplied, metav1.ConditionTrue, 1, "Applied", "applied") + manifestNotAppliedCond = newCondition(placementv1alpha1.ManifestCondTypeApplied, metav1.ConditionFalse, 1, "FailedToApply", "failed to apply") + manifestAvailableCond = newCondition(placementv1alpha1.ManifestCondTypeAvailable, metav1.ConditionTrue, 1, "Available", "available") + manifestNotAvailableCond = newCondition(placementv1alpha1.ManifestCondTypeAvailable, metav1.ConditionFalse, 1, "NotAvailable", "not available") +) + +// newCondition returns a condition with the fixed last transition time. +func newCondition(condType string, status metav1.ConditionStatus, observedGeneration int64, reason, message string) metav1.Condition { + return metav1.Condition{ + Type: condType, + Status: status, + ObservedGeneration: observedGeneration, + Reason: reason, + Message: message, + LastTransitionTime: unitTestLTT, + } +} + +// newManifestStatus returns the status of a manifest (a ConfigMap of the given name) with the given conditions. +func newManifestStatus(ordinal int, configMapName string, conds ...metav1.Condition) placementv1alpha1.PerManifestStatus { + return placementv1alpha1.PerManifestStatus{ + Identifier: configMapManifestIdentifier(ordinal, unitTestResourceNS, configMapName), + Conditions: conds, + } +} + +// newWorkWithStatus returns a work object of the given generation, linked to the given primary snapshot, with one +// manifest per manifest status and the given work-level conditions. +func newWorkWithStatus( + name, linkedSnapshotName string, + generation int64, + workConds []metav1.Condition, + manifestStatuses ...placementv1alpha1.PerManifestStatus, +) placementv1alpha1.Work { + return placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: fmt.Sprintf("fleet-member-%s", memberCluster1Name), + Name: name, + Generation: generation, + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: linkedSnapshotName, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: make([]placementv1alpha1.Manifest, len(manifestStatuses)), + }, + Status: placementv1alpha1.WorkStatus{ + Conditions: workConds, + Manifests: manifestStatuses, + }, + } +} + +// workConds returns the work-level Applied and Available conditions (each omitted if its status is empty) observing +// the given generation. +func workConds(generation int64, applied, available metav1.ConditionStatus) []metav1.Condition { + var conds []metav1.Condition + if applied != "" { + conds = append(conds, newCondition(placementv1alpha1.WorkCondTypeApplied, applied, generation, "WorkApplied", "work applied")) + } + if available != "" { + conds = append(conds, newCondition(placementv1alpha1.WorkCondTypeAvailable, available, generation, "WorkAvailable", "work available")) + } + return conds +} + +// newFailedResource returns the failed resource that the work generator is expected to report for a manifest status +// and its false condition. +func newFailedResource(manifestStatus placementv1alpha1.PerManifestStatus, falseCond metav1.Condition) placementv1alpha1.FailedResource { + return placementv1alpha1.FailedResource{ + ObjectRef: objectRefFromManifestIdentifier(manifestStatus.Identifier), + Conditions: []metav1.Condition{falseCond}, + DiffDetails: manifestStatus.DiffDetails, + } +} + +// newBindingWithStatus returns a namespace-scoped placement binding of the given generation, bound to the given +// primary snapshot, with the given status. +func newBindingWithStatus(generation int64, snapshotName string, status placementv1alpha1.PlacementBindingStatus) *placementv1alpha1.PlacementBinding { + return &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: appNamespaceName, + Name: unitTestBindingName, + Generation: generation, + }, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: unitTestPolicyName, + ClusterName: memberCluster1Name, + ResourceSnapshotName: snapshotName, + }, + Status: status, + } +} + +// newUnitTestHubClient returns a fake hub client holding the given objects, with the given interceptor functions. +func newUnitTestHubClient(t *testing.T, funcs interceptor.Funcs, objs ...client.Object) client.Client { + t.Helper() + scheme := runtime.NewScheme() + if err := placementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + return fake.NewClientBuilder(). + WithScheme(scheme). + WithObjects(objs...). + WithStatusSubresource(&placementv1alpha1.PlacementBinding{}, &placementv1alpha1.ClusterPlacementBinding{}, &placementv1alpha1.Work{}). + WithInterceptorFuncs(funcs). + Build() +} + +// countStatusUpdates returns interceptor functions that count the status updates, and fail them with updateErr if it +// is set. +func countStatusUpdates(count *int, updateErr error) interceptor.Funcs { + return interceptor.Funcs{ + SubResourceUpdate: func(ctx context.Context, c client.Client, subResourceName string, obj client.Object, opts ...client.SubResourceUpdateOption) error { + *count++ + if updateErr != nil { + return updateErr + } + return c.SubResource(subResourceName).Update(ctx, obj, opts...) + }, + } +} + +// getUnitTestBinding retrieves the placement binding from the fake hub client. +func getUnitTestBinding(t *testing.T, c client.Client) *placementv1alpha1.PlacementBinding { + t.Helper() + binding := &placementv1alpha1.PlacementBinding{} + if err := c.Get(context.Background(), client.ObjectKey{Namespace: appNamespaceName, Name: unitTestBindingName}, binding); err != nil { + t.Fatalf("Get() = %v, want no error", err) + } + return binding +} + +// TestCountResourcesInWorksByProcessingResults tests the countResourcesInWorksByProcessingResults function. +func TestCountResourcesInWorksByProcessingResults(t *testing.T) { + diffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(2)), + FirstDiffedObservedTimestamp: unitTestLTT, + ObservedDiffs: []placementv1alpha1.PatchDetail{ + {Path: "/data/foo", ValueInMember: "baz", ValueInHub: "bar"}, + }, + } + notAppliedWithDiffs := newManifestStatus(1, "configmap-b", manifestNotAppliedCond) + notAppliedWithDiffs.DiffDetails = diffDetails + notAvailable := newManifestStatus(0, "configmap-a", manifestAppliedCond, manifestNotAvailableCond) + // A ConfigMap in another namespace that sorts before configmap-a by namespace. + notAvailableInOtherNS := placementv1alpha1.PerManifestStatus{ + Identifier: configMapManifestIdentifier(0, "another-ns", "configmap-z"), + Conditions: []metav1.Condition{manifestAppliedCond, manifestNotAvailableCond}, + } + // A Deployment that sorts before the ConfigMaps by API group. + notAvailableDeploy := placementv1alpha1.PerManifestStatus{ + Identifier: deployManifestIdentifier(2, unitTestResourceNS, "deploy-a"), + Conditions: []metav1.Condition{manifestAppliedCond, manifestNotAvailableCond}, + } + + testCases := []struct { + name string + works []placementv1alpha1.Work + wantTotal int32 + wantSynced int32 + wantAvailable int32 + wantFailed []placementv1alpha1.FailedResource + }{ + { + name: "no work objects", + }, + { + name: "manifests not yet processed", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, nil, + newManifestStatus(0, "configmap-a"), + newManifestStatus(1, "configmap-b"), + ), + }, + wantTotal: 2, + }, + { + name: "manifests applied, with and without completed availability checks", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, nil, + newManifestStatus(0, "configmap-a", manifestAppliedCond, manifestAvailableCond), + // The availability check has not completed yet; the manifest is not considered failed. + newManifestStatus(1, "configmap-b", manifestAppliedCond), + ), + }, + wantTotal: 2, + wantSynced: 2, + wantAvailable: 1, + }, + { + name: "manifests failed to be applied or to become available", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, nil, + notAppliedWithDiffs, + notAvailable, + // An Available condition left on a manifest that has failed to be applied is ignored. + newManifestStatus(2, "configmap-c", manifestNotAppliedCond, manifestAvailableCond), + ), + }, + wantTotal: 3, + // configmap-a has been applied. + wantSynced: 1, + wantFailed: []placementv1alpha1.FailedResource{ + newFailedResource(notAvailable, manifestNotAvailableCond), + newFailedResource(notAppliedWithDiffs, manifestNotAppliedCond), + newFailedResource(newManifestStatus(2, "configmap-c"), manifestNotAppliedCond), + }, + }, + { + name: "failed resources across work objects, sorted by group, version, kind, namespace, and name", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, nil, + notAvailable, + newManifestStatus(1, "configmap-ok", manifestAppliedCond, manifestAvailableCond), + ), + newWorkWithStatus("work-1", unitTestSnapshotName0, 1, nil, + notAvailableDeploy, + notAvailableInOtherNS, + ), + }, + wantTotal: 4, + wantSynced: 4, + wantAvailable: 1, + wantFailed: []placementv1alpha1.FailedResource{ + newFailedResource(notAvailableInOtherNS, manifestNotAvailableCond), + newFailedResource(notAvailable, manifestNotAvailableCond), + newFailedResource(notAvailableDeploy, manifestNotAvailableCond), + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotTotal, gotSynced, gotAvailable, gotFailed := countResourcesInWorksByProcessingResults(tc.works) + if gotTotal != tc.wantTotal || gotSynced != tc.wantSynced || gotAvailable != tc.wantAvailable { + t.Errorf("countResourcesInWorksByProcessingResults() counters = (%d, %d, %d), want (%d, %d, %d)", + gotTotal, gotSynced, gotAvailable, tc.wantTotal, tc.wantSynced, tc.wantAvailable) + } + if diff := cmp.Diff(gotFailed, tc.wantFailed); diff != "" { + t.Errorf("countResourcesInWorksByProcessingResults() failed resources mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// TestFailedResourceFromManifestStatus tests the failedResourceFromManifestStatus function. +func TestFailedResourceFromManifestStatus(t *testing.T) { + diffDetails := &placementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(1)), + FirstDiffedObservedTimestamp: unitTestLTT, + } + manifestStatus := newManifestStatus(3, "configmap-a", manifestNotAppliedCond) + manifestStatus.DiffDetails = diffDetails + + testCases := []struct { + name string + falseCond *metav1.Condition + want placementv1alpha1.FailedResource + }{ + { + name: "with the false condition", + falseCond: &manifestNotAppliedCond, + want: placementv1alpha1.FailedResource{ + ObjectRef: placementv1alpha1.ObjectReference{ + Namespace: unitTestResourceNS, + Name: "configmap-a", + APIVersion: "v1", + Kind: configMapKind, + }, + Conditions: []metav1.Condition{manifestNotAppliedCond}, + DiffDetails: diffDetails, + }, + }, + { + name: "without the false condition", + want: placementv1alpha1.FailedResource{ + ObjectRef: placementv1alpha1.ObjectReference{ + Namespace: unitTestResourceNS, + Name: "configmap-a", + APIVersion: "v1", + Kind: configMapKind, + }, + DiffDetails: diffDetails, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := failedResourceFromManifestStatus(&manifestStatus, tc.falseCond) + if diff := cmp.Diff(got, tc.want); diff != "" { + t.Errorf("failedResourceFromManifestStatus() mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// TestRefreshPlacementBindingSyncCond tests the refreshPlacementBindingSyncCond function. +func TestRefreshPlacementBindingSyncCond(t *testing.T) { + existingConds := []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionFalse, 2, + placementv1alpha1.PlacementBindingSynchronizedCondReasonWaitingForSynchronization, "waiting"), + } + + testCases := []struct { + name string + works []placementv1alpha1.Work + wantNotReady bool + wantConds []metav1.Condition + }{ + { + name: "a work object has not been applied yet", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, "")), + newWorkWithStatus("work-1", unitTestSnapshotName0, 1, nil), + }, + wantNotReady: true, + wantConds: existingConds, + }, + { + name: "a work object has a stale Applied condition", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 2, workConds(1, metav1.ConditionTrue, "")), + }, + wantNotReady: true, + wantConds: existingConds, + }, + { + name: "all work objects have been applied", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, "")), + newWorkWithStatus("work-1", unitTestSnapshotName0, 3, workConds(3, metav1.ConditionTrue, "")), + }, + wantConds: []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionTrue, 2, + placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + "All resources have been synchronized to the target cluster"), + }, + }, + { + name: "a work object has failed to be applied", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, "")), + newWorkWithStatus("work-1", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionFalse, "")), + }, + wantConds: []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionFalse, 2, + placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + "Some resources might be out of sync in the target cluster"), + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + binding := newBindingWithStatus(2, unitTestSnapshotName0, placementv1alpha1.PlacementBindingStatus{ + Conditions: append([]metav1.Condition{}, existingConds...), + }) + gotNotReady := refreshPlacementBindingSyncCond(binding, tc.works) + if gotNotReady != tc.wantNotReady { + t.Errorf("refreshPlacementBindingSyncCond() = %t, want %t", gotNotReady, tc.wantNotReady) + } + if diff := cmp.Diff(binding.Status.Conditions, tc.wantConds, ignoreConditionLTT); diff != "" { + t.Errorf("refreshPlacementBindingSyncCond() conditions mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// TestRefreshPlacementBindingAvailableCond tests the refreshPlacementBindingAvailableCond function. +func TestRefreshPlacementBindingAvailableCond(t *testing.T) { + existingConds := []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionUnknown, 2, + placementv1alpha1.PlacementBindingAvailableCondReasonWaitingForAvailabilityCheck, "waiting"), + } + + testCases := []struct { + name string + works []placementv1alpha1.Work + wantNotReady bool + wantConds []metav1.Condition + }{ + { + name: "a work object has not been checked for availability yet", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, metav1.ConditionTrue)), + newWorkWithStatus("work-1", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, "")), + }, + wantNotReady: true, + wantConds: existingConds, + }, + { + name: "a work object has a stale Available condition", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 2, workConds(1, metav1.ConditionTrue, metav1.ConditionTrue)), + }, + wantNotReady: true, + wantConds: existingConds, + }, + { + name: "all work objects are available", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, metav1.ConditionTrue)), + newWorkWithStatus("work-1", unitTestSnapshotName0, 3, workConds(3, metav1.ConditionTrue, metav1.ConditionTrue)), + }, + wantConds: []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionTrue, 2, + placementv1alpha1.PlacementBindingAvailableCondReasonAllResourcesAvailable, + "All resources are available in the target cluster"), + }, + }, + { + name: "a work object is not available", + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, metav1.ConditionTrue)), + newWorkWithStatus("work-1", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, metav1.ConditionFalse)), + }, + wantConds: []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionFalse, 2, + placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + "Some resources might be unavailable in the target cluster"), + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + binding := newBindingWithStatus(2, unitTestSnapshotName0, placementv1alpha1.PlacementBindingStatus{ + Conditions: append([]metav1.Condition{}, existingConds...), + }) + gotNotReady := refreshPlacementBindingAvailableCond(binding, tc.works) + if gotNotReady != tc.wantNotReady { + t.Errorf("refreshPlacementBindingAvailableCond() = %t, want %t", gotNotReady, tc.wantNotReady) + } + if diff := cmp.Diff(binding.Status.Conditions, tc.wantConds, ignoreConditionLTT); diff != "" { + t.Errorf("refreshPlacementBindingAvailableCond() conditions mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// failedToApplyWorkWithManyManifests returns a work object, linked to the given snapshot, with more manifests than the +// failed resource reporting limit, all of which have failed to be applied; the manifests are listed in descending +// order of their names. It also returns the failed resources that the work generator is expected to report (the +// first ones by name, up to the limit). +func failedToApplyWorkWithManyManifests(linkedSnapshotName string) (placementv1alpha1.Work, []placementv1alpha1.FailedResource) { + manifestCount := maxFailedResourcesToReport + 1 + manifestStatuses := make([]placementv1alpha1.PerManifestStatus, manifestCount) + for idx := 0; idx < manifestCount; idx++ { + manifestStatuses[idx] = newManifestStatus(idx, fmt.Sprintf(paddedConfigMapNameTemplate, manifestCount-1-idx), manifestNotAppliedCond) + } + wantFailed := make([]placementv1alpha1.FailedResource, maxFailedResourcesToReport) + for num := 0; num < maxFailedResourcesToReport; num++ { + wantFailed[num] = newFailedResource(newManifestStatus(0, fmt.Sprintf(paddedConfigMapNameTemplate, num)), manifestNotAppliedCond) + } + work := newWorkWithStatus("work-0", linkedSnapshotName, 1, workConds(1, metav1.ConditionFalse, metav1.ConditionFalse), manifestStatuses...) + return work, wantFailed +} + +// TestRefreshPlacementBindingStatus tests the refreshPlacementBindingStatus method. +func TestRefreshPlacementBindingStatus(t *testing.T) { + const bindingGeneration = 2 + + waitingStatus := placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionFalse, bindingGeneration, + placementv1alpha1.PlacementBindingSynchronizedCondReasonWaitingForSynchronization, "waiting"), + newCondition(placementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionUnknown, bindingGeneration, + placementv1alpha1.PlacementBindingAvailableCondReasonWaitingForAvailabilityCheck, "waiting"), + }, + SelectedResources: ptr.To(int32(3)), + LastProcessedResourceSnapshotName: ptr.To(unitTestSnapshotName0), + } + syncedStatus := placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionTrue, bindingGeneration, + placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + "All resources have been synchronized to the target cluster"), + newCondition(placementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionTrue, bindingGeneration, + placementv1alpha1.PlacementBindingAvailableCondReasonAllResourcesAvailable, + "All resources are available in the target cluster"), + }, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(3)), + AvailableResources: ptr.To(int32(3)), + LastProcessedResourceSnapshotName: ptr.To(unitTestSnapshotName0), + } + failedConds := []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionFalse, bindingGeneration, + placementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources, + "Some resources might be out of sync in the target cluster"), + newCondition(placementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionFalse, bindingGeneration, + placementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable, + "Some resources might be unavailable in the target cluster"), + } + + syncedWorks := []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, metav1.ConditionTrue), + newManifestStatus(0, "configmap-a", manifestAppliedCond, manifestAvailableCond), + newManifestStatus(1, "configmap-b", manifestAppliedCond, manifestAvailableCond), + ), + newWorkWithStatus("work-1", unitTestSnapshotName0, 3, workConds(3, metav1.ConditionTrue, metav1.ConditionTrue), + newManifestStatus(0, "configmap-c", manifestAppliedCond, manifestAvailableCond), + ), + } + notApplied := newManifestStatus(0, "configmap-b", manifestNotAppliedCond) + notAvailable := newManifestStatus(1, "configmap-a", manifestAppliedCond, manifestNotAvailableCond) + someFailedWorks := []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionFalse, metav1.ConditionFalse), + notApplied, notAvailable), + newWorkWithStatus("work-1", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, metav1.ConditionTrue), + newManifestStatus(0, "configmap-c", manifestAppliedCond, manifestAvailableCond), + ), + } + manyFailedWork, wantManyFailed := failedToApplyWorkWithManyManifests(unitTestSnapshotName0) + + testCases := []struct { + name string + existingStatus placementv1alpha1.PlacementBindingStatus + works []placementv1alpha1.Work + updateErr error + wantStatus placementv1alpha1.PlacementBindingStatus + wantWrites int + wantErr bool + }{ + { + name: "a work object has not been applied yet", + existingStatus: waitingStatus, + works: []placementv1alpha1.Work{ + syncedWorks[0], + newWorkWithStatus("work-1", unitTestSnapshotName0, 1, nil, newManifestStatus(0, "configmap-c")), + }, + wantStatus: waitingStatus, + }, + { + name: "a work object has a stale Applied condition", + existingStatus: waitingStatus, + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 2, workConds(1, metav1.ConditionTrue, metav1.ConditionTrue), + newManifestStatus(0, "configmap-a", manifestAppliedCond, manifestAvailableCond), + ), + }, + wantStatus: waitingStatus, + }, + { + name: "a work object has been applied, but not yet checked for availability", + existingStatus: waitingStatus, + works: []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, ""), + newManifestStatus(0, "configmap-a", manifestAppliedCond), + ), + }, + wantStatus: waitingStatus, + }, + { + name: "all resources synchronized and available", + existingStatus: waitingStatus, + works: syncedWorks, + wantStatus: syncedStatus, + wantWrites: 1, + }, + { + name: "some resources failed to be applied or to become available", + existingStatus: waitingStatus, + works: someFailedWorks, + wantStatus: placementv1alpha1.PlacementBindingStatus{ + Conditions: failedConds, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []placementv1alpha1.FailedResource{ + newFailedResource(notAvailable, manifestNotAvailableCond), + newFailedResource(notApplied, manifestNotAppliedCond), + }, + LastProcessedResourceSnapshotName: ptr.To(unitTestSnapshotName0), + }, + wantWrites: 1, + }, + { + name: "status unchanged", + existingStatus: syncedStatus, + works: syncedWorks, + wantStatus: syncedStatus, + }, + { + name: "more failed resources than the reporting limit", + existingStatus: waitingStatus, + works: []placementv1alpha1.Work{manyFailedWork}, + wantStatus: placementv1alpha1.PlacementBindingStatus{ + Conditions: failedConds, + SelectedResources: ptr.To(int32(maxFailedResourcesToReport + 1)), + SynchronizedResources: ptr.To(int32(0)), + AvailableResources: ptr.To(int32(0)), + FailedResources: wantManyFailed, + LastProcessedResourceSnapshotName: ptr.To(unitTestSnapshotName0), + }, + wantWrites: 1, + }, + { + name: "status update fails", + existingStatus: waitingStatus, + works: syncedWorks, + updateErr: apierrors.NewServiceUnavailable("the API server is unavailable"), + wantStatus: waitingStatus, + wantWrites: 1, + wantErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + writes := 0 + c := newUnitTestHubClient(t, countStatusUpdates(&writes, tc.updateErr), + newBindingWithStatus(bindingGeneration, unitTestSnapshotName0, *tc.existingStatus.DeepCopy())) + r := New(c, 1, nil) + + err := r.refreshPlacementBindingStatus(ctx, getUnitTestBinding(t, c), tc.works) + if (err != nil) != tc.wantErr { + t.Fatalf("refreshPlacementBindingStatus() = %v, want error: %t", err, tc.wantErr) + } + if err != nil && kferrors.Category(err) != kferrors.ErrCategoryAPIServer { + t.Errorf("refreshPlacementBindingStatus() error category = %q, want %q", kferrors.Category(err), kferrors.ErrCategoryAPIServer) + } + if writes != tc.wantWrites { + t.Errorf("refreshPlacementBindingStatus() made %d status updates, want %d", writes, tc.wantWrites) + } + if diff := cmp.Diff(getUnitTestBinding(t, c).Status, tc.wantStatus, ignoreConditionLTT); diff != "" { + t.Errorf("refreshPlacementBindingStatus() status mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// TestMarkPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots tests the +// markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots method. +func TestMarkPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots(t *testing.T) { + const ( + errDetails = "placement resource snapshot \"policy-snapshot-1\" not found" + otherErrDetails = "found placement resource snapshots with duplicate sub-indices" + ) + invalidSnapshotsCond := func(generation int64, details string) metav1.Condition { + return newCondition(placementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionFalse, generation, + placementv1alpha1.PlacementBindingSynchronizedCondReasonInvalidSnapshots, + fmt.Sprintf("Failed to synchronize resources as the given placement resource snapshots are invalid: %s", details)) + } + incompleteSetOfResourcesCond := func(generation int64) metav1.Condition { + return newCondition(placementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionFalse, generation, + placementv1alpha1.PlacementBindingAvailableCondReasonIncompleteSetOfResources, + "Cannot check the availability of resources as the given placement resource snapshots are invalid; some resources might be missing from the target cluster") + } + // The status of a placement binding whose resources (from snapshot 0) have been synchronized and become available + // at generation 2. + syncedStatus := placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + newCondition(placementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionTrue, 2, + placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, "synchronized"), + newCondition(placementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionTrue, 2, + placementv1alpha1.PlacementBindingAvailableCondReasonAllResourcesAvailable, "available"), + }, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(3)), + AvailableResources: ptr.To(int32(3)), + LastProcessedResourceSnapshotName: ptr.To(unitTestSnapshotName0), + } + + notAvailable := newManifestStatus(1, "configmap-b", manifestAppliedCond, manifestNotAvailableCond) + // The work objects linked to snapshot 0 (the last processed one), plus one linked to snapshot 1, which should not + // be counted. + works := []placementv1alpha1.Work{ + newWorkWithStatus("work-0", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, metav1.ConditionFalse), + newManifestStatus(0, "configmap-a", manifestAppliedCond, manifestAvailableCond), + notAvailable, + ), + newWorkWithStatus("work-1", unitTestSnapshotName0, 1, workConds(1, metav1.ConditionTrue, metav1.ConditionTrue), + newManifestStatus(0, "configmap-c", manifestAppliedCond, manifestAvailableCond), + ), + newWorkWithStatus("work-2", unitTestSnapshotName1, 1, nil, + newManifestStatus(0, "configmap-d", manifestNotAppliedCond), + ), + } + // The status after rolling out to snapshot 1 (at generation 3) with the given error details. + rolledOutStatus := func(details string) placementv1alpha1.PlacementBindingStatus { + return placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{invalidSnapshotsCond(3, details), syncedStatus.Conditions[1]}, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(3)), + AvailableResources: ptr.To(int32(2)), + FailedResources: []placementv1alpha1.FailedResource{newFailedResource(notAvailable, manifestNotAvailableCond)}, + LastProcessedResourceSnapshotName: ptr.To(unitTestSnapshotName0), + } + } + statusWithoutAvailability := *syncedStatus.DeepCopy() + statusWithoutAvailability.Conditions = statusWithoutAvailability.Conditions[:1] + rolledOutStatusWithoutAvailability := rolledOutStatus(errDetails) + rolledOutStatusWithoutAvailability.Conditions = rolledOutStatusWithoutAvailability.Conditions[:1] + manyFailedWork, wantManyFailed := failedToApplyWorkWithManyManifests(unitTestSnapshotName0) + + testCases := []struct { + name string + bindingGeneration int64 + snapshotName string + existingStatus placementv1alpha1.PlacementBindingStatus + works []placementv1alpha1.Work + errDetails string + updateErr error + wantStatus placementv1alpha1.PlacementBindingStatus + wantWrites int + wantErr bool + }{ + { + name: "rolled out to an invalid snapshot", + bindingGeneration: 3, + snapshotName: unitTestSnapshotName1, + existingStatus: syncedStatus, + works: works, + errDetails: errDetails, + wantStatus: rolledOutStatus(errDetails), + wantWrites: 1, + }, + { + name: "rolled out with no Available condition", + bindingGeneration: 3, + snapshotName: unitTestSnapshotName1, + existingStatus: statusWithoutAvailability, + works: works, + errDetails: errDetails, + wantStatus: rolledOutStatusWithoutAvailability, + wantWrites: 1, + }, + { + name: "no snapshot processed yet", + bindingGeneration: 1, + snapshotName: unitTestSnapshotName0, + // The work objects (e.g., left over from a previous binding of the same name) are not counted. + works: works, + errDetails: errDetails, + wantStatus: placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{invalidSnapshotsCond(1, errDetails), incompleteSetOfResourcesCond(1)}, + SelectedResources: ptr.To(int32(0)), + SynchronizedResources: ptr.To(int32(0)), + AvailableResources: ptr.To(int32(0)), + }, + wantWrites: 1, + }, + { + name: "the processed snapshot has become invalid", + bindingGeneration: 2, + snapshotName: unitTestSnapshotName0, + existingStatus: syncedStatus, + works: works, + errDetails: errDetails, + wantStatus: placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{invalidSnapshotsCond(2, errDetails), incompleteSetOfResourcesCond(2)}, + SelectedResources: ptr.To(int32(3)), + SynchronizedResources: ptr.To(int32(3)), + AvailableResources: ptr.To(int32(2)), + FailedResources: []placementv1alpha1.FailedResource{newFailedResource(notAvailable, manifestNotAvailableCond)}, + LastProcessedResourceSnapshotName: ptr.To(unitTestSnapshotName0), + }, + wantWrites: 1, + }, + { + name: "more failed resources than the reporting limit", + bindingGeneration: 3, + snapshotName: unitTestSnapshotName1, + existingStatus: syncedStatus, + works: []placementv1alpha1.Work{manyFailedWork}, + errDetails: errDetails, + wantStatus: placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{invalidSnapshotsCond(3, errDetails), syncedStatus.Conditions[1]}, + SelectedResources: ptr.To(int32(maxFailedResourcesToReport + 1)), + SynchronizedResources: ptr.To(int32(0)), + AvailableResources: ptr.To(int32(0)), + FailedResources: wantManyFailed, + LastProcessedResourceSnapshotName: ptr.To(unitTestSnapshotName0), + }, + wantWrites: 1, + }, + { + name: "already marked with the same error details", + bindingGeneration: 3, + snapshotName: unitTestSnapshotName1, + existingStatus: rolledOutStatus(errDetails), + works: works, + errDetails: errDetails, + wantStatus: rolledOutStatus(errDetails), + }, + { + name: "already marked with different error details", + bindingGeneration: 3, + snapshotName: unitTestSnapshotName1, + existingStatus: rolledOutStatus(otherErrDetails), + works: works, + errDetails: errDetails, + wantStatus: rolledOutStatus(errDetails), + wantWrites: 1, + }, + { + name: "status update fails", + bindingGeneration: 3, + snapshotName: unitTestSnapshotName1, + existingStatus: syncedStatus, + works: works, + errDetails: errDetails, + updateErr: apierrors.NewConflict(schema.GroupResource{Group: placementv1alpha1.GroupVersion.Group, Resource: "placementbindings"}, unitTestBindingName, fmt.Errorf("conflict")), + wantStatus: syncedStatus, + wantWrites: 1, + wantErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + writes := 0 + c := newUnitTestHubClient(t, countStatusUpdates(&writes, tc.updateErr), + newBindingWithStatus(tc.bindingGeneration, tc.snapshotName, *tc.existingStatus.DeepCopy())) + r := New(c, 1, nil) + + err := r.markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots(ctx, getUnitTestBinding(t, c), tc.works, tc.errDetails) + if (err != nil) != tc.wantErr { + t.Fatalf("markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots() = %v, want error: %t", err, tc.wantErr) + } + if err != nil && kferrors.Category(err) != kferrors.ErrCategoryAPIServer { + t.Errorf("markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots() error category = %q, want %q", + kferrors.Category(err), kferrors.ErrCategoryAPIServer) + } + if writes != tc.wantWrites { + t.Errorf("markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots() made %d status updates, want %d", writes, tc.wantWrites) + } + if diff := cmp.Diff(getUnitTestBinding(t, c).Status, tc.wantStatus, ignoreConditionLTT); diff != "" { + t.Errorf("markPlacementBindingAsFailedToSynchronizeDueToInvalidSnapshots() status mismatch (-got +want):\n%s", diff) + } + }) + } +} diff --git a/pkg/v1/controllers/workgenerator/suite_test.go b/pkg/v1/controllers/workgenerator/suite_test.go new file mode 100644 index 000000000..c7ef9286d --- /dev/null +++ b/pkg/v1/controllers/workgenerator/suite_test.go @@ -0,0 +1,177 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "context" + "flag" + "fmt" + "log" + "os" + "path/filepath" + "testing" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/manager" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + memberCluster1Name = "member-1" + memberCluster2Name = "member-2" + + // The namespace in which namespace-scoped placement objects (e.g., PlacementBinding and + // PlacementResourceSnapshot objects) are created. + appNamespaceName = "app" + + workerCount = 4 + maxConcurrentReconciles = 4 + + // The base and maximum delays of the rate limiter for requeues of failed reconciliations. + // + // The work generator does not watch placement resource snapshots (or namespaces); when it fails to process a + // placement binding due to invalid snapshots (or a missing reserved namespace), it picks up later changes only when + // it retries the reconciliation. The default rate limiter backs off exponentially up to 1000 seconds, so the suite + // caps the delay to keep such retries (and the recoveries that depend on them) within seconds. + requeueBaseDelay = 5 * time.Millisecond + requeueMaxDelay = time.Second + + // The environment variable that the CI pipeline sets to tell which runner invokes the tests. + testRunnerNameEnvVarName = "KUBEFLEET_CI_TEST_RUNNER_NAME" + // The runner (`go test`, via `make test`) with which the tests are skipped in the CI pipeline; the pipeline runs + // this package in a separate step with the Ginkgo CLI (in parallel) instead. + runnerNameToSkipTestsInCI = "default" +) + +var ( + hubCfg *rest.Config + hubEnv *envtest.Environment + hubClient client.Client + mgr manager.Manager + + ctx context.Context + cancel context.CancelFunc + + memberClusterNames = []string{memberCluster1Name, memberCluster2Name} +) + +func TestMain(m *testing.M) { + // Skip the tests if in the CI environment the tests are invoked with `go test` instead of the Ginkgo CLI. + // This has no effect outside the CI environment. + if v := os.Getenv(testRunnerNameEnvVarName); v == runnerNameToSkipTestsInCI { + log.Println("Skipping the tests in CI as they are not run with the expected runner") + os.Exit(0) + } + + os.Exit(m.Run()) +} + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Work Generator Integration Test Suite") +} + +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + By("Bootstrapping test environment") + hubEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("../../../../", "config", "crd", "bases")}, + ErrorIfCRDPathMissing: true, + } + + var err error + hubCfg, err = hubEnv.Start() + Expect(err).Should(Succeed()) + Expect(hubCfg).NotTo(BeNil()) + + By("Setting up the scheme") + Expect(placementv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + By("Starting the controller manager") + mgr, err = ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{ + BindAddress: "0", + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).Should(Succeed()) + + // Use the same (cache-backed) client as the controller to avoid cache delays when verifying results. + By("Setting the hub client to the one used by the controller manager") + hubClient = mgr.GetClient() + + // The field indexes must be set up before the manager starts. + By("Setting up the field indexes") + Expect(fieldindexers.SetupWithHubAgentControllerManager(ctx, mgr)).Should(Succeed()) + + By("Setting up the work generator") + Expect(New(mgr.GetClient(), workerCount, + workqueue.NewTypedItemExponentialFailureRateLimiter[reconcile.Request](requeueBaseDelay, requeueMaxDelay), + ).SetupWithManager(mgr, maxConcurrentReconciles)).Should(Succeed()) + + go func() { + defer GinkgoRecover() + Expect(mgr.Start(ctx)).Should(Succeed(), "Failed to run manager") + }() + Expect(mgr.GetCache().WaitForCacheSync(ctx)).Should(BeTrue(), "Failed to sync the manager cache") + + By("Creating the application namespace") + Expect(hubClient.Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: appNamespaceName}, + })).Should(Succeed(), "Failed to create the application namespace") + + By("Creating the reserved namespaces for member clusters") + for _, clusterName := range memberClusterNames { + Expect(hubClient.Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf(utils.NamespaceNameFormat, clusterName)}, + })).Should(Succeed(), "Failed to create the reserved namespace for member cluster %s", clusterName) + } +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + + cancel() + + By("Tearing down the test environment") + Expect(hubEnv.Stop()).Should(Succeed()) +}) diff --git a/pkg/v1/controllers/workgenerator/uniquename_test.go b/pkg/v1/controllers/workgenerator/uniquename_test.go new file mode 100644 index 000000000..9f815b5a3 --- /dev/null +++ b/pkg/v1/controllers/workgenerator/uniquename_test.go @@ -0,0 +1,272 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "strings" + "testing" + + "k8s.io/apimachinery/pkg/util/validation" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// dottedName returns a name of the given length with a dot as every 10th character, starting from the given offset +// (0-9), except for the last character, e.g., dottedName(23, 9) = "abcdefghi.klmnopqrs.uvw". +func dottedName(length, dotOffset int) string { + var sb strings.Builder + for idx := 0; idx < length; idx++ { + if idx%10 == dotOffset && idx != length-1 { + sb.WriteByte('.') + continue + } + sb.WriteByte(byte('a' + idx%26)) + } + return sb.String() +} + +// customDerivedFromSourceFormatter is a derivedFromSourceFormatter with fully customizable values. +type customDerivedFromSourceFormatter struct { + sourceType string + sourceID string + dnsLabel string +} + +func (f *customDerivedFromSourceFormatter) SourceID() string { return f.sourceID } +func (f *customDerivedFromSourceFormatter) SourceType() string { return f.sourceType } +func (f *customDerivedFromSourceFormatter) StrictDNSLabel() string { return f.dnsLabel } + +func snapshotFormatter(subIdx string) derivedFromSourceFormatter { + return &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: subIdx} +} + +func bindingForUniqueName(namespace, policyName string) placementv1alpha1.PlacementBindingAccessor { + binding := newPlacementBindingObj(namespace, "binding") + binding.SetSpec(placementv1alpha1.PlacementBindingSpec{PlacementPolicyName: policyName}) + return binding +} + +// TestUniqueNameForWorkDerivedFromPlacementResourceSnapshot tests the +// uniqueNameForWorkDerivedFromPlacementResourceSnapshot function. +// +// The hashes below are the first 12 hex characters of the SHA-256 digests of the hash inputs documented in +// uniquename.go (e.g., `team-a/red-work`), computed independently of the implementation. +func TestUniqueNameForWorkDerivedFromPlacementResourceSnapshot(t *testing.T) { + // The dot-free form of a dotted name, truncated to 233 characters (251 - len("-work-") - 12). + dottedPolicyName := dottedName(300, 5) + wantTruncatedDottedPolicyName := strings.ReplaceAll(dottedPolicyName, ".", "")[:233] + + testCases := []struct { + name string + namespace string + policyName string + isPrimary bool + formatter derivedFromSourceFormatter + wantName string + }{ + { + name: "primary, namespace-scoped policy", + namespace: "team-a", + policyName: "red", + isPrimary: true, + formatter: snapshotFormatter("0"), + wantName: "team-a-red-work-25dda90d22a3", + }, + { + name: "primary, cluster-scoped policy", + namespace: "", + policyName: "red", + isPrimary: true, + formatter: snapshotFormatter("0"), + wantName: "red-work-830565d28775", + }, + { + name: "secondary, sub-index 1", + namespace: "team-a", + policyName: "red", + formatter: snapshotFormatter("1"), + wantName: "team-a-red-work-1-309264462839", + }, + { + name: "secondary, sub-index 12", + namespace: "team-a", + policyName: "red", + formatter: snapshotFormatter("12"), + wantName: "team-a-red-work-12-14a0ae508b07", + }, + { + // `red` in `team-a` yields the same name segment as `a-red` in `team`; only the hash tells them apart. + name: "primary, name segment shared with another namespace/name combination", + namespace: "team", + policyName: "a-red", + isPrimary: true, + formatter: snapshotFormatter("0"), + wantName: "team-a-red-work-0d3acb940e0a", + }, + { + // Dots are removed from the name segment, but the hash is computed over the original values. + name: "primary, names with dots", + namespace: "ns.one", + policyName: "my.policy.v1", + isPrimary: true, + formatter: snapshotFormatter("0"), + wantName: "nsone-mypolicyv1-work-81ba5d26981e", + }, + { + name: "primary, exactly at the length limit (not truncated)", + namespace: "", + policyName: strings.Repeat("a", 233), + isPrimary: true, + formatter: snapshotFormatter("0"), + wantName: strings.Repeat("a", 233) + "-work-049cbf283748", + }, + { + name: "primary, one character over the length limit (truncated)", + namespace: "", + policyName: strings.Repeat("a", 234), + isPrimary: true, + formatter: snapshotFormatter("0"), + wantName: strings.Repeat("a", 233) + "-work-4c7c1e597554", + }, + { + name: "primary, namespace and policy name combined over the length limit (truncated)", + namespace: strings.Repeat("n", 100), + policyName: strings.Repeat("p", 200), + isPrimary: true, + formatter: snapshotFormatter("0"), + // The namespaced name segment is `[NAMESPACE]-[NAME]`, truncated to 233 characters. + wantName: strings.Repeat("n", 100) + "-" + strings.Repeat("p", 132) + "-work-d43a46802b5b", + }, + { + // Truncation happens after the dots are removed, so the name never ends with a dot. + name: "primary, dotted policy name over the length limit (truncated)", + namespace: "", + policyName: dottedPolicyName, + isPrimary: true, + formatter: snapshotFormatter("0"), + wantName: wantTruncatedDottedPolicyName + "-work-fadf9e3d4a62", + }, + { + // The available length (251 - len("-work--") - 12 = 232) is split evenly between the name segment + // and the source label; the short label fits as is. + name: "secondary, over the length limit (name segment truncated)", + namespace: "", + policyName: strings.Repeat("a", 300), + formatter: snapshotFormatter("1"), + wantName: strings.Repeat("a", 116) + "-work-1-c3937b0a58ec", + }, + { + name: "secondary, name segment and source label both over the length limit (both truncated)", + namespace: "", + policyName: strings.Repeat("p", 200), + formatter: &customDerivedFromSourceFormatter{ + sourceType: "test-source", + sourceID: "id-1", + dnsLabel: strings.Repeat("l", 200), + }, + wantName: strings.Repeat("p", 116) + "-work-" + strings.Repeat("l", 116) + "-c17475bfc3d0", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + binding := bindingForUniqueName(tc.namespace, tc.policyName) + gotName := uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, tc.isPrimary, tc.formatter) + if gotName != tc.wantName { + t.Errorf("uniqueNameForWorkDerivedFromPlacementResourceSnapshot(%q, %q, %t) = %q, want %q", + tc.namespace, tc.policyName, tc.isPrimary, gotName, tc.wantName) + } + if len(gotName) > nameLenLimit { + t.Errorf("len(uniqueNameForWorkDerivedFromPlacementResourceSnapshot(%q, %q, %t)) = %d, want <= %d", + tc.namespace, tc.policyName, tc.isPrimary, len(gotName), nameLenLimit) + } + if errs := validation.IsDNS1123Subdomain(gotName); len(errs) > 0 { + t.Errorf("uniqueNameForWorkDerivedFromPlacementResourceSnapshot(%q, %q, %t) = %q, which is not a valid DNS subdomain: %v", + tc.namespace, tc.policyName, tc.isPrimary, gotName, errs) + } + }) + } +} + +// TestUniqueNameForWorkDerivedFromPlacementResourceSnapshotUniqueness verifies that the generated names are +// deterministic and distinct across sources and across namespace/name combinations. +func TestUniqueNameForWorkDerivedFromPlacementResourceSnapshotUniqueness(t *testing.T) { + type input struct { + namespace string + policyName string + isPrimary bool + subIdx string + } + inputs := []input{ + {namespace: "team-a", policyName: "red", isPrimary: true, subIdx: "0"}, + {namespace: "team-a", policyName: "red", subIdx: "1"}, + {namespace: "team-a", policyName: "red", subIdx: "2"}, + {namespace: "team-a", policyName: "red", subIdx: "11"}, + {namespace: "team", policyName: "a-red", isPrimary: true, subIdx: "0"}, + {namespace: "team", policyName: "a-red", subIdx: "1"}, + {namespace: "", policyName: "team-a-red", isPrimary: true, subIdx: "0"}, + {namespace: "", policyName: "team-a-red", subIdx: "1"}, + // Names that differ only beyond the truncation point. + {namespace: "", policyName: strings.Repeat("a", 240) + "x", isPrimary: true, subIdx: "0"}, + {namespace: "", policyName: strings.Repeat("a", 240) + "y", isPrimary: true, subIdx: "0"}, + {namespace: "", policyName: strings.Repeat("a", 240) + "x", subIdx: "1"}, + {namespace: "", policyName: strings.Repeat("a", 240) + "y", subIdx: "1"}, + } + + seen := make(map[string]input, len(inputs)) + for _, in := range inputs { + gotName := uniqueNameForWorkDerivedFromPlacementResourceSnapshot( + bindingForUniqueName(in.namespace, in.policyName), in.isPrimary, snapshotFormatter(in.subIdx)) + gotAgain := uniqueNameForWorkDerivedFromPlacementResourceSnapshot( + bindingForUniqueName(in.namespace, in.policyName), in.isPrimary, snapshotFormatter(in.subIdx)) + if gotName != gotAgain { + t.Errorf("uniqueNameForWorkDerivedFromPlacementResourceSnapshot(%+v) = %q then %q, want a deterministic result", in, gotName, gotAgain) + } + if other, found := seen[gotName]; found { + t.Errorf("uniqueNameForWorkDerivedFromPlacementResourceSnapshot(%+v) = %q, which collides with the name for %+v", in, gotName, other) + } + seen[gotName] = in + } +} + +// TestPlacementResourceSnapshotDerivedFromSourceFormatter tests the formatter for placement resource snapshots. +func TestPlacementResourceSnapshotDerivedFromSourceFormatter(t *testing.T) { + testCases := []struct { + subIdx string + wantSourceID string + wantSourceType string + wantStrictDNSLabel string + }{ + {subIdx: "0", wantSourceID: "0", wantSourceType: "placement-resource-snapshot", wantStrictDNSLabel: "0"}, + {subIdx: "12", wantSourceID: "12", wantSourceType: "placement-resource-snapshot", wantStrictDNSLabel: "12"}, + } + + for _, tc := range testCases { + t.Run("sub-index "+tc.subIdx, func(t *testing.T) { + f := snapshotFormatter(tc.subIdx) + if got := f.SourceID(); got != tc.wantSourceID { + t.Errorf("SourceID() = %q, want %q", got, tc.wantSourceID) + } + if got := f.SourceType(); got != tc.wantSourceType { + t.Errorf("SourceType() = %q, want %q", got, tc.wantSourceType) + } + if got := f.StrictDNSLabel(); got != tc.wantStrictDNSLabel { + t.Errorf("StrictDNSLabel() = %q, want %q", got, tc.wantStrictDNSLabel) + } + }) + } +} diff --git a/pkg/v1/controllers/workgenerator/utils_integration_test.go b/pkg/v1/controllers/workgenerator/utils_integration_test.go new file mode 100644 index 000000000..542c0b6f4 --- /dev/null +++ b/pkg/v1/controllers/workgenerator/utils_integration_test.go @@ -0,0 +1,1685 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "crypto/sha256" + "fmt" + "slices" + "strconv" + "strings" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" + testutilsresource "go.goms.io/fleet/test/utils/resource" +) + +const ( + placementBindingNameTemplate = "binding-%s" + placementPolicyNameTemplate = "policy-%s" + clusterPlacementBindingNameTemplate = "cluster-binding-%s" + clusterPlacementPolicyNameTemplate = "cluster-policy-%s" + placementResourceSnapshotNameTemplate = "%s-snapshot-%d" + // The name template for secondary placement resource snapshots (those with a non-zero sub-index). + subIndexedPlacementResourceSnapshotNameTemplate = "%s-snapshot-%d-%d" + // The name template for stray placement resource snapshots (see bindingFixture.createStraySnapshot). + straySnapshotNameTemplate = "%s-snapshot-%d-%d-stray-%s" + nsNameTemplate = "ns-%s" + + deployName = "deploy-1" + configMapName = "configmap-1" + + indexedNSNameTemplate = "ns-%d" + deployNameTemplate = "deploy-%d" + configMapNameTemplate = "configmap-%d" + // A zero-padded ConfigMap name template, so that the names sort in the order of their numbers. + paddedConfigMapNameTemplate = "configmap-%02d" + + workKind = "Work" + nsKind = "Namespace" + deployKind = "Deployment" + configMapKind = "ConfigMap" + nginxName = "nginx" + + appsAPIGroup = "apps" + + configMapDataKey = "foo" + configMapDataValue = "bar" + // Alternative ConfigMap data values, used to simulate changes or drifts. + altConfigMapDataValue1 = "baz" + altConfigMapDataValue2 = "qux" + + dummyContentsHash = "dummy-hash" + + nsResource = "namespaces" + deployResource = "deployments" + configMapResource = "configmaps" + + markedAsAppliedReason = "MarkedAsApplied" + markedAsAvailableReason = "MarkedAsAvailable" + markedAsUnavailableReason = "MarkedAsUnavailable" + markedAsFailedToApplyReason = "MarkedAsFailedToApply" + + markedAsAppliedManifestMsg = "Manifest has been marked as applied" + markedAsAppliedWorkMsg = "Work has been marked as applied" + + // Substrings of the messages that the work generator reports (in the Synchronized condition of a placement binding) + // when the placement resource snapshots that the placement binding points to are invalid. + invalidSnapshotsMsgPrefix = "Failed to synchronize resources as the given placement resource snapshots are invalid" + fewerSnapshotsErrMsg = "there are fewer placement resource snapshots than the count label indicates" + duplicateSubIdxErrMsg = "found placement resource snapshots with duplicate sub-indices" + inconsistentSubIdxErrMsg = "found placement resource snapshots with inconsistent sub-indices" + // The format of the message substring reporting that a placement resource snapshot (by name) is not found. + snapshotNotFoundErrMsgFmt = "%q not found" +) + +// The namespace of cluster-scoped objects (i.e., none); the test helpers use it to tell cluster-scoped placement +// bindings and placement resource snapshots apart from their namespace-scoped counterparts. +const clusterScopedNS = "" + +const ( + eventuallyDuration = time.Second * 10 + eventuallyInterval = time.Second * 1 + consistentlyDuration = time.Second * 5 + consistentlyInterval = time.Millisecond * 500 +) + +var ( + ignoreFieldObjectMetaAutoGenFields = cmpopts.IgnoreFields(metav1.ObjectMeta{}, "CreationTimestamp", "Generation", "ResourceVersion", "SelfLink", "UID", "ManagedFields") + ignoreFieldWorkTypeMetaAndStatus = cmpopts.IgnoreFields(placementv1alpha1.Work{}, "TypeMeta", "Status") + ignoreFieldConditionLTTMsg = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "Message") + + lessFuncWork = func(a, b placementv1alpha1.Work) bool { + return a.Name < b.Name + } + // The work generator collects failed resources work object by work object, in the order the work objects are + // listed from the cache, which is not deterministic; failed resources are sorted before comparison. + lessFuncFailedResource = func(a, b placementv1alpha1.FailedResource) bool { + keyA := fmt.Sprintf("%s/%s/%s/%s", a.ObjectRef.APIGroup, a.ObjectRef.Kind, a.ObjectRef.Namespace, a.ObjectRef.Name) + keyB := fmt.Sprintf("%s/%s/%s/%s", b.ObjectRef.APIGroup, b.ObjectRef.Kind, b.ObjectRef.Namespace, b.ObjectRef.Name) + return keyA < keyB + } +) + +var ( + ns = &corev1.Namespace{ + TypeMeta: metav1.TypeMeta{ + Kind: nsKind, + APIVersion: "v1", + }, + } + + deploy = &appsv1.Deployment{ + TypeMeta: metav1.TypeMeta{ + Kind: deployKind, + APIVersion: "apps/v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: deployName, + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(int32(1)), + Selector: &metav1.LabelSelector{ + MatchLabels: map[string]string{ + "app": nginxName, + }, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{ + "app": nginxName, + }, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: nginxName, + Image: nginxName, + Ports: []corev1.ContainerPort{ + { + ContainerPort: 80, + }, + }, + }, + }, + }, + }, + }, + } + + configMap = &corev1.ConfigMap{ + TypeMeta: metav1.TypeMeta{ + Kind: configMapKind, + APIVersion: "v1", + }, + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + }, + Data: map[string]string{ + configMapDataKey: configMapDataValue, + }, + } +) + +func marshalK8sObjJSON(obj runtime.Object) []byte { + json, err := testutilsresource.MarshalRuntimeObjToJSONForTest(obj) + Expect(err).To(BeNil(), "Failed to marshal the k8s object to JSON") + return json +} + +// nsSnapshottedResource returns a snapshotted resource for a NS object of the given name. +func nsSnapshottedResource(nsName string) placementv1alpha1.SnapshottedResource { + regularNS := ns.DeepCopy() + regularNS.Name = nsName + return placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + }, + Manifest: runtime.RawExtension{Raw: marshalK8sObjJSON(regularNS)}, + } +} + +// deploySnapshottedResource returns a snapshotted resource for a Deployment object of the given namespace and name. +func deploySnapshottedResource(nsName, deployName string) placementv1alpha1.SnapshottedResource { + regularDeploy := deploy.DeepCopy() + regularDeploy.Namespace = nsName + regularDeploy.Name = deployName + return placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + }, + Manifest: runtime.RawExtension{Raw: marshalK8sObjJSON(regularDeploy)}, + } +} + +// configMapSnapshottedResource returns a snapshotted resource for a ConfigMap object of the given namespace and name. +func configMapSnapshottedResource(nsName, configMapName string) placementv1alpha1.SnapshottedResource { + return configMapSnapshottedResourceWithData(nsName, configMapName, nil) +} + +// configMapSnapshottedResourceWithData returns a snapshotted resource for a ConfigMap object of the given namespace +// and name, with the given data (or the data from the template if the given data is nil). +func configMapSnapshottedResourceWithData(nsName, configMapName string, data map[string]string) placementv1alpha1.SnapshottedResource { + regularConfigMap := configMap.DeepCopy() + regularConfigMap.Namespace = nsName + regularConfigMap.Name = configMapName + if data != nil { + regularConfigMap.Data = data + } + return placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{ + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + }, + Manifest: runtime.RawExtension{Raw: marshalK8sObjJSON(regularConfigMap)}, + } +} + +// nsManifestIdentifier returns the manifest identifier for a NS object of the given name at the given ordinal. +func nsManifestIdentifier(ordinal int, nsName string) placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: ordinal, + Name: nsName, + APIVersion: "v1", + Kind: nsKind, + Resource: nsResource, + } +} + +// deployManifestIdentifier returns the manifest identifier for a Deployment object of the given namespace and name +// at the given ordinal. +func deployManifestIdentifier(ordinal int, nsName, deployName string) placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: ordinal, + Namespace: nsName, + Name: deployName, + APIGroup: appsAPIGroup, + APIVersion: "v1", + Kind: deployKind, + Resource: deployResource, + } +} + +// configMapManifestIdentifier returns the manifest identifier for a ConfigMap object of the given namespace and name +// at the given ordinal. +func configMapManifestIdentifier(ordinal int, nsName, configMapName string) placementv1alpha1.ManifestIdentifier { + return placementv1alpha1.ManifestIdentifier{ + Ordinal: ordinal, + Namespace: nsName, + Name: configMapName, + APIVersion: "v1", + Kind: configMapKind, + Resource: configMapResource, + } +} + +// objectRefFromManifestIdentifier returns the object reference (as used in the failed resources of placement binding +// status) for the resource identified by the given manifest identifier. +func objectRefFromManifestIdentifier(identifier placementv1alpha1.ManifestIdentifier) placementv1alpha1.ObjectReference { + return placementv1alpha1.ObjectReference{ + Namespace: identifier.Namespace, + Name: identifier.Name, + APIGroup: identifier.APIGroup, + APIVersion: identifier.APIVersion, + Kind: identifier.Kind, + } +} + +// newPlacementBindingObj returns a placement binding object of the appropriate scope with only the namespace and name +// set: a ClusterPlacementBinding object if the namespace is empty, or a PlacementBinding object otherwise. +func newPlacementBindingObj(bindingNS, bindingName string) placementv1alpha1.PlacementBindingAccessor { + if bindingNS == "" { + return &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: bindingName, + }, + } + } + return &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: bindingNS, + Name: bindingName, + }, + } +} + +// newPlacementResourceSnapshotObj returns a placement resource snapshot object of the appropriate scope with only the +// namespace and name set: a ClusterPlacementResourceSnapshot object if the namespace is empty, or a +// PlacementResourceSnapshot object otherwise. +func newPlacementResourceSnapshotObj(snapshotNS, snapshotName string) placementv1alpha1.PlacementResourceSnapshotAccessor { + if snapshotNS == "" { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + }, + } + } + return &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: snapshotNS, + Name: snapshotName, + }, + } +} + +// getPlacementBinding retrieves a placement binding (cluster-scoped if the namespace is empty). +func getPlacementBinding(bindingNS, bindingName string) (placementv1alpha1.PlacementBindingAccessor, error) { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); err != nil { + return nil, fmt.Errorf("failed to retrieve the placement binding: %w", err) + } + return binding, nil +} + +// createPlacementResourceSnapshot creates a primary placement resource snapshot (sub-index 0) of the given index +// with the given resources, as the only snapshot of its index; it returns the snapshot as written to the API server. +// +// If the namespace is empty, a ClusterPlacementResourceSnapshot object is created; otherwise, a +// PlacementResourceSnapshot object is created. +func createPlacementResourceSnapshot( + snapshotNS, snapshotName, ownerPlacementPolicyName string, + index int, + resources ...placementv1alpha1.SnapshottedResource, +) placementv1alpha1.PlacementResourceSnapshotAccessor { + return createSubIndexedPlacementResourceSnapshot(snapshotNS, snapshotName, ownerPlacementPolicyName, index, 0, 1, resources...) +} + +// createSubIndexedPlacementResourceSnapshot creates a placement resource snapshot of the given index and sub-index +// with the given resources; it returns the snapshot as written to the API server. +// +// The count of snapshots sharing the same index is set (as a label) only on the primary snapshot (sub-index 0); +// all snapshots of the same index share the same contents hash. If the namespace is empty, a +// ClusterPlacementResourceSnapshot object is created; otherwise, a PlacementResourceSnapshot object is created. +func createSubIndexedPlacementResourceSnapshot( + snapshotNS, snapshotName, ownerPlacementPolicyName string, + index, subIndex, subIndexedSnapshotCount int, + resources ...placementv1alpha1.SnapshottedResource, +) placementv1alpha1.PlacementResourceSnapshotAccessor { + return createSubIndexedPlacementResourceSnapshotWithHash(snapshotNS, snapshotName, ownerPlacementPolicyName, + index, subIndex, subIndexedSnapshotCount, dummyContentsHash, resources...) +} + +// createSubIndexedPlacementResourceSnapshotWithHash is the same as createSubIndexedPlacementResourceSnapshot, except +// that it allows specifying the contents hash annotation of the snapshot. +func createSubIndexedPlacementResourceSnapshotWithHash( + snapshotNS, snapshotName, ownerPlacementPolicyName string, + index, subIndex, subIndexedSnapshotCount int, + contentsHash string, + resources ...placementv1alpha1.SnapshottedResource, +) placementv1alpha1.PlacementResourceSnapshotAccessor { + snapshot := newPlacementResourceSnapshotObj(snapshotNS, snapshotName) + labels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: snapshotOwnerLabelValue(ownerPlacementPolicyName), + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: strconv.Itoa(index), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: strconv.Itoa(subIndex), + } + if subIndex == 0 { + labels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = strconv.Itoa(subIndexedSnapshotCount) + } + snapshot.SetLabels(labels) + snapshot.SetAnnotations(map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: contentsHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: ownerPlacementPolicyName, + }) + snapshot.SetSpec(placementv1alpha1.PlacementResourceSnapshotSpec{ + Resources: resources, + }) + Expect(hubClient.Create(ctx, snapshot)).To(Succeed(), "Failed to create the placement resource snapshot") + return snapshot +} + +// snapshotOwnerLabelValue returns the value of the owner label on placement resource snapshots for the given placement +// policy name, mirroring the placement resource snapshot manager: names over 61 characters are truncated and appended +// with a hash suffix, so that the value stays within the label value length limit. The work generator does not read +// this label (it looks up snapshots by the owner annotation, which always carries the full name). +func snapshotOwnerLabelValue(placementPolicyName string) string { + const lenLimit, hashLen = 61, 12 + if len(placementPolicyName) <= lenLimit { + return placementPolicyName + } + hash := fmt.Sprintf("%x", sha256.Sum256([]byte(placementPolicyName)))[:hashLen] + return fmt.Sprintf("%s-%s", placementPolicyName[:lenLimit-hashLen-1], hash) +} + +// longDottedName returns a random name of exactly the given length that starts with the given prefix, and contains +// dots every 50 characters or so; it is a valid DNS subdomain name as long as the length does not exceed 253. +func longDottedName(prefix string, length int) string { + name := fmt.Sprintf("%s-%s", prefix, utils.RandStr()) + for len(name) < length { + remaining := length - len(name) + if remaining == 1 { + // Avoid a trailing dot. + name += "a" + break + } + name += "." + strings.Repeat("a", min(remaining-1, 50)) + } + return name +} + +// createPlacementBinding creates a placement binding that binds a placement resource snapshot to a member cluster; +// it returns the binding as written to the API server. +// +// If the namespace is empty, a ClusterPlacementBinding object is created; otherwise, a PlacementBinding object +// is created. +func createPlacementBinding(bindingNS, bindingName, placementPolicyName, clusterName, snapshotName string) placementv1alpha1.PlacementBindingAccessor { + binding := newPlacementBindingObj(bindingNS, bindingName) + binding.SetSpec(placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: placementPolicyName, + ClusterName: clusterName, + ResourceSnapshotName: snapshotName, + }) + Expect(hubClient.Create(ctx, binding)).To(Succeed(), "Failed to create the placement binding") + return binding +} + +// updatePlacementBindingResourceSnapshot points a placement binding to a different placement resource snapshot, +// as if a rollout has been made. +func updatePlacementBindingResourceSnapshot(bindingNS, bindingName, snapshotName string) { + Eventually(func() error { + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + binding.GetSpec().ResourceSnapshotName = snapshotName + if err := hubClient.Update(ctx, binding); err != nil { + return fmt.Errorf("failed to update the placement binding: %w", err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the placement binding with a new placement resource snapshot") +} + +// wantWorkForPrimaryPlacementResourceSnapshot builds the work object that the work generator is expected to create +// for a placement binding (cluster-scoped if the namespace is empty) from its primary placement resource snapshot, +// when the snapshot is the only one of its index. +func wantWorkForPrimaryPlacementResourceSnapshot( + workNS, workName, bindingNS, bindingName, placementPolicyName string, + snapshot placementv1alpha1.PlacementResourceSnapshotAccessor, +) placementv1alpha1.Work { + return wantWorkForPlacementResourceSnapshot(workNS, workName, bindingNS, bindingName, placementPolicyName, + snapshot.GetName(), 0, 1, nil, snapshot) +} + +// wantWorkForPlacementResourceSnapshot builds the work object that the work generator is expected to create for a +// placement binding (cluster-scoped if the namespace is empty) from a placement resource snapshot of the given +// sub-index. +// +// The linked work count applies only to the primary work object (sub-index 0); the owner reference (which should +// point to the primary work object) applies only to secondary work objects. +func wantWorkForPlacementResourceSnapshot( + workNS, workName, bindingNS, bindingName, placementPolicyName, primarySnapshotName string, + subIndex, linkedWorkCount int, + ownerRef *metav1.OwnerReference, + snapshot placementv1alpha1.PlacementResourceSnapshotAccessor, +) placementv1alpha1.Work { + // The manifests in the work object should match those in the snapshot as written to the API server. + resources := snapshot.GetSpec().Resources + manifests := make([]placementv1alpha1.Manifest, len(resources)) + for idx := range resources { + manifests[idx] = placementv1alpha1.Manifest{RawExtension: resources[idx].Manifest} + } + + work := placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workName, + Labels: map[string]string{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: bindingNS, + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: workOwnerLabelValue(placementPolicyName), + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: workOwnerLabelValue(bindingName), + }, + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: primarySnapshotName, + placementv1alpha1.WorkDerivedFromSourceAnnotationKey: fmt.Sprintf("placement-resource-snapshot/%d", subIndex), + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: placementPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: bindingName, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: manifests, + }, + } + if subIndex == 0 { + work.Annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = strconv.Itoa(linkedWorkCount) + } else if ownerRef != nil { + work.OwnerReferences = []metav1.OwnerReference{*ownerRef} + } + return work +} + +// wantPlacementBindingStatusWaitingForSync builds the placement binding status that the work generator is expected +// to report after it has (re-)generated the work objects, but before the work objects have been processed. +func wantPlacementBindingStatusWaitingForSync(selectedResources int32, snapshotName string) *placementv1alpha1.PlacementBindingStatus { + return &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonWaitingForSynchronization, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionUnknown, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonWaitingForAvailabilityCheck, + }, + }, + SelectedResources: ptr.To(selectedResources), + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } +} + +// wantPlacementBindingStatusSyncedAndAvailable builds the placement binding status that the work generator is +// expected to report after all the work objects have been applied and become available. +func wantPlacementBindingStatusSyncedAndAvailable(selectedResources int32, snapshotName string) *placementv1alpha1.PlacementBindingStatus { + return &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonAllResourcesAvailable, + }, + }, + SelectedResources: ptr.To(selectedResources), + SynchronizedResources: ptr.To(selectedResources), + AvailableResources: ptr.To(selectedResources), + LastProcessedResourceSnapshotName: ptr.To(snapshotName), + } +} + +// wantPlacementBindingStatusInvalidSnapshots builds the placement binding status that the work generator is expected +// to report when the placement resource snapshots that a placement binding has been rolled out to are invalid. The +// resource counters are tallied from the work objects of the last processed placement resource snapshot, all of +// which are expected to have been applied and become available. For a rollout to a different snapshot, previousStatus +// supplies the Available condition to preserve; nil expects an incomplete set of resources instead. +func wantPlacementBindingStatusInvalidSnapshots(placedResources int32, lastProcessedSnapshotName string, previousStatus *placementv1alpha1.PlacementBindingStatus) *placementv1alpha1.PlacementBindingStatus { + wantStatus := &placementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.PlacementBindingCondTypeSynchronized, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingSynchronizedCondReasonInvalidSnapshots, + }, + { + Type: placementv1alpha1.PlacementBindingCondTypeAvailable, + Status: metav1.ConditionFalse, + Reason: placementv1alpha1.PlacementBindingAvailableCondReasonIncompleteSetOfResources, + }, + }, + SelectedResources: ptr.To(placedResources), + SynchronizedResources: ptr.To(placedResources), + AvailableResources: ptr.To(placedResources), + LastProcessedResourceSnapshotName: ptr.To(lastProcessedSnapshotName), + } + if previousStatus != nil { + wantStatus.Conditions = wantStatus.Conditions[:1] + if availableCond := meta.FindStatusCondition(previousStatus.Conditions, placementv1alpha1.PlacementBindingCondTypeAvailable); availableCond != nil { + wantStatus.Conditions = append(wantStatus.Conditions, *availableCond) + } + } + return wantStatus +} + +func placementBindingFinalizerAddedActual(bindingNS, bindingName string) func() error { + return func() error { + // Retrieve the placement binding. + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + // Check that the cleanup finalizer has been added. + if !controllerutil.ContainsFinalizer(binding, workGeneratorCleanupFinalizer) { + return fmt.Errorf("cleanup finalizer has not been added") + } + return nil + } +} + +func worksGeneratedActual(clusterName, bindingNS, bindingName string, wantWorks []placementv1alpha1.Work) func() error { + return func() error { + // List the work objects owned by the placement binding (the owner label carries a hashed value for long or + // dotted placement binding names). + workList := &placementv1alpha1.WorkList{} + listOptions := []client.ListOption{ + client.InNamespace(fmt.Sprintf(utils.NamespaceNameFormat, clusterName)), + client.MatchingLabels{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: bindingNS, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: workOwnerLabelValue(bindingName), + }, + } + if err := hubClient.List(ctx, workList, listOptions...); err != nil { + return fmt.Errorf("failed to list work objects: %w", err) + } + + // Check that the work objects have been generated as expected. + if diff := cmp.Diff( + workList.Items, wantWorks, + ignoreFieldObjectMetaAutoGenFields, + ignoreFieldWorkTypeMetaAndStatus, + cmpopts.EquateEmpty(), + cmpopts.SortSlices(lessFuncWork), + ); diff != "" { + return fmt.Errorf("works diff (-got +want):\n%s", diff) + } + return nil + } +} + +func placementBindingStatusUpdatedActual(bindingNS, bindingName string, wantStatus *placementv1alpha1.PlacementBindingStatus) func() error { + return func() error { + // Retrieve the placement binding. + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + // Prepare the expected placement binding status; update the conditions with the observed generation. + wantStatusWithObsGen := wantStatus.DeepCopy() + for idx := range wantStatusWithObsGen.Conditions { + wantStatusWithObsGen.Conditions[idx].ObservedGeneration = binding.GetGeneration() + } + + // Check that the placement binding status has been updated as expected. + if diff := cmp.Diff( + *binding.GetStatus(), *wantStatusWithObsGen, + ignoreFieldConditionLTTMsg, + cmpopts.SortSlices(lessFuncFailedResource), + ); diff != "" { + return fmt.Errorf("placement binding status diff (-got +want):\n%s", diff) + } + return nil + } +} + +// placementBindingStatusMatchesActual verifies that the status of a placement binding matches the given one as is, +// i.e., unlike placementBindingStatusUpdatedActual, the observed generations of the conditions are not updated with +// the current generation of the placement binding. This is useful for verifying that a placement binding status has +// not been refreshed after the placement binding spec changes. +func placementBindingStatusMatchesActual(bindingNS, bindingName string, wantStatus *placementv1alpha1.PlacementBindingStatus) func() error { + return func() error { + // Retrieve the placement binding. + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + // Check that the placement binding status matches the expected one. + if diff := cmp.Diff( + *binding.GetStatus(), *wantStatus, + ignoreFieldConditionLTTMsg, + cmpopts.SortSlices(lessFuncFailedResource), + ); diff != "" { + return fmt.Errorf("placement binding status diff (-got +want):\n%s", diff) + } + return nil + } +} + +// placementBindingSyncCondMessageActual verifies that the message of the Synchronized condition on a placement binding +// contains the given substring; the status comparison helpers ignore condition messages. +func placementBindingSyncCondMessageActual(bindingNS, bindingName, wantMsgSubstr string) func() error { + return func() error { + // Retrieve the placement binding. + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + syncCond := meta.FindStatusCondition(binding.GetStatus().Conditions, placementv1alpha1.PlacementBindingCondTypeSynchronized) + if syncCond == nil { + return fmt.Errorf("the Synchronized condition is not found on the placement binding") + } + if !strings.Contains(syncCond.Message, wantMsgSubstr) { + return fmt.Errorf("the Synchronized condition message = %q, want it to contain %q", syncCond.Message, wantMsgSubstr) + } + return nil + } +} + +// markWorkAsAppliedAndAvailable updates the status of a work object to mark it (and all of its manifests) as +// applied and available, as the KubeFleet member agent would do. +func markWorkAsAppliedAndAvailable(workNS, workName string, manifestIdentifiers []placementv1alpha1.ManifestIdentifier) { + // Retrieve the work object. + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work)).To(Succeed(), "Failed to retrieve the work object") + + // Mark the work object as applied and available. + now := metav1.Now() + manifestStatuses := make([]placementv1alpha1.PerManifestStatus, len(manifestIdentifiers)) + for idx := range manifestIdentifiers { + manifestStatuses[idx] = placementv1alpha1.PerManifestStatus{ + Identifier: manifestIdentifiers[idx], + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAppliedReason, + Message: markedAsAppliedManifestMsg, + LastTransitionTime: now, + }, + { + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAvailableReason, + Message: "Manifest has been marked as available", + LastTransitionTime: now, + }, + }, + } + } + work.Status = placementv1alpha1.WorkStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: work.Generation, + Reason: markedAsAppliedReason, + Message: markedAsAppliedWorkMsg, + LastTransitionTime: now, + }, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionTrue, + ObservedGeneration: work.Generation, + Reason: markedAsAvailableReason, + Message: "Work has been marked as available", + LastTransitionTime: now, + }, + }, + Manifests: manifestStatuses, + } + Expect(hubClient.Status().Update(ctx, work)).To(Succeed(), "Failed to mark the work object as applied and available") +} + +// markWorkManifestAsUnavailable updates the status of a work object to mark one of its manifests (identified by +// its ordinal), and consequently the work object itself, as unavailable, as the KubeFleet member agent would do when +// an applied resource fails its availability check. +func markWorkManifestAsUnavailable(workNS, workName string, ordinal int) { + // Retrieve the work object. + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work)).To(Succeed(), "Failed to retrieve the work object") + + // Mark the manifest as unavailable. + found := false + for idx := range work.Status.Manifests { + manifestStatus := &work.Status.Manifests[idx] + if manifestStatus.Identifier.Ordinal != ordinal { + continue + } + meta.SetStatusCondition(&manifestStatus.Conditions, metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + Message: "Manifest has been marked as unavailable", + }) + found = true + } + Expect(found).To(BeTrue(), "Failed to find the status of the manifest with ordinal %d", ordinal) + + // Mark the work object as unavailable. + meta.SetStatusCondition(&work.Status.Conditions, metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: work.Generation, + Reason: markedAsUnavailableReason, + Message: "Work has been marked as unavailable", + }) + Expect(hubClient.Status().Update(ctx, work)).To(Succeed(), "Failed to mark the work object as unavailable") +} + +// markWorkManifestsAsFailed updates the status of a work object to mark some of its manifests (identified by their +// ordinals) as failed to be applied (with the given diff details, if any), and some as applied but failed to become +// available; all the other manifests are marked as applied and available. This simulates what the KubeFleet member +// agent would do when it cannot apply some resources (e.g., when it fails to take over existing resources with diffs) +// and some applied resources fail their availability checks. +func markWorkManifestsAsFailed( + workNS, workName string, + manifestIdentifiers []placementv1alpha1.ManifestIdentifier, + failedToApplyOrdinals map[int]*placementv1alpha1.DiffDetails, + failedToBecomeAvailableOrdinals sets.Set[int], +) { + // Retrieve the work object. + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work)).To(Succeed(), "Failed to retrieve the work object") + + now := metav1.Now() + appliedCond := metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAppliedReason, + Message: markedAsAppliedManifestMsg, + LastTransitionTime: now, + } + availableCond := metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAvailableReason, + Message: "Manifest has been marked as available", + LastTransitionTime: now, + } + failedToApplyCond := metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsFailedToApplyReason, + Message: "Manifest has been marked as failed to be applied", + LastTransitionTime: now, + } + unavailableCond := metav1.Condition{ + Type: placementv1alpha1.ManifestCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: 1, + Reason: markedAsUnavailableReason, + Message: "Manifest has been marked as unavailable", + LastTransitionTime: now, + } + + manifestStatuses := make([]placementv1alpha1.PerManifestStatus, len(manifestIdentifiers)) + for idx := range manifestIdentifiers { + ordinal := manifestIdentifiers[idx].Ordinal + manifestStatuses[idx] = placementv1alpha1.PerManifestStatus{ + Identifier: manifestIdentifiers[idx], + } + + if diffDetails, failedToApply := failedToApplyOrdinals[ordinal]; failedToApply { + // Mark the manifest as failed to be applied; the availability check is skipped. + manifestStatuses[idx].Conditions = []metav1.Condition{failedToApplyCond} + manifestStatuses[idx].DiffDetails = diffDetails + continue + } + + if failedToBecomeAvailableOrdinals.Has(ordinal) { + // Mark the manifest as applied but unavailable. + manifestStatuses[idx].Conditions = []metav1.Condition{appliedCond, unavailableCond} + continue + } + + // Mark the manifest as applied and available. + manifestStatuses[idx].Conditions = []metav1.Condition{appliedCond, availableCond} + } + + // Mark the work object as failed to be applied (if applicable) and unavailable. + workAppliedCond := metav1.Condition{ + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: work.Generation, + Reason: markedAsAppliedReason, + Message: markedAsAppliedWorkMsg, + LastTransitionTime: now, + } + if len(failedToApplyOrdinals) > 0 { + workAppliedCond.Status = metav1.ConditionFalse + workAppliedCond.Reason = markedAsFailedToApplyReason + workAppliedCond.Message = "Work has been marked as failed to be applied" + } + work.Status = placementv1alpha1.WorkStatus{ + Conditions: []metav1.Condition{ + workAppliedCond, + { + Type: placementv1alpha1.WorkCondTypeAvailable, + Status: metav1.ConditionFalse, + ObservedGeneration: work.Generation, + Reason: markedAsUnavailableReason, + Message: "Work has been marked as unavailable", + LastTransitionTime: now, + }, + }, + Manifests: manifestStatuses, + } + Expect(hubClient.Status().Update(ctx, work)).To(Succeed(), "Failed to mark some manifests in the work object as failed") +} + +func removePlacementBinding(bindingNS, bindingName string) { + binding := newPlacementBindingObj(bindingNS, bindingName) + Expect(hubClient.Delete(ctx, binding)).To(Succeed(), "Failed to delete the placement binding") +} + +// updatePlacementBindingSuspended suspends or resumes a placement binding. +func updatePlacementBindingSuspended(bindingNS, bindingName string, suspended bool) { + Eventually(func() error { + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + binding.GetSpec().Suspended = suspended + if err := hubClient.Update(ctx, binding); err != nil { + return fmt.Errorf("failed to update the placement binding: %w", err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the suspended flag of the placement binding") +} + +// updatePlacementBindingSyncStrategy sets (or, if nil, unsets) the sync strategy of a placement binding. +func updatePlacementBindingSyncStrategy(bindingNS, bindingName string, syncStrategy *placementv1alpha1.SyncStrategy) { + Eventually(func() error { + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + binding.GetSpec().SyncStrategy = syncStrategy.DeepCopy() + if err := hubClient.Update(ctx, binding); err != nil { + return fmt.Errorf("failed to update the placement binding: %w", err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the sync strategy of the placement binding") +} + +// updatePlacementBindingSpec updates the spec of a placement binding with the given mutation in one write. +func updatePlacementBindingSpec(bindingNS, bindingName string, mutate func(spec *placementv1alpha1.PlacementBindingSpec)) { + Eventually(func() error { + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + mutate(binding.GetSpec()) + if err := hubClient.Update(ctx, binding); err != nil { + return fmt.Errorf("failed to update the placement binding: %w", err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the spec of the placement binding") +} + +// placementBindingFinalizerRemovedActual verifies that the cleanup finalizer is absent from a placement binding. +func placementBindingFinalizerRemovedActual(bindingNS, bindingName string) func() error { + return func() error { + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + if controllerutil.ContainsFinalizer(binding, workGeneratorCleanupFinalizer) { + return fmt.Errorf("cleanup finalizer is still present") + } + return nil + } +} + +// placementBindingLastProcessedSnapshotUnsetActual verifies that a placement binding has no last processed +// placement resource snapshot name in its status. +func placementBindingLastProcessedSnapshotUnsetActual(bindingNS, bindingName string) func() error { + return func() error { + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + if lastProcessed := binding.GetStatus().LastProcessedResourceSnapshotName; lastProcessed != nil { + return fmt.Errorf("last processed resource snapshot name is set to %q", *lastProcessed) + } + return nil + } +} + +// deleteWork deletes a work object; it is not an error if the work object is already gone. +func deleteWork(workNS, workName string) { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workName, + }, + } + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, work))).To(Succeed(), "Failed to delete the work object") +} + +// markWorkAsApplied updates the status of a work object to mark it (and all of its manifests) as applied, as the +// KubeFleet member agent would do before it completes the availability checks; no Available conditions are set. +func markWorkAsApplied(workNS, workName string, manifestIdentifiers []placementv1alpha1.ManifestIdentifier) { + // Retrieve the work object. + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work)).To(Succeed(), "Failed to retrieve the work object") + + now := metav1.Now() + manifestStatuses := make([]placementv1alpha1.PerManifestStatus, len(manifestIdentifiers)) + for idx := range manifestIdentifiers { + manifestStatuses[idx] = placementv1alpha1.PerManifestStatus{ + Identifier: manifestIdentifiers[idx], + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.ManifestCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: 1, + Reason: markedAsAppliedReason, + Message: markedAsAppliedManifestMsg, + LastTransitionTime: now, + }, + }, + } + } + work.Status = placementv1alpha1.WorkStatus{ + Conditions: []metav1.Condition{ + { + Type: placementv1alpha1.WorkCondTypeApplied, + Status: metav1.ConditionTrue, + ObservedGeneration: work.Generation, + Reason: markedAsAppliedReason, + Message: markedAsAppliedWorkMsg, + LastTransitionTime: now, + }, + }, + Manifests: manifestStatuses, + } + Expect(hubClient.Status().Update(ctx, work)).To(Succeed(), "Failed to mark the work object as applied") +} + +// touchWorkStatus updates the status of a work object without changing any processing result, by giving the +// work-level Applied condition a new message; the update still counts as a status change (i.e., it is an event the +// work generator reacts to). +func touchWorkStatus(workNS, workName string) { + Eventually(func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work); err != nil { + return fmt.Errorf("failed to retrieve the work object: %w", err) + } + + appliedCond := meta.FindStatusCondition(work.Status.Conditions, placementv1alpha1.WorkCondTypeApplied) + if appliedCond == nil { + return fmt.Errorf("the work object has no Applied condition") + } + appliedCond.Message = fmt.Sprintf("Work has been marked as applied (touched: %s)", utils.RandStr()) + if err := hubClient.Status().Update(ctx, work); err != nil { + return fmt.Errorf("failed to update the work object status: %w", err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to touch the work object status") +} + +// setWorkAnnotation sets an annotation on a work object (a metadata-only change). +func setWorkAnnotation(workNS, workName, key, value string) { + Eventually(func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work); err != nil { + return fmt.Errorf("failed to retrieve the work object: %w", err) + } + + annotations := work.GetAnnotations() + if annotations == nil { + annotations = map[string]string{} + } + annotations[key] = value + work.SetAnnotations(annotations) + if err := hubClient.Update(ctx, work); err != nil { + return fmt.Errorf("failed to update the work object: %w", err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to set the annotation on the work object") +} + +// workAnnotationActual verifies that a work object has the given annotation value. +func workAnnotationActual(workNS, workName, key, wantValue string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work); err != nil { + return fmt.Errorf("failed to retrieve the work object: %w", err) + } + if got := work.GetAnnotations()[key]; got != wantValue { + return fmt.Errorf("work object annotation %s = %q, want %q", key, got, wantValue) + } + return nil + } +} + +// createReservedNamespace creates the reserved namespace of a member cluster on the hub cluster. +func createReservedNamespace(clusterName string) { + Expect(hubClient.Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf(utils.NamespaceNameFormat, clusterName)}, + })).Should(Succeed(), "Failed to create the reserved namespace for member cluster %s", clusterName) +} + +// placementBindingStatusInOrderActual is the same as placementBindingStatusUpdatedActual, except that the failed +// resources are compared in order (i.e., the order that the work generator reports them in is verified). +func placementBindingStatusInOrderActual(bindingNS, bindingName string, wantStatus *placementv1alpha1.PlacementBindingStatus) func() error { + return func() error { + binding, err := getPlacementBinding(bindingNS, bindingName) + if err != nil { + return err + } + + wantStatusWithObsGen := wantStatus.DeepCopy() + for idx := range wantStatusWithObsGen.Conditions { + wantStatusWithObsGen.Conditions[idx].ObservedGeneration = binding.GetGeneration() + } + if diff := cmp.Diff(*binding.GetStatus(), *wantStatusWithObsGen, ignoreFieldConditionLTTMsg); diff != "" { + return fmt.Errorf("placement binding status diff (-got +want):\n%s", diff) + } + return nil + } +} + +// createForeignWork creates a work object with arbitrary labels and annotations, i.e., one that is not +// generated by the work generator. +func createForeignWork(workNS, workName string, labels, annotations map[string]string) { + work := &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workName, + Labels: labels, + Annotations: annotations, + }, + } + Expect(hubClient.Create(ctx, work)).To(Succeed(), "Failed to create the work object") +} + +// placementResourceSnapshotRemovedActual deletes a placement resource snapshot, and verifies that it is gone. +func placementResourceSnapshotRemovedActual(snapshotNS, snapshotName string) func() error { + return func() error { + snapshot := newPlacementResourceSnapshotObj(snapshotNS, snapshotName) + if err := hubClient.Delete(ctx, snapshot); err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("failed to delete the placement resource snapshot: %w", err) + } + + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: snapshotNS, Name: snapshotName}, snapshot); !apierrors.IsNotFound(err) { + return fmt.Errorf("placement resource snapshot still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func placementBindingRemovedActual(bindingNS, bindingName string) func() error { + return func() error { + binding := newPlacementBindingObj(bindingNS, bindingName) + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: bindingNS, Name: bindingName}, binding); !apierrors.IsNotFound(err) { + return fmt.Errorf("placement binding still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +func workRemovedActual(workNS, workName string) func() error { + return func() error { + work := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: workNS, Name: workName}, work); !apierrors.IsNotFound(err) { + return fmt.Errorf("work object still exists or an unexpected error occurred: %w", err) + } + return nil + } +} + +// scope describes the scope (namespace-scoped or cluster-scoped) of the placement bindings (and their placement +// resource snapshots) that a test runs against. +type scope struct { + name string + bindingNS string + bindingNameTemplate string + policyNameTemplate string +} + +var scopes = []scope{ + { + name: "namespace-scoped", + bindingNS: appNamespaceName, + bindingNameTemplate: placementBindingNameTemplate, + policyNameTemplate: placementPolicyNameTemplate, + }, + { + name: "cluster-scoped", + bindingNS: clusterScopedNS, + bindingNameTemplate: clusterPlacementBindingNameTemplate, + policyNameTemplate: clusterPlacementPolicyNameTemplate, + }, +} + +// Every placement resource snapshot that the binding fixture creates carries a NS, a Deployment, and a ConfigMap. +const resourcesPerSnapshot = 3 + +// snapshotSet is a set of placement resource snapshots that share the same index; the first one is the +// primary snapshot (sub-index 0). +type snapshotSet struct { + index int + names []string + resources [][]placementv1alpha1.SnapshottedResource + snapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + identifiers [][]placementv1alpha1.ManifestIdentifier +} + +func (s *snapshotSet) primaryName() string { + return s.names[0] +} + +// resourceCount returns the number of resources across all the snapshots in the set. +func (s *snapshotSet) resourceCount() int32 { + total := 0 + for idx := range s.identifiers { + total += len(s.identifiers[idx]) + } + return int32(total) //nolint:gosec // safe: the counts are tiny. +} + +// bindingFixture bundles the objects and the expectations of a placement binding bound to a member cluster. +// Its methods that touch the API server must be called at spec run time (not at tree construction time). +type bindingFixture struct { + scope scope + clusterName string + bindingName string + policyName string + workNS string + + // The snapshot sets created (or prepared) so far, keyed by snapshot index. + sets map[int]*snapshotSet + // The names of the stray snapshots created so far; see createStraySnapshot. + strayNames []string + // The UIDs of the work objects, keyed by work name; recorded by tests that check if work objects are + // updated in place or re-created. + workUIDs map[string]types.UID + // The status and the resource version of the binding; recorded by tests that check if the binding is left + // untouched. + status *placementv1alpha1.PlacementBindingStatus + resourceVersion string +} + +func newBindingFixture(sc scope, clusterName string) *bindingFixture { + return newBindingFixtureWithNames(sc, clusterName, + fmt.Sprintf(sc.bindingNameTemplate, utils.RandStr()), fmt.Sprintf(sc.policyNameTemplate, utils.RandStr())) +} + +// newBindingFixtureWithNames is the same as newBindingFixture, except that it uses the given binding and placement +// policy names. +func newBindingFixtureWithNames(sc scope, clusterName, bindingName, policyName string) *bindingFixture { + return &bindingFixture{ + scope: sc, + clusterName: clusterName, + bindingName: bindingName, + policyName: policyName, + workNS: fmt.Sprintf(utils.NamespaceNameFormat, clusterName), + sets: map[int]*snapshotSet{}, + } +} + +// newFixtureOfSamePolicy returns a fixture for another binding of the same placement policy, bound to the given member +// cluster. The two fixtures share their snapshot sets, i.e., snapshots created via one are visible to the other. +func (f *bindingFixture) newFixtureOfSamePolicy(clusterName string) *bindingFixture { + other := newBindingFixtureWithNames(f.scope, clusterName, fmt.Sprintf(f.scope.bindingNameTemplate, utils.RandStr()), f.policyName) + other.sets = f.sets + return other +} + +// createSnapshotWithResources creates a single placement resource snapshot of the given index with the given resources +// (and their expected manifest identifiers), as the only snapshot of its index. +func (f *bindingFixture) createSnapshotWithResources( + index int, + resources []placementv1alpha1.SnapshottedResource, + identifiers []placementv1alpha1.ManifestIdentifier, +) *snapshotSet { + set := &snapshotSet{ + index: index, + names: []string{fmt.Sprintf(placementResourceSnapshotNameTemplate, f.policyName, index)}, + resources: [][]placementv1alpha1.SnapshottedResource{resources}, + snapshots: make([]placementv1alpha1.PlacementResourceSnapshotAccessor, 1), + identifiers: [][]placementv1alpha1.ManifestIdentifier{identifiers}, + } + f.sets[index] = set + f.createSnapshotsInSet(set) + return set +} + +// createSnapshots creates `count` placement resource snapshots of the given index, each with its own NS, Deployment, +// and ConfigMap. +func (f *bindingFixture) createSnapshots(index, count int) *snapshotSet { + set := f.newSnapshotSet(index, count) + f.createSnapshotsInSet(set) + return set +} + +// newSnapshotSet prepares (but does not create) `count` placement resource snapshots of the given index, each with its +// own NS, Deployment, and ConfigMap; use createSnapshotsInSet to create them. +func (f *bindingFixture) newSnapshotSet(index, count int) *snapshotSet { + set := &snapshotSet{ + index: index, + names: make([]string, count), + resources: make([][]placementv1alpha1.SnapshottedResource, count), + snapshots: make([]placementv1alpha1.PlacementResourceSnapshotAccessor, count), + identifiers: make([][]placementv1alpha1.ManifestIdentifier, count), + } + suffix := utils.RandStr() + for subIdx := 0; subIdx < count; subIdx++ { + if subIdx == 0 { + set.names[subIdx] = fmt.Sprintf(placementResourceSnapshotNameTemplate, f.policyName, index) + } else { + set.names[subIdx] = fmt.Sprintf(subIndexedPlacementResourceSnapshotNameTemplate, f.policyName, index, subIdx) + } + + nsName := fmt.Sprintf("ns-%s-%d", suffix, subIdx) + deployName := fmt.Sprintf(deployNameTemplate, subIdx) + configMapName := fmt.Sprintf(configMapNameTemplate, subIdx) + set.resources[subIdx] = []placementv1alpha1.SnapshottedResource{ + nsSnapshottedResource(nsName), + deploySnapshottedResource(nsName, deployName), + configMapSnapshottedResource(nsName, configMapName), + } + set.identifiers[subIdx] = []placementv1alpha1.ManifestIdentifier{ + nsManifestIdentifier(0, nsName), + deployManifestIdentifier(1, nsName, deployName), + configMapManifestIdentifier(2, nsName, configMapName), + } + } + f.sets[index] = set + return set +} + +// createSnapshotsInSet creates the placement resource snapshots of the given sub-indices (all of them if none is +// given) in a prepared snapshot set. The snapshots are created in descending order of their sub-indices, so that the +// primary snapshot comes last, as the placement resource snapshot manager would do. It can also re-create snapshots +// that have been deleted. +func (f *bindingFixture) createSnapshotsInSet(set *snapshotSet, subIdxs ...int) { + if len(subIdxs) == 0 { + for subIdx := range set.names { + subIdxs = append(subIdxs, subIdx) + } + } + slices.Sort(subIdxs) + slices.Reverse(subIdxs) + for _, subIdx := range subIdxs { + set.snapshots[subIdx] = createSubIndexedPlacementResourceSnapshot(f.scope.bindingNS, set.names[subIdx], f.policyName, + set.index, subIdx, len(set.names), set.resources[subIdx]...) + } +} + +// deleteSnapshot deletes a placement resource snapshot of a snapshot set, and waits until the deletion is observed +// in the cache. +func (f *bindingFixture) deleteSnapshot(setIdx, subIdx int) { + snapshotName := f.sets[setIdx].names[subIdx] + Eventually(placementResourceSnapshotRemovedActual(f.scope.bindingNS, snapshotName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") +} + +// createStraySnapshot creates a stray placement resource snapshot of the given index and sub-index under a name +// different from the ones the fixture uses, e.g., one left over from a previous placement policy of the same name. +// If it is a primary snapshot (sub-index 0), it carries the given count label. It returns the name of the snapshot. +func (f *bindingFixture) createStraySnapshot(index, subIdx, count int, contentsHash string) string { + snapshotName := fmt.Sprintf(straySnapshotNameTemplate, f.policyName, index, subIdx, utils.RandStr()) + createSubIndexedPlacementResourceSnapshotWithHash(f.scope.bindingNS, snapshotName, f.policyName, index, subIdx, count, contentsHash, + nsSnapshottedResource(fmt.Sprintf(nsNameTemplate, utils.RandStr()))) + f.strayNames = append(f.strayNames, snapshotName) + return snapshotName +} + +// deleteStraySnapshot deletes a stray placement resource snapshot, and waits until the deletion is observed in the +// cache. +func (f *bindingFixture) deleteStraySnapshot(snapshotName string) { + Eventually(placementResourceSnapshotRemovedActual(f.scope.bindingNS, snapshotName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the stray placement resource snapshot") +} + +// createBinding creates the placement binding, bound to the primary snapshot of the given snapshot set. +func (f *bindingFixture) createBinding(setIdx int) { + createPlacementBinding(f.scope.bindingNS, f.bindingName, f.policyName, f.clusterName, f.sets[setIdx].primaryName()) +} + +// workName returns the name of the work object derived from the snapshot with the given sub-index. +func (f *bindingFixture) workName(subIdx int) string { + binding := newPlacementBindingObj(f.scope.bindingNS, f.bindingName) + binding.SetSpec(placementv1alpha1.PlacementBindingSpec{PlacementPolicyName: f.policyName}) + return uniqueNameForWorkDerivedFromPlacementResourceSnapshot(binding, subIdx == 0, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: strconv.Itoa(subIdx)}) +} + +// worksMatchActual verifies that the work objects of the binding are exactly those derived from the given snapshot +// set, with the given sync strategy. +func (f *bindingFixture) worksMatchActual(setIdx int, syncStrategy *placementv1alpha1.SyncStrategy) func() error { + return func() error { + set := f.sets[setIdx] + + // The secondary work objects should be owned by the primary work object; retrieve its current UID as it + // changes whenever the primary work object is re-created. + primaryWork := &placementv1alpha1.Work{} + if err := hubClient.Get(ctx, client.ObjectKey{Namespace: f.workNS, Name: f.workName(0)}, primaryWork); err != nil { + return fmt.Errorf("failed to retrieve the primary work object: %w", err) + } + ownerRef := &metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: workKind, + Name: primaryWork.Name, + UID: primaryWork.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + } + + wantWorks := make([]placementv1alpha1.Work, len(set.names)) + for subIdx := range set.names { + wantWorks[subIdx] = wantWorkForPlacementResourceSnapshot(f.workNS, f.workName(subIdx), f.scope.bindingNS, f.bindingName, f.policyName, + set.primaryName(), subIdx, len(set.names), ownerRef, set.snapshots[subIdx]) + wantWorks[subIdx].Spec.SyncStrategy = syncStrategy.DeepCopy() + } + return worksGeneratedActual(f.clusterName, f.scope.bindingNS, f.bindingName, wantWorks)() + } +} + +// noWorksActual verifies that there is no work object for the binding. +func (f *bindingFixture) noWorksActual() func() error { + return worksGeneratedActual(f.clusterName, f.scope.bindingNS, f.bindingName, nil) +} + +func (f *bindingFixture) waitingStatusActual(setIdx int) func() error { + set := f.sets[setIdx] + return placementBindingStatusUpdatedActual(f.scope.bindingNS, f.bindingName, + wantPlacementBindingStatusWaitingForSync(set.resourceCount(), set.primaryName())) +} + +func (f *bindingFixture) syncedStatusActual(setIdx int) func() error { + set := f.sets[setIdx] + return placementBindingStatusUpdatedActual(f.scope.bindingNS, f.bindingName, + wantPlacementBindingStatusSyncedAndAvailable(set.resourceCount(), set.primaryName())) +} + +// markWorks marks the work objects with the given sub-indices (all of them if none is given) as applied and +// available. +func (f *bindingFixture) markWorks(setIdx int, subIdxs ...int) { + set := f.sets[setIdx] + if len(subIdxs) == 0 { + for subIdx := range set.names { + subIdxs = append(subIdxs, subIdx) + } + } + for _, subIdx := range subIdxs { + markWorkAsAppliedAndAvailable(f.workNS, f.workName(subIdx), set.identifiers[subIdx]) + } +} + +// listWorkUIDs returns the UIDs of the work objects of the binding, keyed by work name. +func (f *bindingFixture) listWorkUIDs() (map[string]types.UID, error) { + workList := &placementv1alpha1.WorkList{} + if err := hubClient.List(ctx, workList, + client.InNamespace(f.workNS), + client.MatchingLabels{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: f.scope.bindingNS, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: workOwnerLabelValue(f.bindingName), + }, + ); err != nil { + return nil, fmt.Errorf("failed to list work objects: %w", err) + } + uids := make(map[string]types.UID, len(workList.Items)) + for idx := range workList.Items { + uids[workList.Items[idx].Name] = workList.Items[idx].UID + } + return uids, nil +} + +// recordWorkUIDs records the current UIDs of the work objects of the binding. +func (f *bindingFixture) recordWorkUIDs() { + uids, err := f.listWorkUIDs() + Expect(err).To(Succeed(), "Failed to list the work objects") + f.workUIDs = uids +} + +// recordStatus records the current status and resource version of the binding. +func (f *bindingFixture) recordStatus() { + binding, err := getPlacementBinding(f.scope.bindingNS, f.bindingName) + Expect(err).To(Succeed(), "Failed to retrieve the placement binding") + f.status = binding.GetStatus().DeepCopy() + f.resourceVersion = binding.GetResourceVersion() +} + +// statusUnchangedActual verifies that the status of the binding matches the one last recorded, including the +// observed generations of the conditions. +func (f *bindingFixture) statusUnchangedActual() func() error { + return func() error { + return placementBindingStatusMatchesActual(f.scope.bindingNS, f.bindingName, f.status)() + } +} + +// resourceVersionUnchangedActual verifies that the binding has the resource version last recorded, i.e., that it +// has not been written to (spec or status) since. +func (f *bindingFixture) resourceVersionUnchangedActual() func() error { + return func() error { + binding, err := getPlacementBinding(f.scope.bindingNS, f.bindingName) + if err != nil { + return err + } + if binding.GetResourceVersion() != f.resourceVersion { + return fmt.Errorf("placement binding resource version = %s, want %s", binding.GetResourceVersion(), f.resourceVersion) + } + return nil + } +} + +// invalidSnapshotsStatusActual verifies the given status, preserving explicitly supplied observed generations, and +// checks that the Synchronized condition reports invalid placement resource snapshots with the given details. +func (f *bindingFixture) invalidSnapshotsStatusActual(wantStatus *placementv1alpha1.PlacementBindingStatus, wantMsgSubstr string) func() error { + return func() error { + binding, err := getPlacementBinding(f.scope.bindingNS, f.bindingName) + if err != nil { + return err + } + wantStatusWithObsGen := wantStatus.DeepCopy() + for idx := range wantStatusWithObsGen.Conditions { + if wantStatusWithObsGen.Conditions[idx].ObservedGeneration == 0 { + wantStatusWithObsGen.Conditions[idx].ObservedGeneration = binding.GetGeneration() + } + } + if err := placementBindingStatusMatchesActual(f.scope.bindingNS, f.bindingName, wantStatusWithObsGen)(); err != nil { + return err + } + if err := placementBindingSyncCondMessageActual(f.scope.bindingNS, f.bindingName, invalidSnapshotsMsgPrefix)(); err != nil { + return err + } + return placementBindingSyncCondMessageActual(f.scope.bindingNS, f.bindingName, wantMsgSubstr)() + } +} + +// worksMatchBindingStrategyActual is the same as worksMatchActual, except that the work objects are expected to carry +// the sync strategy of the binding as written to the API server (which might carry defaulted fields). +func (f *bindingFixture) worksMatchBindingStrategyActual(setIdx int) func() error { + return func() error { + binding, err := getPlacementBinding(f.scope.bindingNS, f.bindingName) + if err != nil { + return err + } + syncStrategy := binding.GetSpec().SyncStrategy + if syncStrategy == nil { + return fmt.Errorf("the sync strategy has not been set on the placement binding") + } + return f.worksMatchActual(setIdx, syncStrategy)() + } +} + +// expectWorksNotRecreated verifies that the work objects recorded earlier still exist with the same UIDs. +func (f *bindingFixture) expectWorksNotRecreated() { + uids, err := f.listWorkUIDs() + Expect(err).To(Succeed(), "Failed to list the work objects") + for workName, wantUID := range f.workUIDs { + if uids[workName] != wantUID { + Fail(fmt.Sprintf("work object %s has been re-created or removed (UID %s, want %s)", workName, uids[workName], wantUID)) + } + } +} + +// updateSpec updates the spec of the binding with the given mutation in one write. +func (f *bindingFixture) updateSpec(mutate func(spec *placementv1alpha1.PlacementBindingSpec)) { + updatePlacementBindingSpec(f.scope.bindingNS, f.bindingName, mutate) +} + +// registerWorkGenerationSpecs registers the specs that generate the work objects from the given snapshot set and bring +// the binding to the waiting-for-synchronization state. +func (f *bindingFixture) registerWorkGenerationSpecs(setIdx int) { + It("should add cleanup finalizer to the placement binding", func() { + finalizerAddedActual := placementBindingFinalizerAddedActual(f.scope.bindingNS, f.bindingName) + Eventually(finalizerAddedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add cleanup finalizer to the placement binding") + }) + + It("should generate the work objects", func() { + Eventually(f.worksMatchActual(setIdx, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + }) + + It("should refresh the placement binding status", func() { + Eventually(f.waitingStatusActual(setIdx), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + }) +} + +// registerBaselineSpecs registers the specs that generate the work objects from the given snapshot set and bring the +// binding to the synchronized and available state. +func (f *bindingFixture) registerBaselineSpecs(setIdx int) { + f.registerWorkGenerationSpecs(setIdx) + + It("can mark the work objects as applied and available", func() { + f.markWorks(setIdx) + }) + + It("should refresh the placement binding status", func() { + Eventually(f.syncedStatusActual(setIdx), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + f.recordStatus() + }) +} + +// markWorksAsApplied marks all the work objects of the given snapshot set as applied (but not yet checked for +// availability). +func (f *bindingFixture) markWorksAsApplied(setIdx int) { + set := f.sets[setIdx] + for subIdx := range set.names { + markWorkAsApplied(f.workNS, f.workName(subIdx), set.identifiers[subIdx]) + } +} + +// workGeneration returns the generation of the work object derived from the snapshot with the given sub-index. +func (f *bindingFixture) workGeneration(subIdx int) int64 { + work := &placementv1alpha1.Work{} + Expect(hubClient.Get(ctx, client.ObjectKey{Namespace: f.workNS, Name: f.workName(subIdx)}, work)).To(Succeed(), "Failed to retrieve the work object") + return work.Generation +} + +// syncCondTransitionedSinceRecordActual verifies that the Synchronized condition of the binding has transitioned +// (i.e., has a later last transition time) since the status was last recorded. +func (f *bindingFixture) syncCondTransitionedSinceRecordActual() func() error { + return func() error { + recordedCond := meta.FindStatusCondition(f.status.Conditions, placementv1alpha1.PlacementBindingCondTypeSynchronized) + if recordedCond == nil { + return fmt.Errorf("the recorded status has no Synchronized condition") + } + + binding, err := getPlacementBinding(f.scope.bindingNS, f.bindingName) + if err != nil { + return err + } + syncCond := meta.FindStatusCondition(binding.GetStatus().Conditions, placementv1alpha1.PlacementBindingCondTypeSynchronized) + if syncCond == nil { + return fmt.Errorf("the Synchronized condition is not found on the placement binding") + } + if !syncCond.LastTransitionTime.After(recordedCond.LastTransitionTime.Time) { + return fmt.Errorf("the Synchronized condition last transitioned at %v, want a time after %v", + syncCond.LastTransitionTime, recordedCond.LastTransitionTime) + } + return nil + } +} + +// syncWorks generates the work objects from the given snapshot set and brings the binding to the synchronized and +// available state in one go; it is for bindings that are not the focus of a test. +func (f *bindingFixture) syncWorks(setIdx int) { + Eventually(f.worksMatchActual(setIdx, nil), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the work objects") + Eventually(f.waitingStatusActual(setIdx), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") + f.markWorks(setIdx) + Eventually(f.syncedStatusActual(setIdx), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to refresh the placement binding status") +} + +// teardown removes the binding, its leftover work objects, and the snapshots. +func (f *bindingFixture) teardown() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, newPlacementBindingObj(f.scope.bindingNS, f.bindingName)))).To(Succeed(), "Failed to delete the placement binding") + Eventually(placementBindingRemovedActual(f.scope.bindingNS, f.bindingName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement binding") + + // The work generator deletes only the primary work object upon cleanup, and the secondary work objects are left to + // K8s' garbage collection, which does not run in the test environment; delete any leftover explicitly. + uids, err := f.listWorkUIDs() + Expect(err).To(Succeed(), "Failed to list the work objects") + for workName := range uids { + deleteWork(f.workNS, workName) + Eventually(workRemovedActual(f.workNS, workName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the work object") + } + + for _, set := range f.sets { + for _, snapshotName := range set.names { + Eventually(placementResourceSnapshotRemovedActual(f.scope.bindingNS, snapshotName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the placement resource snapshot") + } + } + for _, snapshotName := range f.strayNames { + Eventually(placementResourceSnapshotRemovedActual(f.scope.bindingNS, snapshotName), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the stray placement resource snapshot") + } +} diff --git a/pkg/v1/controllers/workgenerator/works.go b/pkg/v1/controllers/workgenerator/works.go index fe9455b9f..64a1701f1 100644 --- a/pkg/v1/controllers/workgenerator/works.go +++ b/pkg/v1/controllers/workgenerator/works.go @@ -30,6 +30,7 @@ import ( "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" @@ -51,10 +52,37 @@ var ( } ) -// areWorksUpToDate checks if the work objects for a placement binding are up-to-date, i.e., if all the work objects -// needed given the current placement binding spec have been created/updated. If so, the work generation can skip -// the work object create/update ops and skip to refreshing the placement binding status. -func areWorksUpToDate(placementBinding placementv1alpha1.PlacementBindingAccessor, works []placementv1alpha1.Work) (bool, error) { +// workSyncStatus represents the synchronization status of the work objects for a placement binding. +type workSyncStatus string + +const ( + // workSyncStatusInSync signals that all the work objects for the placement binding are in sync, i.e., the last + // processed placement resource snapshot matches the primary snapshot in the placement binding spec, all the work + // objects are linked to that primary snapshot, the number of work objects matches the linked work count, and the + // work objects have the sync strategy dictated by the placement binding. + workSyncStatusInSync workSyncStatus = "InSync" + // workSyncStatusResourcesOutOfSync signals that the work objects for the placement binding need to be + // (re)generated from the placement resource snapshots, i.e., the last processed placement resource snapshot differs + // from the primary snapshot in the placement binding spec, no work objects are found, some work objects are not + // linked to the expected primary placement resource snapshot, or the number of work objects does not match the + // linked work count. + workSyncStatusResourcesOutOfSync workSyncStatus = "ResourcesOutOfSync" + // workSyncStatusStrategyOutOfSync signals that the work objects for the placement binding are in sync resource-wise, + // but some of them have a sync strategy different from the one dictated by the placement binding; only the sync + // strategy needs to be updated, and the placement resource snapshots do not need to be read. + workSyncStatusStrategyOutOfSync workSyncStatus = "StrategyOutOfSync" + // workSyncStatusErred signals that there was an error while checking the synchronization status of the work objects. + // This is a value added for completeness reasons only; it is not checked. + workSyncStatusErred workSyncStatus = "Erred" +) + +// areWorksInSync checks if the work objects for a placement binding are in sync with the current placement binding +// spec, i.e., if all the work objects needed have been created/updated. It returns workSyncStatusInSync if so, in +// which case the work generation can skip the work object create/update ops and go straight to refreshing the +// placement binding status; workSyncStatusResourcesOutOfSync if the work objects need to be (re)generated; +// workSyncStatusStrategyOutOfSync if only the sync strategy of some work objects needs to be updated; or +// workSyncStatusErred (along with an error) if the work objects are in an unexpected state. +func areWorksInSync(placementBinding placementv1alpha1.PlacementBindingAccessor, works []placementv1alpha1.Work) (workSyncStatus, error) { // Check if the last processed placement resource snapshot name in the placement binding status matches the // primary placement resource snapshot name in the placement binding spec. If so, it ensures that all the needed // work objects have been created/updated for the placement binding. @@ -65,26 +93,39 @@ func areWorksUpToDate(placementBinding placementv1alpha1.PlacementBindingAccesso primarySnapshotName := placementBinding.GetSpec().ResourceSnapshotName if lastProcessedSnapshotName != primarySnapshotName { - return false, nil + klog.V(2).InfoS("Last processed placement resource snapshot does not match the primary snapshot in the placement binding spec", + "placementBinding", klog.KObj(placementBinding), + "lastProcessedSnapshotName", lastProcessedSnapshotName, "observedPrimarySnapshotName", primarySnapshotName, + "controller", controllerName) + return workSyncStatusResourcesOutOfSync, nil } // Do some sanity checks, just to make sure that the cache is up to date. if len(works) == 0 { - // No work objects exist for the placement binding. The cache might not have caught up yet. - return false, errors.NewTransientError(nil, "no work objects are found; the cache might be stale") + // No work objects exist for the placement binding. It might be that the cache is stale, or the work objects + // have been tampered with (deleted). Either way, report the works as out of sync; the work generator will + // (re)create them, and a stale cache will self-correct through retries. + klog.V(2).InfoS("No work objects are found for the placement binding; the cache might be stale or the work objects might be tampered with", + "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBinding.GetSpec().ClusterName, + "controller", controllerName) + return workSyncStatusResourcesOutOfSync, nil } // Check if all the work objects have been linked to the expected placement resource snapshot (in the spec), // and verify that the linked work count recorded on the primary work matches the number of listed works. linkedWorkCount := -1 + foundResOutOfSyncWorks := false for idx := range works { work := &works[idx] annotations := work.GetAnnotations() if linked := annotations[placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey]; linked != primarySnapshotName { // The work object is linked to a different placement resource snapshot than the one in the placement binding spec. // This might happen if the cache is stale. - return false, errors.NewTransientError(nil, "found a work object that is not linked to the expected primary placement resource snapshot", - "work", klog.KObj(work), "linkedPlacementResourceSnapshotName", linked, "expectedPlacementResourceSnapshotName", primarySnapshotName) + klog.V(2).InfoS("Found a work object that is not linked to the expected primary placement resource snapshot", + "work", klog.KObj(work), "placementBinding", klog.KObj(placementBinding), + "linkedPlacementResourceSnapshotName", linked, "expectedPlacementResourceSnapshotName", primarySnapshotName) + foundResOutOfSyncWorks = true + continue } linkedWorkCountStr, found := annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] @@ -92,21 +133,28 @@ func areWorksUpToDate(placementBinding placementv1alpha1.PlacementBindingAccesso continue } if linkedWorkCount != -1 { - // At any time there should be exactly one primary placement resource snapshot that has the + // At any time there should be exactly one primary work object that has the // linked work count annotation. - return false, errors.NewUnexpectedError(nil, "multiple primary placement resource snapshots have the linked work count annotation", + return workSyncStatusErred, errors.NewUnexpectedError(nil, "multiple work objects have the linked work count annotation", "work", klog.KObj(work), "linkedWorkCount", linkedWorkCountStr) } var err error linkedWorkCount, err = strconv.Atoi(linkedWorkCountStr) if err != nil || linkedWorkCount < 1 { - return false, errors.NewUnexpectedError(err, "invalid linked work count annotation on work object", + return workSyncStatusErred, errors.NewUnexpectedError(err, "invalid linked work count annotation on work object", "work", klog.KObj(work), "linkedWorkCount", linkedWorkCountStr) } } + + if foundResOutOfSyncWorks { + return workSyncStatusResourcesOutOfSync, nil + } if linkedWorkCount != len(works) { - return false, errors.NewTransientError(nil, "the number of work objects is not as expected", - "expectedWorkCount", linkedWorkCount, "actualWorkCount", len(works)) + klog.V(2).InfoS("The number of work objects is not as expected", + "expectedWorkCount", linkedWorkCount, "actualWorkCount", len(works), + "placementBinding", klog.KObj(placementBinding), "targetCluster", placementBinding.GetSpec().ClusterName, + "controller", controllerName) + return workSyncStatusResourcesOutOfSync, nil } // Check if the sync strategy of the placement binding still matches that on the work objects. @@ -114,19 +162,23 @@ func areWorksUpToDate(placementBinding placementv1alpha1.PlacementBindingAccesso for idx := range works { work := &works[idx] if !equality.Semantic.DeepEqual(work.Spec.SyncStrategy, syncStrategy) { - return false, nil + return workSyncStatusStrategyOutOfSync, nil } } - return true, nil + return workSyncStatusInSync, nil } +// refreshWorks creates, updates, and deletes the work objects for a placement binding as needed. Besides the +// created/updated work objects, it also reports whether any work object has been created or has had its spec +// changed; metadata-only changes (e.g., re-linking a work object to a new placement resource snapshot with +// the same content) are not counted. func (r *Reconciler) refreshWorks(ctx context.Context, placementBinding placementv1alpha1.PlacementBindingAccessor, sortedPlacementResourceSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, works []placementv1alpha1.Work, ) ([]*placementv1alpha1.Work, bool, error) { - writtenToStorage := false + workSpecsChanged := false worksToDelete := []*placementv1alpha1.Work{} // Build an index of work objects by their names. @@ -196,14 +248,14 @@ func (r *Reconciler) refreshWorks(ctx context.Context, // Create the primary work object first. This is needed as the controller needs its object UID to set // owner references on the secondary work objects. - createdOrUpdatedWorks, primaryWorkObjWrittenToStorage, err := r.createOrUpdateWorkObjects(ctx, + createdOrUpdatedWorks, primaryWorkSpecChanged, err := r.createOrUpdateWorkObjects(ctx, []*placementv1alpha1.Work{primaryWorkToCreateOrUpdate}, placementBinding) if err != nil { return nil, false, errors.Wraps(err, "failed to create or update work object for primary placement resource snapshot", "primaryPlacementResourceSnapshot", klog.KObj(primaryPlacementResourceSnapshot)) } ownerWorkObjRef := metav1.NewControllerRef(createdOrUpdatedWorks[0], workGVK) - writtenToStorage = primaryWorkObjWrittenToStorage + workSpecsChanged = primaryWorkSpecChanged // Set the owner reference on all secondary work objects. for idx := range additionalWorksToCreateOrUpdate { @@ -212,16 +264,16 @@ func (r *Reconciler) refreshWorks(ctx context.Context, } // Issue the create or update ops for the secondary work objects in parallel. - additionalCreatedOrUpdatedWorks, additionalCreatedOrUpdated, err := r.createOrUpdateWorkObjects(ctx, additionalWorksToCreateOrUpdate, placementBinding) + additionalCreatedOrUpdatedWorks, additionalWorkSpecsChanged, err := r.createOrUpdateWorkObjects(ctx, additionalWorksToCreateOrUpdate, placementBinding) if err != nil { return nil, false, errors.Wraps(err, "failed to create or update additional work objects for secondary placement resource snapshots") } createdOrUpdatedWorks = append(createdOrUpdatedWorks, additionalCreatedOrUpdatedWorks...) - if !writtenToStorage { - writtenToStorage = additionalCreatedOrUpdated + if !workSpecsChanged { + workSpecsChanged = additionalWorkSpecsChanged } - return createdOrUpdatedWorks, writtenToStorage, nil + return createdOrUpdatedWorks, workSpecsChanged, nil } func buildWorkObjectFor( @@ -321,7 +373,7 @@ func (r *Reconciler) createOrUpdateWorkObjects( createdOrUpdatedWorks := make([]*placementv1alpha1.Work, len(worksToCreateOrUpdate)) errFlag := parallelizer.NewErrorFlag() - createdOrUpdated := atomic.Bool{} + specsChanged := atomic.Bool{} r.parallelizer.ParallelizeUntil(childCtx, len(worksToCreateOrUpdate), func(idx int) { work := worksToCreateOrUpdate[idx] @@ -331,7 +383,15 @@ func (r *Reconciler) createOrUpdateWorkObjects( Name: work.GetName(), }, } + specChanged := false resOp, err := controllerutil.CreateOrUpdate(childCtx, r.hubClient, createdOrUpdatedWork, func() error { + // Check if the spec of the work object has changed; this is used to track if the controller + // here performs a metadata-only change, which can happen upon same-content rollouts. + // + // Note that this is different from the comparison automatically performed by + // the CreateOrUpdate method. + specChanged = !equality.Semantic.DeepEqual(createdOrUpdatedWork.Spec, work.Spec) + // Work objects are considered to be fully internal KubeFleet resources; for this reason // here the control loop chooses to overwrite the spec, labels, annotations, and owner references of // the work object with the latest values instead of attempting to do a merge. @@ -350,9 +410,9 @@ func (r *Reconciler) createOrUpdateWorkObjects( } createdOrUpdatedWorks[idx] = createdOrUpdatedWork - if resOp != controllerutil.OperationResultNone { - // The work object has been created or updated. - createdOrUpdated.CompareAndSwap(false, true) + if specChanged { + // The work object has been created or has its spec updated. Metadata-only changes are ignored. + specsChanged.CompareAndSwap(false, true) } klog.V(2).InfoS("Successfully created or updated work object", "work", klog.KObj(createdOrUpdatedWork), "resOp", resOp, @@ -361,7 +421,7 @@ func (r *Reconciler) createOrUpdateWorkObjects( if err := errFlag.Lower(); err != nil { return nil, false, err } - return createdOrUpdatedWorks, createdOrUpdated.Load(), nil + return createdOrUpdatedWorks, specsChanged.Load(), nil } func (r *Reconciler) deleteWorkObjects( @@ -402,3 +462,38 @@ func workOwnerLabelValue(ownerName string) string { } return fmt.Sprintf("%s-%s", name, hash) } + +// refreshSyncStrategyOnlyForWorks patches only the sync strategy of out-of-sync work objects in parallel. +func (r *Reconciler) refreshSyncStrategyOnlyForWorks( + ctx context.Context, + placementBinding placementv1alpha1.PlacementBindingAccessor, + works []placementv1alpha1.Work, +) error { + childCtx, childCancel := context.WithCancel(ctx) + defer childCancel() + + syncStrategy := placementBinding.GetSpec().SyncStrategy + errFlag := parallelizer.NewErrorFlag() + r.parallelizer.ParallelizeUntil(childCtx, len(works), func(idx int) { + work := &works[idx] + if equality.Semantic.DeepEqual(work.Spec.SyncStrategy, syncStrategy) { + return + } + + patchedWork := work.DeepCopy() + patchedWork.Spec.SyncStrategy = syncStrategy.DeepCopy() + if err := r.hubClient.Patch(childCtx, patchedWork, client.MergeFromWithOptions(work, client.MergeFromWithOptimisticLock{})); err != nil { + wrappedErr := errors.NewAPIServerError(err, "failed to update sync strategy of work object", false, + "work", klog.KObj(work), "placementBinding", klog.KObj(placementBinding)) + errFlag.Raise(wrappedErr) + childCancel() + return + } + klog.V(2).InfoS("Successfully patched sync strategy of work object", + "work", klog.KObj(patchedWork), "placementBinding", klog.KObj(placementBinding)) + }, "refreshSyncStrategyOnlyForWorks") + if err := errFlag.Lower(); err != nil { + return err + } + return ctx.Err() +} diff --git a/pkg/v1/controllers/workgenerator/works_test.go b/pkg/v1/controllers/workgenerator/works_test.go new file mode 100644 index 000000000..24b2f2410 --- /dev/null +++ b/pkg/v1/controllers/workgenerator/works_test.go @@ -0,0 +1,741 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package workgenerator + +import ( + "context" + "fmt" + "slices" + "strings" + "sync" + "testing" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" +) + +// TestWorkOwnerLabelValue tests the workOwnerLabelValue function. +// +// The hashes below are the first 12 hex characters of the SHA-256 digests of the original (dotted) owner names, +// computed independently of the implementation. +func TestWorkOwnerLabelValue(t *testing.T) { + dottedOwnerName := dottedName(100, 3) + // The dot-free form of the name, truncated to 48 characters (61 - 12 - 1). + wantDottedOwnerNamePrefix := strings.ReplaceAll(dottedOwnerName, ".", "")[:48] + + testCases := []struct { + name string + ownerName string + want string + }{ + { + name: "short name (unchanged)", + ownerName: "binding-1", + want: "binding-1", + }, + { + name: "name exactly at the length limit (unchanged)", + ownerName: strings.Repeat("a", 61), + want: strings.Repeat("a", 61), + }, + { + name: "name one character over the length limit (truncated, with hash)", + ownerName: strings.Repeat("a", 62), + want: strings.Repeat("a", 48) + "-f506898cc7c2", + }, + { + name: "short name with dots (dots removed, with hash)", + ownerName: "my.binding", + want: "mybinding-d602fbfb398e", + }, + { + name: "long name with dots (dots removed, truncated, with hash)", + ownerName: dottedOwnerName, + want: wantDottedOwnerNamePrefix + "-f996857ab27a", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got := workOwnerLabelValue(tc.ownerName) + if got != tc.want { + t.Errorf("workOwnerLabelValue(%q) = %q, want %q", tc.ownerName, got, tc.want) + } + if errs := validation.IsValidLabelValue(got); len(errs) > 0 { + t.Errorf("workOwnerLabelValue(%q) = %q, which is not a valid label value: %v", tc.ownerName, got, errs) + } + }) + } +} + +// TestWorkOwnerLabelValueDistinct verifies that owner names that reduce to the same prefix still map to different +// label values, as the hash is computed over the original name. +func TestWorkOwnerLabelValueDistinct(t *testing.T) { + ownerNames := []string{ + "a.bc", + "ab.c", + "abc.", + strings.Repeat("a", 62), + strings.Repeat("a", 63), + strings.Repeat("a", 48) + "." + strings.Repeat("a", 20), + } + + seen := make(map[string]string, len(ownerNames)) + for _, ownerName := range ownerNames { + got := workOwnerLabelValue(ownerName) + if got != workOwnerLabelValue(ownerName) { + t.Errorf("workOwnerLabelValue(%q) is not deterministic", ownerName) + } + if other, found := seen[got]; found { + t.Errorf("workOwnerLabelValue(%q) = %q, which collides with the value for %q", ownerName, got, other) + } + seen[got] = ownerName + } +} + +// TestAreWorksInSync tests the areWorksInSync function. +func TestAreWorksInSync(t *testing.T) { + const ( + primarySnapshotName = "policy-0-snapshot" + staleSnapshotName = "policy-0-old-snapshot" + ) + serverSideApplyStrategy := &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodServerSideApply, + } + + // A work object linked to the given snapshot; the linked work count annotation is set only if the count is + // not empty (i.e., for the primary work object). + newWork := func(name, linkedSnapshotName, linkedWorkCount string, syncStrategy *placementv1alpha1.SyncStrategy) placementv1alpha1.Work { + annotations := map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: linkedSnapshotName, + } + if len(linkedWorkCount) > 0 { + annotations[placementv1alpha1.LinkedWorkCountAnnotationKey] = linkedWorkCount + } + return placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Annotations: annotations, + }, + Spec: placementv1alpha1.WorkSpec{ + SyncStrategy: syncStrategy, + }, + } + } + newBinding := func(lastProcessedSnapshotName *string, syncStrategy *placementv1alpha1.SyncStrategy) placementv1alpha1.PlacementBindingAccessor { + binding := &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "app", + Name: "binding", + }, + Spec: placementv1alpha1.PlacementBindingSpec{ + ClusterName: "member-1", + ResourceSnapshotName: primarySnapshotName, + SyncStrategy: syncStrategy, + }, + } + binding.Status.LastProcessedResourceSnapshotName = lastProcessedSnapshotName + return binding + } + + testCases := []struct { + name string + binding placementv1alpha1.PlacementBindingAccessor + works []placementv1alpha1.Work + wantStatus workSyncStatus + wantErr bool + }{ + { + name: "last processed snapshot is not set", + binding: newBinding(nil, nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "1", nil), + }, + wantStatus: workSyncStatusResourcesOutOfSync, + }, + { + name: "last processed snapshot differs from the primary snapshot", + binding: newBinding(ptr.To(staleSnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "1", nil), + }, + wantStatus: workSyncStatusResourcesOutOfSync, + }, + { + name: "no work objects", + binding: newBinding(ptr.To(primarySnapshotName), nil), + wantStatus: workSyncStatusResourcesOutOfSync, + }, + { + name: "a work object is linked to another snapshot", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "2", nil), + newWork("work-1", staleSnapshotName, "", nil), + }, + wantStatus: workSyncStatusResourcesOutOfSync, + }, + { + name: "a work object has no link annotation", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "2", nil), + {ObjectMeta: metav1.ObjectMeta{Name: "work-1"}}, + }, + wantStatus: workSyncStatusResourcesOutOfSync, + }, + { + name: "the primary work object is missing (no linked work count)", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-1", primarySnapshotName, "", nil), + }, + wantStatus: workSyncStatusResourcesOutOfSync, + }, + { + name: "fewer work objects than the linked work count", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "2", nil), + }, + wantStatus: workSyncStatusResourcesOutOfSync, + }, + { + name: "more work objects than the linked work count", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "1", nil), + newWork("work-1", primarySnapshotName, "", nil), + }, + wantStatus: workSyncStatusResourcesOutOfSync, + }, + { + name: "sync strategy is set on the binding but not on the work objects", + binding: newBinding(ptr.To(primarySnapshotName), serverSideApplyStrategy), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "1", nil), + }, + wantStatus: workSyncStatusStrategyOutOfSync, + }, + { + name: "sync strategy is unset on the binding but set on a secondary work object", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "2", nil), + newWork("work-1", primarySnapshotName, "", serverSideApplyStrategy), + }, + wantStatus: workSyncStatusStrategyOutOfSync, + }, + { + name: "multiple work objects have the linked work count annotation", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "2", nil), + newWork("work-1", primarySnapshotName, "2", nil), + }, + wantStatus: workSyncStatusErred, + wantErr: true, + }, + { + name: "linked work count annotation is not a number", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "two", nil), + }, + wantStatus: workSyncStatusErred, + wantErr: true, + }, + { + name: "linked work count annotation is not positive", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "0", nil), + }, + wantStatus: workSyncStatusErred, + wantErr: true, + }, + { + name: "in sync (single work object)", + binding: newBinding(ptr.To(primarySnapshotName), nil), + works: []placementv1alpha1.Work{ + newWork("work-0", primarySnapshotName, "1", nil), + }, + wantStatus: workSyncStatusInSync, + }, + { + name: "in sync (multiple work objects, with sync strategy)", + binding: newBinding(ptr.To(primarySnapshotName), serverSideApplyStrategy), + works: []placementv1alpha1.Work{ + newWork("work-1", primarySnapshotName, "", serverSideApplyStrategy.DeepCopy()), + newWork("work-0", primarySnapshotName, "2", serverSideApplyStrategy.DeepCopy()), + }, + wantStatus: workSyncStatusInSync, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotStatus, err := areWorksInSync(tc.binding, tc.works) + if (err != nil) != tc.wantErr { + t.Fatalf("areWorksInSync() = (%q, %v), want error: %t", gotStatus, err, tc.wantErr) + } + if gotStatus != tc.wantStatus { + t.Errorf("areWorksInSync() = %q, want %q", gotStatus, tc.wantStatus) + } + }) + } +} + +// configMapManifestForUnitTest returns the manifest of a ConfigMap of the given name. +func configMapManifestForUnitTest(name string) runtime.RawExtension { + return runtime.RawExtension{Raw: fmt.Appendf(nil, `{"apiVersion":"v1","kind":"ConfigMap","metadata":{"name":%q,"namespace":%q}}`, name, unitTestResourceNS)} +} + +// configMapResourcesForUnitTest returns the snapshotted resources for ConfigMaps of the given names, and the +// manifests that the work generator is expected to build from them. +func configMapResourcesForUnitTest(names ...string) ([]placementv1alpha1.SnapshottedResource, []placementv1alpha1.Manifest) { + resources := make([]placementv1alpha1.SnapshottedResource, len(names)) + manifests := make([]placementv1alpha1.Manifest, len(names)) + for idx, name := range names { + resources[idx] = placementv1alpha1.SnapshottedResource{ + Identifier: placementv1alpha1.ObjectReference{Namespace: unitTestResourceNS, Name: name, APIVersion: "v1", Kind: configMapKind}, + Manifest: configMapManifestForUnitTest(name), + } + manifests[idx] = placementv1alpha1.Manifest{RawExtension: configMapManifestForUnitTest(name)} + } + return resources, manifests +} + +// TestBuildWorkObjectFor tests the buildWorkObjectFor function. +func TestBuildWorkObjectFor(t *testing.T) { + serverSideApplyStrategy := &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodServerSideApply, + } + newSnapshot := func(subIdx string, resources []placementv1alpha1.SnapshottedResource) placementv1alpha1.PlacementResourceSnapshotAccessor { + labels := map[string]string{} + if subIdx != "" { + labels[placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey] = subIdx + } + return &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{Namespace: appNamespaceName, Name: "snapshot", Labels: labels}, + Spec: placementv1alpha1.PlacementResourceSnapshotSpec{Resources: resources}, + } + } + + // A namespace-scoped binding with short names and a sync strategy. + nsBinding := &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{Namespace: appNamespaceName, Name: unitTestBindingName}, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: unitTestPolicyName, + ClusterName: memberCluster1Name, + ResourceSnapshotName: unitTestSnapshotName1, + SyncStrategy: serverSideApplyStrategy, + }, + } + // A cluster-scoped binding with long, dotted names and no sync strategy. + longBindingName := dottedName(100, 3) + longPolicyName := dottedName(120, 5) + clusterBinding := &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{Name: longBindingName}, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: longPolicyName, + ClusterName: memberCluster2Name, + ResourceSnapshotName: unitTestSnapshotName1, + }, + } + primaryResources, primaryManifests := configMapResourcesForUnitTest("configmap-a", "configmap-b") + secondaryResources, secondaryManifests := configMapResourcesForUnitTest("configmap-c") + + testCases := []struct { + name string + snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + binding placementv1alpha1.PlacementBindingAccessor + want *placementv1alpha1.Work + wantErr bool + }{ + { + name: "primary snapshot, namespace-scoped binding with a sync strategy", + snapshot: newSnapshot("0", primaryResources), + binding: nsBinding, + want: &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: fmt.Sprintf("fleet-member-%s", memberCluster1Name), + Name: uniqueNameForWorkDerivedFromPlacementResourceSnapshot(nsBinding, true, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "0"}), + Labels: map[string]string{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: appNamespaceName, + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: unitTestPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: unitTestBindingName, + }, + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: unitTestSnapshotName1, + placementv1alpha1.WorkDerivedFromSourceAnnotationKey: "placement-resource-snapshot/0", + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: unitTestPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: unitTestBindingName, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: primaryManifests, + SyncStrategy: serverSideApplyStrategy.DeepCopy(), + }, + }, + }, + { + name: "secondary snapshot, cluster-scoped binding with long dotted names", + snapshot: newSnapshot("2", secondaryResources), + binding: clusterBinding, + want: &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: fmt.Sprintf("fleet-member-%s", memberCluster2Name), + Name: uniqueNameForWorkDerivedFromPlacementResourceSnapshot(clusterBinding, false, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "2"}), + Labels: map[string]string{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: "", + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: workOwnerLabelValue(longPolicyName), + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: workOwnerLabelValue(longBindingName), + }, + // The annotations carry the full names. + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: unitTestSnapshotName1, + placementv1alpha1.WorkDerivedFromSourceAnnotationKey: "placement-resource-snapshot/2", + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: longPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: longBindingName, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: secondaryManifests, + }, + }, + }, + { + name: "snapshot without a sub-index label", + snapshot: newSnapshot("", primaryResources), + binding: nsBinding, + wantErr: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := buildWorkObjectFor(tc.snapshot, tc.binding, unitTestSnapshotName1) + if (err != nil) != tc.wantErr { + t.Fatalf("buildWorkObjectFor() = (%v, %v), want error: %t", got, err, tc.wantErr) + } + if err != nil { + if kferrors.Category(err) != kferrors.ErrCategoryUnexpected { + t.Errorf("buildWorkObjectFor() error category = %q, want %q", kferrors.Category(err), kferrors.ErrCategoryUnexpected) + } + return + } + if diff := cmp.Diff(got, tc.want); diff != "" { + t.Errorf("buildWorkObjectFor() mismatch (-got +want):\n%s", diff) + } + if got.Spec.SyncStrategy != nil && got.Spec.SyncStrategy == tc.binding.GetSpec().SyncStrategy { + t.Errorf("buildWorkObjectFor() shares the sync strategy with the placement binding, want a copy") + } + }) + } +} + +// TestUpdateWorkObjectMetadataAndSpec tests the updateWorkObjectMetadataAndSpec function. +func TestUpdateWorkObjectMetadataAndSpec(t *testing.T) { + serverSideApplyStrategy := &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodServerSideApply, + } + _, oldManifests := configMapResourcesForUnitTest("configmap-old-a", "configmap-old-b") + newResources, newManifests := configMapResourcesForUnitTest("configmap-new") + workNS := fmt.Sprintf("fleet-member-%s", memberCluster1Name) + const ( + workName = "work" + wantDerivedFromID = "placement-resource-snapshot/1" + ) + + testCases := []struct { + name string + work *placementv1alpha1.Work + syncStrategy *placementv1alpha1.SyncStrategy + want *placementv1alpha1.Work + }{ + { + name: "work object without metadata or spec", + work: &placementv1alpha1.Work{ObjectMeta: metav1.ObjectMeta{Namespace: workNS, Name: workName}}, + syncStrategy: serverSideApplyStrategy, + want: &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workName, + Labels: map[string]string{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: appNamespaceName, + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: unitTestPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: unitTestBindingName, + }, + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: unitTestSnapshotName1, + placementv1alpha1.WorkDerivedFromSourceAnnotationKey: wantDerivedFromID, + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: unitTestPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: unitTestBindingName, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: newManifests, + SyncStrategy: serverSideApplyStrategy, + }, + }, + }, + { + name: "work object with existing metadata and spec", + work: &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workName, + Labels: map[string]string{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: appNamespaceName, + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: unitTestPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: unitTestBindingName, + "unrelated-label": "foo", + }, + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: unitTestSnapshotName0, + placementv1alpha1.WorkDerivedFromSourceAnnotationKey: wantDerivedFromID, + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: unitTestPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: unitTestBindingName, + placementv1alpha1.LinkedWorkCountAnnotationKey: "2", + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: oldManifests, + SyncStrategy: serverSideApplyStrategy, + }, + }, + // The owned labels and annotations, the manifests, and the sync strategy are overwritten (the sync + // strategy is unset); the other labels and annotations are kept. + want: &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: workNS, + Name: workName, + Labels: map[string]string{ + placementv1alpha1.WorkOwnerNamespaceLabelKey: appNamespaceName, + placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey: unitTestPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingLabelKey: unitTestBindingName, + "unrelated-label": "foo", + }, + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: unitTestSnapshotName1, + placementv1alpha1.WorkDerivedFromSourceAnnotationKey: wantDerivedFromID, + placementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey: unitTestPolicyName, + placementv1alpha1.WorkOwnedByPlacementBindingAnnotationKey: unitTestBindingName, + placementv1alpha1.LinkedWorkCountAnnotationKey: "2", + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: newManifests, + }, + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + work := tc.work.DeepCopy() + updateWorkObjectMetadataAndSpec(work, appNamespaceName, unitTestPolicyName, unitTestBindingName, unitTestSnapshotName1, + &placementResourceSnapshotDerivedFromSourceFormatter{snapshotSubIdx: "1"}, newResources, tc.syncStrategy) + if diff := cmp.Diff(work, tc.want); diff != "" { + t.Errorf("updateWorkObjectMetadataAndSpec() mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// TestRefreshSyncStrategyOnlyForWorks tests the refreshSyncStrategyOnlyForWorks method. +func TestRefreshSyncStrategyOnlyForWorks(t *testing.T) { + serverSideApplyStrategy := &placementv1alpha1.SyncStrategy{ + ApplyMethod: placementv1alpha1.ApplyMethodServerSideApply, + } + _, manifests := configMapResourcesForUnitTest("configmap-a") + const ( + workName0 = "work-0" + workName1 = "work-1" + workName2 = "work-2" + ) + newStoredWork := func(name string, syncStrategy *placementv1alpha1.SyncStrategy) *placementv1alpha1.Work { + return &placementv1alpha1.Work{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: fmt.Sprintf("fleet-member-%s", memberCluster1Name), + Name: name, + Annotations: map[string]string{ + placementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey: unitTestSnapshotName0, + }, + }, + Spec: placementv1alpha1.WorkSpec{ + Manifests: manifests, + SyncStrategy: syncStrategy.DeepCopy(), + }, + } + } + + testCases := []struct { + name string + bindingStrategy *placementv1alpha1.SyncStrategy + storedWorks []*placementv1alpha1.Work + // The name of a work object that is updated out of band after it has been read, i.e., the copy passed to + // the method is stale. + staleWorkName string + wantPatchedNames []string + // The sync strategies of the work objects as stored afterwards, keyed by work name. + wantStrategies map[string]*placementv1alpha1.SyncStrategy + wantConflict bool + }{ + { + name: "set a sync strategy", + bindingStrategy: serverSideApplyStrategy, + storedWorks: []*placementv1alpha1.Work{ + newStoredWork(workName0, nil), + newStoredWork(workName1, nil), + newStoredWork(workName2, serverSideApplyStrategy), + }, + wantPatchedNames: []string{workName0, workName1}, + wantStrategies: map[string]*placementv1alpha1.SyncStrategy{ + workName0: serverSideApplyStrategy, + workName1: serverSideApplyStrategy, + workName2: serverSideApplyStrategy, + }, + }, + { + name: "unset the sync strategy", + storedWorks: []*placementv1alpha1.Work{ + newStoredWork(workName0, serverSideApplyStrategy), + newStoredWork(workName1, nil), + }, + wantPatchedNames: []string{workName0}, + wantStrategies: map[string]*placementv1alpha1.SyncStrategy{ + workName0: nil, + workName1: nil, + }, + }, + { + name: "all work objects in sync", + bindingStrategy: serverSideApplyStrategy, + storedWorks: []*placementv1alpha1.Work{ + newStoredWork(workName0, serverSideApplyStrategy), + }, + wantStrategies: map[string]*placementv1alpha1.SyncStrategy{ + workName0: serverSideApplyStrategy, + }, + }, + { + name: "a work object has been updated since it was read", + bindingStrategy: serverSideApplyStrategy, + storedWorks: []*placementv1alpha1.Work{ + newStoredWork(workName0, nil), + }, + staleWorkName: workName0, + wantPatchedNames: []string{workName0}, + wantStrategies: map[string]*placementv1alpha1.SyncStrategy{ + workName0: nil, + }, + wantConflict: true, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + + var mu sync.Mutex + var gotPatchedNames []string + objs := make([]client.Object, len(tc.storedWorks)) + for idx := range tc.storedWorks { + objs[idx] = tc.storedWorks[idx] + } + c := newUnitTestHubClient(t, interceptor.Funcs{ + Patch: func(ctx context.Context, c client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + mu.Lock() + gotPatchedNames = append(gotPatchedNames, obj.GetName()) + mu.Unlock() + return c.Patch(ctx, obj, patch, opts...) + }, + }, objs...) + + // Read the work objects, as the work generator would do. + workList := &placementv1alpha1.WorkList{} + if err := c.List(ctx, workList); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + if tc.staleWorkName != "" { + work := &placementv1alpha1.Work{} + if err := c.Get(ctx, client.ObjectKey{Namespace: fmt.Sprintf("fleet-member-%s", memberCluster1Name), Name: tc.staleWorkName}, work); err != nil { + t.Fatalf("Get() = %v, want no error", err) + } + work.Labels = map[string]string{"updated-out-of-band": "true"} + if err := c.Update(ctx, work); err != nil { + t.Fatalf("Update() = %v, want no error", err) + } + } + + binding := newBindingWithStatus(1, unitTestSnapshotName0, placementv1alpha1.PlacementBindingStatus{}) + binding.Spec.SyncStrategy = tc.bindingStrategy.DeepCopy() + r := New(c, 2, nil) + err := r.refreshSyncStrategyOnlyForWorks(ctx, binding, workList.Items) + if (err != nil) != tc.wantConflict { + t.Fatalf("refreshSyncStrategyOnlyForWorks() = %v, want error: %t", err, tc.wantConflict) + } + if err != nil { + if !apierrors.IsConflict(err) { + t.Errorf("refreshSyncStrategyOnlyForWorks() = %v, want a conflict error", err) + } + if kferrors.Category(err) != kferrors.ErrCategoryAPIServer { + t.Errorf("refreshSyncStrategyOnlyForWorks() error category = %q, want %q", kferrors.Category(err), kferrors.ErrCategoryAPIServer) + } + } + + slices.Sort(gotPatchedNames) + if diff := cmp.Diff(gotPatchedNames, tc.wantPatchedNames, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("refreshSyncStrategyOnlyForWorks() patched work objects mismatch (-got +want):\n%s", diff) + } + + // Only the sync strategies should have changed. + gotWorkList := &placementv1alpha1.WorkList{} + if err := c.List(ctx, gotWorkList); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + if len(gotWorkList.Items) != len(tc.storedWorks) { + t.Fatalf("List() returned %d work objects, want %d", len(gotWorkList.Items), len(tc.storedWorks)) + } + for idx := range gotWorkList.Items { + gotWork := &gotWorkList.Items[idx] + wantSpec := newStoredWork(gotWork.Name, tc.wantStrategies[gotWork.Name]).Spec + if diff := cmp.Diff(gotWork.Spec, wantSpec); diff != "" { + t.Errorf("work object %s spec mismatch (-got +want):\n%s", gotWork.Name, diff) + } + wantAnnotations := newStoredWork(gotWork.Name, nil).Annotations + if diff := cmp.Diff(gotWork.Annotations, wantAnnotations); diff != "" { + t.Errorf("work object %s annotations mismatch (-got +want):\n%s", gotWork.Name, diff) + } + } + }) + } +} diff --git a/pkg/v1/managers/placementresourcesnapshot/corner_cases_integration_test.go b/pkg/v1/managers/placementresourcesnapshot/corner_cases_integration_test.go new file mode 100644 index 000000000..1009b34df --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/corner_cases_integration_test.go @@ -0,0 +1,1869 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementresourcesnapshot + +import ( + "fmt" + "slices" + "strings" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + quorumReadNewerSnapshotErrMsg = "a newer snapshot already exists (found via quorum reads)" + quorumReadPrimaryExistsErrMsg = "a primary placement resource snapshot already exists at the given index (found via quorum read)" + orphansCleanedUpErrMsg = "cleaned up orphaned secondary placement resource snapshots" + primaryCreationFailedErrMsg = "failed to create the primary placement resource snapshot" + secondaryCreationFailedErrMsg = "failed to create a secondary placement resource snapshot" + fewerSubIndexedSnapshotsErrMsg = "there are fewer sub-indexed placement resource snapshots than the count label indicates" + mismatchedContentsHashErrMsg = "a sub-indexed placement resource snapshot has a different contents hash than the primary placement resource snapshot" + staleOwnerSnapshotErrMsg = "found a placement resource snapshot owned by a different placement policy of the same name" + leftoverSnapshotsErrMsg = "leftover resource snapshots are detected" + unexpectedOwnerRefsErrMsg = "that has unexpected owner references" + + // The expected limits on generated snapshot names and owned-by label values; they are spelled out here (rather + // than reusing the constants of the package) so that the tests catch unintended changes. + wantMaxSnapshotNameLen = 251 + wantOwnedByLabelValLen = 61 + wantOwnedByLabelValPrefixLen = 48 + wantHashSuffixPattern = `^[0-9a-f]{12}$` + wantHashedPrimaryNamePattern = `-resource-snapshot-0-[0-9a-f]{12}$` + wantHashedSecondaryNamePattern = `-resource-snapshot-0-1-[0-9a-f]{12}$` + // The snapshot names minus the hash suffix, should the placement policy name need no sanitization. + wantUnsanitizedPrimaryNamePrefixFmt = "%s-resource-snapshot-%d-" + wantUnsanitizedSecondaryNamePrefixFmt = "%s-resource-snapshot-%d-%d-" + concurrentSnapshotCallerCnt = 8 + malformedPrimarySnapshotSuffix = "-malformed" + placementPolicyNameFillerLetter = "a" +) + +// decoySnapshotSpec describes a hand-crafted placement resource snapshot (a decoy). +type decoySnapshotSpec struct { + name string + labels map[string]string + annotations map[string]string +} + +var _ = Describe("corner cases (stale cache and inconsistent states)", func() { + // The cache has yet to observe a primary placement resource snapshot at the next index; the scenario is + // run for both placement policies and cluster placement policies. + for _, clusterScoped := range []bool{false, true} { + desc := "newer primary snapshot visible only to quorum reads" + if clusterScoped { + desc = "newer cluster primary snapshot visible only to quorum reads" + } + + Context(desc, Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + var placementPolicy placementv1alpha1.PlacementPolicyAccessor + if clusterScoped { + selector := configMapNameSelector(configMapName) + selector.Namespace = appNamespaceName + placementPolicy = newClusterPlacementPolicy(selector) + } else { + placementPolicy = newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + } + snapshotNS := placementPolicy.GetNamespace() + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var primarySnapshotName0, primarySnapshotName1 string + + var hash string + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + primarySnapshotName0 = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0) + primarySnapshotName1 = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 1) + }) + + It("can create a placement resource snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + hash = snapshotContentsHash(getPrimarySnapshotFromCache(placementPolicy, 0)) + }) + + It("can create an unlabeled decoy at the primary snapshot name of the next index", func() { + createDecoySnapshot(snapshotNS, primarySnapshotName1, nil, nil, nil) + }) + + It("should return a transient error when checking for staleness", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, quorumReadNewerSnapshotErrMsg) + }) + + It("should return a transient error even if only the existing snapshot is requested", func() { + // This is expected; upon retry the manager finds the newer snapshot once the cache catches up. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, quorumReadNewerSnapshotErrMsg) + }) + + It("can simulate the cache catching up by turning the decoy into an indexed primary snapshot", func() { + decoy := verifySnapshotExists(snapshotNS, primarySnapshotName1) + decoy.SetLabels(decoyPrimaryLabels(placementPolicy, "1", "1")) + // A snapshot that the manager creates is always owned by the placement policy. + decoy.SetOwnerReferences(decoyOwnerRefs(placementPolicy)) + annotations := decoyOwnedByAnnotations(placementPolicy) + // The selected resources have not changed; a real snapshot at the next index would carry the same hash. + annotations[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] = hash + decoy.SetAnnotations(annotations) + Expect(hubClient.Update(ctx, decoy)).To(Succeed(), "Failed to update the decoy placement resource snapshot") + }) + + It("should return the newer snapshot on retry", func() { + // Until the cache catches up, the manager keeps returning the transient error. + Eventually(func() error { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + if err != nil { + return err + } + if diff := cmp.Diff(snapshotNamesOf(gotSnapshots), []string{primarySnapshotName1}); diff != "" || !isUpToDate { + return fmt.Errorf("unexpected snapshots (up-to-date %t), names diff (-got +want):\n%s", isUpToDate, diff) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to retrieve the newer placement resource snapshot") + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, []string{primarySnapshotName1}) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 2) + }) + + AfterAll(func() { + deleteObjectsAndWait([]client.Object{ + placementResourceSnapshotObj(snapshotNS, primarySnapshotName0), + placementResourceSnapshotObj(snapshotNS, primarySnapshotName1), + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + }) + }) + }) + } + + Context("primary snapshot name taken", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + // The snapshot name depends on the UID of the placement policy; it is set once the policy is created. + var primarySnapshotName string + + var decoyUID types.UID + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + primarySnapshotName = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0) + + // This mimics the situation where the manager has created the primary snapshot, yet the cache has + // not observed it. + decoyUID = createDecoySnapshot(appNamespaceName, primarySnapshotName, nil, nil, nil).GetUID() + }) + + It("should return an API server error when the primary snapshot name is taken", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryAPIServer, primaryCreationFailedErrMsg) + Expect(errors.IsAlreadyExists(err)).To(BeTrue(), "The error does not wrap an already exists error: %v", err) + }) + + It("should not modify the existing object", func() { + decoy := verifySnapshotExists(appNamespaceName, primarySnapshotName) + Expect(decoy.GetUID()).To(Equal(decoyUID), "The decoy placement resource snapshot has been re-created") + Expect(decoy.GetLabels()).To(BeEmpty(), "The decoy placement resource snapshot has been modified") + }) + + It("can remove the decoy", func() { + deleteObjectsAndWait([]client.Object{placementResourceSnapshotObj(appNamespaceName, primarySnapshotName)}) + waitForSnapshotGoneFromCache(appNamespaceName, primarySnapshotName) + }) + + It("can create the placement resource snapshot once the name is free", func() { + _, wantHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + verifySnapshotMetadata(getPrimarySnapshotFromCache(placementPolicy, 0), placementPolicy, 0, 0, 1, wantHash) + }) + + AfterAll(func() { + deleteObjectsAndWait([]client.Object{ + placementResourceSnapshotObj(appNamespaceName, primarySnapshotName), + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + }) + }) + }) + + Context("secondary snapshot name taken (partial provisioning)", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + // Seven resources, with at most three resources per snapshot: one primary and two secondary snapshots. + placementPolicy := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupLabels)) + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var wantSnapshotNames []string + + var wantHash string + + BeforeAll(func() { + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 7, groupLabels) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + wantSnapshotNames = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 2), + } + createDecoySnapshot(appNamespaceName, wantSnapshotNames[1], nil, nil, nil) + }) + + It("can compute the hash of selected resources", func() { + _, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + wantHash = hash + }) + + It("should return an API server error when a secondary snapshot name is taken", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryAPIServer, secondaryCreationFailedErrMsg) + Expect(errors.IsAlreadyExists(err)).To(BeTrue(), "The error does not wrap an already exists error: %v", err) + }) + + It("should have left a partially provisioned snapshot group behind", func() { + // The secondary snapshots are created from the last sub-index down; the creation stops at the decoy. + verifySnapshotExists(appNamespaceName, wantSnapshotNames[2]) + verifySnapshotNotFound(appNamespaceName, wantSnapshotNames[0]) + decoy := verifySnapshotExists(appNamespaceName, wantSnapshotNames[1]) + Expect(decoy.GetLabels()).To(BeEmpty(), "The decoy placement resource snapshot has been modified") + }) + + It("can remove the decoy", func() { + deleteObjectsAndWait([]client.Object{placementResourceSnapshotObj(appNamespaceName, wantSnapshotNames[1])}) + waitForSnapshotGoneFromCache(appNamespaceName, wantSnapshotNames[1]) + // Wait until the cache has observed the orphaned secondary snapshot. + getSnapshotFromCache(appNamespaceName, wantSnapshotNames[2]) + }) + + It("should clean up the orphaned secondary snapshot and return a transient error", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, orphansCleanedUpErrMsg) + verifySnapshotNotFound(appNamespaceName, wantSnapshotNames[2]) + }) + + It("can create all the snapshots once the orphan is gone", func() { + waitForSnapshotGoneFromCache(appNamespaceName, wantSnapshotNames[2]) + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + for subIdx, name := range wantSnapshotNames { + verifySnapshotMetadata(getSnapshotFromCache(appNamespaceName, name), placementPolicy, 0, subIdx, len(wantSnapshotNames), wantHash) + } + }) + + AfterAll(func() { + objs := make([]client.Object, 0, len(wantSnapshotNames)+1+len(configMapNames)) + for _, name := range wantSnapshotNames { + objs = append(objs, placementResourceSnapshotObj(appNamespaceName, name)) + } + objs = append(objs, placementPolicy) + objs = append(objs, configMapObjs(configMapNames)...) + deleteObjectsAndWait(objs) + }) + }) + + Context("orphan clean-up deferred by a quorum read", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var primarySnapshotName, secondarySnapshotName string + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + primarySnapshotName = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0) + secondarySnapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1) + + // An indexed secondary snapshot (the orphan candidate), and an unindexed primary snapshot at the + // same index; the latter mimics a primary snapshot that the cache has yet to observe. + createDecoySnapshot(appNamespaceName, secondarySnapshotName, + decoySecondaryLabels(placementPolicy, "0", "1"), decoyOwnedByAnnotations(placementPolicy), nil) + createDecoySnapshot(appNamespaceName, primarySnapshotName, nil, nil, nil) + getSnapshotFromCache(appNamespaceName, secondarySnapshotName) + }) + + It("should return a transient error without deleting the secondary snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, quorumReadPrimaryExistsErrMsg) + + secondarySnapshot := verifySnapshotExists(appNamespaceName, secondarySnapshotName) + Expect(secondarySnapshot.GetDeletionTimestamp()).To(BeNil(), "The secondary placement resource snapshot has been deleted") + }) + + It("can remove the primary decoy", func() { + deleteObjectsAndWait([]client.Object{placementResourceSnapshotObj(appNamespaceName, primarySnapshotName)}) + waitForSnapshotGoneFromCache(appNamespaceName, primarySnapshotName) + }) + + It("should clean up the orphaned secondary snapshot now", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, orphansCleanedUpErrMsg) + verifySnapshotNotFound(appNamespaceName, secondarySnapshotName) + }) + + It("can create the placement resource snapshot", func() { + waitForSnapshotGoneFromCache(appNamespaceName, secondarySnapshotName) + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, []string{primarySnapshotName}) + }) + + AfterAll(func() { + deleteObjectsAndWait([]client.Object{ + placementResourceSnapshotObj(appNamespaceName, primarySnapshotName), + placementResourceSnapshotObj(appNamespaceName, secondarySnapshotName), + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + }) + }) + }) + + Context("orphaned secondary snapshot pending deletion", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var primarySnapshotName, secondarySnapshotName string + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + primarySnapshotName = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0) + secondarySnapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1) + + // An orphaned secondary snapshot that cannot be deleted right away due to a finalizer. + createDecoySnapshot(appNamespaceName, secondarySnapshotName, + decoySecondaryLabels(placementPolicy, "0", "1"), decoyOwnedByAnnotations(placementPolicy), []string{testFinalizer}) + getSnapshotFromCache(appNamespaceName, secondarySnapshotName) + }) + + It("should delete the orphaned secondary snapshot and return a transient error", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, orphansCleanedUpErrMsg) + + secondarySnapshot := verifySnapshotExists(appNamespaceName, secondarySnapshotName) + Expect(secondarySnapshot.GetDeletionTimestamp()).ToNot(BeNil(), "The secondary placement resource snapshot has not been marked for deletion") + }) + + It("should keep returning a transient error while the orphan is being deleted", func() { + Eventually(func() error { + secondarySnapshot := placementResourceSnapshotObj(appNamespaceName, secondarySnapshotName) + if err := hubClient.Get(ctx, client.ObjectKeyFromObject(secondarySnapshot), secondarySnapshot); err != nil { + return err + } + if secondarySnapshot.GetDeletionTimestamp() == nil { + return fmt.Errorf("the cache has not observed the deletion timestamp yet") + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the deletion timestamp in the cache") + + Consistently(func() error { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + return checkSnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, orphansCleanedUpErrMsg) + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The manager did not keep returning the transient error") + verifySnapshotNotFound(appNamespaceName, primarySnapshotName) + }) + + It("can remove the finalizer", func() { + removeFinalizersAndDelete(placementResourceSnapshotObj(appNamespaceName, secondarySnapshotName)) + waitForSnapshotGoneFromCache(appNamespaceName, secondarySnapshotName) + }) + + It("can create the placement resource snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, []string{primarySnapshotName}) + }) + + AfterAll(func() { + // The decoy is normally gone by now; remove the finalizer anyway in case a spec failed midway. + removeFinalizersAndDelete(placementResourceSnapshotObj(appNamespaceName, secondarySnapshotName)) + deleteObjectsAndWait([]client.Object{ + placementResourceSnapshotObj(appNamespaceName, primarySnapshotName), + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + }) + }) + }) + + Context("more sub-indexed snapshots than the count label indicates", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + // Four resources, with at most three resources per snapshot: one primary and one secondary snapshot. + placementPolicy := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupLabels)) + + // A placement policy that selects a single resource; its snapshot features a count label of 1. + singleGroupPlacementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapNamePrefix+"-0")) + + // The snapshot names depend on the UIDs of the placement policies; they are set once the policies are created. + var wantSnapshotNames []string + var extraSecondarySnapshotName string + var singleGroupPrimarySnapshotName, singleGroupExtraSecondarySnapshotName string + + BeforeAll(func() { + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 4, groupLabels) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + Expect(hubClient.Create(ctx, singleGroupPlacementPolicy)).To(Succeed(), "Failed to create the placement policy") + + wantSnapshotNames = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1), + } + extraSecondarySnapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 2) + singleGroupPrimarySnapshotName = uniquename.PrimaryPlacementResourceSnapshotName(singleGroupPlacementPolicy.Name, singleGroupPlacementPolicy.UID, 0) + singleGroupExtraSecondarySnapshotName = uniquename.SecondaryPlacementResourceSnapshotName(singleGroupPlacementPolicy.Name, singleGroupPlacementPolicy.UID, 0, 1) + }) + + It("can create a primary and a secondary snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + for _, name := range wantSnapshotNames { + getSnapshotFromCache(appNamespaceName, name) + } + }) + + It("can create an extra secondary snapshot at the latest index", func() { + createDecoySnapshot(appNamespaceName, extraSecondarySnapshotName, + decoySecondaryLabels(placementPolicy, "0", "2"), decoyOwnedByAnnotations(placementPolicy), nil) + getSnapshotFromCache(appNamespaceName, extraSecondarySnapshotName) + }) + + It("should return only the snapshots dictated by the count label", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + + gotSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + It("can create a single-group snapshot and an extra secondary snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, singleGroupPlacementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, []string{singleGroupPrimarySnapshotName}) + getSnapshotFromCache(appNamespaceName, singleGroupPrimarySnapshotName) + + createDecoySnapshot(appNamespaceName, singleGroupExtraSecondarySnapshotName, + decoySecondaryLabels(singleGroupPlacementPolicy, "0", "1"), decoyOwnedByAnnotations(singleGroupPlacementPolicy), nil) + getSnapshotFromCache(appNamespaceName, singleGroupExtraSecondarySnapshotName) + }) + + It("should ignore extra secondary snapshots when the count label is 1", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, singleGroupPlacementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, []string{singleGroupPrimarySnapshotName}) + }) + + AfterAll(func() { + deleteObjectsAndWait(slices.Concat( + snapshotObjs(appNamespaceName, wantSnapshotNames[0], wantSnapshotNames[1], extraSecondarySnapshotName, + singleGroupPrimarySnapshotName, singleGroupExtraSecondarySnapshotName), + []client.Object{placementPolicy, singleGroupPlacementPolicy}, + configMapObjs(configMapNames), + )) + }) + }) + + Context("fewer sub-indexed snapshots than the count label indicates", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + placementPolicy := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupLabels)) + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var wantSnapshotNames []string + + var originalPrimarySnapshotUID types.UID + + BeforeAll(func() { + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 4, groupLabels) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + wantSnapshotNames = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1), + } + }) + + It("can create a primary and a secondary snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + originalPrimarySnapshotUID = gotSnapshots[0].GetUID() + for _, name := range wantSnapshotNames { + getSnapshotFromCache(appNamespaceName, name) + } + }) + + It("can delete the secondary snapshot", func() { + // Note that this is for simulation purposes only; the manager never deletes a single secondary snapshot. + deleteObjectsAndWait([]client.Object{placementResourceSnapshotObj(appNamespaceName, wantSnapshotNames[1])}) + waitForSnapshotGoneFromCache(appNamespaceName, wantSnapshotNames[1]) + }) + + It("should return an unexpected error", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryUnexpected, fewerSubIndexedSnapshotsErrMsg) + + gotSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryUnexpected, fewerSubIndexedSnapshotsErrMsg) + }) + + It("can update a selected config map", func() { + updateConfigMapData(appNamespaceName, configMapNames[0], "changed") + }) + + It("should still return an unexpected error after a resource change alone", func() { + // The latest snapshots are retrieved (and verified) before the selected resources are; a resource change + // alone does not help the manager recover. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryUnexpected, fewerSubIndexedSnapshotsErrMsg) + }) + + It("can manually clean up the broken primary snapshot", func() { + deleteObjectsAndWait([]client.Object{placementResourceSnapshotObj(appNamespaceName, wantSnapshotNames[0])}) + waitForSnapshotGoneFromCache(appNamespaceName, wantSnapshotNames[0]) + }) + + It("should create a new snapshot set after the manual clean-up", func() { + _, wantHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + + // No other primary snapshot exists; the snapshots are re-created at index 0. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + Expect(gotSnapshots[0].GetUID()).ToNot(Equal(originalPrimarySnapshotUID), "The primary placement resource snapshot has not been re-created") + for subIdx, name := range wantSnapshotNames { + verifySnapshotMetadata(getSnapshotFromCache(appNamespaceName, name), placementPolicy, 0, subIdx, len(wantSnapshotNames), wantHash) + } + }) + + AfterAll(func() { + deleteObjectsAndWait(slices.Concat( + snapshotObjs(appNamespaceName, wantSnapshotNames...), + []client.Object{placementPolicy}, + configMapObjs(configMapNames), + )) + }) + }) + + Context("malformed snapshot labels", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + // The hash of the currently selected resources; every placement policy below selects the same config map. + var currentHash string + + primaryDecoy := func(placementPolicy placementv1alpha1.PlacementPolicyAccessor, name, idx, cnt string, extraAnnotations map[string]string) decoySnapshotSpec { + annotations := decoyOwnedByAnnotations(placementPolicy) + for k, v := range extraAnnotations { + annotations[k] = v + } + return decoySnapshotSpec{name: name, labels: decoyPrimaryLabels(placementPolicy, idx, cnt), annotations: annotations} + } + secondaryDecoy := func(placementPolicy placementv1alpha1.PlacementPolicyAccessor, name, idx, subIdx string, extraAnnotations map[string]string) decoySnapshotSpec { + annotations := decoyOwnedByAnnotations(placementPolicy) + for k, v := range extraAnnotations { + annotations[k] = v + } + return decoySnapshotSpec{name: name, labels: decoySecondaryLabels(placementPolicy, idx, subIdx), annotations: annotations} + } + primaryName := func(placementPolicy placementv1alpha1.PlacementPolicyAccessor) string { + return uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0) + } + secondaryName := func(placementPolicy placementv1alpha1.PlacementPolicyAccessor) string { + return uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0, 1) + } + withHash := func(hash string) map[string]string { + return map[string]string{placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: hash} + } + + type malformedSnapshotLabelsTestCase struct { + desc string + placementPolicy *placementv1alpha1.PlacementPolicy + // decoys returns the decoys to create; it is evaluated in BeforeAll, after the current hash is known. + decoys func(placementPolicy placementv1alpha1.PlacementPolicyAccessor) []decoySnapshotSpec + useIfStale bool + wantErrMsgSubstrs []string + } + + testCases := []malformedSnapshotLabelsTestCase{ + { + desc: "the count label is not an integer", + placementPolicy: newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + decoys: func(pp placementv1alpha1.PlacementPolicyAccessor) []decoySnapshotSpec { + return []decoySnapshotSpec{primaryDecoy(pp, primaryName(pp), "0", "abc", nil)} + }, + wantErrMsgSubstrs: []string{"failed to convert sub-indexed placement resource snapshot count label to integer"}, + }, + { + desc: "the count label is less than 1", + placementPolicy: newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + decoys: func(pp placementv1alpha1.PlacementPolicyAccessor) []decoySnapshotSpec { + return []decoySnapshotSpec{primaryDecoy(pp, primaryName(pp), "0", "0", nil)} + }, + wantErrMsgSubstrs: []string{"sub-indexed placement resource snapshot count label is less than 1"}, + }, + { + desc: "the index label of one of the primary snapshots is not an integer", + placementPolicy: newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + decoys: func(pp placementv1alpha1.PlacementPolicyAccessor) []decoySnapshotSpec { + return []decoySnapshotSpec{ + primaryDecoy(pp, primaryName(pp), "0", "1", nil), + primaryDecoy(pp, pp.GetName()+malformedPrimarySnapshotSuffix, "abc", "1", nil), + } + }, + wantErrMsgSubstrs: []string{"failed to sort primary placement resource snapshots"}, + }, + { + desc: "the sub-index label of a secondary snapshot is not an integer", + placementPolicy: newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + decoys: func(pp placementv1alpha1.PlacementPolicyAccessor) []decoySnapshotSpec { + return []decoySnapshotSpec{ + primaryDecoy(pp, primaryName(pp), "0", "2", nil), + secondaryDecoy(pp, secondaryName(pp), "0", "abc", nil), + } + }, + wantErrMsgSubstrs: []string{"failed to sort sub-indexed placement resource snapshots"}, + }, + { + // The hash does not match; the manager attempts to create a new snapshot. + desc: "the index label of the latest primary snapshot is not an integer and the snapshot is stale", + placementPolicy: newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + decoys: func(pp placementv1alpha1.PlacementPolicyAccessor) []decoySnapshotSpec { + return []decoySnapshotSpec{primaryDecoy(pp, primaryName(pp), "abc", "1", nil)} + }, + useIfStale: true, + wantErrMsgSubstrs: []string{"failed to convert last seen primary placement resource snapshot index label to integer"}, + }, + { + // The hash matches; the manager attempts to verify (via a quorum read) that the snapshot is up-to-date. + desc: "the index label of the latest primary snapshot is not an integer and the hash matches", + placementPolicy: newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + decoys: func(pp placementv1alpha1.PlacementPolicyAccessor) []decoySnapshotSpec { + return []decoySnapshotSpec{primaryDecoy(pp, primaryName(pp), "abc", "1", withHash(currentHash))} + }, + wantErrMsgSubstrs: []string{"failed to convert index label to integer"}, + }, + { + desc: "a secondary snapshot has a different contents hash than the primary snapshot", + placementPolicy: newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + decoys: func(pp placementv1alpha1.PlacementPolicyAccessor) []decoySnapshotSpec { + return []decoySnapshotSpec{ + primaryDecoy(pp, primaryName(pp), "0", "2", withHash("hash-a")), + secondaryDecoy(pp, secondaryName(pp), "0", "1", withHash("hash-b")), + } + }, + wantErrMsgSubstrs: []string{mismatchedContentsHashErrMsg}, + }, + } + // The index of the test case whose latest primary snapshot is stale and features a malformed index label. + const staleMalformedIndexTestCaseIdx = 4 + + var decoyNames [][]string + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + _, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, testCases[0].placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + currentHash = hash + + decoyNames = make([][]string, len(testCases)) + for idx, tc := range testCases { + Expect(hubClient.Create(ctx, tc.placementPolicy)).To(Succeed(), "Failed to create the placement policy (%s)", tc.desc) + for _, decoy := range tc.decoys(tc.placementPolicy) { + createOwnedDecoySnapshot(tc.placementPolicy, decoy.name, decoy.labels, decoy.annotations) + getSnapshotFromCache(appNamespaceName, decoy.name) + decoyNames[idx] = append(decoyNames[idx], decoy.name) + } + } + }) + + for _, tc := range testCases { + It(fmt.Sprintf("should return an unexpected error when %s", tc.desc), func() { + snapshotFn := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists + if tc.useIfStale { + snapshotFn = resourceSnapshotManager.SnapshotResourcesIfStale + } + gotSnapshots, isUpToDate, err := snapshotFn(ctx, tc.placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryUnexpected, tc.wantErrMsgSubstrs...) + }) + } + + It("should return a malformed snapshot without error when only an existing snapshot is requested", func() { + // The hash does not match, which short-circuits the up-to-date check before the index label is parsed. + tc := testCases[staleMalformedIndexTestCaseIdx] + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, tc.placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeFalse(), "The malformed placement resource snapshot is reported as up-to-date") + if diff := cmp.Diff(snapshotNamesOf(gotSnapshots), []string{primaryName(tc.placementPolicy)}); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot names diff (-got +want):\n%s", diff)) + } + }) + + AfterAll(func() { + var objs []client.Object + for idx, tc := range testCases { + for _, name := range decoyNames[idx] { + objs = append(objs, placementResourceSnapshotObj(appNamespaceName, name)) + } + objs = append(objs, tc.placementPolicy) + } + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + deleteObjectsAndWait(objs) + }) + }) + + Context("concurrent callers", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + // Four resources, with at most three resources per snapshot: one primary and one secondary snapshot. + placementPolicy := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupLabels)) + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var wantSnapshotNames0, wantSnapshotNames1 []string + + BeforeAll(func() { + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 4, groupLabels) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + wantSnapshotNames0 = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1), + } + wantSnapshotNames1 = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 1, 1), + } + }) + + It("should create exactly one set of snapshots when called concurrently", func() { + results := snapshotConcurrently(concurrentSnapshotCallerCnt, placementPolicy) + verifyConcurrentSnapshotCalls(results, wantSnapshotNames0) + verifyOwnedSnapshots(placementPolicy, wantSnapshotNames0) + }) + + It("should converge to the up-to-date snapshots", func() { + for _, name := range wantSnapshotNames0 { + getSnapshotFromCache(appNamespaceName, name) + } + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + It("can update a selected config map", func() { + updateConfigMapData(appNamespaceName, configMapNames[0], "changed") + }) + + It("should create exactly one new set of snapshots when called concurrently", func() { + results := snapshotConcurrently(concurrentSnapshotCallerCnt, placementPolicy) + verifyConcurrentSnapshotCalls(results, wantSnapshotNames1) + verifyOwnedSnapshots(placementPolicy, append(slices.Clone(wantSnapshotNames0), wantSnapshotNames1...)) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 2) + }) + + AfterAll(func() { + deleteObjectsAndWait(slices.Concat( + snapshotObjs(appNamespaceName, slices.Concat(wantSnapshotNames0, wantSnapshotNames1)...), + []client.Object{placementPolicy}, + configMapObjs(configMapNames), + )) + }) + }) +}) + +var _ = Describe("corner cases (long and dotted names)", func() { + Context("name length boundaries", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + type nameLengthTestCase struct { + desc string + placementPolicy placementv1alpha1.PlacementPolicyAccessor + wantLabelTruncated bool + wantPrimarySanitized bool + wantSecondarySanitized bool + } + + // Every placement policy selects the same four config maps, which yields one primary and one secondary + // snapshot; this covers the naming of both kinds of snapshots. + newNamedPlacementPolicy := func(nameLen int) placementv1alpha1.PlacementPolicyAccessor { + placementPolicy := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupLabels)) + placementPolicy.Name = placementPolicyNameOfLen(nameLen) + return placementPolicy + } + newNamedClusterPlacementPolicy := func(nameLen int) placementv1alpha1.PlacementPolicyAccessor { + selector := configMapLabelSelector(groupLabels) + selector.Namespace = appNamespaceName + placementPolicy := newClusterPlacementPolicy(selector) + placementPolicy.Name = placementPolicyNameOfLen(nameLen) + return placementPolicy + } + + testCases := []nameLengthTestCase{ + { + desc: "the longest name with an untruncated owned-by label (61 characters)", + placementPolicy: newNamedPlacementPolicy(61), + }, + { + desc: "the shortest name with a truncated owned-by label (62 characters)", + placementPolicy: newNamedPlacementPolicy(62), + wantLabelTruncated: true, + }, + { + desc: "the longest name with untruncated snapshot names (216 characters)", + placementPolicy: newNamedPlacementPolicy(216), + wantLabelTruncated: true, + }, + { + desc: "the shortest name with a truncated secondary snapshot name (217 characters)", + placementPolicy: newNamedPlacementPolicy(217), + wantLabelTruncated: true, + wantSecondarySanitized: true, + }, + { + desc: "the longest name with an untruncated primary snapshot name (218 characters)", + placementPolicy: newNamedPlacementPolicy(218), + wantLabelTruncated: true, + wantSecondarySanitized: true, + }, + { + desc: "the shortest name with a truncated primary snapshot name (219 characters)", + placementPolicy: newNamedPlacementPolicy(219), + wantLabelTruncated: true, + wantPrimarySanitized: true, + wantSecondarySanitized: true, + }, + { + desc: "the longest valid name (253 characters)", + placementPolicy: newNamedPlacementPolicy(253), + wantLabelTruncated: true, + wantPrimarySanitized: true, + wantSecondarySanitized: true, + }, + { + desc: "the longest valid name of a cluster placement policy (253 characters)", + placementPolicy: newNamedClusterPlacementPolicy(253), + wantLabelTruncated: true, + wantPrimarySanitized: true, + wantSecondarySanitized: true, + }, + } + + BeforeAll(func() { + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 4, groupLabels) + for _, tc := range testCases { + Expect(hubClient.Create(ctx, tc.placementPolicy)).To(Succeed(), "Failed to create the placement policy (%s)", tc.desc) + } + }) + + for _, tc := range testCases { + It(fmt.Sprintf("should create and find snapshots for %s", tc.desc), func() { + placementPolicy := tc.placementPolicy + name := placementPolicy.GetName() + snapshotNS := placementPolicy.GetNamespace() + wantSnapshotNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(name, placementPolicy.GetUID(), 0), + uniquename.SecondaryPlacementResourceSnapshotName(name, placementPolicy.GetUID(), 0, 1), + } + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + + verifySnapshotName(wantSnapshotNames[0], fmt.Sprintf(wantUnsanitizedPrimaryNamePrefixFmt, name, 0), + wantHashedPrimaryNamePattern, tc.wantPrimarySanitized) + verifySnapshotName(wantSnapshotNames[1], fmt.Sprintf(wantUnsanitizedSecondaryNamePrefixFmt, name, 0, 1), + wantHashedSecondaryNamePattern, tc.wantSecondarySanitized) + verifyOwnedByLabelVal(placementPolicy, tc.wantLabelTruncated) + + // The labels (possibly truncated) and the owned-by annotation (always the full name) on the snapshots. + hash := snapshotContentsHash(gotSnapshots[0]) + for subIdx, snapshotName := range wantSnapshotNames { + verifySnapshotMetadata(getSnapshotFromCache(snapshotNS, snapshotName), placementPolicy, 0, subIdx, len(wantSnapshotNames), hash) + } + + // The field index lookups (keyed by the owned-by annotation) and the quorum reads (by name) should + // find the existing snapshots. + gotSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + } + + AfterAll(func() { + objs := make([]client.Object, 0, 3*len(testCases)+len(configMapNames)) + for _, tc := range testCases { + name := tc.placementPolicy.GetName() + uid := tc.placementPolicy.GetUID() + snapshotNS := tc.placementPolicy.GetNamespace() + objs = append(objs, + placementResourceSnapshotObj(snapshotNS, uniquename.PrimaryPlacementResourceSnapshotName(name, uid, 0)), + placementResourceSnapshotObj(snapshotNS, uniquename.SecondaryPlacementResourceSnapshotName(name, uid, 0, 1)), + tc.placementPolicy, + ) + } + objs = append(objs, configMapObjs(configMapNames)...) + deleteObjectsAndWait(objs) + }) + }) + + Context("long name lifecycle and garbage collection", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + placementPolicy := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupLabels)) + placementPolicy.Name = placementPolicyNameOfLen(240) + // Keep only the latest snapshots so that garbage collection kicks in as soon as possible. + placementPolicy.Spec.ResourceRevisionHistoryLimit = ptr.To(int32(1)) + + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var wantSnapshotNames0, wantSnapshotNames1 []string + + // placementBindingCntInCache returns the number of placement bindings that use the primary snapshot at + // index 0, as found via the cache (and its field index). + placementBindingCntInCache := func() (int, error) { + placementBindingList := &placementv1alpha1.PlacementBindingList{} + err := hubClient.List(ctx, placementBindingList, client.InNamespace(appNamespaceName), client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, placementPolicy.Name, wantSnapshotNames0[0]), + }) + return len(placementBindingList.Items), err + } + + BeforeAll(func() { + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 4, groupLabels) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + wantSnapshotNames0 = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1), + } + wantSnapshotNames1 = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 1, 1), + } + }) + + It("can create a primary and a secondary snapshot with truncated names", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames0) + verifySnapshotName(wantSnapshotNames0[0], fmt.Sprintf(wantUnsanitizedPrimaryNamePrefixFmt, placementPolicy.Name, 0), + wantHashedPrimaryNamePattern, true) + verifySnapshotName(wantSnapshotNames0[1], fmt.Sprintf(wantUnsanitizedSecondaryNamePrefixFmt, placementPolicy.Name, 0, 1), + wantHashedSecondaryNamePattern, true) + for _, name := range wantSnapshotNames0 { + getSnapshotFromCache(appNamespaceName, name) + } + }) + + It("should return the existing snapshots when they are not stale", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + It("can create a placement binding that uses the primary snapshot at index 0", func() { + placementBinding := &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementBindingName, + Namespace: appNamespaceName, + OwnerReferences: []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.PlacementPolicyKind, + Name: placementPolicy.Name, + UID: placementPolicy.UID, + }, + }, + }, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: placementPolicy.Name, + ClusterName: "member-1", + ResourceSnapshotName: wantSnapshotNames0[0], + }, + } + Expect(hubClient.Create(ctx, placementBinding)).To(Succeed(), "Failed to create the placement binding") + + // Wait until the cache (and its field index) has observed the placement binding; the garbage collection + // process relies on it to tell if a snapshot is still in use. + Eventually(placementBindingCntInCache, eventuallyDuration, eventuallyInterval).Should(Equal(1), "Failed to find the placement binding in the cache") + }) + + It("can update a selected config map", func() { + updateConfigMapData(appNamespaceName, configMapNames[0], "changed") + }) + + It("should create new snapshots at index 1", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames1) + for _, name := range wantSnapshotNames1 { + getSnapshotFromCache(appNamespaceName, name) + } + }) + + It("should not garbage collect the snapshots at index 0 as they are in use", func() { + // The snapshots at index 0 are now past the revision history limit; this call enqueues them for + // garbage collection, which should skip them as a placement binding is using them. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames1) + + Consistently(func() error { + for _, name := range wantSnapshotNames0 { + if err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: name}, &placementv1alpha1.PlacementResourceSnapshot{}); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The snapshots at index 0 have been garbage collected unexpectedly") + }) + + It("can delete the placement binding", func() { + deleteObjectsAndWait([]client.Object{ + &placementv1alpha1.PlacementBinding{ObjectMeta: metav1.ObjectMeta{Name: placementBindingName, Namespace: appNamespaceName}}, + }) + Eventually(placementBindingCntInCache, eventuallyDuration, eventuallyInterval).Should(Equal(0), "The placement binding is still found in the cache") + }) + + It("should garbage collect the primary and secondary snapshots at index 0", func() { + // This call enqueues the snapshots at index 0 for garbage collection again. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames1) + + for _, name := range wantSnapshotNames0 { + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: name}, &placementv1alpha1.PlacementResourceSnapshot{})) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Placement resource snapshot %s has not been garbage collected", name) + } + for _, name := range wantSnapshotNames1 { + verifySnapshotExists(appNamespaceName, name) + } + }) + + AfterAll(func() { + deleteObjectsAndWait(slices.Concat( + snapshotObjs(appNamespaceName, slices.Concat(wantSnapshotNames0, wantSnapshotNames1)...), + []client.Object{ + &placementv1alpha1.PlacementBinding{ObjectMeta: metav1.ObjectMeta{Name: placementBindingName, Namespace: appNamespaceName}}, + placementPolicy, + }, + configMapObjs(configMapNames), + )) + }) + }) + + Context("dotted names", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + // The names share the same random suffix; `ab.c` and `a.bc` collide after dot removal. + suffix := utils.RandStr() + + type dottedNameTestCase struct { + placementPolicy *placementv1alpha1.PlacementPolicy + wantSanitized bool + // The wanted prefix of the (sanitized) primary snapshot name. + wantPrimaryNamePrefix string + } + + newNamedPlacementPolicy := func(name string) *placementv1alpha1.PlacementPolicy { + placementPolicy := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupLabels)) + placementPolicy.Name = name + return placementPolicy + } + + testCases := []dottedNameTestCase{ + { + placementPolicy: newNamedPlacementPolicy("policy." + suffix + ".v1"), + wantSanitized: true, + wantPrimaryNamePrefix: "policy" + suffix + "v1-resource-snapshot-0-", + }, + { + placementPolicy: newNamedPlacementPolicy("ab.c" + suffix), + wantSanitized: true, + wantPrimaryNamePrefix: "abc" + suffix + "-resource-snapshot-0-", + }, + { + placementPolicy: newNamedPlacementPolicy("a.bc" + suffix), + wantSanitized: true, + wantPrimaryNamePrefix: "abc" + suffix + "-resource-snapshot-0-", + }, + { + placementPolicy: newNamedPlacementPolicy("abc" + suffix), + }, + } + + wantSnapshotNamesOf := func(placementPolicy placementv1alpha1.PlacementPolicyAccessor) []string { + return []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0, 1), + } + } + + BeforeAll(func() { + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 4, groupLabels) + for _, tc := range testCases { + Expect(hubClient.Create(ctx, tc.placementPolicy)).To(Succeed(), "Failed to create placement policy %s", tc.placementPolicy.Name) + } + }) + + for _, tc := range testCases { + It(fmt.Sprintf("can create snapshots for a placement policy named %s", tc.placementPolicy.Name), func() { + placementPolicy := tc.placementPolicy + wantSnapshotNames := wantSnapshotNamesOf(placementPolicy) + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + + verifySnapshotName(wantSnapshotNames[0], fmt.Sprintf(wantUnsanitizedPrimaryNamePrefixFmt, placementPolicy.Name, 0), + wantHashedPrimaryNamePattern, tc.wantSanitized) + verifySnapshotName(wantSnapshotNames[1], fmt.Sprintf(wantUnsanitizedSecondaryNamePrefixFmt, placementPolicy.Name, 0, 1), + wantHashedSecondaryNamePattern, tc.wantSanitized) + if tc.wantSanitized { + Expect(wantSnapshotNames[0]).To(HavePrefix(tc.wantPrimaryNamePrefix), "Unexpected prefix of the sanitized snapshot name") + } + + // The name is short enough; both the owned-by label (dots are allowed in label values) and the + // owned-by annotation should be the name itself. + verifyOwnedByLabelVal(placementPolicy, false) + hash := snapshotContentsHash(gotSnapshots[0]) + for subIdx, name := range wantSnapshotNames { + verifySnapshotMetadata(getSnapshotFromCache(appNamespaceName, name), placementPolicy, 0, subIdx, len(wantSnapshotNames), hash) + } + }) + } + + It("should keep the snapshots of names that collide after dot removal apart", func() { + allSnapshotNames := sets.New[string]() + for _, tc := range testCases { + allSnapshotNames.Insert(wantSnapshotNamesOf(tc.placementPolicy)...) + } + Expect(allSnapshotNames.Len()).To(Equal(2*len(testCases)), "Some placement resource snapshot names collide") + + for _, tc := range testCases { + wantSnapshotNames := wantSnapshotNamesOf(tc.placementPolicy) + verifyOwnedSnapshots(tc.placementPolicy, wantSnapshotNames) + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, tc.placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + } + }) + + AfterAll(func() { + var objs []client.Object + for _, tc := range testCases { + for _, name := range wantSnapshotNamesOf(tc.placementPolicy) { + objs = append(objs, placementResourceSnapshotObj(appNamespaceName, name)) + } + objs = append(objs, tc.placementPolicy) + } + objs = append(objs, configMapObjs(configMapNames)...) + deleteObjectsAndWait(objs) + }) + }) + + Context("long names sharing a prefix", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + // The names (240 characters) share the first 229 characters, which is more than what the owned-by label keeps. + prefix := strings.Repeat("b", 229) + placementPolicies := []*placementv1alpha1.PlacementPolicy{ + newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)), + } + for _, placementPolicy := range placementPolicies { + placementPolicy.Name = prefix + "-" + utils.RandStr() + } + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + for _, placementPolicy := range placementPolicies { + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + } + }) + + It("can create snapshots for both placement policies", func() { + for _, placementPolicy := range placementPolicies { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + verifySnapshotName(gotSnapshots[0].GetName(), fmt.Sprintf(wantUnsanitizedPrimaryNamePrefixFmt, placementPolicy.Name, 0), + wantHashedPrimaryNamePattern, true) + } + }) + + It("should give each placement policy distinct owned-by labels and annotations", func() { + labelVals := make([]string, 0, len(placementPolicies)) + for _, placementPolicy := range placementPolicies { + verifyOwnedByLabelVal(placementPolicy, true) + labelVals = append(labelVals, placementPolicyOwnerLabelVal(placementPolicy)) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + verifySnapshotMetadata(snapshot, placementPolicy, 0, 0, 1, snapshotContentsHash(snapshot)) + } + Expect(labelVals[0]).ToNot(Equal(labelVals[1]), "The placement policies share the same owned-by label value") + }) + + It("should keep the snapshots apart", func() { + for _, placementPolicy := range placementPolicies { + wantSnapshotNames := []string{uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0)} + verifyOwnedSnapshots(placementPolicy, wantSnapshotNames) + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNames) + } + }) + + AfterAll(func() { + objs := make([]client.Object, 0, 2*len(placementPolicies)+1) + for _, placementPolicy := range placementPolicies { + objs = append(objs, primarySnapshotObjs(placementPolicy, 1)...) + objs = append(objs, placementPolicy) + } + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + deleteObjectsAndWait(objs) + }) + }) + + Context("same names in different namespaces and scopes", Ordered, func() { + name := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + namespaceName := fmt.Sprintf(namespaceNameTemplate, utils.RandStr()) + // The config maps in the two namespaces share the same name. + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + appPlacementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + appPlacementPolicy.Name = name + nsPlacementPolicy := newPlacementPolicy(namespaceName, configMapNameSelector(configMapName)) + nsPlacementPolicy.Name = name + clusterSelector := configMapNameSelector(configMapName) + clusterSelector.Namespace = namespaceName + clusterPlacementPolicy := newClusterPlacementPolicy(clusterSelector) + clusterPlacementPolicy.Name = name + + placementPolicies := []placementv1alpha1.PlacementPolicyAccessor{appPlacementPolicy, nsPlacementPolicy, clusterPlacementPolicy} + // The placement policies share the same name, but their snapshot names differ, as the names factor in + // the UIDs of the placement policies. + primarySnapshotNameOf := func(placementPolicy placementv1alpha1.PlacementPolicyAccessor, idx int) string { + return uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), idx) + } + + BeforeAll(func() { + createNamespace(namespaceName) + for _, ns := range []string{appNamespaceName, namespaceName} { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: ns}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map in namespace %s", ns) + } + for _, placementPolicy := range placementPolicies { + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create placement policy %T", placementPolicy) + } + }) + + It("can create snapshots for all three placement policies", func() { + // The snapshots live in different namespaces (or at the cluster scope). + for _, placementPolicy := range placementPolicies { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, []string{primarySnapshotNameOf(placementPolicy, 0)}) + getPrimarySnapshotFromCache(placementPolicy, 0) + } + }) + + It("can update the config map in the app namespace", func() { + updateConfigMapData(appNamespaceName, configMapName, "changed") + }) + + It("should create a new snapshot only for the placement policy that selects the updated config map", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, appPlacementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, []string{primarySnapshotNameOf(appPlacementPolicy, 1)}) + + for _, placementPolicy := range []placementv1alpha1.PlacementPolicyAccessor{nsPlacementPolicy, clusterPlacementPolicy} { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, []string{primarySnapshotNameOf(placementPolicy, 0)}) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + } + }) + + AfterAll(func() { + objs := primarySnapshotObjs(appPlacementPolicy, 2) + objs = append(objs, primarySnapshotObjs(nsPlacementPolicy, 1)...) + objs = append(objs, primarySnapshotObjs(clusterPlacementPolicy, 1)...) + for _, placementPolicy := range placementPolicies { + objs = append(objs, placementPolicy) + } + objs = append(objs, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: namespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("owned-by label collision", Ordered, func() { + groupALabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + groupBLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefixA := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + configMapNamePrefixB := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + // Placement policy A has a long name, and placement policy B is named after the (truncated) owned-by label + // value of placement policy A; the two share the same owned-by label value. + placementPolicyA := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupALabels)) + placementPolicyA.Name = placementPolicyNameOfLen(240) + placementPolicyB := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupBLabels)) + placementPolicyB.Name = placementPolicyOwnerLabelVal(placementPolicyA) + + // The snapshot names depend on the UIDs of the placement policies; they are set once the policies are created. + var wantSnapshotNamesA0, wantSnapshotNamesA1, wantSnapshotNamesB0 []string + + BeforeAll(func() { + configMapNames = append(createLabeledConfigMaps(configMapNamePrefixA, 4, groupALabels), + createLabeledConfigMaps(configMapNamePrefixB, 1, groupBLabels)...) + Expect(hubClient.Create(ctx, placementPolicyA)).To(Succeed(), "Failed to create placement policy A") + Expect(hubClient.Create(ctx, placementPolicyB)).To(Succeed(), "Failed to create placement policy B") + + wantSnapshotNamesA0 = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyA.Name, placementPolicyA.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyA.Name, placementPolicyA.UID, 0, 1), + } + wantSnapshotNamesA1 = []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyA.Name, placementPolicyA.UID, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyA.Name, placementPolicyA.UID, 1, 1), + } + wantSnapshotNamesB0 = []string{uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyB.Name, placementPolicyB.UID, 0)} + }) + + It("can create snapshots for the long-named placement policy", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicyA) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNamesA0) + for _, name := range wantSnapshotNamesA0 { + getSnapshotFromCache(appNamespaceName, name) + } + }) + + It("should give both placement policies the same owned-by label but different annotations", func() { + Expect(placementPolicyOwnerLabelVal(placementPolicyB)).To(Equal(placementPolicyOwnerLabelVal(placementPolicyA)), + "The placement policies do not share the same owned-by label value") + snapshot := getSnapshotFromCache(appNamespaceName, wantSnapshotNamesA0[0]) + Expect(snapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey]).To(Equal(placementPolicyB.Name), + "Unexpected owned-by label value") + Expect(snapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey]).To(Equal(placementPolicyA.Name), + "Unexpected owned-by annotation value") + }) + + It("should not return the snapshots of the other placement policy", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicyB) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNamesB0) + getSnapshotFromCache(appNamespaceName, wantSnapshotNamesB0[0]) + }) + + It("should keep each placement policy's snapshot indices independent", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicyA) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNamesA0) + gotSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicyB) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNamesB0) + + verifyOwnedSnapshots(placementPolicyA, wantSnapshotNamesA0) + verifyOwnedSnapshots(placementPolicyB, wantSnapshotNamesB0) + }) + + It("can update a config map selected by the long-named placement policy", func() { + updateConfigMapData(appNamespaceName, configMapNamePrefixA+"-0", "changed") + }) + + It("should create new snapshots only for that placement policy", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicyA) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNamesA1) + for _, name := range wantSnapshotNamesA1 { + getSnapshotFromCache(appNamespaceName, name) + } + + gotSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicyB) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantSnapshotNamesB0) + verifyNoPrimarySnapshotAtIdx(placementPolicyB, 1) + }) + + AfterAll(func() { + deleteObjectsAndWait(slices.Concat( + snapshotObjs(appNamespaceName, slices.Concat(wantSnapshotNamesA0, wantSnapshotNamesA1, wantSnapshotNamesB0)...), + []client.Object{placementPolicyA, placementPolicyB}, + configMapObjs(configMapNames), + )) + }) + }) +}) + +var _ = Describe("corner cases (leftover snapshots from previous placement policies)", func() { + // The envtest environment does not run the Kubernetes GC: deleting a placement policy leaves its placement resource + // snapshots behind, still owned by the deleted placement policy, which is exactly the state where the Kubernetes GC + // has yet to collect them. Where needed, the test cases below stand in for the Kubernetes GC: they delete the + // snapshots by hand, or remove their owner references to simulate a deletion with the orphan propagation policy. + + Context("placement policy re-created before the snapshots of the previous one are gone", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var placementPolicy placementv1alpha1.PlacementPolicyAccessor = newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + + var pausedMgr *Manager + // The snapshot name depends on the UID of the (previous) placement policy; it is set once the policy is created. + var previousSnapshotName string + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + previousSnapshotName = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0) + pausedMgr = newPausedManager() + }) + + It("can create a snapshot", func() { + snapshotAndVerify(pausedMgr, placementPolicy, []string{previousSnapshotName}) + }) + + It("can delete and re-create the placement policy", func() { + placementPolicy = recreatePlacementPolicy(placementPolicy) + // The snapshot is still owned by the previous placement policy. + verifySnapshotExists(appNamespaceName, previousSnapshotName) + }) + + It("should block snapshotting with a transient error", func() { + gotSnapshots, isUpToDate, err := pausedMgr.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, staleOwnerSnapshotErrMsg) + gotSnapshots, isUpToDate, err = pausedMgr.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, staleOwnerSnapshotErrMsg) + + // The snapshot of the previous placement policy is left to the Kubernetes GC; no GC request is enqueued. + verifyGCRequests(drainGCRequests(pausedMgr), nil) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 0) + Expect(checkOwnedSnapshots(placementPolicy, []string{previousSnapshotName})).To(Succeed()) + }) + + It("should keep blocking snapshotting after a resource change", func() { + updateConfigMapData(appNamespaceName, configMapName, "changed") + gotSnapshots, isUpToDate, err := pausedMgr.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, staleOwnerSnapshotErrMsg) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + It("can delete the snapshot of the previous placement policy (in place of the Kubernetes GC)", func() { + deleteObjectsAndWait(snapshotObjs(appNamespaceName, previousSnapshotName)) + waitForSnapshotGoneFromCache(appNamespaceName, previousSnapshotName) + }) + + It("should create a snapshot owned by the re-created placement policy, under a new name", func() { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0) + Expect(snapshotName).ToNot(Equal(previousSnapshotName), "The snapshot name of the previous placement policy is reused") + + snapshotAndVerify(pausedMgr, placementPolicy, []string{snapshotName}) + if diff := cmp.Diff(getPrimarySnapshotFromCache(placementPolicy, 0).GetOwnerReferences(), wantPlacementPolicyOwnerRefs(placementPolicy)); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot %s owner references diff (-got +want):\n%s", snapshotName, diff)) + } + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + AfterAll(func() { + if pausedMgr != nil { + pausedMgr.gcwq.ShutDown() + } + objs := snapshotObjs(appNamespaceName, + previousSnapshotName, + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0), + ) + objs = append(objs, placementPolicy, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + deleteObjectsAndWait(objs) + }) + }) + + // The scenario is run for both placement policies and cluster placement policies. + for _, clusterScoped := range []bool{false, true} { + desc := "placement policy re-created after the previous one is deleted with the orphan propagation policy" + if clusterScoped { + desc = "cluster placement policy re-created after the previous one is deleted with the orphan propagation policy" + } + + Context(desc, Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + var configMapNames []string + + resourceSelector := configMapLabelSelector(groupLabels) + var placementPolicy placementv1alpha1.PlacementPolicyAccessor + if clusterScoped { + resourceSelector.Namespace = appNamespaceName + placementPolicy = newClusterPlacementPolicy(resourceSelector) + } else { + placementPolicy = newPlacementPolicy(appNamespaceName, resourceSelector) + } + snapshotNS := placementPolicy.GetNamespace() + + var pausedMgr *Manager + // The snapshots of the previous placement policy; the names are set once the policy is created. + var previousSnapshotNames []string + var previousPrimarySnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + + BeforeAll(func() { + // Four config maps make two groups, as each snapshot can hold at most three resources. + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 4, groupLabels) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + previousSnapshotNames = slices.Concat(snapshotSetNames(placementPolicy, 0, 2), snapshotSetNames(placementPolicy, 1, 2)) + pausedMgr = newPausedManager() + }) + + It("can create snapshots at index 0 and 1", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, snapshotSetNames(placementPolicy, 0, 2)) + updateConfigMapData(appNamespaceName, configMapNames[0], "changed") + snapshotAndVerify(resourceSnapshotManager, placementPolicy, snapshotSetNames(placementPolicy, 1, 2)) + previousPrimarySnapshots = []placementv1alpha1.PlacementResourceSnapshotAccessor{ + getPrimarySnapshotFromCache(placementPolicy, 0), + getPrimarySnapshotFromCache(placementPolicy, 1), + } + }) + + It("can create a placement binding without an owner reference that uses the snapshot at index 1", func() { + // The placement binding mimics one orphaned along with the snapshots. + createPlacementBinding(placementPolicy, placementBindingName, previousPrimarySnapshots[1].GetName(), nil) + waitForPlacementBindingCntInCache(placementPolicy, previousPrimarySnapshots[1].GetName(), 1) + }) + + It("can orphan the snapshots and re-create the placement policy", func() { + for _, name := range previousSnapshotNames { + setSnapshotOwnerRefs(snapshotNS, name, nil) + } + placementPolicy = recreatePlacementPolicy(placementPolicy) + }) + + It("should block snapshotting and enqueue the orphaned primary snapshots for GC", func() { + wantGCRequests := []snapshotGarbageCollectionRequest{ + gcRequestFor(placementPolicy, previousPrimarySnapshots[0]), + gcRequestFor(placementPolicy, previousPrimarySnapshots[1]), + } + + gotSnapshots, isUpToDate, err := pausedMgr.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, leftoverSnapshotsErrMsg) + verifyGCRequests(drainGCRequests(pausedMgr), wantGCRequests) + + gotSnapshots, isUpToDate, err = pausedMgr.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, leftoverSnapshotsErrMsg) + verifyGCRequests(drainGCRequests(pausedMgr), wantGCRequests) + + verifyNoPrimarySnapshotAtIdx(placementPolicy, 0) + }) + + It("should garbage collect all the orphaned snapshots, even though a placement binding uses one of them", func() { + // Unlike the paused manager, the manager under test runs its GC worker. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, leftoverSnapshotsErrMsg) + Eventually(func() error { + return checkOwnedSnapshots(placementPolicy, nil) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect the orphaned snapshots") + }) + + It("should create snapshots owned by the re-created placement policy, under new names", func() { + wantNames := snapshotSetNames(placementPolicy, 0, 2) + for _, name := range wantNames { + Expect(previousSnapshotNames).ToNot(ContainElement(name), "A snapshot name of the previous placement policy is reused") + } + + // The cache might take a moment to observe the removal of the orphaned snapshots. + Eventually(func() error { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + if err != nil { + return err + } + if diff := cmp.Diff(snapshotNamesOf(gotSnapshots), wantNames); diff != "" || !isUpToDate { + return fmt.Errorf("unexpected snapshots (up-to-date %t), names diff (-got +want):\n%s", isUpToDate, diff) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create snapshots for the re-created placement policy") + Expect(checkOwnedSnapshots(placementPolicy, wantNames)).To(Succeed()) + if diff := cmp.Diff(getPrimarySnapshotFromCache(placementPolicy, 0).GetOwnerReferences(), wantPlacementPolicyOwnerRefs(placementPolicy)); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot %s owner references diff (-got +want):\n%s", wantNames[0], diff)) + } + }) + + It("should leave the placement binding untouched", func() { + placementBinding := placementBindingObj(snapshotNS, placementBindingName) + Expect(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(placementBinding), placementBinding)).To(Succeed(), + "Failed to get the placement binding") + Expect(placementBinding.GetSpec().ResourceSnapshotName).To(Equal(previousPrimarySnapshots[1].GetName()), + "The placement binding has been changed") + }) + + AfterAll(func() { + if pausedMgr != nil { + pausedMgr.gcwq.ShutDown() + } + objs := snapshotObjs(snapshotNS, slices.Concat(previousSnapshotNames, snapshotSetNames(placementPolicy, 0, 2))...) + objs = append(objs, placementBindingObj(snapshotNS, placementBindingName), placementPolicy) + objs = append(objs, configMapObjs(configMapNames)...) + deleteObjectsAndWait(objs) + }) + }) + } + + Context("orphaned primary snapshot pending deletion", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var placementPolicy placementv1alpha1.PlacementPolicyAccessor = newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + + // The snapshot name depends on the UID of the (previous) placement policy; it is set once the policy is created. + var previousSnapshotName string + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + previousSnapshotName = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0) + }) + + It("can create a snapshot", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{previousSnapshotName}) + }) + + It("can orphan the snapshot, block its deletion with a finalizer, and re-create the placement policy", func() { + addSnapshotFinalizer(appNamespaceName, previousSnapshotName) + setSnapshotOwnerRefs(appNamespaceName, previousSnapshotName, nil) + placementPolicy = recreatePlacementPolicy(placementPolicy) + }) + + It("should keep blocking snapshotting while the orphaned snapshot is pending deletion", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, leftoverSnapshotsErrMsg) + + // The GC process deletes the orphaned snapshot, which the finalizer keeps around. + Eventually(func() error { + snapshot := placementResourceSnapshotObj(appNamespaceName, previousSnapshotName) + if err := hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: previousSnapshotName}, snapshot); err != nil { + return err + } + if snapshot.GetDeletionTimestamp().IsZero() { + return fmt.Errorf("the orphaned snapshot has not been marked for deletion yet") + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the orphaned snapshot for deletion") + + Consistently(func() error { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + return checkSnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, leftoverSnapshotsErrMsg) + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Snapshotting is not blocked while the orphaned snapshot is pending deletion") + verifyNoPrimarySnapshotAtIdx(placementPolicy, 0) + }) + + It("should create a snapshot once the orphaned snapshot is gone", func() { + removeFinalizersAndDelete(placementResourceSnapshotObj(appNamespaceName, previousSnapshotName)) + waitForSnapshotGoneFromCache(appNamespaceName, previousSnapshotName) + + snapshotAndVerify(resourceSnapshotManager, placementPolicy, + []string{uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0)}) + }) + + AfterAll(func() { + removeFinalizersAndDelete(placementResourceSnapshotObj(appNamespaceName, previousSnapshotName)) + objs := snapshotObjs(appNamespaceName, uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), 0)) + objs = append(objs, placementPolicy, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + deleteObjectsAndWait(objs) + }) + }) + + Context("snapshot with unexpected owner references", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + + var pausedMgr *Manager + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var snapshotName0, snapshotName1 string + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + snapshotName0 = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0) + snapshotName1 = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 1) + pausedMgr = newPausedManager() + }) + + It("can create a snapshot", func() { + snapshotAndVerify(pausedMgr, placementPolicy, []string{snapshotName0}) + }) + + It("can add a second owner reference to the snapshot", func() { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + ownerRefs := append(wantPlacementPolicyOwnerRefs(placementPolicy), metav1.OwnerReference{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMap.Name, + UID: configMap.UID, + }) + setSnapshotOwnerRefs(appNamespaceName, snapshotName0, ownerRefs) + }) + + It("should block snapshotting with a user error that names the snapshot", func() { + gotSnapshots, isUpToDate, err := pausedMgr.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryUser, unexpectedOwnerRefsErrMsg, snapshotName0) + gotSnapshots, isUpToDate, err = pausedMgr.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryUser, unexpectedOwnerRefsErrMsg, snapshotName0) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + It("should keep blocking snapshotting after a resource change", func() { + updateConfigMapData(appNamespaceName, configMapName, "changed") + gotSnapshots, isUpToDate, err := pausedMgr.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryUser, unexpectedOwnerRefsErrMsg, snapshotName0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + It("can remove the extra owner reference", func() { + setSnapshotOwnerRefs(appNamespaceName, snapshotName0, wantPlacementPolicyOwnerRefs(placementPolicy)) + }) + + It("should create a snapshot for the resource change", func() { + snapshotAndVerify(pausedMgr, placementPolicy, []string{snapshotName1}) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + AfterAll(func() { + if pausedMgr != nil { + pausedMgr.gcwq.ShutDown() + } + objs := snapshotObjs(appNamespaceName, snapshotName0, snapshotName1) + objs = append(objs, placementPolicy, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + deleteObjectsAndWait(objs) + }) + }) + + Context("leftover secondary snapshots without a primary snapshot", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + + // The leftover secondary snapshots mimic ones created by a previous placement policy of the same name in + // interrupted snapshot creation attempts, and then orphaned; they are named after the previous placement policy. + previousPlacementPolicyUID := types.UID(utils.RandStr()) + leftoverSnapshotName0 := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, previousPlacementPolicyUID, 0, 1) + leftoverSnapshotName1 := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, previousPlacementPolicyUID, 1, 1) + + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var snapshotName0, snapshotName1 string + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + // The leftover secondary snapshots are indexed (they carry the owned-by annotation and the index labels), + // but have no owners. + createDecoySnapshot(appNamespaceName, leftoverSnapshotName0, decoySecondaryLabels(placementPolicy, "0", "1"), decoyOwnedByAnnotations(placementPolicy), nil) + createDecoySnapshot(appNamespaceName, leftoverSnapshotName1, decoySecondaryLabels(placementPolicy, "1", "1"), decoyOwnedByAnnotations(placementPolicy), nil) + getSnapshotFromCache(appNamespaceName, leftoverSnapshotName0) + getSnapshotFromCache(appNamespaceName, leftoverSnapshotName1) + + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + snapshotName0 = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0) + snapshotName1 = uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 1) + }) + + It("should not block snapshotting, but clean up the leftover secondary snapshot at index 0 first", func() { + // Only primary snapshots are checked for leftovers; the leftover secondary snapshot at index 0 is cleaned + // up as an orphan right before the manager creates snapshots at index 0. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, orphansCleanedUpErrMsg) + waitForSnapshotGoneFromCache(appNamespaceName, leftoverSnapshotName0) + verifySnapshotExists(appNamespaceName, leftoverSnapshotName1) + }) + + It("should create a snapshot at index 0, leaving the leftover secondary snapshot at index 1 behind", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{snapshotName0}) + verifySnapshotExists(appNamespaceName, leftoverSnapshotName1) + }) + + It("should clean up the leftover secondary snapshot at index 1 before creating a snapshot there", func() { + updateConfigMapData(appNamespaceName, configMapName, "changed") + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, orphansCleanedUpErrMsg) + waitForSnapshotGoneFromCache(appNamespaceName, leftoverSnapshotName1) + + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{snapshotName1}) + }) + + AfterAll(func() { + objs := snapshotObjs(appNamespaceName, leftoverSnapshotName0, leftoverSnapshotName1, snapshotName0, snapshotName1) + objs = append(objs, placementPolicy, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + deleteObjectsAndWait(objs) + }) + }) +}) diff --git a/pkg/v1/managers/placementresourcesnapshot/gc.go b/pkg/v1/managers/placementresourcesnapshot/gc.go new file mode 100644 index 000000000..e3f3d722a --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/gc.go @@ -0,0 +1,395 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementresourcesnapshot + +import ( + "context" + "fmt" + "strconv" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/klog/v2" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + errors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +type snapshotGarbageCollectionRequest struct { + snapshotNamespacedName types.NamespacedName + snapshotUID types.UID + + placementPolicyNamespacedName types.NamespacedName +} + +// enqueueOldResourceSnapshotsForGC enqueues old resource snapshots for garbage collection based on the +// revision history limit of the placement policy. +func (m *Manager) enqueueOldResourceSnapshotsForGC( + sortedSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, + placementPolicyAccessor placementv1alpha1.PlacementPolicyAccessor, +) { + // Exclude the most recent snapshot from the history. + history := sortedSnapshots[:len(sortedSnapshots)-1] + revLimit := defaultRevisionHistoryLimit + if placementPolicyAccessor.GetSpec().ResourceRevisionHistoryLimit != nil { + revLimit = *placementPolicyAccessor.GetSpec().ResourceRevisionHistoryLimit + } + if revLimit <= 0 { + // Do a sanity check. + revLimit = 1 + } + + if len(history)+1 <= int(revLimit) { + // No old snapshots to enqueue for GC. + return + } + history = history[:len(history)+1-int(revLimit)] + for _, snapshot := range history { + if !snapshot.GetDeletionTimestamp().IsZero() { + // Skip snapshots that are already marked for deletion. + continue + } + m.gcwq.Add(snapshotGarbageCollectionRequest{ + snapshotNamespacedName: types.NamespacedName{ + Namespace: snapshot.GetNamespace(), + Name: snapshot.GetName(), + }, + snapshotUID: snapshot.GetUID(), + placementPolicyNamespacedName: types.NamespacedName{ + Namespace: placementPolicyAccessor.GetNamespace(), + Name: placementPolicyAccessor.GetName(), + }, + }) + } +} + +// checkForInconsistentResourceSnapshots scans the given (sorted) primary resource snapshots of a placement policy for +// snapshots that are not owned by the placement policy, e.g., those left over from a previous placement policy of +// the same name. It returns the leftover (ownerless) primary resource snapshots found, which are enqueued for +// garbage collection (unless already marked for deletion). +// +// Note that only primary resource snapshots are checked here. A secondary resource snapshot without a primary, i.e., +// one created by a previous placement policy of the same name in an interrupted snapshot creation attempt and then +// orphaned (e.g., the placement policy is deleted with the orphan propagation policy), will not be detected; nor will +// the Kubernetes GC remove it, as it has no owners. This is considered acceptable, as such a snapshot is harmless: +// it is not referenced by any object, and it never gets picked up when listing snapshots by index, as any secondary +// snapshots left over at an index are always cleaned up (see cleanUpOrphanedSecondarySnapshots) before the placement +// policy creates new snapshots at that index. The snapshot will linger until then (or until it is removed manually). +func (m *Manager) checkForInconsistentResourceSnapshots( + sortedSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, + placementPolicyAccessor placementv1alpha1.PlacementPolicyAccessor, +) (leftovers []klog.ObjectRef, err error) { + for idx := range sortedSnapshots { + snapshot := sortedSnapshots[idx] + + // Check if the snapshot has an owner reference to the same named placement policy, but with a different UID. + // + // This signals that we might be observing an inconsistent state where the Kubernetes garbage collector (run by + // kube-controller-manager) has yet to finish; return a transient error and wait for the situation to resolve. + ownerRef := placementPolicyOwnerRef(snapshot, placementPolicyAccessor.GetName()) + if ownerRef != nil && ownerRef.UID != placementPolicyAccessor.GetUID() { + // Err out right away; no need to keep scanning. + return leftovers, errors.NewTransientError(nil, + "found a placement resource snapshot owned by a different placement policy of the same name (e.g., one that has been deleted); waiting for garbage collection to complete", + "placementResourceSnapshot", klog.KObj(snapshot), "observedUID", ownerRef.UID, "expectedUID", placementPolicyAccessor.GetUID()) + } + + // Check if the snapshot has no owners. If so, we might have caught a situation where the snapshot is orphaned. + // Enqueue the snapshot for garbage collection, unless it is already marked for deletion; either way, + // snapshotting stays blocked until the snapshot is gone. + if len(snapshot.GetOwnerReferences()) == 0 { + leftovers = append(leftovers, klog.KObj(snapshot)) + if snapshot.GetDeletionTimestamp().IsZero() { + m.gcwq.Add(snapshotGarbageCollectionRequest{ + snapshotNamespacedName: types.NamespacedName{ + Namespace: snapshot.GetNamespace(), + Name: snapshot.GetName(), + }, + snapshotUID: snapshot.GetUID(), + placementPolicyNamespacedName: types.NamespacedName{ + Namespace: placementPolicyAccessor.GetNamespace(), + Name: placementPolicyAccessor.GetName(), + }, + }) + } + } + + // Handle the unlikely case where the snapshot has other owners (differently named placement policies, etc.). + // This is considered a user error. + if ownerRefs := snapshot.GetOwnerReferences(); len(ownerRefs) > 0 && (ownerRef == nil || len(ownerRefs) > 1) { + return leftovers, errors.NewUserError(nil, + fmt.Sprintf("found a placement resource snapshot (%s) that has unexpected owner references", snapshot.GetName()), + "placementResourceSnapshot", klog.KObj(snapshot), "ownerReferences", ownerRefs, + "placementPolicy", klog.KObj(placementPolicyAccessor)) + } + } + return leftovers, nil +} + +// garbageCollect processes a garbage collection request. +func (m *Manager) garbageCollect(ctx context.Context, snapshotGCRequest snapshotGarbageCollectionRequest) error { + // GC the resource snapshots (the primary and its secondaries). + // + // A GC request is issued for either: + // + // * an outdated primary resource snapshot (one beyond the revision history limit), which is only enqueued after + // the resource snapshot manager has seen at least one newer snapshot; or + // * a primary resource snapshot left over from a previous placement policy of the same name (one without any + // owner references), which might be the latest snapshot that the previous placement policy has. + // + // In both cases, no more snapshots will be created at the index of the primary resource snapshot: KubeFleet + // always creates the snapshots of an index in a specific order (secondaries first, primary last) and only moves + // forward to newer indices afterwards; and the resource snapshot manager blocks any snapshot creation as long as + // there is a primary resource snapshot not owned by the placement policy. This guarantees that the cache can be + // reliably used to list a primary resource snapshot and its secondaries - no snapshots will be missed. + // + // The same mechanisms also ensure that, in normal operations, all the snapshots listed at the index come from + // the same placement policy as the primary resource snapshot: snapshots of a re-created placement policy can only + // be created after all the primary snapshots left over from previous placement policies are gone, and any + // secondary snapshots left over at an index are cleaned up before new snapshots are created at that index (see + // cleanUpOrphanedSecondarySnapshots). + // + // Note, however, that these mechanisms rely on the cache; as snapshot names factor in the UID of the placement + // policy, the API server itself does not prevent snapshots of different placement policies of the same name from + // co-existing at the same index. Under heavy cache lag, the snapshots listed at the index might include those of + // a re-created placement policy. As a sanity check, the GC process skips any listed snapshot that is another + // primary snapshot, or that does not share the same owner (or lack of one) as the primary resource snapshot. + + // Retrieve the primary resource snapshot. + var primarySnapshot placementv1alpha1.PlacementResourceSnapshotAccessor + if snapshotGCRequest.snapshotNamespacedName.Namespace == "" { + primarySnapshot = &placementv1alpha1.ClusterPlacementResourceSnapshot{} + } else { + primarySnapshot = &placementv1alpha1.PlacementResourceSnapshot{} + } + if err := m.hubClient.Get(ctx, snapshotGCRequest.snapshotNamespacedName, primarySnapshot); err != nil { + if apierrors.IsNotFound(err) { + // The primary snapshot has been garbage collected already. + klog.V(2).InfoS("The primary placement resource snapshot is not found; skipping garbage collection", + "snapshotGarbageCollectionRequest", snapshotGCRequest) + return nil + } + return errors.NewAPIServerError(err, "failed to retrieve the primary placement resource snapshot", true) + } + + // Do a sanity check; verify that the UID of the primary snapshot matches the expected UID from the garbage collection request. + if primarySnapshot.GetUID() != snapshotGCRequest.snapshotUID { + // The snapshot has been re-created; consider the GC attempt completed. + klog.V(2).InfoS("The UID of the primary placement resource snapshot does not match the expected UID; skipping garbage collection", + "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot), "expectedUID", snapshotGCRequest.snapshotUID, "observedUID", primarySnapshot.GetUID()) + return nil + } + + if !primarySnapshot.GetDeletionTimestamp().IsZero() { + // The snapshot is already marked for deletion; no further action is needed. + // + // This can happen either because a) a GC attempt has been made, which guarantees that all secondary snapshots + // are also marked for deletion already, or b) the placement policy itself is marked for deletion, which + // would trigger the deletion of all associated snapshots, primary or secondary. + klog.V(2).InfoS("The primary placement resource snapshot is already marked for deletion; skipping garbage collection", + "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot)) + return nil + } + + // Verify that the snapshot is no longer in use by any placement bindings. + // + // This is a best-effort check; if there is an on-going rollout that uses a to-be-GC'd resource snapshot, the + // GC process does not guarantee that the snapshot will be saved. Note that a binding, once synchronized, + // no longer depends on its target resource snapshot; errors will only occur when the binding has not gotten + // a chance to read the GC'd resource snapshot yet, and users will be notified that a newer rollout is needed to + // fix the situation. + // + // TO-DO (chenyu1): evaluate if it is needed to add additional safeguards against such + // premature garbage collection; the chances are quite low in normal ops. + inUse, err := m.isSnapshotInUseByAnyPlacementBinding(ctx, primarySnapshot, snapshotGCRequest.placementPolicyNamespacedName) + if err != nil { + return errors.Wraps(err, "failed to check if the placement resource snapshot is in use by any placement bindings") + } + if inUse { + // The snapshot will be left alone for now; the manager will check again at the next GC attempt. + klog.V(2).InfoS("The placement resource snapshot is still in use by some placement bindings; skipping garbage collection", + "snapshotGarbageCollectionRequest", snapshotGCRequest) + return nil + } + + // Retrieve the index of the primary resource snapshot. + primarySnapshotIdxStr := primarySnapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotIndexLabelKey] + primarySnapshotIdx, err := strconv.Atoi(primarySnapshotIdxStr) + if err != nil { + return errors.NewUnexpectedError(err, "failed to parse the index label of the primary placement resource snapshot", + "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot), "observedIdx", primarySnapshotIdxStr) + } + + // List all the resource snapshots with the same index. + ownedBy := primarySnapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] + if ownedBy == "" { + return errors.NewUnexpectedError(nil, "the primary placement resource snapshot is missing the owned-by annotation", + "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot)) + } + fieldMatchers := client.MatchingFields{ + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, + ownedBy, strconv.Itoa(primarySnapshotIdx)), + } + var snapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + var listErr error + if snapshotGCRequest.snapshotNamespacedName.Namespace == "" { + // The snapshots are cluster-scoped; list cluster placement resource snapshots. + snapshots, listErr = m.listPlacementResourceSnapshots(ctx, true, []client.ListOption{fieldMatchers}) + } else { + // The snapshots are namespace-scoped; list placement resource snapshots in the same namespace. + snapshots, listErr = m.listPlacementResourceSnapshots(ctx, false, []client.ListOption{ + client.InNamespace(snapshotGCRequest.snapshotNamespacedName.Namespace), + fieldMatchers, + }) + } + if listErr != nil { + return listErr + } + + // Retrieve the owner reference of the primary resource snapshot that points to its placement policy (nil if the + // primary resource snapshot has been orphaned); it is used to verify the ownership of the listed snapshots. + primarySnapshotOwnerRef := placementPolicyOwnerRef(primarySnapshot, ownedBy) + + // Delete all the resource snapshots that are not the primary first. + for _, snapshot := range snapshots { + if snapshot.GetName() == primarySnapshot.GetName() { + continue + } + + // As a sanity check, skip any snapshot that is another primary snapshot (sub-index 0), or that does not + // share the same owner (or lack of one) as the primary resource snapshot, e.g., a snapshot of a re-created + // placement policy of the same name. See the notes at the beginning of this method; this can only happen + // under heavy cache lag. + // + // A skipped snapshot is left to its owner (or the Kubernetes GC); should it end up as an ownerless secondary + // snapshot, it is handled the same way as other leftover secondary snapshots (see the notes on + // checkForInconsistentResourceSnapshots). + subIdxStr := snapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey] + if subIdxStr == "0" || !isSamePlacementPolicyOwner(placementPolicyOwnerRef(snapshot, ownedBy), primarySnapshotOwnerRef) { + klog.V(2).InfoS("Found a placement resource snapshot at the same index that is another primary snapshot or has a different owner (the cache might be lagging); skipping its deletion", + "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot), + "skippedPlacementResourceSnapshot", klog.KObj(snapshot), "observedSubIndex", subIdxStr) + continue + } + + if err := m.hubClient.Delete(ctx, snapshot, client.Preconditions{UID: ptr.To(snapshot.GetUID())}); err != nil && !apierrors.IsNotFound(err) { + return errors.NewAPIServerError(err, "failed to delete a secondary placement resource snapshot", false, + "secondaryPlacementResourceSnapshot", klog.KObj(snapshot)) + } + } + + // Delete the primary placement resource snapshot last. + if err := m.hubClient.Delete(ctx, primarySnapshot, client.Preconditions{UID: ptr.To(primarySnapshot.GetUID())}); err != nil && !apierrors.IsNotFound(err) { + return errors.NewAPIServerError(err, "failed to delete the primary placement resource snapshot", false, + "primaryPlacementResourceSnapshot", klog.KObj(primarySnapshot)) + } + klog.V(2).InfoS("Garbage collected the placement resource snapshots", + "snapshotGarbageCollectionRequest", snapshotGCRequest) + + return nil +} + +// isSnapshotInUseByAnyPlacementBinding checks if any placement binding of the placement policy (i.e., one whose spec +// names the placement policy; binding owners are not checked) is using the given (primary) placement resource snapshot. +// +// Note that this method reads from the cache, as the custom field index is only available there. +func (m *Manager) isSnapshotInUseByAnyPlacementBinding(ctx context.Context, + primarySnapshot placementv1alpha1.PlacementResourceSnapshotAccessor, + placementPolicyNamespacedName types.NamespacedName, +) (bool, error) { + placementPolicyName := placementPolicyNamespacedName.Name + snapshotOwnerRef := placementPolicyOwnerRef(primarySnapshot, placementPolicyName) + if snapshotOwnerRef == nil { + // The primary resource snapshot is not owned by the placement policy; always consider such snapshots to + // be safe for GC. + return false, nil + } + + placementPolicyNamespace := placementPolicyNamespacedName.Namespace + fieldMatchers := client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, + placementPolicyName, + primarySnapshot.GetName(), + ), + } + // Only one matching binding is needed to tell if the snapshot is in use. + limit := client.Limit(1) + + if placementPolicyNamespace == "" { + // The placement policy is cluster-scoped; list cluster placement bindings. + placementBindingList := &placementv1alpha1.ClusterPlacementBindingList{} + if err := m.hubClient.List(ctx, placementBindingList, fieldMatchers, limit); err != nil { + return false, errors.NewAPIServerError(err, "failed to list cluster placement bindings", true) + } + if len(placementBindingList.Items) == 0 { + return false, nil + } + } else { + // The placement policy is namespace-scoped; list placement bindings in the same namespace. + placementBindingList := &placementv1alpha1.PlacementBindingList{} + if err := m.hubClient.List(ctx, placementBindingList, + client.InNamespace(placementPolicyNamespace), fieldMatchers, limit); err != nil { + return false, errors.NewAPIServerError(err, "failed to list placement bindings", true) + } + if len(placementBindingList.Items) == 0 { + return false, nil + } + } + // A binding has been found to reference the primary resource snapshot; it is considered in use. + return true, nil +} + +// placementPolicyOwnerRef returns the owner reference on the given object that points to the placement policy +// of the given name; it returns nil if no such owner reference is found. +// +// Note that the placement policy is always of the same scope as the given object. +func placementPolicyOwnerRef(obj client.Object, placementPolicyName string) *metav1.OwnerReference { + wantKind := placementv1alpha1.PlacementPolicyKind + if obj.GetNamespace() == "" { + wantKind = placementv1alpha1.ClusterPlacementPolicyKind + } + + ownerRefs := obj.GetOwnerReferences() + for idx := range ownerRefs { + ownerRef := &ownerRefs[idx] + gv, err := schema.ParseGroupVersion(ownerRef.APIVersion) + if err != nil { + // Skip owner references with malformed API versions. + continue + } + if gv.Group == placementv1alpha1.GroupVersion.Group && ownerRef.Kind == wantKind && ownerRef.Name == placementPolicyName { + return ownerRef + } + } + return nil +} + +// isSamePlacementPolicyOwner returns true if the two given owner references (as returned by placementPolicyOwnerRef) +// point to the same placement policy, or if both are nil (i.e., neither object is owned by the placement policy). +func isSamePlacementPolicyOwner(a, b *metav1.OwnerReference) bool { + if a == nil || b == nil { + return a == nil && b == nil + } + return a.UID == b.UID +} diff --git a/pkg/v1/managers/placementresourcesnapshot/gc_test.go b/pkg/v1/managers/placementresourcesnapshot/gc_test.go new file mode 100644 index 000000000..36a8092c7 --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/gc_test.go @@ -0,0 +1,890 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementresourcesnapshot + +import ( + "context" + "errors" + "fmt" + "slices" + "strconv" + "sync/atomic" + "testing" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/wait" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +// TestCheckForInconsistentResourceSnapshots tests the checkForInconsistentResourceSnapshots method. +func TestCheckForInconsistentResourceSnapshots(t *testing.T) { + const ( + policyName = "policy" + otherPolicyName = "other-policy" + namespaceName = "app" + ) + policyUID := types.UID("policy-uid") + + clusterPolicy := &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: policyName, + UID: policyUID, + }, + } + namespacedPolicy := &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: policyName, + Namespace: namespaceName, + UID: policyUID, + }, + } + + ownerRef := func(kind, name string, uid types.UID) metav1.OwnerReference { + return metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: kind, + Name: name, + UID: uid, + } + } + clusterPolicyOwnerRef := ownerRef(placementv1alpha1.ClusterPlacementPolicyKind, policyName, policyUID) + namespacedPolicyOwnerRef := ownerRef(placementv1alpha1.PlacementPolicyKind, policyName, policyUID) + // The owner reference to a placement policy that has been deleted and then re-created with the same name. + staleClusterPolicyOwnerRef := ownerRef(placementv1alpha1.ClusterPlacementPolicyKind, policyName, "stale-policy-uid") + otherClusterPolicyOwnerRef := ownerRef(placementv1alpha1.ClusterPlacementPolicyKind, otherPolicyName, "other-policy-uid") + + snapshotName := func(idx int) string { + return uniquename.PrimaryPlacementResourceSnapshotName(policyName, policyUID, idx) + } + snapshotUID := func(idx int) types.UID { + return types.UID(fmt.Sprintf("snapshot-%d-uid", idx)) + } + clusterSnapshot := func(idx int, ownerRefs ...metav1.OwnerReference) *placementv1alpha1.ClusterPlacementResourceSnapshot { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName(idx), + UID: snapshotUID(idx), + OwnerReferences: ownerRefs, + }, + } + } + namespacedSnapshot := func(idx int, ownerRefs ...metav1.OwnerReference) *placementv1alpha1.PlacementResourceSnapshot { + return &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName(idx), + Namespace: namespaceName, + UID: snapshotUID(idx), + OwnerReferences: ownerRefs, + }, + } + } + deletionTimestamp := metav1.NewTime(time.Date(2026, time.January, 1, 0, 0, 0, 0, time.UTC)) + // markedForDeletion marks the given cluster placement resource snapshot for deletion. + markedForDeletion := func(snapshot *placementv1alpha1.ClusterPlacementResourceSnapshot) *placementv1alpha1.ClusterPlacementResourceSnapshot { + snapshot.DeletionTimestamp = &deletionTimestamp + return snapshot + } + gcRequest := func(namespace string, idx int) snapshotGarbageCollectionRequest { + return snapshotGarbageCollectionRequest{ + snapshotNamespacedName: types.NamespacedName{Namespace: namespace, Name: snapshotName(idx)}, + snapshotUID: snapshotUID(idx), + placementPolicyNamespacedName: types.NamespacedName{Namespace: namespace, Name: policyName}, + } + } + leftover := func(namespace string, idx int) klog.ObjectRef { + return klog.ObjectRef{Namespace: namespace, Name: snapshotName(idx)} + } + + testCases := []struct { + name string + placementPolicy placementv1alpha1.PlacementPolicyAccessor + sortedSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + wantLeftovers []klog.ObjectRef + // The category of the error that is expected to be returned; leave empty if no error is expected. + wantErrCategory kferrors.ErrCategory + wantGCRequests []snapshotGarbageCollectionRequest + }{ + { + name: "no snapshots", + placementPolicy: clusterPolicy, + }, + { + name: "all snapshots owned by the placement policy", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + clusterSnapshot(0, clusterPolicyOwnerRef), + clusterSnapshot(1, clusterPolicyOwnerRef), + }, + }, + { + name: "orphaned snapshots", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + clusterSnapshot(0), + clusterSnapshot(1, clusterPolicyOwnerRef), + clusterSnapshot(2), + }, + wantLeftovers: []klog.ObjectRef{leftover("", 0), leftover("", 2)}, + wantGCRequests: []snapshotGarbageCollectionRequest{gcRequest("", 0), gcRequest("", 2)}, + }, + { + name: "orphaned snapshot marked for deletion", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + markedForDeletion(clusterSnapshot(0)), + }, + // Snapshotting stays blocked, but no GC request is enqueued again. + wantLeftovers: []klog.ObjectRef{leftover("", 0)}, + }, + { + name: "orphaned snapshots, one of which is marked for deletion", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + markedForDeletion(clusterSnapshot(0)), + clusterSnapshot(1), + clusterSnapshot(2, clusterPolicyOwnerRef), + }, + wantLeftovers: []klog.ObjectRef{leftover("", 0), leftover("", 1)}, + wantGCRequests: []snapshotGarbageCollectionRequest{gcRequest("", 1)}, + }, + { + name: "snapshot owned by a deleted placement policy with the same name", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + clusterSnapshot(0, staleClusterPolicyOwnerRef), + clusterSnapshot(1, clusterPolicyOwnerRef), + }, + wantErrCategory: kferrors.ErrCategoryTransient, + }, + { + name: "orphaned snapshot followed by a snapshot owned by a deleted placement policy with the same name", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + clusterSnapshot(0), + clusterSnapshot(1, staleClusterPolicyOwnerRef), + }, + wantLeftovers: []klog.ObjectRef{leftover("", 0)}, + wantErrCategory: kferrors.ErrCategoryTransient, + wantGCRequests: []snapshotGarbageCollectionRequest{gcRequest("", 0)}, + }, + { + name: "snapshot owned by a differently named placement policy", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + clusterSnapshot(0, otherClusterPolicyOwnerRef), + }, + wantErrCategory: kferrors.ErrCategoryUser, + }, + { + name: "snapshot owned by the placement policy and another owner", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + clusterSnapshot(0, clusterPolicyOwnerRef, otherClusterPolicyOwnerRef), + }, + wantErrCategory: kferrors.ErrCategoryUser, + }, + { + name: "snapshot with an owner reference of a mismatched kind", + placementPolicy: clusterPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + clusterSnapshot(0, namespacedPolicyOwnerRef), + }, + wantErrCategory: kferrors.ErrCategoryUser, + }, + { + name: "namespaced snapshots owned by the placement policy", + placementPolicy: namespacedPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + namespacedSnapshot(0, namespacedPolicyOwnerRef), + namespacedSnapshot(1, namespacedPolicyOwnerRef), + }, + }, + { + name: "orphaned namespaced snapshot", + placementPolicy: namespacedPolicy, + sortedSnapshots: []placementv1alpha1.PlacementResourceSnapshotAccessor{ + namespacedSnapshot(0), + namespacedSnapshot(1, namespacedPolicyOwnerRef), + }, + wantLeftovers: []klog.ObjectRef{leftover(namespaceName, 0)}, + wantGCRequests: []snapshotGarbageCollectionRequest{gcRequest(namespaceName, 0)}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gcwq := workqueue.NewTypedRateLimitingQueue(workqueue.DefaultTypedControllerRateLimiter[snapshotGarbageCollectionRequest]()) + defer gcwq.ShutDown() + m := &Manager{gcwq: gcwq} + + gotLeftovers, err := m.checkForInconsistentResourceSnapshots(tc.sortedSnapshots, tc.placementPolicy) + switch { + case tc.wantErrCategory != "" && err == nil: + t.Errorf("checkForInconsistentResourceSnapshots() = %v, nil, want error of category %v", gotLeftovers, tc.wantErrCategory) + case tc.wantErrCategory != "": + verifyErrCategory(t, "checkForInconsistentResourceSnapshots()", err, tc.wantErrCategory) + case err != nil: + t.Errorf("checkForInconsistentResourceSnapshots() = %v, want no error", err) + } + if diff := cmp.Diff(gotLeftovers, tc.wantLeftovers, cmpopts.EquateEmpty()); diff != "" { + t.Errorf("checkForInconsistentResourceSnapshots() leftovers mismatch (-got +want):\n%s", diff) + } + + gotGCRequests := make([]snapshotGarbageCollectionRequest, 0, gcwq.Len()) + for gcwq.Len() > 0 { + req, _ := gcwq.Get() + gotGCRequests = append(gotGCRequests, req) + gcwq.Done(req) + } + if diff := cmp.Diff(gotGCRequests, tc.wantGCRequests, + cmp.AllowUnexported(snapshotGarbageCollectionRequest{}), cmpopts.EquateEmpty()); diff != "" { + t.Errorf("checkForInconsistentResourceSnapshots() enqueued GC requests mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// TestIsSnapshotInUseByAnyPlacementBinding tests the isSnapshotInUseByAnyPlacementBinding method. +func TestIsSnapshotInUseByAnyPlacementBinding(t *testing.T) { + const ( + policyName = "policy" + otherPolicyName = "other-policy" + bindingName = "binding" + namespaceName = "app" + otherNamespace = "other-app" + clusterName = "member-1" + ) + policyUID := types.UID("policy-uid") + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(policyName, policyUID, 0) + otherSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(policyName, policyUID, 1) + + policyOwnerRef := func(kind, name string) metav1.OwnerReference { + return metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: kind, + Name: name, + UID: policyUID, + } + } + clusterPolicyOwnerRef := policyOwnerRef(placementv1alpha1.ClusterPlacementPolicyKind, policyName) + namespacedPolicyOwnerRef := policyOwnerRef(placementv1alpha1.PlacementPolicyKind, policyName) + + clusterSnapshot := func(ownerRefs ...metav1.OwnerReference) *placementv1alpha1.ClusterPlacementResourceSnapshot { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + OwnerReferences: ownerRefs, + }, + } + } + clusterBinding := func(resourceSnapshotName string) *placementv1alpha1.ClusterPlacementBinding { + return &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: bindingName, + OwnerReferences: []metav1.OwnerReference{clusterPolicyOwnerRef}, + }, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: policyName, + ClusterName: clusterName, + ResourceSnapshotName: resourceSnapshotName, + }, + } + } + namespacedSnapshot := &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: snapshotName, + Namespace: namespaceName, + OwnerReferences: []metav1.OwnerReference{namespacedPolicyOwnerRef}, + }, + } + namespacedBinding := func(namespace string) *placementv1alpha1.PlacementBinding { + return &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: bindingName, + Namespace: namespace, + OwnerReferences: []metav1.OwnerReference{namespacedPolicyOwnerRef}, + }, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: policyName, + ClusterName: clusterName, + ResourceSnapshotName: snapshotName, + }, + } + } + + clusterPolicyNamespacedName := types.NamespacedName{Name: policyName} + namespacedPolicyNamespacedName := types.NamespacedName{Namespace: namespaceName, Name: policyName} + + testCases := []struct { + name string + snapshot placementv1alpha1.PlacementResourceSnapshotAccessor + placementPolicyNamespacedName types.NamespacedName + bindings []client.Object + interceptorFuncs interceptor.Funcs + want bool + // The category of the error that is expected to be returned; leave empty if no error is expected. + wantErrCategory kferrors.ErrCategory + }{ + { + name: "no placement bindings", + snapshot: clusterSnapshot(clusterPolicyOwnerRef), + placementPolicyNamespacedName: clusterPolicyNamespacedName, + want: false, + }, + { + name: "placement binding using a different snapshot", + snapshot: clusterSnapshot(clusterPolicyOwnerRef), + placementPolicyNamespacedName: clusterPolicyNamespacedName, + bindings: []client.Object{clusterBinding(otherSnapshotName)}, + want: false, + }, + { + name: "placement binding using the snapshot", + snapshot: clusterSnapshot(clusterPolicyOwnerRef), + placementPolicyNamespacedName: clusterPolicyNamespacedName, + bindings: []client.Object{clusterBinding(snapshotName)}, + want: true, + }, + { + name: "orphaned snapshot (no owner references) referenced by a placement binding", + snapshot: clusterSnapshot(), + placementPolicyNamespacedName: clusterPolicyNamespacedName, + bindings: []client.Object{clusterBinding(snapshotName)}, + want: false, + }, + { + name: "snapshot owned by a differently named placement policy", + snapshot: clusterSnapshot(policyOwnerRef(placementv1alpha1.ClusterPlacementPolicyKind, otherPolicyName)), + placementPolicyNamespacedName: clusterPolicyNamespacedName, + bindings: []client.Object{clusterBinding(snapshotName)}, + want: false, + }, + { + name: "snapshot with an owner reference of a mismatched kind", + snapshot: clusterSnapshot(namespacedPolicyOwnerRef), + placementPolicyNamespacedName: clusterPolicyNamespacedName, + bindings: []client.Object{clusterBinding(snapshotName)}, + want: false, + }, + { + name: "failed to list placement bindings", + snapshot: clusterSnapshot(clusterPolicyOwnerRef), + placementPolicyNamespacedName: clusterPolicyNamespacedName, + bindings: []client.Object{clusterBinding(snapshotName)}, + interceptorFuncs: interceptor.Funcs{ + List: func(_ context.Context, _ client.WithWatch, _ client.ObjectList, _ ...client.ListOption) error { + return errors.New("simulated list failure") + }, + }, + wantErrCategory: kferrors.ErrCategoryAPIServer, + }, + { + name: "namespaced placement binding using the snapshot", + snapshot: namespacedSnapshot, + placementPolicyNamespacedName: namespacedPolicyNamespacedName, + bindings: []client.Object{namespacedBinding(namespaceName)}, + want: true, + }, + { + name: "namespaced placement binding in a different namespace", + snapshot: namespacedSnapshot, + placementPolicyNamespacedName: namespacedPolicyNamespacedName, + bindings: []client.Object{namespacedBinding(otherNamespace)}, + want: false, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + m := &Manager{hubClient: interceptor.NewClient(newFakeHubClientWithIndexes(t, tc.bindings...), tc.interceptorFuncs)} + + got, err := m.isSnapshotInUseByAnyPlacementBinding(context.Background(), tc.snapshot, tc.placementPolicyNamespacedName) + if tc.wantErrCategory != "" { + if err == nil { + t.Fatalf("isSnapshotInUseByAnyPlacementBinding() = %v, nil, want error of category %v", got, tc.wantErrCategory) + } + verifyErrCategory(t, "isSnapshotInUseByAnyPlacementBinding()", err, tc.wantErrCategory) + return + } + if err != nil { + t.Fatalf("isSnapshotInUseByAnyPlacementBinding() = %v, want no error", err) + } + if got != tc.want { + t.Errorf("isSnapshotInUseByAnyPlacementBinding() = %v, want %v", got, tc.want) + } + }) + } +} + +// TestGarbageCollect tests the garbageCollect method. +func TestGarbageCollect(t *testing.T) { + const ( + policyName = "policy" + bindingName = "binding" + clusterName = "member-1" + ) + policyUID := types.UID("policy-uid") + recreatedPolicyUID := types.UID("recreated-policy-uid") + primaryName := uniquename.PrimaryPlacementResourceSnapshotName(policyName, policyUID, 0) + secondaryName := uniquename.SecondaryPlacementResourceSnapshotName(policyName, policyUID, 0, 1) + tertiaryName := uniquename.SecondaryPlacementResourceSnapshotName(policyName, policyUID, 0, 2) + // Unexpected primary and secondary snapshots at the same index as the primary snapshot specified in the GC + // request, named as those of a re-created placement policy of the same name. + otherPrimaryName := uniquename.PrimaryPlacementResourceSnapshotName(policyName, recreatedPolicyUID, 0) + otherSecondaryName := uniquename.SecondaryPlacementResourceSnapshotName(policyName, recreatedPolicyUID, 0, 1) + primaryUID := types.UID("primary-uid") + recreatedPrimaryUID := types.UID("recreated-primary-uid") + policyOwnerRef := metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.ClusterPlacementPolicyKind, + Name: policyName, + UID: policyUID, + } + recreatedPolicyOwnerRef := policyOwnerRef + recreatedPolicyOwnerRef.UID = recreatedPolicyUID + + snapshot := func(name string, uid types.UID, subIdx int, ownerRefs ...metav1.OwnerReference) *placementv1alpha1.ClusterPlacementResourceSnapshot { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + UID: uid, + Labels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: policyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: strconv.Itoa(subIdx), + }, + Annotations: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: policyName, + }, + OwnerReferences: ownerRefs, + }, + } + } + bindingUsingPrimary := &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: bindingName, + OwnerReferences: []metav1.OwnerReference{policyOwnerRef}, + }, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: policyName, + ClusterName: clusterName, + ResourceSnapshotName: primaryName, + }, + } + gcRequest := snapshotGarbageCollectionRequest{ + snapshotNamespacedName: types.NamespacedName{Name: primaryName}, + snapshotUID: primaryUID, + placementPolicyNamespacedName: types.NamespacedName{Name: policyName}, + } + + testCases := []struct { + name string + objs []client.Object + interceptorFuncs interceptor.Funcs + // The category of the error that is expected to be returned; leave empty if no error is expected. + wantErrCategory kferrors.ErrCategory + wantRemainingSnapshots []string + }{ + { + name: "primary and secondary snapshots owned by the placement policy", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0, policyOwnerRef), + snapshot(secondaryName, "secondary-uid", 1, policyOwnerRef), + snapshot(tertiaryName, "tertiary-uid", 2, policyOwnerRef), + }, + }, + { + name: "orphaned primary and secondary snapshots", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0), + snapshot(secondaryName, "secondary-uid", 1), + }, + }, + { + name: "another primary snapshot at the same index", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0, policyOwnerRef), + snapshot(secondaryName, "secondary-uid", 1, policyOwnerRef), + snapshot(otherPrimaryName, "other-primary-uid", 0, policyOwnerRef), + }, + wantRemainingSnapshots: []string{otherPrimaryName}, + }, + { + name: "orphaned primary snapshot, with snapshots of a re-created placement policy at the same index", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0), + snapshot(secondaryName, "secondary-uid", 1), + snapshot(otherPrimaryName, "other-primary-uid", 0, recreatedPolicyOwnerRef), + snapshot(otherSecondaryName, "other-secondary-uid", 1, recreatedPolicyOwnerRef), + }, + wantRemainingSnapshots: []string{otherPrimaryName, otherSecondaryName}, + }, + { + name: "secondary snapshot owned by a different placement policy of the same name", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0, policyOwnerRef), + snapshot(secondaryName, "secondary-uid", 1, recreatedPolicyOwnerRef), + snapshot(tertiaryName, "tertiary-uid", 2, policyOwnerRef), + }, + wantRemainingSnapshots: []string{secondaryName}, + }, + { + name: "orphaned secondary snapshot with a primary snapshot owned by the placement policy", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0, policyOwnerRef), + snapshot(secondaryName, "secondary-uid", 1), + }, + wantRemainingSnapshots: []string{secondaryName}, + }, + { + name: "primary snapshot in use by a placement binding", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0, policyOwnerRef), + snapshot(secondaryName, "secondary-uid", 1, policyOwnerRef), + bindingUsingPrimary, + }, + wantRemainingSnapshots: []string{primaryName, secondaryName}, + }, + { + name: "orphaned primary snapshot referenced by a placement binding", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0), + snapshot(secondaryName, "secondary-uid", 1), + bindingUsingPrimary, + }, + }, + { + name: "primary snapshot has been re-created", + // Snapshot names factor in the UID of the placement policy; only the same placement policy can re-create + // a snapshot under the same name (e.g., after the snapshot has been deleted by hand). + objs: []client.Object{ + snapshot(primaryName, recreatedPrimaryUID, 0, policyOwnerRef), + snapshot(secondaryName, "secondary-uid", 1, policyOwnerRef), + }, + wantRemainingSnapshots: []string{primaryName, secondaryName}, + }, + { + name: "primary snapshot not found", + objs: []client.Object{ + snapshot(secondaryName, "secondary-uid", 1, policyOwnerRef), + }, + wantRemainingSnapshots: []string{secondaryName}, + }, + { + name: "failed to retrieve the primary snapshot", + objs: []client.Object{ + snapshot(primaryName, primaryUID, 0, policyOwnerRef), + snapshot(secondaryName, "secondary-uid", 1, policyOwnerRef), + }, + interceptorFuncs: interceptor.Funcs{ + Get: func(_ context.Context, _ client.WithWatch, _ client.ObjectKey, _ client.Object, _ ...client.GetOption) error { + return errors.New("simulated get failure") + }, + }, + wantErrCategory: kferrors.ErrCategoryAPIServer, + wantRemainingSnapshots: []string{primaryName, secondaryName}, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + fakeClient := newFakeHubClientWithIndexes(t, tc.objs...) + m := &Manager{hubClient: interceptor.NewClient(fakeClient, tc.interceptorFuncs)} + + err := m.garbageCollect(context.Background(), gcRequest) + switch { + case tc.wantErrCategory != "" && err == nil: + t.Errorf("garbageCollect() = nil, want error of category %v", tc.wantErrCategory) + case tc.wantErrCategory != "": + verifyErrCategory(t, "garbageCollect()", err, tc.wantErrCategory) + case err != nil: + t.Errorf("garbageCollect() = %v, want no error", err) + } + + snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} + if err := fakeClient.List(context.Background(), snapshotList); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + gotRemainingSnapshots := make([]string, 0, len(snapshotList.Items)) + for idx := range snapshotList.Items { + gotRemainingSnapshots = append(gotRemainingSnapshots, snapshotList.Items[idx].Name) + } + if diff := cmp.Diff(gotRemainingSnapshots, tc.wantRemainingSnapshots, + cmpopts.EquateEmpty(), cmpopts.SortSlices(func(a, b string) bool { return a < b })); diff != "" { + t.Errorf("garbageCollect() remaining snapshots mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// TestStart tests the GC worker loop that the Start method runs. +func TestStart(t *testing.T) { + const policyName = "policy" + policyUID := types.UID("policy-uid") + primaryName0 := uniquename.PrimaryPlacementResourceSnapshotName(policyName, policyUID, 0) + secondaryName := uniquename.SecondaryPlacementResourceSnapshotName(policyName, policyUID, 0, 1) + primaryName1 := uniquename.PrimaryPlacementResourceSnapshotName(policyName, policyUID, 1) + // The primary snapshot at index 2; it is given a malformed index label below. + malformedName := uniquename.PrimaryPlacementResourceSnapshotName(policyName, policyUID, 2) + policyOwnerRef := metav1.OwnerReference{ + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.ClusterPlacementPolicyKind, + Name: policyName, + UID: policyUID, + } + + // snapshot returns a cluster placement resource snapshot owned by the placement policy; the index label is a + // string so that a malformed value can be injected. + snapshot := func(name, idxLabel string, subIdx int) *placementv1alpha1.ClusterPlacementResourceSnapshot { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + UID: types.UID(name + "-uid"), + Labels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: policyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: idxLabel, + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: strconv.Itoa(subIdx), + }, + Annotations: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: policyName, + }, + OwnerReferences: []metav1.OwnerReference{policyOwnerRef}, + }, + } + } + gcRequestOf := func(primaryName string) snapshotGarbageCollectionRequest { + return snapshotGarbageCollectionRequest{ + snapshotNamespacedName: types.NamespacedName{Name: primaryName}, + snapshotUID: types.UID(primaryName + "-uid"), + placementPolicyNamespacedName: types.NamespacedName{Name: policyName}, + } + } + remainingSnapshotNames := func(t *testing.T, c client.Client) []string { + t.Helper() + + snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} + if err := c.List(context.Background(), snapshotList); err != nil { + t.Fatalf("List() = %v, want no error", err) + } + names := make([]string, 0, len(snapshotList.Items)) + for idx := range snapshotList.Items { + names = append(names, snapshotList.Items[idx].Name) + } + return names + } + + t.Run("retryable failure is retried with backoff until it succeeds", func(t *testing.T) { + fakeClient := newFakeHubClientWithIndexes(t, snapshot(primaryName0, "0", 0), snapshot(secondaryName, "0", 1)) + var failDeletes atomic.Bool + failDeletes.Store(true) + m := &Manager{ + hubClient: interceptor.NewClient(fakeClient, interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + if failDeletes.Load() { + return errors.New("simulated delete failure") + } + return c.Delete(ctx, obj, opts...) + }, + }), + gcwq: newTestGCQueue(), + } + stop := startGCWorker(t, m) + + gcRequest := gcRequestOf(primaryName0) + m.gcwq.Add(gcRequest) + pollUntil(t, "the failed GC request is retried at least twice", func() bool { + return m.gcwq.NumRequeues(gcRequest) >= 2 + }) + if got := remainingSnapshotNames(t, fakeClient); len(got) != 2 { + t.Errorf("remaining snapshots = %v, want both snapshots kept while deletions fail", got) + } + + failDeletes.Store(false) + pollUntil(t, "the snapshots are garbage collected once deletions succeed", func() bool { + return len(remainingSnapshotNames(t, fakeClient)) == 0 + }) + // The backoff is reset once the GC request succeeds. + pollUntil(t, "the GC request is forgotten once it succeeds", func() bool { + return m.gcwq.NumRequeues(gcRequest) == 0 + }) + + if err := stop(); err != nil { + t.Errorf("Start() = %v, want no error", err) + } + }) + + t.Run("unexpected failure is dropped, and later requests are still processed", func(t *testing.T) { + fakeClient := newFakeHubClientWithIndexes(t, snapshot(malformedName, "malformed", 0), snapshot(primaryName1, "1", 0)) + var malformedGetCnt atomic.Int32 + m := &Manager{ + hubClient: interceptor.NewClient(fakeClient, interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if key.Name == malformedName { + malformedGetCnt.Add(1) + } + return c.Get(ctx, key, obj, opts...) + }, + }), + gcwq: newTestGCQueue(), + } + stop := startGCWorker(t, m) + + malformedGCRequest := gcRequestOf(malformedName) + m.gcwq.Add(malformedGCRequest) + pollUntil(t, "the GC request with a malformed index label is processed", func() bool { + return malformedGetCnt.Load() >= 1 + }) + + m.gcwq.Add(gcRequestOf(primaryName1)) + pollUntil(t, "a later GC request is processed", func() bool { + return !slices.Contains(remainingSnapshotNames(t, fakeClient), primaryName1) + }) + + // With the backoff of the test workqueue, a retry would have happened long before this. + time.Sleep(100 * time.Millisecond) + if got := malformedGetCnt.Load(); got != 1 { + t.Errorf("GC attempts on the snapshot with a malformed index label = %d, want 1 (no retries)", got) + } + if got := m.gcwq.NumRequeues(malformedGCRequest); got != 0 { + t.Errorf("NumRequeues() = %d, want 0", got) + } + if diff := cmp.Diff(remainingSnapshotNames(t, fakeClient), []string{malformedName}); diff != "" { + t.Errorf("remaining snapshots mismatch (-got +want):\n%s", diff) + } + + if err := stop(); err != nil { + t.Errorf("Start() = %v, want no error", err) + } + }) + + t.Run("canceling the context stops the loop", func(t *testing.T) { + m := &Manager{hubClient: newFakeHubClientWithIndexes(t), gcwq: newTestGCQueue()} + stop := startGCWorker(t, m) + + if err := stop(); err != nil { + t.Errorf("Start() = %v, want no error", err) + } + if !m.gcwq.ShuttingDown() { + t.Errorf("ShuttingDown() = false, want true") + } + }) +} + +// newTestGCQueue returns a GC workqueue with a short backoff, so that retries happen quickly in tests. +func newTestGCQueue() workqueue.TypedRateLimitingInterface[snapshotGarbageCollectionRequest] { + return workqueue.NewTypedRateLimitingQueue( + workqueue.NewTypedItemExponentialFailureRateLimiter[snapshotGarbageCollectionRequest](time.Millisecond, 10*time.Millisecond)) +} + +// startGCWorker runs the GC worker loop of the given manager in the background; the returned function cancels the +// context, waits for the loop to exit, and returns the error that Start returns. +func startGCWorker(t *testing.T, m *Manager) func() error { + t.Helper() + + ctx, cancel := context.WithCancel(context.Background()) + errCh := make(chan error, 1) + go func() { + errCh <- m.Start(ctx) + }() + return func() error { + cancel() + select { + case err := <-errCh: + return err + case <-time.After(10 * time.Second): + t.Fatalf("Start() did not return after the context was canceled") + return nil + } + } +} + +// pollUntil polls the given condition until it holds; it fails the test if the condition does not hold in time. +func pollUntil(t *testing.T, desc string, cond func() bool) { + t.Helper() + + err := wait.PollUntilContextTimeout(context.Background(), 5*time.Millisecond, 10*time.Second, true, + func(context.Context) (bool, error) { + return cond(), nil + }) + if err != nil { + t.Fatalf("Timed out waiting until %s: %v", desc, err) + } +} + +// newFakeHubClientWithIndexes returns a fake client with the custom field indexes that the garbage collection +// process relies on. +func newFakeHubClientWithIndexes(t *testing.T, objs ...client.Object) client.WithWatch { + t.Helper() + + scheme := runtime.NewScheme() + if err := placementv1alpha1.AddToScheme(scheme); err != nil { + t.Fatalf("AddToScheme() = %v, want no error", err) + } + + // Mirror the field indexes that the hub controller manager sets up. + bindingExtractor := func(obj client.Object) []string { + spec := obj.(placementv1alpha1.PlacementBindingAccessor).GetSpec() + return []string{fmt.Sprintf(fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, + spec.PlacementPolicyName, spec.ResourceSnapshotName)} + } + snapshotExtractor := func(obj client.Object) []string { + return []string{fmt.Sprintf(fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, + obj.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey], + obj.GetLabels()[placementv1alpha1.PlacementResourceSnapshotIndexLabelKey])} + } + return fake.NewClientBuilder(). + WithScheme(scheme). + WithObjects(objs...). + WithIndex(&placementv1alpha1.ClusterPlacementBinding{}, + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName, bindingExtractor). + WithIndex(&placementv1alpha1.PlacementBinding{}, + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName, bindingExtractor). + WithIndex(&placementv1alpha1.ClusterPlacementResourceSnapshot{}, + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName, snapshotExtractor). + WithIndex(&placementv1alpha1.PlacementResourceSnapshot{}, + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName, snapshotExtractor). + Build() +} + +// verifyErrCategory verifies that the given error is a KubeFleet error of the wanted category. +func verifyErrCategory(t *testing.T, funcName string, err error, wantErrCategory kferrors.ErrCategory) { + t.Helper() + + // The error category is the first key-value pair in the error attributes. + gotErrAttrs := kferrors.Args(err) + if len(gotErrAttrs) < 2 { + t.Fatalf("%s error = %v, want a KubeFleet error", funcName, err) + } + wantErrCategoryAttrs := []interface{}{"errCategory", wantErrCategory} + if diff := cmp.Diff(gotErrAttrs[:2], wantErrCategoryAttrs); diff != "" { + t.Errorf("%s error category attributes mismatch (-got +want):\n%s\nerror: %v", funcName, diff, err) + } +} diff --git a/pkg/v1/managers/placementresourcesnapshot/manager.go b/pkg/v1/managers/placementresourcesnapshot/manager.go index a81b8a71e..68a014621 100644 --- a/pkg/v1/managers/placementresourcesnapshot/manager.go +++ b/pkg/v1/managers/placementresourcesnapshot/manager.go @@ -14,25 +14,40 @@ See the License for the specific language governing permissions and limitations under the License. */ +// Package placementresourcesnapshot provides the manager for handling placement resource snapshots in KubeFleet. package placementresourcesnapshot import ( + "context" "fmt" "hash/fnv" "sync" + "time" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/client-go/dynamic" + "k8s.io/client-go/util/workqueue" + "k8s.io/klog/v2" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/manager" placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" errors "go.goms.io/fleet/pkg/utils/errors" "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" ) const ( managerName = "placementresourcesnapshot" + + defaultRevisionHistoryLimit = int32(3) + snapshotGCRateLimiterBaseDelay = time.Second + snapshotGCRateLimiterMaxDelay = 600 * time.Second +) + +var ( + _ manager.Runnable = (*Manager)(nil) ) const ( @@ -50,35 +65,124 @@ type Manager struct { hubDynamicClient dynamic.Interface hubDynamicInformerManager informer.Manager + resourceEligibilityChecker resourceeligibility.Checker + + gcwq workqueue.TypedRateLimitingInterface[snapshotGarbageCollectionRequest] + restMapper meta.RESTMapper mus []sync.Mutex muSlotCnt uint32 + + maxPerSnapshotResourceDataSizeBytes int + maxPerSnapshotResourceCnt int } // New returns a new Manager. func New(mgr ctrl.Manager, hubDynamicClient dynamic.Interface, hubDynamicInformerManager informer.Manager, + resourceEligibilityChecker resourceeligibility.Checker, restMapper meta.RESTMapper, muSlotCnt int32, + maxPerSnapshotResourceDataSizeBytes int, + maxPerSnapshotResourceCnt int, ) (*Manager, error) { if muSlotCnt < minSlotCnt { return nil, errors.NewUserError(nil, "mu slot size must be greater than or equal to the minimum limit", "manager", managerName, "limit", minSlotCnt, "actual", muSlotCnt) } + if maxPerSnapshotResourceDataSizeBytes < MinPerSnapshotResourceDataSizeBytes || maxPerSnapshotResourceDataSizeBytes > MaxPerSnapshotResourceDataSizeBytes { + return nil, errors.NewUserError(nil, "an inappropriate max per snapshot resource data size is set", + "manager", managerName, "limitRange", fmt.Sprintf("%d~%d", MinPerSnapshotResourceDataSizeBytes, MaxPerSnapshotResourceDataSizeBytes), "actual", maxPerSnapshotResourceDataSizeBytes) + } + if maxPerSnapshotResourceCnt < MinPerSnapshotResourceCnt || maxPerSnapshotResourceCnt > MaxPerSnapshotResourceCnt { + return nil, errors.NewUserError(nil, "an inappropriate max per snapshot resource count is set", + "manager", managerName, "limitRange", fmt.Sprintf("%d~%d", MinPerSnapshotResourceCnt, MaxPerSnapshotResourceCnt), "actual", maxPerSnapshotResourceCnt) + } + + if resourceEligibilityChecker == nil { + return nil, errors.NewUnexpectedError(nil, "a resource eligibility checker must be set", + "manager", managerName) + } + + // Set up the resource snapshot GC workqueue. + // + // The work queue uses an exponential backoff rate limiter (power of 2, starting at 1 second and capped + // at 600 seconds). + gcwq := workqueue.NewTypedRateLimitingQueueWithConfig( + workqueue.NewTypedItemExponentialFailureRateLimiter[snapshotGarbageCollectionRequest](snapshotGCRateLimiterBaseDelay, snapshotGCRateLimiterMaxDelay), + workqueue.TypedRateLimitingQueueConfig[snapshotGarbageCollectionRequest]{ + Name: "placementresourcesnapshotmanager-garbage-collection", + }) + return &Manager{ - hubClient: mgr.GetClient(), - hubUncachedReader: mgr.GetAPIReader(), - hubDynamicClient: hubDynamicClient, - hubDynamicInformerManager: hubDynamicInformerManager, - restMapper: restMapper, - mus: make([]sync.Mutex, muSlotCnt), - muSlotCnt: uint32(muSlotCnt), + hubClient: mgr.GetClient(), + hubUncachedReader: mgr.GetAPIReader(), + hubDynamicClient: hubDynamicClient, + hubDynamicInformerManager: hubDynamicInformerManager, + resourceEligibilityChecker: resourceEligibilityChecker, + gcwq: gcwq, + restMapper: restMapper, + mus: make([]sync.Mutex, muSlotCnt), + muSlotCnt: uint32(muSlotCnt), + maxPerSnapshotResourceDataSizeBytes: maxPerSnapshotResourceDataSizeBytes, + maxPerSnapshotResourceCnt: maxPerSnapshotResourceCnt, }, nil } +// Start starts the GC process for the placement resource snapshots. +func (m *Manager) Start(ctx context.Context) error { + var wg sync.WaitGroup + + // Start a goroutine to shutdown the GC workqueue when the main context is canceled. + wg.Add(1) + go func() { + defer wg.Done() + <-ctx.Done() + m.gcwq.ShutDown() + }() + + // Start a goroutine to process garbage collection of stale resource snapshots. + wg.Add(1) + go func() { + defer wg.Done() + + for { + snapshotGCRequest, shutdown := m.gcwq.Get() + if shutdown { + return + } + if ctx.Err() != nil { + // The Get() method signals that the workqueue has been shut down only when the queue is empty. + // Here we just short-circuit the processing loop if the context has been canceled. + m.gcwq.Done(snapshotGCRequest) + return + } + + if err := m.garbageCollect(ctx, snapshotGCRequest); err != nil { + wrappedErr := errors.Wraps(err, "", "snapshotGarbageCollectionRequest", snapshotGCRequest) + klog.ErrorS(wrappedErr, "Failed to garbage collect placement resource snapshot", errors.Args(wrappedErr)...) + m.gcwq.Done(snapshotGCRequest) + + // No need to requeue if the manager sees an unexpected error. + if errors.Category(err) == errors.ErrCategoryUnexpected { + m.gcwq.Forget(snapshotGCRequest) + continue + } + m.gcwq.AddRateLimited(snapshotGCRequest) + continue + } + m.gcwq.Done(snapshotGCRequest) + m.gcwq.Forget(snapshotGCRequest) + } + }() + + wg.Wait() + return nil +} + // acquireLock acquires a mutex for a given placement policy. // // The placement resource snapshot manager features a slot-based locking mechanism to ensure that KubeFleet always diff --git a/pkg/v1/managers/placementresourcesnapshot/ops.go b/pkg/v1/managers/placementresourcesnapshot/ops.go index 7e70e490c..dc0f539bc 100644 --- a/pkg/v1/managers/placementresourcesnapshot/ops.go +++ b/pkg/v1/managers/placementresourcesnapshot/ops.go @@ -27,6 +27,7 @@ import ( placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" errors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" "go.goms.io/fleet/pkg/v1/utils/fieldindexers" ) @@ -72,6 +73,8 @@ func (m *Manager) snapshotResources(ctx context.Context, } // Retrieve the currently selected resources and their hash based on the placement policy. + // + // At this point, KubeFleet allows empty resource snapshots. currentResources, currentHash, err := m.retrieveAndHashSelectedResources(ctx, placementPolicy) if err != nil { return nil, false, errors.Wraps(err, "failed to retrieve and hash the selected resources") @@ -127,29 +130,21 @@ func (m *Manager) retrieveLatestSnapshot(ctx context.Context, placementPolicy pl // Retrieve the primary resource snapshots associated with the placement policy. var snapshots []placementv1alpha1.PlacementResourceSnapshotAccessor fieldMatchers := client.MatchingFields{ - fieldindexers.PlacementResourceSnapshotOwnedByAndSubIndexedCustomFieldName: fmt.Sprintf(fieldindexers.PlacementResourceSnapshotOwnedByAndSubIndexedCustomFieldValFmt, placementPolicyOwnerLabelVal(placementPolicy), "0"), + fieldindexers.PlacementResourceSnapshotOwnedByAndSubIndexedCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementResourceSnapshotOwnedByAndSubIndexedCustomFieldValFmt, placementPolicy.GetName(), "0"), } if placementPolicy.GetNamespace() == "" { // The placement policy is cluster-scoped; list cluster placement resource snapshots. - placementResourceSnapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} - if err := m.hubClient.List(ctx, placementResourceSnapshotList, fieldMatchers); err != nil { - return nil, errors.NewAPIServerError(err, "failed to list cluster placement resource snapshots", true) - } - snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(placementResourceSnapshotList.Items)) - for i := range placementResourceSnapshotList.Items { - snapshots[i] = &placementResourceSnapshotList.Items[i] - } + snapshots, err = m.listPlacementResourceSnapshots(ctx, true, []client.ListOption{fieldMatchers}) } else { // The placement policy is namespace-scoped; list placement resource snapshots in the same namespace. - placementResourceSnapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} - if err := m.hubClient.List(ctx, placementResourceSnapshotList, - client.InNamespace(placementPolicy.GetNamespace()), fieldMatchers); err != nil { - return nil, errors.NewAPIServerError(err, "failed to list placement resource snapshots", true) - } - snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(placementResourceSnapshotList.Items)) - for i := range placementResourceSnapshotList.Items { - snapshots[i] = &placementResourceSnapshotList.Items[i] - } + snapshots, err = m.listPlacementResourceSnapshots(ctx, false, []client.ListOption{ + client.InNamespace(placementPolicy.GetNamespace()), + fieldMatchers, + }) + } + if err != nil { + return nil, err } if len(snapshots) == 0 { @@ -180,7 +175,27 @@ func (m *Manager) retrieveLatestSnapshot(ctx context.Context, placementPolicy pl return nil, errors.NewUnexpectedError(nil, "failed to sort primary placement resource snapshots", "errs", sortErrs) } + // Check for inconsistent resource snapshots. + leftovers, err := m.checkForInconsistentResourceSnapshots(snapshots, placementPolicy) + if err != nil { + return nil, errors.Wraps(err, "observed inconsistent resource snapshots") + } + if len(leftovers) > 0 { + // Leftover resource snapshots have been found; wait for them to disappear before proceeding. + return nil, errors.NewTransientError(nil, "leftover resource snapshots are detected; waiting for garbage collection to complete", + "placementPolicy", klog.KObj(placementPolicy), "leftoverPlacementResourceSnapshots", leftovers) + } + + // Enqueue the outdated primary resource snapshots for GC. + m.enqueueOldResourceSnapshotsForGC(snapshots, placementPolicy) + latestPrimarySnapshot := snapshots[len(snapshots)-1] + // Verify if the latest primary snapshot has been marked for deletion. If so, wait for its deletion to complete. + if !latestPrimarySnapshot.GetDeletionTimestamp().IsZero() { + return nil, errors.NewTransientError(nil, "the latest primary placement resource snapshot is marked for deletion", + "placementResourceSnapshot", klog.KObj(latestPrimarySnapshot)) + } + // Check if there are snapshots with the same index. subIndexedSnapshotCntStr := latestPrimarySnapshot.GetLabels()[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] if subIndexedSnapshotCntStr == "1" { @@ -202,31 +217,23 @@ func (m *Manager) retrieveLatestSnapshot(ctx context.Context, placementPolicy pl // There are sub-indexed placement resource snapshots with the same index; retrieve them. latestIndex := latestPrimarySnapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotIndexLabelKey] fieldMatchers = client.MatchingFields{ - fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName: fmt.Sprintf(fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, placementPolicyOwnerLabelVal(placementPolicy), latestIndex), + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName: fmt.Sprintf(fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, placementPolicy.GetName(), latestIndex), } var subIndexedSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor if placementPolicy.GetNamespace() == "" { // The placement policy is cluster-scoped; list cluster placement resource snapshots. - placementResourceSnapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} - if err := m.hubClient.List(ctx, placementResourceSnapshotList, fieldMatchers); err != nil { - return nil, errors.NewAPIServerError(err, "failed to list cluster placement resource snapshots", true) - } - subIndexedSnapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(placementResourceSnapshotList.Items)) - for i := range placementResourceSnapshotList.Items { - subIndexedSnapshots[i] = &placementResourceSnapshotList.Items[i] - } + subIndexedSnapshots, err = m.listPlacementResourceSnapshots(ctx, true, []client.ListOption{fieldMatchers}) } else { // The placement policy is namespace-scoped; list placement resource snapshots in the same namespace. - placementResourceSnapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} - if err := m.hubClient.List(ctx, placementResourceSnapshotList, - client.InNamespace(placementPolicy.GetNamespace()), fieldMatchers); err != nil { - return nil, errors.NewAPIServerError(err, "failed to list placement resource snapshots", true) - } - subIndexedSnapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(placementResourceSnapshotList.Items)) - for i := range placementResourceSnapshotList.Items { - subIndexedSnapshots[i] = &placementResourceSnapshotList.Items[i] - } + subIndexedSnapshots, err = m.listPlacementResourceSnapshots(ctx, false, []client.ListOption{ + client.InNamespace(placementPolicy.GetNamespace()), + fieldMatchers, + }) } + if err != nil { + return nil, err + } + // Sort the sub-indexed snapshots by their sub-indices. sortErrs = nil sort.Slice(subIndexedSnapshots, func(i, j int) bool { @@ -272,7 +279,21 @@ func (m *Manager) retrieveLatestSnapshot(ctx context.Context, placementPolicy pl len(subIndexedSnapshots), subIndexedSnapshotCnt, klog.KObj(placementPolicy), subIndexedSnapshotCnt) } - return subIndexedSnapshots[:subIndexedSnapshotCnt], nil + resSnapshots := subIndexedSnapshots[:subIndexedSnapshotCnt] + // Verify that all snapshots in the result have the same resource hash. This is a sanity check that should always + // pass in normal ops. + primaryHash := latestPrimarySnapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] + for idx := range resSnapshots { + snapshot := resSnapshots[idx] + snapshotHash := snapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] + if snapshotHash != primaryHash { + return nil, errors.NewUnexpectedError(nil, "a sub-indexed placement resource snapshot has a different contents hash than the primary placement resource snapshot", + "primaryPlacementResourceSnapshot", klog.KObj(latestPrimarySnapshot), "primaryContentsHash", primaryHash, + "placementResourceSnapshot", klog.KObj(snapshot), "contentsHash", snapshotHash) + } + } + + return resSnapshots, nil } // isSnapshotUpToDate checks if the given placement resource snapshot is up-to-date, i.e., the snapshot is @@ -323,7 +344,7 @@ func (m *Manager) isSnapshotUpToDate( // A primary placement resource snapshot already exists at the given index; the currently observed primary // resource snapshot is not up-to-date. return false, errors.NewTransientError(nil, "a newer snapshot already exists (found via quorum reads); the client cache might be stale", - "primaryPlacementResourceSnapshotName", uniqueNameForPrimaryPlacementResourceSnapshot(placementPolicy.GetName(), nextIdx), + "primaryPlacementResourceSnapshotName", uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), nextIdx), "snapshotIndex", nextIdx) } @@ -379,7 +400,7 @@ func (m *Manager) createResourceSnapshotAnyway( // Split the resources into size-controlled groups. Each group corresponds to a placement resource snapshot // that will be created. - resGroups, err := splitResourcesIntoSizeControlledGroups(currentResources) + resGroups, err := m.splitResourcesIntoSizeControlledGroups(currentResources) if err != nil { return nil, errors.Wraps(err, "failed to split the selected resources into size-controlled groups") } @@ -393,7 +414,7 @@ func (m *Manager) createResourceSnapshotAnyway( // Create the secondary placement resource snapshots first. Start with the last resource group and work // backwards, so that the primary snapshot (which carries the count label) is created last. for subIdx := len(resGroups) - 1; subIdx >= 1; subIdx-- { - secondaryName := uniqueNameForSecondaryPlacementResourceSnapshot(placementPolicy.GetName(), nextSnapshotIdx, subIdx) + secondaryName := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), nextSnapshotIdx, subIdx) secondarySnapshot, err := secondaryPlacementResourceSnapshot( placementPolicy.GetNamespace(), secondaryName, placementPolicy, nextSnapshotIdx, subIdx, resGroups[subIdx], currentHash, m.hubClient.Scheme()) if err != nil { @@ -413,7 +434,7 @@ func (m *Manager) createResourceSnapshotAnyway( } // Create the primary placement resource snapshot last, with the count label. - primaryName := uniqueNameForPrimaryPlacementResourceSnapshot(placementPolicy.GetName(), nextSnapshotIdx) + primaryName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), nextSnapshotIdx) primarySnapshot, err := primaryPlacementResourceSnapshot( placementPolicy.GetNamespace(), primaryName, placementPolicy, nextSnapshotIdx, resGroups[0], currentHash, len(resGroups), m.hubClient.Scheme()) if err != nil { @@ -443,31 +464,23 @@ func (m *Manager) cleanUpOrphanedSecondarySnapshots( ) (bool, error) { // List all placement resource snapshots at the given index. fieldMatchers := client.MatchingFields{ - fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName: fmt.Sprintf(fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, placementPolicyOwnerLabelVal(placementPolicy), strconv.Itoa(nextSnapshotIdx)), + fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName: fmt.Sprintf(fieldindexers.PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, placementPolicy.GetName(), strconv.Itoa(nextSnapshotIdx)), } var snapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + var err error if placementPolicy.GetNamespace() == "" { // The placement policy is cluster-scoped; list cluster placement resource snapshots. - placementResourceSnapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} - if err := m.hubClient.List(ctx, placementResourceSnapshotList, fieldMatchers); err != nil { - return false, errors.NewAPIServerError(err, "failed to list cluster placement resource snapshots", true) - } - snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(placementResourceSnapshotList.Items)) - for i := range placementResourceSnapshotList.Items { - snapshots[i] = &placementResourceSnapshotList.Items[i] - } + snapshots, err = m.listPlacementResourceSnapshots(ctx, true, []client.ListOption{fieldMatchers}) } else { // The placement policy is namespace-scoped; list placement resource snapshots in the same namespace. - placementResourceSnapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} - if err := m.hubClient.List(ctx, placementResourceSnapshotList, - client.InNamespace(placementPolicy.GetNamespace()), fieldMatchers); err != nil { - return false, errors.NewAPIServerError(err, "failed to list placement resource snapshots", true) - } - snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(placementResourceSnapshotList.Items)) - for i := range placementResourceSnapshotList.Items { - snapshots[i] = &placementResourceSnapshotList.Items[i] - } + snapshots, err = m.listPlacementResourceSnapshots(ctx, false, []client.ListOption{ + client.InNamespace(placementPolicy.GetNamespace()), + fieldMatchers, + }) + } + if err != nil { + return false, err } if len(snapshots) == 0 { @@ -480,8 +493,8 @@ func (m *Manager) cleanUpOrphanedSecondarySnapshots( snapshot := snapshots[idx] subIdxStr := snapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey] if subIdxStr == "0" { - // This normally should never occur. - return false, errors.NewUnexpectedError(nil, + // This might occur if the cache is lagging behind. The caller should retry the snapshotting operation. + return false, errors.NewTransientError(nil, "found a primary placement resource snapshot at the given index while cleaning up orphaned secondary snapshots", "primaryPlacementResourceSnapshot", klog.KObj(snapshot)) } @@ -498,7 +511,7 @@ func (m *Manager) cleanUpOrphanedSecondarySnapshots( // A primary placement resource snapshot already exists at the given index; the secondary snapshots found // here are not orphans. Report this as an error; the caller should requeue and wait for the cache to catch up. return false, errors.NewTransientError(nil, "a primary placement resource snapshot already exists at the given index (found via quorum read); the client cache might be stale", - "primaryPlacementResourceSnapshotName", uniqueNameForPrimaryPlacementResourceSnapshot(placementPolicy.GetName(), nextSnapshotIdx), + "primaryPlacementResourceSnapshotName", uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), nextSnapshotIdx), "snapshotIndex", nextSnapshotIdx) } @@ -517,3 +530,56 @@ func (m *Manager) cleanUpOrphanedSecondarySnapshots( } return true, nil } + +// IsSnapshotSynchronized checks whether the given primary placement resource snapshot (and its secondaries) is +// still synchronized, i.e., the resources in the snapshot(s) match with the current state of selected resources +// in the cluster. +// +// Important: this method will NOT verify if the given primary placement resource snapshot is still the latest +// one. Call SnapshotResourcesIfNoSnapshotExists instead. +func (m *Manager) IsSnapshotSynchronized(ctx context.Context, + placementResourceSnapshot placementv1alpha1.PlacementResourceSnapshotAccessor, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, +) (bool, error) { + // Do some sanity checks; verify if the given snapshot is a primary placement resource snapshot for the given + // placement policy. + if placementResourceSnapshot == nil || placementPolicy == nil { + return false, errors.NewUnexpectedError(nil, "placement resource snapshot or placement policy is nil", "manager", managerName) + } + + subIdxStr := placementResourceSnapshot.GetLabels()[placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey] + if subIdxStr != "0" { + return false, errors.NewUnexpectedError(nil, "the given placement resource snapshot is not a primary placement resource snapshot", + "placementResourceSnapshot", klog.KObj(placementResourceSnapshot), "observedSubIndex", subIdxStr) + } + + // Owner references are only valid within the same namespace; verify the namespace as well. + ownerRef := placementPolicyOwnerRef(placementResourceSnapshot, placementPolicy.GetName()) + if ownerRef == nil || placementResourceSnapshot.GetNamespace() != placementPolicy.GetNamespace() { + return false, errors.NewUnexpectedError(nil, "the primary placement resource snapshot is not owned by the given placement policy", + "primaryPlacementResourceSnapshot", klog.KObj(placementResourceSnapshot), "placementPolicy", klog.KObj(placementPolicy)) + } + if ownerRef.UID != placementPolicy.GetUID() { + // The placement policy might have been deleted and re-created with the same name, and the cache might + // still hold snapshots owned by the previous generation of the placement policy. + return false, errors.NewTransientError(nil, "observed an inconsistent state: the primary placement resource snapshot is owned by a different placement policy with the same name", + "primaryPlacementResourceSnapshot", klog.KObj(placementResourceSnapshot), "primaryPlacementResourceSnapshotOwnerUID", ownerRef.UID, + "placementPolicy", klog.KObj(placementPolicy), "placementPolicyUID", placementPolicy.GetUID()) + } + + // Retrieve the resource hash annotated on the given placement resource snapshot. + snapshotHash := placementResourceSnapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] + if snapshotHash == "" { + return false, errors.NewUnexpectedError(nil, "the primary placement resource snapshot is missing the contents hash annotation", + "primaryPlacementResourceSnapshot", klog.KObj(placementResourceSnapshot)) + } + + // Retrieve the currently selected resources and their hash based on the placement policy. + _, currentHash, err := m.retrieveAndHashSelectedResources(ctx, placementPolicy) + if err != nil { + return false, errors.Wraps(err, "failed to retrieve and hash the selected resources") + } + + // Compare the hashes. + return snapshotHash == currentHash, nil +} diff --git a/pkg/v1/managers/placementresourcesnapshot/ops_gc_integration_test.go b/pkg/v1/managers/placementresourcesnapshot/ops_gc_integration_test.go new file mode 100644 index 000000000..36ba03dca --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/ops_gc_integration_test.go @@ -0,0 +1,4263 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementresourcesnapshot + +import ( + "encoding/json" + "fmt" + "slices" + "strings" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + eventuallyDuration = time.Second * 10 + eventuallyInterval = time.Second * 1 + + consistentlyDuration = time.Second * 5 + consistentlyInterval = time.Second * 1 + + placementPolicyNameTemplate = "placement-policy-%s" + configMapNameTemplate = "configmap-%s" + deploymentNameTemplate = "deployment-%s" + placementBindingNameTemplate = "placement-binding-%s" + + latestSnapshotPendingDeletionErrMsg = "the latest primary placement resource snapshot is marked for deletion" + revisionHistoryLimitFieldPath = "spec.resourceRevisionHistoryLimit" +) + +var _ = Describe("core ops (placement resource snapshots)", func() { + Context("primary placement resource snapshot only (name-based selection)", Ordered, func() { + // The environment prepared by the envtest package does not support namespace deletion; the + // test case creates its objects in the shared app namespace, with unique names. + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + deploymentName := fmt.Sprintf(deploymentNameTemplate, utils.RandStr()) + + var placementPolicy *placementv1alpha1.PlacementPolicy + var wantHash string + + BeforeAll(func() { + // Create a config map and a deployment in the app namespace. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + Labels: map[string]string{"app": "demo"}, + }, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + deployment := &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: deploymentName, + Namespace: appNamespaceName, + Labels: map[string]string{"app": "demo"}, + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(int32(2)), + Selector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"app": "demo"}, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{"app": "demo"}, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "nginx", + Image: "nginx:1.27", + }, + }, + }, + }, + }, + } + Expect(hubClient.Create(ctx, deployment)).To(Succeed(), "Failed to create the Deployment") + waitForDeploymentsInCache(deployment) + + // Create a placement policy that selects the two resources. + placementPolicy = &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + Namespace: appNamespaceName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMapName, + }, + { + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Name: deploymentName, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a primary placement resource snapshot", func() { + createdSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to create the placement resource snapshot") + Expect(createdSnapshots).To(HaveLen(1), "Unexpected number of created placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The created placement resource snapshot is not up-to-date") + }) + + It("can compute the hash of selected resources", func() { + _, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + wantHash = hash + }) + + It("should persist the placement resource snapshot as expected", func() { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot") + + // Verify the metadata. + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot labels diff (-got +want):\n%s", diff)) + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot annotations diff (-got +want):\n%s", diff)) + } + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "PlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot owner references diff (-got +want):\n%s", diff)) + } + + // Verify the snapshotted resources; they are sorted by their unique IDs, i.e., the config map comes first. + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: appNamespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Namespace: appNamespaceName, + Name: deploymentName, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + }, + } + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("Snapshotted resource identifiers diff (-got +want):\n%s", diff)) + } + + // Verify the config map manifest. + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[0].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + wantConfigMap := &corev1.ConfigMap{ + TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "ConfigMap"}, + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + Labels: map[string]string{"app": "demo"}, + }, + Data: map[string]string{"key": "value"}, + } + if diff := cmp.Diff(gotConfigMap, wantConfigMap); diff != "" { + Fail(fmt.Sprintf("Snapshotted config map diff (-got +want):\n%s", diff)) + } + + // Verify the Deployment manifest; the spec should match that of the live object (which carries the + // defaulted fields), while the read-only metadata and status fields should have been stripped. + liveDeployment := &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: deploymentName}, liveDeployment)).To(Succeed(), + "Failed to get the Deployment") + gotDeployment := &appsv1.Deployment{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[1].Manifest.Raw, gotDeployment)).To(Succeed(), "Failed to unmarshal the Deployment manifest") + wantDeployment := &appsv1.Deployment{ + TypeMeta: metav1.TypeMeta{APIVersion: "apps/v1", Kind: "Deployment"}, + ObjectMeta: metav1.ObjectMeta{ + Name: deploymentName, + Namespace: appNamespaceName, + Labels: map[string]string{"app": "demo"}, + }, + Spec: liveDeployment.Spec, + } + if diff := cmp.Diff(gotDeployment, wantDeployment, cmpopts.EquateEmpty()); diff != "" { + Fail(fmt.Sprintf("Snapshotted Deployment diff (-got +want):\n%s", diff)) + } + }) + + It("should return the existing snapshot when no snapshot is needed", func() { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + wantSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + // The previous spec has waited for the snapshot to appear in the cache. + Expect(hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: snapshotName}, wantSnapshot)).To(Succeed(), + "Failed to get the placement resource snapshot") + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The existing placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of placement resource snapshots") + + gotSnapshot, ok := gotSnapshots[0].(*placementv1alpha1.PlacementResourceSnapshot) + Expect(ok).To(BeTrue(), "Unexpected type of the returned placement resource snapshot: %T", gotSnapshots[0]) + if diff := cmp.Diff(gotSnapshot, wantSnapshot, cmpopts.IgnoreFields(metav1.TypeMeta{}, "APIVersion", "Kind")); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot diff (-got +want):\n%s", diff)) + } + + // No new snapshot should have been created. + nextSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + err = hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: nextSnapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + Expect(errors.IsNotFound(err)).To(BeTrue(), "A new placement resource snapshot was created unexpectedly: %v", err) + }) + + It("should return the existing snapshot when it is not stale", func() { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + wantSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Expect(hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: snapshotName}, wantSnapshot)).To(Succeed(), + "Failed to get the placement resource snapshot") + + // The selected resources have not changed since the snapshot was taken. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The existing placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of placement resource snapshots") + + gotSnapshot, ok := gotSnapshots[0].(*placementv1alpha1.PlacementResourceSnapshot) + Expect(ok).To(BeTrue(), "Unexpected type of the returned placement resource snapshot: %T", gotSnapshots[0]) + if diff := cmp.Diff(gotSnapshot, wantSnapshot, cmpopts.IgnoreFields(metav1.TypeMeta{}, "APIVersion", "Kind")); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot diff (-got +want):\n%s", diff)) + } + + // No new snapshot should have been created. + nextSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + err = hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: nextSnapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + Expect(errors.IsNotFound(err)).To(BeTrue(), "A new placement resource snapshot was created unexpectedly: %v", err) + }) + + It("can update the selected resources", func() { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = "new-value" + configMap.Data["extra-key"] = "extra-value" + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + + deployment := &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: deploymentName}, deployment)).To(Succeed(), + "Failed to get the Deployment") + deployment.Spec.Replicas = ptr.To(int32(3)) + deployment.Spec.Template.Spec.Containers[0].Image = "nginx:1.28" + Expect(hubClient.Update(ctx, deployment)).To(Succeed(), "Failed to update the Deployment") + waitForDeploymentsInCache(deployment) + }) + + It("can create a new snapshot when the existing one is stale", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of placement resource snapshots") + + newSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + Expect(gotSnapshots[0].GetName()).To(Equal(newSnapshotName), "Unexpected name of the new placement resource snapshot") + }) + + It("should create a new snapshot with updated resources", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after the updates") + + newSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: newSnapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the new placement resource snapshot") + + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "1", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("New placement resource snapshot labels diff (-got +want):\n%s", diff)) + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: newHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("New placement resource snapshot annotations diff (-got +want):\n%s", diff)) + } + + // Verify that the new snapshot carries the updated resources. + Expect(gotSnapshot.Spec.Resources).To(HaveLen(2), "Unexpected number of snapshotted resources") + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[0].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + wantConfigMapData := map[string]string{"key": "new-value", "extra-key": "extra-value"} + if diff := cmp.Diff(gotConfigMap.Data, wantConfigMapData); diff != "" { + Fail(fmt.Sprintf("Snapshotted config map data diff (-got +want):\n%s", diff)) + } + gotDeployment := &appsv1.Deployment{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[1].Manifest.Raw, gotDeployment)).To(Succeed(), "Failed to unmarshal the Deployment manifest") + Expect(gotDeployment.Spec.Replicas).To(Equal(ptr.To(int32(3))), "Unexpected replica count in the snapshotted Deployment") + Expect(gotDeployment.Spec.Template.Spec.Containers[0].Image).To(Equal("nginx:1.28"), "Unexpected image in the snapshotted Deployment") + + // The previous snapshot should be left intact. + oldSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{ + Namespace: appNamespaceName, + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + }, oldSnapshot)).To(Succeed(), "Failed to get the previous placement resource snapshot") + Expect(oldSnapshot.Annotations[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey]).To(Equal(wantHash), + "The previous placement resource snapshot has been modified") + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; remove all the objects explicitly. + objs := []client.Object{ + &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + Namespace: appNamespaceName, + }}, + &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1), + Namespace: appNamespaceName, + }}, + placementPolicy, + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: deploymentName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + } + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) + + Context("primary and secondary placement resource snapshots (too many resources) (label-based selection)", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + deploymentNamePrefix := fmt.Sprintf(deploymentNameTemplate, utils.RandStr()) + + // The names are in ascending order, which is also the order in which the selected resources are + // snapshotted. + deploymentNames := make([]string, 5) + for idx := range deploymentNames { + deploymentNames[idx] = fmt.Sprintf("%s-%d", deploymentNamePrefix, idx) + } + + // The names of the Deployments added later; they sort after the ones above. + extraDeploymentNames := make([]string, 5) + for idx := range extraDeploymentNames { + extraDeploymentNames[idx] = fmt.Sprintf("%s-%d", deploymentNamePrefix, idx+len(deploymentNames)) + } + + allDeploymentNames := append(slices.Clone(deploymentNames), extraDeploymentNames...) + + deploymentIdentifiers := func(names []string) []placementv1alpha1.ObjectReference { + identifiers := make([]placementv1alpha1.ObjectReference, 0, len(names)) + for _, name := range names { + identifiers = append(identifiers, placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: name, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + }) + } + return identifiers + } + + var placementPolicy *placementv1alpha1.PlacementPolicy + var wantHash string + + BeforeAll(func() { + // Create five Deployments with the same label in the app namespace. + deployments := make([]*appsv1.Deployment, 0, len(deploymentNames)) + for _, name := range deploymentNames { + deployment := newDeployment(name, appNamespaceName) + deployment.Labels = map[string]string{"placement-group": "foo"} + Expect(hubClient.Create(ctx, deployment)).To(Succeed(), "Failed to create the Deployment") + deployments = append(deployments, deployment) + } + waitForDeploymentsInCache(deployments...) + + // Create a placement policy that selects the Deployments by their label. + placementPolicy = &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + Namespace: appNamespaceName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"placement-group": "foo"}, + }, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(resources).To(HaveLen(len(deploymentNames)), "Unexpected number of selected resources") + wantHash = hash + }) + + It("can create a primary and a secondary placement resource snapshot", func() { + createdSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The created placement resource snapshots are not up-to-date") + Expect(createdSnapshots).To(HaveLen(2), "Unexpected number of created placement resource snapshots") + + // The snapshots are ordered by their sub-indices. + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + } + gotNames := []string{createdSnapshots[0].GetName(), createdSnapshots[1].GetName()} + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Created placement resource snapshot names diff (-got +want):\n%s", diff)) + } + }) + + It("should persist the placement resource snapshots as expected", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "PlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + testCases := []struct { + name string + snapshotName string + wantLabels map[string]string + wantIdentifiers []placementv1alpha1.ObjectReference + }{ + { + // Only the primary snapshot carries the count label. + name: "primary", + snapshotName: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + wantLabels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "2", + }, + wantIdentifiers: deploymentIdentifiers(deploymentNames[:3]), + }, + { + name: "secondary", + snapshotName: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + wantLabels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "1", + }, + wantIdentifiers: deploymentIdentifiers(deploymentNames[3:]), + }, + } + for _, tc := range testCases { + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: tc.snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the %s placement resource snapshot", tc.name) + + if diff := cmp.Diff(gotSnapshot.Labels, tc.wantLabels); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot labels diff (-got +want):\n%s", tc.name, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot annotations diff (-got +want):\n%s", tc.name, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot owner references diff (-got +want):\n%s", tc.name, diff)) + } + + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, tc.wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot resource identifiers diff (-got +want):\n%s", tc.name, diff)) + } + } + }) + + It("can create more Deployments", func() { + deployments := make([]*appsv1.Deployment, 0, len(extraDeploymentNames)) + for _, name := range extraDeploymentNames { + deployment := newDeployment(name, appNamespaceName) + deployment.Labels = map[string]string{"placement-group": "foo"} + Expect(hubClient.Create(ctx, deployment)).To(Succeed(), "Failed to create the Deployment") + deployments = append(deployments, deployment) + } + waitForDeploymentsInCache(deployments...) + }) + + It("should not create new snapshots if snapshots already exist", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + // The selected resources have changed since the snapshots were taken. + Expect(isUpToDate).To(BeFalse(), "The existing placement resource snapshots are unexpectedly up-to-date") + + // The existing snapshots (at index 0) are returned, ordered by their sub-indices. + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + } + gotNames := make([]string, 0, len(gotSnapshots)) + for _, snapshot := range gotSnapshots { + gotNames = append(gotNames, snapshot.GetName()) + } + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot names diff (-got +want):\n%s", diff)) + } + + // No snapshots should have been created at the next index. + nextSnapshotNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1, 1), + } + for _, name := range nextSnapshotNames { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: name}, &placementv1alpha1.PlacementResourceSnapshot{}) + Expect(errors.IsNotFound(err)).To(BeTrue(), "Placement resource snapshot %s was created unexpectedly: %v", name, err) + } + }) + + // verifySnapshotsOfDeployments verifies that the given snapshots, created at the given index, are + // the primary and secondary placement resource snapshots that cover the given Deployments, and returns + // the snapshots as persisted. + // + // The resources are spread across the snapshots in groups of three, in the order of their names. + verifySnapshotsOfDeployments := func( + snapshotIdx int, + gotSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, + wantHash string, + wantDeploymentNames []string, + ) []*placementv1alpha1.PlacementResourceSnapshot { + wantSnapshotCnt := (len(wantDeploymentNames) + 2) / 3 + Expect(gotSnapshots).To(HaveLen(wantSnapshotCnt), "Unexpected number of placement resource snapshots") + + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "PlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + + persistedSnapshots := make([]*placementv1alpha1.PlacementResourceSnapshot, len(gotSnapshots)) + for subIdx := range gotSnapshots { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, snapshotIdx) + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: fmt.Sprintf("%d", snapshotIdx), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: fmt.Sprintf("%d", subIdx), + } + if subIdx == 0 { + // Only the primary snapshot carries the count label. + wantLabels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = fmt.Sprintf("%d", wantSnapshotCnt) + } else { + snapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, snapshotIdx, subIdx) + } + Expect(gotSnapshots[subIdx].GetName()).To(Equal(snapshotName), "Unexpected name of the placement resource snapshot at sub-index %d", subIdx) + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot at sub-index %d", subIdx) + persistedSnapshots[subIdx] = gotSnapshot + + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) labels diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) annotations diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) owner references diff (-got +want):\n%s", subIdx, diff)) + } + + groupStart := subIdx * 3 + groupEnd := min(groupStart+3, len(wantDeploymentNames)) + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, deploymentIdentifiers(wantDeploymentNames[groupStart:groupEnd])); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) resource identifiers diff (-got +want):\n%s", subIdx, diff)) + } + } + return persistedSnapshots + } + + It("should create new snapshots if the existing ones are stale", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after adding Deployments") + wantHash = newHash + + // Ten resources, with at most three resources per snapshot: one primary and three secondary snapshots. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshots are not up-to-date") + verifySnapshotsOfDeployments(1, gotSnapshots, wantHash, allDeploymentNames) + }) + + It("can update the last Deployment", func() { + lastDeploymentName := extraDeploymentNames[len(extraDeploymentNames)-1] + deployment := &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: lastDeploymentName}, deployment)).To(Succeed(), + "Failed to get the Deployment") + deployment.Spec.Replicas = ptr.To(int32(3)) + deployment.Spec.Template.Spec.Containers[0].Image = "nginx:1.28" + Expect(hubClient.Update(ctx, deployment)).To(Succeed(), "Failed to update the Deployment") + waitForDeploymentsInCache(deployment) + }) + + It("should create new snapshots again if the existing ones are stale", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after updating the Deployment") + wantHash = newHash + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshots are not up-to-date") + persistedSnapshots := verifySnapshotsOfDeployments(2, gotSnapshots, wantHash, allDeploymentNames) + + // The last Deployment lives in the last secondary snapshot; it should carry the updated content. + lastSnapshot := persistedSnapshots[len(persistedSnapshots)-1] + Expect(lastSnapshot.Spec.Resources).To(HaveLen(1), "Unexpected number of resources in the last secondary snapshot") + gotDeployment := &appsv1.Deployment{} + Expect(json.Unmarshal(lastSnapshot.Spec.Resources[0].Manifest.Raw, gotDeployment)).To(Succeed(), "Failed to unmarshal the Deployment manifest") + Expect(gotDeployment.Spec.Replicas).To(Equal(ptr.To(int32(3))), "Unexpected replica count in the snapshotted Deployment") + Expect(gotDeployment.Spec.Template.Spec.Containers[0].Image).To(Equal("nginx:1.28"), "Unexpected image in the snapshotted Deployment") + }) + + It("can delete some Deployments", func() { + deletedIdxs := []int{2, 5, 9} + deployments := make([]*appsv1.Deployment, 0, len(deletedIdxs)) + for _, idx := range deletedIdxs { + deployment := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: allDeploymentNames[idx], Namespace: appNamespaceName}} + Expect(hubClient.Delete(ctx, deployment)).To(Succeed(), "Failed to delete the Deployment") + deployments = append(deployments, deployment) + } + waitForDeploymentsRemovedFromCache(deployments...) + }) + + It("should create new snapshots if some selected resources have been removed", func() { + // Seven resources are left, with at most three resources per snapshot: one primary and two secondary snapshots. + var wantDeploymentNames []string + for idx, name := range allDeploymentNames { + if idx != 2 && idx != 5 && idx != 9 { + wantDeploymentNames = append(wantDeploymentNames, name) + } + } + + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after deleting the Deployments") + wantHash = newHash + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshots are not up-to-date") + verifySnapshotsOfDeployments(3, gotSnapshots, wantHash, wantDeploymentNames) + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; remove all the objects explicitly. + objs := []client.Object{ + &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + Namespace: appNamespaceName, + }}, + &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + Namespace: appNamespaceName, + }}, + placementPolicy, + } + var snapshotNames []string + for snapshotIdx := 1; snapshotIdx <= 3; snapshotIdx++ { + snapshotNames = append(snapshotNames, uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, snapshotIdx)) + for subIdx := 1; subIdx <= 3; subIdx++ { + snapshotNames = append(snapshotNames, uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, snapshotIdx, subIdx)) + } + } + for _, name := range snapshotNames { + objs = append(objs, &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: appNamespaceName}}) + } + for _, name := range allDeploymentNames { + objs = append(objs, &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: appNamespaceName}}) + } + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) + + Context("primary and secondary placement resource snapshots (resources too large) (label-based selection)", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + // The name sorts after that of the first config map. + secondConfigMapName := configMapName + "-2" + + var placementPolicy *placementv1alpha1.PlacementPolicy + var wantHash string + + BeforeAll(func() { + // Create a config map that is larger than the per-snapshot size limit of the manager under test. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + Labels: map[string]string{"placement-group": "foo"}, + }, + Data: map[string]string{"key": strings.Repeat("a", 15000)}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + // Create a placement policy that selects the config map by its label. + placementPolicy = &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + Namespace: appNamespaceName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"placement-group": "foo"}, + }, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("should return a user error when a selected resource is too large", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).To(HaveOccurred(), "Snapshotting resources succeeded unexpectedly") + Expect(gotSnapshots).To(BeEmpty(), "Placement resource snapshots are returned unexpectedly") + Expect(isUpToDate).To(BeFalse(), "The up-to-date flag is set unexpectedly") + + // The error category is the first key-value pair in the error attributes. + gotErrAttrs := kferrors.Args(err) + Expect(gotErrAttrs).ToNot(BeEmpty(), "The error is not a KubeFleet error: %v", err) + wantErrCategoryAttrs := []interface{}{"errCategory", kferrors.ErrCategoryUser} + if diff := cmp.Diff(gotErrAttrs[:2], wantErrCategoryAttrs); diff != "" { + Fail(fmt.Sprintf("Error category attributes diff (-got +want):\n%s\nerror: %v", diff, err)) + } + }) + + It("should not create any placement resource snapshot", func() { + snapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} + Expect(hubUncachedReader.List(ctx, snapshotList, client.InNamespace(appNamespaceName), client.MatchingLabels{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + })).To(Succeed(), "Failed to list placement resource snapshots") + Expect(snapshotList.Items).To(BeEmpty(), "Placement resource snapshots are created unexpectedly") + }) + + It("can resize the config map and create another config map", func() { + // Each config map now fits in a snapshot, but the two of them together do not. + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = strings.Repeat("a", 7500) + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + + secondConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: secondConfigMapName, + Namespace: appNamespaceName, + Labels: map[string]string{"placement-group": "foo"}, + }, + Data: map[string]string{"key": strings.Repeat("b", 7500)}, + } + Expect(hubClient.Create(ctx, secondConfigMap)).To(Succeed(), "Failed to create the second config map") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(resources).To(HaveLen(2), "Unexpected number of selected resources") + wantHash = hash + }) + + It("can create a primary and a secondary placement resource snapshot", func() { + createdSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The created placement resource snapshots are not up-to-date") + Expect(createdSnapshots).To(HaveLen(2), "Unexpected number of created placement resource snapshots") + + // The snapshots are ordered by their sub-indices. + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + } + gotNames := []string{createdSnapshots[0].GetName(), createdSnapshots[1].GetName()} + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Created placement resource snapshot names diff (-got +want):\n%s", diff)) + } + }) + + It("should persist the placement resource snapshots as expected", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "PlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + + testCases := []struct { + name string + snapshotName string + wantLabels map[string]string + wantConfigMap string + wantConfigData string + }{ + { + // Only the primary snapshot carries the count label. + name: "primary", + snapshotName: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + wantLabels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "2", + }, + wantConfigMap: configMapName, + wantConfigData: strings.Repeat("a", 7500), + }, + { + name: "secondary", + snapshotName: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + wantLabels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "1", + }, + wantConfigMap: secondConfigMapName, + wantConfigData: strings.Repeat("b", 7500), + }, + } + for _, tc := range testCases { + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: tc.snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the %s placement resource snapshot", tc.name) + + if diff := cmp.Diff(gotSnapshot.Labels, tc.wantLabels); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot labels diff (-got +want):\n%s", tc.name, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot annotations diff (-got +want):\n%s", tc.name, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot owner references diff (-got +want):\n%s", tc.name, diff)) + } + + // Each snapshot holds exactly one of the two config maps. + Expect(gotSnapshot.Spec.Resources).To(HaveLen(1), "Unexpected number of resources in the %s placement resource snapshot", tc.name) + wantIdentifier := placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: tc.wantConfigMap, + APIVersion: "v1", + Kind: "ConfigMap", + } + if diff := cmp.Diff(gotSnapshot.Spec.Resources[0].Identifier, wantIdentifier); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot resource identifier diff (-got +want):\n%s", tc.name, diff)) + } + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[0].Manifest.Raw, gotConfigMap)).To(Succeed(), + "Failed to unmarshal the config map manifest in the %s placement resource snapshot", tc.name) + if diff := cmp.Diff(gotConfigMap.Data, map[string]string{"key": tc.wantConfigData}); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot config map data diff (-got +want):\n%s", tc.name, diff)) + } + } + }) + + It("can shrink the config maps", func() { + // The two config maps now fit in a single snapshot together. + for _, name := range []string{configMapName, secondConfigMapName} { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: name}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = strings.Repeat("c", 4000) + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + } + }) + + It("should create a primary placement resource snapshot only if the existing snapshots are stale", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after shrinking the config maps") + wantHash = newHash + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of created placement resource snapshots") + wantName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + Expect(gotSnapshots[0].GetName()).To(Equal(wantName), "Unexpected name of the new placement resource snapshot") + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: wantName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the new placement resource snapshot") + + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "1", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot labels diff (-got +want):\n%s", diff)) + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot annotations diff (-got +want):\n%s", diff)) + } + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "PlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot owner references diff (-got +want):\n%s", diff)) + } + + // Both config maps live in the primary snapshot, with the updated content. + Expect(gotSnapshot.Spec.Resources).To(HaveLen(2), "Unexpected number of resources in the new placement resource snapshot") + for idx, name := range []string{configMapName, secondConfigMapName} { + wantIdentifier := placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: name, + APIVersion: "v1", + Kind: "ConfigMap", + } + if diff := cmp.Diff(gotSnapshot.Spec.Resources[idx].Identifier, wantIdentifier); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot resource identifier (index %d) diff (-got +want):\n%s", idx, diff)) + } + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[idx].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + if diff := cmp.Diff(gotConfigMap.Data, map[string]string{"key": strings.Repeat("c", 4000)}); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot config map data (index %d) diff (-got +want):\n%s", idx, diff)) + } + } + + // No secondary snapshot should have been created at the new index. + secondaryName := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1, 1) + err = hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: secondaryName}, &placementv1alpha1.PlacementResourceSnapshot{}) + Expect(errors.IsNotFound(err)).To(BeTrue(), "A secondary placement resource snapshot was created unexpectedly: %v", err) + }) + + AfterAll(func() { + objs := []client.Object{ + &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + Namespace: appNamespaceName, + }}, + &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + Namespace: appNamespaceName, + }}, + &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1), + Namespace: appNamespaceName, + }}, + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: secondConfigMapName, Namespace: appNamespaceName}}, + } + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) + + Context("partial placement resource snapshot provisioning", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + // The names are in ascending order, which is also the order in which the selected resources are + // snapshotted. + configMapNames := make([]string, 10) + for idx := range configMapNames { + configMapNames[idx] = fmt.Sprintf("%s-%d", configMapNamePrefix, idx) + } + + var placementPolicy *placementv1alpha1.PlacementPolicy + var wantHash string + // The UIDs of the secondary placement resource snapshots (by sub-index minus one), recorded for + // later verification. + var secondarySnapshotUIDs []types.UID + + // verifyOrphanedSecondariesCleanedUp verifies that the manager cleans up the orphaned secondary placement + // resource snapshots (the primary snapshot is missing) and returns a transient error. + // + // The manager treats the secondary snapshots left without a primary snapshot as orphans: it deletes them + // and returns a transient error, asking the caller to requeue before creating the snapshots anew. + verifyOrphanedSecondariesCleanedUp := func() { + // Other errors (e.g., those caused by a stale cache) might surface first; retry until the expected + // transient error is observed. + Eventually(func() error { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + if err == nil { + return fmt.Errorf("snapshotting resources succeeded unexpectedly (snapshots %d, up-to-date %t)", len(gotSnapshots), isUpToDate) + } + if !strings.Contains(err.Error(), "cleaned up orphaned secondary placement resource snapshots") { + return fmt.Errorf("unexpected error message: %w", err) + } + if len(gotSnapshots) != 0 || isUpToDate { + return fmt.Errorf("unexpected return values alongside the error (snapshots %d, up-to-date %t)", len(gotSnapshots), isUpToDate) + } + + // The error category is the first key-value pair in the error attributes. + gotErrAttrs := kferrors.Args(err) + if len(gotErrAttrs) < 2 { + return fmt.Errorf("the error is not a KubeFleet error: %w", err) + } + wantErrCategoryAttrs := []interface{}{"errCategory", kferrors.ErrCategoryTransient} + if diff := cmp.Diff(gotErrAttrs[:2], wantErrCategoryAttrs); diff != "" { + return fmt.Errorf("error category attributes diff (-got +want):\n%s\nerror: %w", diff, err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the transient error from the clean-up of orphaned secondary placement resource snapshots") + + // The orphaned secondary snapshots should have been deleted. + for subIdx := 1; subIdx <= 3; subIdx++ { + secondarySnapshotName := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: secondarySnapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove the orphaned secondary placement resource snapshot at sub-index %d", subIdx) + } + } + + BeforeAll(func() { + // Create ten config maps with the same label in the app namespace. + for idx := range configMapNames { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapNames[idx], + Namespace: appNamespaceName, + Labels: map[string]string{"placement-group": "foo"}, + }, + Data: map[string]string{"key": fmt.Sprintf("value-%d", idx)}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + } + + // Create a placement policy that selects the config maps by their label. + placementPolicy = &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + Namespace: appNamespaceName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"placement-group": "foo"}, + }, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(resources).To(HaveLen(len(configMapNames)), "Unexpected number of selected resources") + wantHash = hash + }) + + It("can create a primary and three secondary placement resource snapshots", func() { + // Ten resources, with at most three resources per snapshot: one primary and three secondary snapshots. + createdSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The created placement resource snapshots are not up-to-date") + Expect(createdSnapshots).To(HaveLen(4), "Unexpected number of created placement resource snapshots") + + // The snapshots are ordered by their sub-indices. + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 2), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 3), + } + gotNames := make([]string, 0, len(createdSnapshots)) + for _, snapshot := range createdSnapshots { + gotNames = append(gotNames, snapshot.GetName()) + } + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Created placement resource snapshot names diff (-got +want):\n%s", diff)) + } + + // Write down the UIDs of the secondary placement resource snapshots. + secondarySnapshotUIDs = make([]types.UID, 0, len(createdSnapshots)-1) + for _, snapshot := range createdSnapshots[1:] { + Expect(snapshot.GetUID()).ToNot(BeEmpty(), "The secondary placement resource snapshot %s has no UID", snapshot.GetName()) + secondarySnapshotUIDs = append(secondarySnapshotUIDs, snapshot.GetUID()) + } + }) + + It("should persist the placement resource snapshots as expected", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "PlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + + for subIdx := 0; subIdx < 4; subIdx++ { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: fmt.Sprintf("%d", subIdx), + } + if subIdx == 0 { + // Only the primary snapshot carries the count label. + wantLabels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = "4" + } else { + snapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx) + } + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot at sub-index %d", subIdx) + + if subIdx > 0 { + Expect(gotSnapshot.UID).To(Equal(secondarySnapshotUIDs[subIdx-1]), "Unexpected UID of the placement resource snapshot at sub-index %d", subIdx) + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) labels diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) annotations diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) owner references diff (-got +want):\n%s", subIdx, diff)) + } + + // The resources are spread across the snapshots in groups of three, in the order of their names. + groupStart := subIdx * 3 + groupEnd := min(groupStart+3, len(configMapNames)) + wantIdentifiers := make([]placementv1alpha1.ObjectReference, 0, groupEnd-groupStart) + for _, name := range configMapNames[groupStart:groupEnd] { + wantIdentifiers = append(wantIdentifiers, placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: name, + APIVersion: "v1", + Kind: "ConfigMap", + }) + } + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) resource identifiers diff (-got +want):\n%s", subIdx, diff)) + } + } + }) + + It("can delete the primary placement resource snapshot", func() { + // Note that this is for simulation purposes only: it mimics a partial provisioning, where the primary + // snapshot is missing while its secondary snapshots remain. The manager never deletes only the primary + // snapshot in normal operations, so this situation will not occur in practice. + primarySnapshot := &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + Namespace: appNamespaceName, + }, + } + Expect(hubClient.Delete(ctx, primarySnapshot)).To(Succeed(), "Failed to delete the primary placement resource snapshot") + + // Wait until the cache has observed the deletion. + Eventually(func() bool { + return errors.IsNotFound(hubClient.Get(ctx, client.ObjectKeyFromObject(primarySnapshot), primarySnapshot)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove the primary placement resource snapshot") + }) + + It("should clean up the orphaned secondary placement resource snapshots and return a transient error", func() { + verifyOrphanedSecondariesCleanedUp() + }) + + It("can re-create the primary and secondary placement resource snapshots", func() { + // The orphaned secondary snapshots have been cleaned up; the manager can now create the snapshots anew. + // The call is retried as the cache might not have observed the clean-up yet. + var recreatedSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + var isUpToDate bool + Eventually(func() error { + var err error + recreatedSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + return err + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The re-created placement resource snapshots are not up-to-date") + + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 2), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 3), + } + gotNames := make([]string, 0, len(recreatedSnapshots)) + for _, snapshot := range recreatedSnapshots { + gotNames = append(gotNames, snapshot.GetName()) + } + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Re-created placement resource snapshot names diff (-got +want):\n%s", diff)) + } + }) + + It("should have re-created the primary snapshot and the secondary snapshots with new UIDs", func() { + // The primary snapshot has been re-created (its UID is not checked). + primarySnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + primarySnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: primarySnapshotName}, primarySnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the primary placement resource snapshot") + Expect(primarySnapshot.Annotations).To(HaveKeyWithValue(placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey, wantHash), + "Unexpected contents hash of the re-created primary placement resource snapshot") + Expect(primarySnapshot.Labels).To(HaveKeyWithValue(placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey, "4"), + "Unexpected snapshot count of the re-created primary placement resource snapshot") + + // All the secondary snapshots have been re-created with new UIDs. + for subIdx := 1; subIdx <= 3; subIdx++ { + secondarySnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + secondarySnapshotName := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx) + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: secondarySnapshotName}, secondarySnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the secondary placement resource snapshot at sub-index %d", subIdx) + Expect(secondarySnapshot.UID).ToNot(Equal(secondarySnapshotUIDs[subIdx-1]), + "The secondary placement resource snapshot at sub-index %d has not been re-created (UID unchanged)", subIdx) + Expect(secondarySnapshot.Annotations).To(HaveKeyWithValue(placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey, wantHash), + "Unexpected contents hash of the re-created secondary placement resource snapshot at sub-index %d", subIdx) + } + }) + + It("can delete the primary placement resource snapshot again", func() { + // Record the UIDs of the secondary placement resource snapshots that were re-created previously. + secondarySnapshotUIDs = make([]types.UID, 0, 3) + for subIdx := 1; subIdx <= 3; subIdx++ { + secondarySnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Expect(hubClient.Get(ctx, types.NamespacedName{ + Namespace: appNamespaceName, + Name: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx), + }, secondarySnapshot)).To(Succeed(), "Failed to get the secondary placement resource snapshot at sub-index %d", subIdx) + secondarySnapshotUIDs = append(secondarySnapshotUIDs, secondarySnapshot.UID) + } + + // Note that this is for simulation purposes only: it mimics a partial provisioning, where the primary + // snapshot is missing while its secondary snapshots remain. The manager never deletes only the primary + // snapshot in normal operations, so this situation will not occur in practice. + primarySnapshot := &placementv1alpha1.PlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + Namespace: appNamespaceName, + }, + } + Expect(hubClient.Delete(ctx, primarySnapshot)).To(Succeed(), "Failed to delete the primary placement resource snapshot") + + // Wait until the cache has observed the deletion. + Eventually(func() bool { + return errors.IsNotFound(hubClient.Get(ctx, client.ObjectKeyFromObject(primarySnapshot), primarySnapshot)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove the primary placement resource snapshot") + }) + + It("can update the last config map", func() { + lastConfigMapName := configMapNames[len(configMapNames)-1] + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: lastConfigMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = "updated-value" + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + }) + + It("should clean up the orphaned secondary placement resource snapshots and return a transient error again", func() { + verifyOrphanedSecondariesCleanedUp() + }) + + It("can re-create the primary and secondary placement resource snapshots with the updated content", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after updating the config map") + wantHash = newHash + + // The orphaned secondary snapshots have been cleaned up; the manager can now create the snapshots anew. + // The call is retried as the cache might not have observed the clean-up yet. + var recreatedSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + var isUpToDate bool + Eventually(func() error { + var err error + recreatedSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + return err + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The re-created placement resource snapshots are not up-to-date") + Expect(recreatedSnapshots).To(HaveLen(4), "Unexpected number of re-created placement resource snapshots") + }) + + It("should have re-created all the snapshots with the expected content", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "PlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + + for subIdx := 0; subIdx < 4; subIdx++ { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: fmt.Sprintf("%d", subIdx), + } + if subIdx == 0 { + // Only the primary snapshot carries the count label. + wantLabels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = "4" + } else { + snapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx) + } + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot at sub-index %d", subIdx) + + if subIdx > 0 { + Expect(gotSnapshot.UID).ToNot(Equal(secondarySnapshotUIDs[subIdx-1]), + "The secondary placement resource snapshot at sub-index %d has not been re-created (UID unchanged)", subIdx) + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) labels diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) annotations diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) owner references diff (-got +want):\n%s", subIdx, diff)) + } + + // The resources are spread across the snapshots in groups of three, in the order of their names. + groupStart := subIdx * 3 + groupEnd := min(groupStart+3, len(configMapNames)) + wantIdentifiers := make([]placementv1alpha1.ObjectReference, 0, groupEnd-groupStart) + for _, name := range configMapNames[groupStart:groupEnd] { + wantIdentifiers = append(wantIdentifiers, placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: name, + APIVersion: "v1", + Kind: "ConfigMap", + }) + } + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) resource identifiers diff (-got +want):\n%s", subIdx, diff)) + } + + // Verify the content of each config map; only the last one has been updated. + for idx := range gotSnapshot.Spec.Resources { + configMapIdx := groupStart + idx + wantData := map[string]string{"key": fmt.Sprintf("value-%d", configMapIdx)} + if configMapIdx == len(configMapNames)-1 { + wantData["key"] = "updated-value" + } + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[idx].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + if diff := cmp.Diff(gotConfigMap.Data, wantData); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) config map data (index %d) diff (-got +want):\n%s", subIdx, idx, diff)) + } + } + } + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; remove all the objects explicitly. + objs := []client.Object{ + &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + Namespace: appNamespaceName, + }}, + placementPolicy, + } + for subIdx := 1; subIdx <= 3; subIdx++ { + objs = append(objs, &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx), + Namespace: appNamespaceName, + }}) + } + for _, name := range configMapNames { + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: appNamespaceName}}) + } + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) + + Context("GC", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + + var placementPolicy *placementv1alpha1.PlacementPolicy + + // updateConfigMapAndSnapshot updates the config map with the given data and calls the manager to snapshot + // the selected resources; it returns the snapshots created. + updateConfigMapAndSnapshot := func(data string) []placementv1alpha1.PlacementResourceSnapshotAccessor { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = data + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of created placement resource snapshots") + + // Wait until the cache has observed the new snapshot; otherwise the next call might not see it. + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKeyFromObject(gotSnapshots[0]), &placementv1alpha1.PlacementResourceSnapshot{}) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the new placement resource snapshot") + return gotSnapshots + } + + BeforeAll(func() { + // Create a config map in the app namespace. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + }, + Data: map[string]string{"key": "initial-value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + // Create a placement policy that selects the config map. + placementPolicy = &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + Namespace: appNamespaceName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMapName, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create four primary placement resource snapshots", func() { + for idx := 0; idx < 4; idx++ { + gotSnapshots := updateConfigMapAndSnapshot(fmt.Sprintf("value-%d", idx)) + wantName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, idx) + Expect(gotSnapshots[0].GetName()).To(Equal(wantName), "Unexpected name of the placement resource snapshot at index %d", idx) + } + }) + + It("should have four primary placement resource snapshots at the expected indexes", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "PlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + + seenHashes := map[string]bool{} + for idx := 0; idx < 4; idx++ { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, idx) + gotSnapshot := &placementv1alpha1.PlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot at index %d", idx) + + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: fmt.Sprintf("%d", idx), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (index %d) labels diff (-got +want):\n%s", idx, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (index %d) owner references diff (-got +want):\n%s", idx, diff)) + } + + // Each snapshot tracks a different state of the config map. + hash := gotSnapshot.Annotations[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] + Expect(hash).ToNot(BeEmpty(), "The placement resource snapshot (index %d) has no contents hash", idx) + Expect(seenHashes).ToNot(HaveKey(hash), "The placement resource snapshot (index %d) has the same contents hash as another snapshot", idx) + seenHashes[hash] = true + + Expect(gotSnapshot.Spec.Resources).To(HaveLen(1), "Unexpected number of resources in the placement resource snapshot (index %d)", idx) + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[0].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + if diff := cmp.Diff(gotConfigMap.Data, map[string]string{"key": fmt.Sprintf("value-%d", idx)}); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (index %d) config map data diff (-got +want):\n%s", idx, diff)) + } + } + + // There should be no other snapshots. + snapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} + Expect(hubUncachedReader.List(ctx, snapshotList, client.InNamespace(appNamespaceName), client.MatchingLabels{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + })).To(Succeed(), "Failed to list placement resource snapshots") + Expect(snapshotList.Items).To(HaveLen(4), "Unexpected number of placement resource snapshots") + }) + + It("should garbage collect the first primary placement resource snapshot", func() { + updateConfigMapAndSnapshot("value-4") + + firstSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: firstSnapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The first primary placement resource snapshot has not been garbage collected") + }) + + It("should garbage collect the second primary placement resource snapshot", func() { + updateConfigMapAndSnapshot("value-5") + + secondSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: secondSnapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The second primary placement resource snapshot has not been garbage collected") + }) + + It("can create a placement binding that uses the third primary placement resource snapshot", func() { + thirdSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 2) + placementBinding := &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementBindingName, + Namespace: appNamespaceName, + OwnerReferences: []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.PlacementPolicyKind, + Name: placementPolicyName, + UID: placementPolicy.UID, + }, + }, + }, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: placementPolicyName, + ClusterName: "member-1", + ResourceSnapshotName: thirdSnapshotName, + }, + } + Expect(hubClient.Create(ctx, placementBinding)).To(Succeed(), "Failed to create the placement binding") + + // Wait until the cache (and its field index) has observed the placement binding; the garbage collection + // process relies on it to tell if a snapshot is still in use. + wantFieldVal := fmt.Sprintf( + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, placementPolicyName, thirdSnapshotName) + Eventually(func() (int, error) { + placementBindingList := &placementv1alpha1.PlacementBindingList{} + err := hubClient.List(ctx, placementBindingList, client.InNamespace(appNamespaceName), client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName: wantFieldVal, + }) + return len(placementBindingList.Items), err + }, eventuallyDuration, eventuallyInterval).Should(Equal(1), "Failed to find the placement binding in the cache") + }) + + It("should not garbage collect the third primary placement resource snapshot as it is in use", func() { + // The third snapshot is now the oldest one that is past the revision history limit; the garbage + // collection process should skip it as a placement binding is using it. + updateConfigMapAndSnapshot("value-6") + + thirdSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 2) + Consistently(func() error { + return hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: thirdSnapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The third primary placement resource snapshot has been garbage collected unexpectedly") + }) + + It("should garbage collect the fourth primary placement resource snapshot but keep the third one", func() { + // The third and the fourth snapshots are now both past the revision history limit; the third one is + // still in use by a placement binding, so only the fourth one should be garbage collected. + updateConfigMapAndSnapshot("value-7") + + fourthSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 3) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: fourthSnapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The fourth primary placement resource snapshot has not been garbage collected") + + // The garbage collection requests are processed in order; the request for the third snapshot has been + // handled by now. + thirdSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 2) + Consistently(func() error { + return hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: thirdSnapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The third primary placement resource snapshot has been garbage collected unexpectedly") + }) + + It("can delete the placement binding", func() { + placementBinding := &placementv1alpha1.PlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementBindingName, + Namespace: appNamespaceName, + }, + } + Expect(hubClient.Delete(ctx, placementBinding)).To(Succeed(), "Failed to delete the placement binding") + + // Wait until the cache (and its field index) has observed the deletion; the garbage collection process + // relies on it to tell if a snapshot is still in use. + thirdSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 2) + Eventually(func() (int, error) { + placementBindingList := &placementv1alpha1.PlacementBindingList{} + err := hubClient.List(ctx, placementBindingList, client.InNamespace(appNamespaceName), client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, placementPolicyName, thirdSnapshotName), + }) + return len(placementBindingList.Items), err + }, eventuallyDuration, eventuallyInterval).Should(Equal(0), "The placement binding is still found in the cache") + }) + + It("should garbage collect the third and the fifth primary placement resource snapshots", func() { + // The third snapshot is no longer in use; both the third and the fifth snapshots are now past the + // revision history limit. + updateConfigMapAndSnapshot("value-8") + + for _, idx := range []int{2, 4} { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, idx) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: snapshotName}, &placementv1alpha1.PlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The primary placement resource snapshot at index %d has not been garbage collected", idx) + } + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; remove all the objects explicitly. + objs := []client.Object{ + &placementv1alpha1.PlacementBinding{ObjectMeta: metav1.ObjectMeta{Name: placementBindingName, Namespace: appNamespaceName}}, + placementPolicy, + } + for idx := 0; idx < 9; idx++ { + objs = append(objs, &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, idx), + Namespace: appNamespaceName, + }}) + } + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) +}) + +var _ = Describe("core ops (cluster placement resource snapshots)", func() { + Context("primary placement resource snapshot only (name-based selection)", Ordered, func() { + // The environment prepared by the envtest package does not support namespace deletion; the + // test case creates its objects in the shared app namespace, with unique names. + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + deploymentName := fmt.Sprintf(deploymentNameTemplate, utils.RandStr()) + + var placementPolicy *placementv1alpha1.ClusterPlacementPolicy + var wantHash string + + BeforeAll(func() { + // Create a config map and a deployment in the app namespace. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + Labels: map[string]string{"app": "demo"}, + }, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + deployment := &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: deploymentName, + Namespace: appNamespaceName, + Labels: map[string]string{"app": "demo"}, + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(int32(2)), + Selector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"app": "demo"}, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{"app": "demo"}, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "nginx", + Image: "nginx:1.27", + }, + }, + }, + }, + }, + } + Expect(hubClient.Create(ctx, deployment)).To(Succeed(), "Failed to create the Deployment") + waitForDeploymentsInCache(deployment) + + // Create a placement policy that selects the two resources. + placementPolicy = &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + Name: configMapName, + }, + { + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Namespace: appNamespaceName, + Name: deploymentName, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a primary placement resource snapshot", func() { + createdSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to create the placement resource snapshot") + Expect(createdSnapshots).To(HaveLen(1), "Unexpected number of created placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The created placement resource snapshot is not up-to-date") + }) + + It("can compute the hash of selected resources", func() { + _, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + wantHash = hash + }) + + It("should persist the placement resource snapshot as expected", func() { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot") + + // Verify the metadata. + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot labels diff (-got +want):\n%s", diff)) + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot annotations diff (-got +want):\n%s", diff)) + } + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "ClusterPlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot owner references diff (-got +want):\n%s", diff)) + } + + // Verify the snapshotted resources; they are sorted by their unique IDs, i.e., the config map comes first. + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: appNamespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Namespace: appNamespaceName, + Name: deploymentName, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + }, + } + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("Snapshotted resource identifiers diff (-got +want):\n%s", diff)) + } + + // Verify the config map manifest. + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[0].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + wantConfigMap := &corev1.ConfigMap{ + TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "ConfigMap"}, + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + Labels: map[string]string{"app": "demo"}, + }, + Data: map[string]string{"key": "value"}, + } + if diff := cmp.Diff(gotConfigMap, wantConfigMap); diff != "" { + Fail(fmt.Sprintf("Snapshotted config map diff (-got +want):\n%s", diff)) + } + + // Verify the Deployment manifest; the spec should match that of the live object (which carries the + // defaulted fields), while the read-only metadata and status fields should have been stripped. + liveDeployment := &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: deploymentName}, liveDeployment)).To(Succeed(), + "Failed to get the Deployment") + gotDeployment := &appsv1.Deployment{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[1].Manifest.Raw, gotDeployment)).To(Succeed(), "Failed to unmarshal the Deployment manifest") + wantDeployment := &appsv1.Deployment{ + TypeMeta: metav1.TypeMeta{APIVersion: "apps/v1", Kind: "Deployment"}, + ObjectMeta: metav1.ObjectMeta{ + Name: deploymentName, + Namespace: appNamespaceName, + Labels: map[string]string{"app": "demo"}, + }, + Spec: liveDeployment.Spec, + } + if diff := cmp.Diff(gotDeployment, wantDeployment, cmpopts.EquateEmpty()); diff != "" { + Fail(fmt.Sprintf("Snapshotted Deployment diff (-got +want):\n%s", diff)) + } + }) + + It("should return the existing snapshot when no snapshot is needed", func() { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + wantSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + // The previous spec has waited for the snapshot to appear in the cache. + Expect(hubClient.Get(ctx, types.NamespacedName{Name: snapshotName}, wantSnapshot)).To(Succeed(), + "Failed to get the placement resource snapshot") + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The existing placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of placement resource snapshots") + + gotSnapshot, ok := gotSnapshots[0].(*placementv1alpha1.ClusterPlacementResourceSnapshot) + Expect(ok).To(BeTrue(), "Unexpected type of the returned placement resource snapshot: %T", gotSnapshots[0]) + if diff := cmp.Diff(gotSnapshot, wantSnapshot, cmpopts.IgnoreFields(metav1.TypeMeta{}, "APIVersion", "Kind")); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot diff (-got +want):\n%s", diff)) + } + + // No new snapshot should have been created. + nextSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + err = hubUncachedReader.Get(ctx, types.NamespacedName{Name: nextSnapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + Expect(errors.IsNotFound(err)).To(BeTrue(), "A new placement resource snapshot was created unexpectedly: %v", err) + }) + + It("should return the existing snapshot when it is not stale", func() { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + wantSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Expect(hubClient.Get(ctx, types.NamespacedName{Name: snapshotName}, wantSnapshot)).To(Succeed(), + "Failed to get the placement resource snapshot") + + // The selected resources have not changed since the snapshot was taken. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The existing placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of placement resource snapshots") + + gotSnapshot, ok := gotSnapshots[0].(*placementv1alpha1.ClusterPlacementResourceSnapshot) + Expect(ok).To(BeTrue(), "Unexpected type of the returned placement resource snapshot: %T", gotSnapshots[0]) + if diff := cmp.Diff(gotSnapshot, wantSnapshot, cmpopts.IgnoreFields(metav1.TypeMeta{}, "APIVersion", "Kind")); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot diff (-got +want):\n%s", diff)) + } + + // No new snapshot should have been created. + nextSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + err = hubUncachedReader.Get(ctx, types.NamespacedName{Name: nextSnapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + Expect(errors.IsNotFound(err)).To(BeTrue(), "A new placement resource snapshot was created unexpectedly: %v", err) + }) + + It("can update the selected resources", func() { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = "new-value" + configMap.Data["extra-key"] = "extra-value" + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + + deployment := &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: deploymentName}, deployment)).To(Succeed(), + "Failed to get the Deployment") + deployment.Spec.Replicas = ptr.To(int32(3)) + deployment.Spec.Template.Spec.Containers[0].Image = "nginx:1.28" + Expect(hubClient.Update(ctx, deployment)).To(Succeed(), "Failed to update the Deployment") + waitForDeploymentsInCache(deployment) + }) + + It("can create a new snapshot when the existing one is stale", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of placement resource snapshots") + + newSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + Expect(gotSnapshots[0].GetName()).To(Equal(newSnapshotName), "Unexpected name of the new placement resource snapshot") + }) + + It("can create a new snapshot when the existing one is stale", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after the updates") + + newSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: newSnapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the new placement resource snapshot") + + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "1", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("New placement resource snapshot labels diff (-got +want):\n%s", diff)) + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: newHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("New placement resource snapshot annotations diff (-got +want):\n%s", diff)) + } + + // Verify that the new snapshot carries the updated resources. + Expect(gotSnapshot.Spec.Resources).To(HaveLen(2), "Unexpected number of snapshotted resources") + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[0].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + wantConfigMapData := map[string]string{"key": "new-value", "extra-key": "extra-value"} + if diff := cmp.Diff(gotConfigMap.Data, wantConfigMapData); diff != "" { + Fail(fmt.Sprintf("Snapshotted config map data diff (-got +want):\n%s", diff)) + } + gotDeployment := &appsv1.Deployment{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[1].Manifest.Raw, gotDeployment)).To(Succeed(), "Failed to unmarshal the Deployment manifest") + Expect(gotDeployment.Spec.Replicas).To(Equal(ptr.To(int32(3))), "Unexpected replica count in the snapshotted Deployment") + Expect(gotDeployment.Spec.Template.Spec.Containers[0].Image).To(Equal("nginx:1.28"), "Unexpected image in the snapshotted Deployment") + + // The previous snapshot should be left intact. + oldSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + }, oldSnapshot)).To(Succeed(), "Failed to get the previous placement resource snapshot") + Expect(oldSnapshot.Annotations[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey]).To(Equal(wantHash), + "The previous placement resource snapshot has been modified") + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; remove all the objects explicitly. + objs := []client.Object{ + &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + }}, + &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1), + }}, + placementPolicy, + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: deploymentName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + } + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) + + Context("primary and secondary placement resource snapshots (too many resources) (label-based selection)", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + deploymentNamePrefix := fmt.Sprintf(deploymentNameTemplate, utils.RandStr()) + + // The names are in ascending order, which is also the order in which the selected resources are + // snapshotted. + deploymentNames := make([]string, 5) + for idx := range deploymentNames { + deploymentNames[idx] = fmt.Sprintf("%s-%d", deploymentNamePrefix, idx) + } + + // The names of the Deployments added later; they sort after the ones above. + extraDeploymentNames := make([]string, 5) + for idx := range extraDeploymentNames { + extraDeploymentNames[idx] = fmt.Sprintf("%s-%d", deploymentNamePrefix, idx+len(deploymentNames)) + } + + allDeploymentNames := append(slices.Clone(deploymentNames), extraDeploymentNames...) + + deploymentIdentifiers := func(names []string) []placementv1alpha1.ObjectReference { + identifiers := make([]placementv1alpha1.ObjectReference, 0, len(names)) + for _, name := range names { + identifiers = append(identifiers, placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: name, + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + }) + } + return identifiers + } + + var placementPolicy *placementv1alpha1.ClusterPlacementPolicy + var wantHash string + + BeforeAll(func() { + // Create five Deployments with the same label in the app namespace. + deployments := make([]*appsv1.Deployment, 0, len(deploymentNames)) + for _, name := range deploymentNames { + deployment := newDeployment(name, appNamespaceName) + deployment.Labels = map[string]string{"placement-group": "foo"} + Expect(hubClient.Create(ctx, deployment)).To(Succeed(), "Failed to create the Deployment") + deployments = append(deployments, deployment) + } + waitForDeploymentsInCache(deployments...) + + // Create a placement policy that selects the Deployments by their label. + placementPolicy = &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Namespace: appNamespaceName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"placement-group": "foo"}, + }, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(resources).To(HaveLen(len(deploymentNames)), "Unexpected number of selected resources") + wantHash = hash + }) + + It("can create a primary and a secondary placement resource snapshot", func() { + createdSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The created placement resource snapshots are not up-to-date") + Expect(createdSnapshots).To(HaveLen(2), "Unexpected number of created placement resource snapshots") + + // The snapshots are ordered by their sub-indices. + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + } + gotNames := []string{createdSnapshots[0].GetName(), createdSnapshots[1].GetName()} + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Created placement resource snapshot names diff (-got +want):\n%s", diff)) + } + }) + + It("should persist the placement resource snapshots as expected", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "ClusterPlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + testCases := []struct { + name string + snapshotName string + wantLabels map[string]string + wantIdentifiers []placementv1alpha1.ObjectReference + }{ + { + // Only the primary snapshot carries the count label. + name: "primary", + snapshotName: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + wantLabels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "2", + }, + wantIdentifiers: deploymentIdentifiers(deploymentNames[:3]), + }, + { + name: "secondary", + snapshotName: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + wantLabels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "1", + }, + wantIdentifiers: deploymentIdentifiers(deploymentNames[3:]), + }, + } + for _, tc := range testCases { + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: tc.snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the %s placement resource snapshot", tc.name) + + if diff := cmp.Diff(gotSnapshot.Labels, tc.wantLabels); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot labels diff (-got +want):\n%s", tc.name, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot annotations diff (-got +want):\n%s", tc.name, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot owner references diff (-got +want):\n%s", tc.name, diff)) + } + + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, tc.wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot resource identifiers diff (-got +want):\n%s", tc.name, diff)) + } + } + }) + + It("can create more Deployments", func() { + deployments := make([]*appsv1.Deployment, 0, len(extraDeploymentNames)) + for _, name := range extraDeploymentNames { + deployment := newDeployment(name, appNamespaceName) + deployment.Labels = map[string]string{"placement-group": "foo"} + Expect(hubClient.Create(ctx, deployment)).To(Succeed(), "Failed to create the Deployment") + deployments = append(deployments, deployment) + } + waitForDeploymentsInCache(deployments...) + }) + + It("should not create new snapshots if snapshots already exist", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + // The selected resources have changed since the snapshots were taken. + Expect(isUpToDate).To(BeFalse(), "The existing placement resource snapshots are unexpectedly up-to-date") + + // The existing snapshots (at index 0) are returned, ordered by their sub-indices. + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + } + gotNames := make([]string, 0, len(gotSnapshots)) + for _, snapshot := range gotSnapshots { + gotNames = append(gotNames, snapshot.GetName()) + } + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot names diff (-got +want):\n%s", diff)) + } + + // No snapshots should have been created at the next index. + nextSnapshotNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1, 1), + } + for _, name := range nextSnapshotNames { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Name: name}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + Expect(errors.IsNotFound(err)).To(BeTrue(), "Placement resource snapshot %s was created unexpectedly: %v", name, err) + } + }) + + // verifySnapshotsOfDeployments verifies that the given snapshots, created at the given index, are + // the primary and secondary placement resource snapshots that cover the given Deployments, and returns + // the snapshots as persisted. + // + // The resources are spread across the snapshots in groups of three, in the order of their names. + verifySnapshotsOfDeployments := func( + snapshotIdx int, + gotSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, + wantHash string, + wantDeploymentNames []string, + ) []*placementv1alpha1.ClusterPlacementResourceSnapshot { + wantSnapshotCnt := (len(wantDeploymentNames) + 2) / 3 + Expect(gotSnapshots).To(HaveLen(wantSnapshotCnt), "Unexpected number of placement resource snapshots") + + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "ClusterPlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + + persistedSnapshots := make([]*placementv1alpha1.ClusterPlacementResourceSnapshot, len(gotSnapshots)) + for subIdx := range gotSnapshots { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, snapshotIdx) + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: fmt.Sprintf("%d", snapshotIdx), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: fmt.Sprintf("%d", subIdx), + } + if subIdx == 0 { + // Only the primary snapshot carries the count label. + wantLabels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = fmt.Sprintf("%d", wantSnapshotCnt) + } else { + snapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, snapshotIdx, subIdx) + } + Expect(gotSnapshots[subIdx].GetName()).To(Equal(snapshotName), "Unexpected name of the placement resource snapshot at sub-index %d", subIdx) + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot at sub-index %d", subIdx) + persistedSnapshots[subIdx] = gotSnapshot + + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) labels diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) annotations diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) owner references diff (-got +want):\n%s", subIdx, diff)) + } + + groupStart := subIdx * 3 + groupEnd := min(groupStart+3, len(wantDeploymentNames)) + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, deploymentIdentifiers(wantDeploymentNames[groupStart:groupEnd])); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) resource identifiers diff (-got +want):\n%s", subIdx, diff)) + } + } + return persistedSnapshots + } + + It("should create new snapshots if the existing ones are stale", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after adding Deployments") + wantHash = newHash + + // Ten resources, with at most three resources per snapshot: one primary and three secondary snapshots. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshots are not up-to-date") + verifySnapshotsOfDeployments(1, gotSnapshots, wantHash, allDeploymentNames) + }) + + It("can update the last Deployment", func() { + lastDeploymentName := extraDeploymentNames[len(extraDeploymentNames)-1] + deployment := &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: lastDeploymentName}, deployment)).To(Succeed(), + "Failed to get the Deployment") + deployment.Spec.Replicas = ptr.To(int32(3)) + deployment.Spec.Template.Spec.Containers[0].Image = "nginx:1.28" + Expect(hubClient.Update(ctx, deployment)).To(Succeed(), "Failed to update the Deployment") + waitForDeploymentsInCache(deployment) + }) + + It("should create new snapshots again if the existing ones are stale", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after updating the Deployment") + wantHash = newHash + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshots are not up-to-date") + persistedSnapshots := verifySnapshotsOfDeployments(2, gotSnapshots, wantHash, allDeploymentNames) + + // The last Deployment lives in the last secondary snapshot; it should carry the updated content. + lastSnapshot := persistedSnapshots[len(persistedSnapshots)-1] + Expect(lastSnapshot.Spec.Resources).To(HaveLen(1), "Unexpected number of resources in the last secondary snapshot") + gotDeployment := &appsv1.Deployment{} + Expect(json.Unmarshal(lastSnapshot.Spec.Resources[0].Manifest.Raw, gotDeployment)).To(Succeed(), "Failed to unmarshal the Deployment manifest") + Expect(gotDeployment.Spec.Replicas).To(Equal(ptr.To(int32(3))), "Unexpected replica count in the snapshotted Deployment") + Expect(gotDeployment.Spec.Template.Spec.Containers[0].Image).To(Equal("nginx:1.28"), "Unexpected image in the snapshotted Deployment") + }) + + It("can delete some Deployments", func() { + deletedIdxs := []int{2, 5, 9} + deployments := make([]*appsv1.Deployment, 0, len(deletedIdxs)) + for _, idx := range deletedIdxs { + deployment := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: allDeploymentNames[idx], Namespace: appNamespaceName}} + Expect(hubClient.Delete(ctx, deployment)).To(Succeed(), "Failed to delete the Deployment") + deployments = append(deployments, deployment) + } + waitForDeploymentsRemovedFromCache(deployments...) + }) + + It("should create new snapshots if some selected resources have been removed", func() { + // Seven resources are left, with at most three resources per snapshot: one primary and two secondary snapshots. + var wantDeploymentNames []string + for idx, name := range allDeploymentNames { + if idx != 2 && idx != 5 && idx != 9 { + wantDeploymentNames = append(wantDeploymentNames, name) + } + } + + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after deleting the Deployments") + wantHash = newHash + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshots are not up-to-date") + verifySnapshotsOfDeployments(3, gotSnapshots, wantHash, wantDeploymentNames) + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; remove all the objects explicitly. + objs := []client.Object{ + &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + }}, + &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + }}, + placementPolicy, + } + var snapshotNames []string + for snapshotIdx := 1; snapshotIdx <= 3; snapshotIdx++ { + snapshotNames = append(snapshotNames, uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, snapshotIdx)) + for subIdx := 1; subIdx <= 3; subIdx++ { + snapshotNames = append(snapshotNames, uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, snapshotIdx, subIdx)) + } + } + for _, name := range snapshotNames { + objs = append(objs, &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{Name: name}}) + } + for _, name := range allDeploymentNames { + objs = append(objs, &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: appNamespaceName}}) + } + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) + + Context("primary and secondary placement resource snapshots (resources too large) (label-based selection)", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + // The name sorts after that of the first config map. + secondConfigMapName := configMapName + "-2" + + var placementPolicy *placementv1alpha1.ClusterPlacementPolicy + var wantHash string + + BeforeAll(func() { + // Create a config map that is larger than the per-snapshot size limit of the manager under test. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + Labels: map[string]string{"placement-group": "foo"}, + }, + Data: map[string]string{"key": strings.Repeat("a", 15000)}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + // Create a placement policy that selects the config map by its label. + placementPolicy = &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"placement-group": "foo"}, + }, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("should return a user error when a selected resource is too large", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).To(HaveOccurred(), "Snapshotting resources succeeded unexpectedly") + Expect(gotSnapshots).To(BeEmpty(), "Placement resource snapshots are returned unexpectedly") + Expect(isUpToDate).To(BeFalse(), "The up-to-date flag is set unexpectedly") + + // The error category is the first key-value pair in the error attributes. + gotErrAttrs := kferrors.Args(err) + Expect(gotErrAttrs).ToNot(BeEmpty(), "The error is not a KubeFleet error: %v", err) + wantErrCategoryAttrs := []interface{}{"errCategory", kferrors.ErrCategoryUser} + if diff := cmp.Diff(gotErrAttrs[:2], wantErrCategoryAttrs); diff != "" { + Fail(fmt.Sprintf("Error category attributes diff (-got +want):\n%s\nerror: %v", diff, err)) + } + }) + + It("should not create any placement resource snapshot", func() { + snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} + Expect(hubUncachedReader.List(ctx, snapshotList, client.MatchingLabels{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + })).To(Succeed(), "Failed to list placement resource snapshots") + Expect(snapshotList.Items).To(BeEmpty(), "Placement resource snapshots are created unexpectedly") + }) + + It("can resize the config map and create another config map", func() { + // Each config map now fits in a snapshot, but the two of them together do not. + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = strings.Repeat("a", 7500) + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + + secondConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: secondConfigMapName, + Namespace: appNamespaceName, + Labels: map[string]string{"placement-group": "foo"}, + }, + Data: map[string]string{"key": strings.Repeat("b", 7500)}, + } + Expect(hubClient.Create(ctx, secondConfigMap)).To(Succeed(), "Failed to create the second config map") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(resources).To(HaveLen(2), "Unexpected number of selected resources") + wantHash = hash + }) + + It("can create a primary and a secondary placement resource snapshot", func() { + createdSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The created placement resource snapshots are not up-to-date") + Expect(createdSnapshots).To(HaveLen(2), "Unexpected number of created placement resource snapshots") + + // The snapshots are ordered by their sub-indices. + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + } + gotNames := []string{createdSnapshots[0].GetName(), createdSnapshots[1].GetName()} + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Created placement resource snapshot names diff (-got +want):\n%s", diff)) + } + }) + + It("should persist the placement resource snapshots as expected", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "ClusterPlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + + testCases := []struct { + name string + snapshotName string + wantLabels map[string]string + wantConfigMap string + wantConfigData string + }{ + { + // Only the primary snapshot carries the count label. + name: "primary", + snapshotName: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + wantLabels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "2", + }, + wantConfigMap: configMapName, + wantConfigData: strings.Repeat("a", 7500), + }, + { + name: "secondary", + snapshotName: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + wantLabels: map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "1", + }, + wantConfigMap: secondConfigMapName, + wantConfigData: strings.Repeat("b", 7500), + }, + } + for _, tc := range testCases { + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: tc.snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the %s placement resource snapshot", tc.name) + + if diff := cmp.Diff(gotSnapshot.Labels, tc.wantLabels); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot labels diff (-got +want):\n%s", tc.name, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot annotations diff (-got +want):\n%s", tc.name, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot owner references diff (-got +want):\n%s", tc.name, diff)) + } + + // Each snapshot holds exactly one of the two config maps. + Expect(gotSnapshot.Spec.Resources).To(HaveLen(1), "Unexpected number of resources in the %s placement resource snapshot", tc.name) + wantIdentifier := placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: tc.wantConfigMap, + APIVersion: "v1", + Kind: "ConfigMap", + } + if diff := cmp.Diff(gotSnapshot.Spec.Resources[0].Identifier, wantIdentifier); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot resource identifier diff (-got +want):\n%s", tc.name, diff)) + } + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[0].Manifest.Raw, gotConfigMap)).To(Succeed(), + "Failed to unmarshal the config map manifest in the %s placement resource snapshot", tc.name) + if diff := cmp.Diff(gotConfigMap.Data, map[string]string{"key": tc.wantConfigData}); diff != "" { + Fail(fmt.Sprintf("The %s placement resource snapshot config map data diff (-got +want):\n%s", tc.name, diff)) + } + } + }) + + It("can shrink the config maps", func() { + // The two config maps now fit in a single snapshot together. + for _, name := range []string{configMapName, secondConfigMapName} { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: name}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = strings.Repeat("c", 4000) + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + } + }) + + It("should create a primary placement resource snapshot only if the existing snapshots are stale", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after shrinking the config maps") + wantHash = newHash + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of created placement resource snapshots") + wantName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + Expect(gotSnapshots[0].GetName()).To(Equal(wantName), "Unexpected name of the new placement resource snapshot") + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: wantName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the new placement resource snapshot") + + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "1", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot labels diff (-got +want):\n%s", diff)) + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot annotations diff (-got +want):\n%s", diff)) + } + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "ClusterPlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot owner references diff (-got +want):\n%s", diff)) + } + + // Both config maps live in the primary snapshot, with the updated content. + Expect(gotSnapshot.Spec.Resources).To(HaveLen(2), "Unexpected number of resources in the new placement resource snapshot") + for idx, name := range []string{configMapName, secondConfigMapName} { + wantIdentifier := placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: name, + APIVersion: "v1", + Kind: "ConfigMap", + } + if diff := cmp.Diff(gotSnapshot.Spec.Resources[idx].Identifier, wantIdentifier); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot resource identifier (index %d) diff (-got +want):\n%s", idx, diff)) + } + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[idx].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + if diff := cmp.Diff(gotConfigMap.Data, map[string]string{"key": strings.Repeat("c", 4000)}); diff != "" { + Fail(fmt.Sprintf("The new placement resource snapshot config map data (index %d) diff (-got +want):\n%s", idx, diff)) + } + } + + // No secondary snapshot should have been created at the new index. + secondaryName := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1, 1) + err = hubUncachedReader.Get(ctx, types.NamespacedName{Name: secondaryName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + Expect(errors.IsNotFound(err)).To(BeTrue(), "A secondary placement resource snapshot was created unexpectedly: %v", err) + }) + + AfterAll(func() { + objs := []client.Object{ + &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + }}, + &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + }}, + &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1), + }}, + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: secondConfigMapName, Namespace: appNamespaceName}}, + } + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) + + Context("partial placement resource snapshot provisioning", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + // The names are in ascending order, which is also the order in which the selected resources are + // snapshotted. + configMapNames := make([]string, 10) + for idx := range configMapNames { + configMapNames[idx] = fmt.Sprintf("%s-%d", configMapNamePrefix, idx) + } + + var placementPolicy *placementv1alpha1.ClusterPlacementPolicy + var wantHash string + // The UIDs of the secondary placement resource snapshots (by sub-index minus one), recorded for + // later verification. + var secondarySnapshotUIDs []types.UID + + // verifyOrphanedSecondariesCleanedUp verifies that the manager cleans up the orphaned secondary placement + // resource snapshots (the primary snapshot is missing) and returns a transient error. + // + // The manager treats the secondary snapshots left without a primary snapshot as orphans: it deletes them + // and returns a transient error, asking the caller to requeue before creating the snapshots anew. + verifyOrphanedSecondariesCleanedUp := func() { + // Other errors (e.g., those caused by a stale cache) might surface first; retry until the expected + // transient error is observed. + Eventually(func() error { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + if err == nil { + return fmt.Errorf("snapshotting resources succeeded unexpectedly (snapshots %d, up-to-date %t)", len(gotSnapshots), isUpToDate) + } + if !strings.Contains(err.Error(), "cleaned up orphaned secondary placement resource snapshots") { + return fmt.Errorf("unexpected error message: %w", err) + } + if len(gotSnapshots) != 0 || isUpToDate { + return fmt.Errorf("unexpected return values alongside the error (snapshots %d, up-to-date %t)", len(gotSnapshots), isUpToDate) + } + + // The error category is the first key-value pair in the error attributes. + gotErrAttrs := kferrors.Args(err) + if len(gotErrAttrs) < 2 { + return fmt.Errorf("the error is not a KubeFleet error: %w", err) + } + wantErrCategoryAttrs := []interface{}{"errCategory", kferrors.ErrCategoryTransient} + if diff := cmp.Diff(gotErrAttrs[:2], wantErrCategoryAttrs); diff != "" { + return fmt.Errorf("error category attributes diff (-got +want):\n%s\nerror: %w", diff, err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to observe the transient error from the clean-up of orphaned secondary placement resource snapshots") + + // The orphaned secondary snapshots should have been deleted. + for subIdx := 1; subIdx <= 3; subIdx++ { + secondarySnapshotName := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Name: secondarySnapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove the orphaned secondary placement resource snapshot at sub-index %d", subIdx) + } + } + + BeforeAll(func() { + // Create ten config maps with the same label in the app namespace. + for idx := range configMapNames { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapNames[idx], + Namespace: appNamespaceName, + Labels: map[string]string{"placement-group": "foo"}, + }, + Data: map[string]string{"key": fmt.Sprintf("value-%d", idx)}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + } + + // Create a placement policy that selects the config maps by their label. + placementPolicy = &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"placement-group": "foo"}, + }, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(resources).To(HaveLen(len(configMapNames)), "Unexpected number of selected resources") + wantHash = hash + }) + + It("can create a primary and three secondary placement resource snapshots", func() { + // Ten resources, with at most three resources per snapshot: one primary and three secondary snapshots. + createdSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The created placement resource snapshots are not up-to-date") + Expect(createdSnapshots).To(HaveLen(4), "Unexpected number of created placement resource snapshots") + + // The snapshots are ordered by their sub-indices. + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 2), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 3), + } + gotNames := make([]string, 0, len(createdSnapshots)) + for _, snapshot := range createdSnapshots { + gotNames = append(gotNames, snapshot.GetName()) + } + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Created placement resource snapshot names diff (-got +want):\n%s", diff)) + } + + // Write down the UIDs of the secondary placement resource snapshots. + secondarySnapshotUIDs = make([]types.UID, 0, len(createdSnapshots)-1) + for _, snapshot := range createdSnapshots[1:] { + Expect(snapshot.GetUID()).ToNot(BeEmpty(), "The secondary placement resource snapshot %s has no UID", snapshot.GetName()) + secondarySnapshotUIDs = append(secondarySnapshotUIDs, snapshot.GetUID()) + } + }) + + It("should persist the placement resource snapshots as expected", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "ClusterPlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + + for subIdx := 0; subIdx < 4; subIdx++ { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: fmt.Sprintf("%d", subIdx), + } + if subIdx == 0 { + // Only the primary snapshot carries the count label. + wantLabels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = "4" + } else { + snapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx) + } + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot at sub-index %d", subIdx) + + if subIdx > 0 { + Expect(gotSnapshot.UID).To(Equal(secondarySnapshotUIDs[subIdx-1]), "Unexpected UID of the placement resource snapshot at sub-index %d", subIdx) + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) labels diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) annotations diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) owner references diff (-got +want):\n%s", subIdx, diff)) + } + + // The resources are spread across the snapshots in groups of three, in the order of their names. + groupStart := subIdx * 3 + groupEnd := min(groupStart+3, len(configMapNames)) + wantIdentifiers := make([]placementv1alpha1.ObjectReference, 0, groupEnd-groupStart) + for _, name := range configMapNames[groupStart:groupEnd] { + wantIdentifiers = append(wantIdentifiers, placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: name, + APIVersion: "v1", + Kind: "ConfigMap", + }) + } + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) resource identifiers diff (-got +want):\n%s", subIdx, diff)) + } + } + }) + + It("can delete the primary placement resource snapshot", func() { + // Note that this is for simulation purposes only: it mimics a partial provisioning, where the primary + // snapshot is missing while its secondary snapshots remain. The manager never deletes only the primary + // snapshot in normal operations, so this situation will not occur in practice. + primarySnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + }, + } + Expect(hubClient.Delete(ctx, primarySnapshot)).To(Succeed(), "Failed to delete the primary placement resource snapshot") + + // Wait until the cache has observed the deletion. + Eventually(func() bool { + return errors.IsNotFound(hubClient.Get(ctx, client.ObjectKeyFromObject(primarySnapshot), primarySnapshot)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove the primary placement resource snapshot") + }) + + It("should clean up the orphaned secondary placement resource snapshots and return a transient error", func() { + verifyOrphanedSecondariesCleanedUp() + }) + + It("can re-create the primary and secondary placement resource snapshots", func() { + // The orphaned secondary snapshots have been cleaned up; the manager can now create the snapshots anew. + // The call is retried as the cache might not have observed the clean-up yet. + var recreatedSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + var isUpToDate bool + Eventually(func() error { + var err error + recreatedSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + return err + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The re-created placement resource snapshots are not up-to-date") + + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 1), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 2), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, 3), + } + gotNames := make([]string, 0, len(recreatedSnapshots)) + for _, snapshot := range recreatedSnapshots { + gotNames = append(gotNames, snapshot.GetName()) + } + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Re-created placement resource snapshot names diff (-got +want):\n%s", diff)) + } + }) + + It("should have re-created the primary snapshot and the secondary snapshots with new UIDs", func() { + // The primary snapshot has been re-created (its UID is not checked). + primarySnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + primarySnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: primarySnapshotName}, primarySnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the primary placement resource snapshot") + Expect(primarySnapshot.Annotations).To(HaveKeyWithValue(placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey, wantHash), + "Unexpected contents hash of the re-created primary placement resource snapshot") + Expect(primarySnapshot.Labels).To(HaveKeyWithValue(placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey, "4"), + "Unexpected snapshot count of the re-created primary placement resource snapshot") + + // All the secondary snapshots have been re-created with new UIDs. + for subIdx := 1; subIdx <= 3; subIdx++ { + secondarySnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + secondarySnapshotName := uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx) + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: secondarySnapshotName}, secondarySnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the secondary placement resource snapshot at sub-index %d", subIdx) + Expect(secondarySnapshot.UID).ToNot(Equal(secondarySnapshotUIDs[subIdx-1]), + "The secondary placement resource snapshot at sub-index %d has not been re-created (UID unchanged)", subIdx) + Expect(secondarySnapshot.Annotations).To(HaveKeyWithValue(placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey, wantHash), + "Unexpected contents hash of the re-created secondary placement resource snapshot at sub-index %d", subIdx) + } + }) + + It("can delete the primary placement resource snapshot again", func() { + // Record the UIDs of the secondary placement resource snapshots that were re-created previously. + secondarySnapshotUIDs = make([]types.UID, 0, 3) + for subIdx := 1; subIdx <= 3; subIdx++ { + secondarySnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Expect(hubClient.Get(ctx, types.NamespacedName{ + Name: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx), + }, secondarySnapshot)).To(Succeed(), "Failed to get the secondary placement resource snapshot at sub-index %d", subIdx) + secondarySnapshotUIDs = append(secondarySnapshotUIDs, secondarySnapshot.UID) + } + + // Note that this is for simulation purposes only: it mimics a partial provisioning, where the primary + // snapshot is missing while its secondary snapshots remain. The manager never deletes only the primary + // snapshot in normal operations, so this situation will not occur in practice. + primarySnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{ + ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + }, + } + Expect(hubClient.Delete(ctx, primarySnapshot)).To(Succeed(), "Failed to delete the primary placement resource snapshot") + + // Wait until the cache has observed the deletion. + Eventually(func() bool { + return errors.IsNotFound(hubClient.Get(ctx, client.ObjectKeyFromObject(primarySnapshot), primarySnapshot)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove the primary placement resource snapshot") + }) + + It("can update the last config map", func() { + lastConfigMapName := configMapNames[len(configMapNames)-1] + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: lastConfigMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = "updated-value" + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + }) + + It("should clean up the orphaned secondary placement resource snapshots and return a transient error again", func() { + verifyOrphanedSecondariesCleanedUp() + }) + + It("can re-create the primary and secondary placement resource snapshots with the updated content", func() { + _, newHash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(newHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed after updating the config map") + wantHash = newHash + + // The orphaned secondary snapshots have been cleaned up; the manager can now create the snapshots anew. + // The call is retried as the cache might not have observed the clean-up yet. + var recreatedSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + var isUpToDate bool + Eventually(func() error { + var err error + recreatedSnapshots, isUpToDate, err = resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + return err + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to re-create the placement resource snapshots") + Expect(isUpToDate).To(BeTrue(), "The re-created placement resource snapshots are not up-to-date") + Expect(recreatedSnapshots).To(HaveLen(4), "Unexpected number of re-created placement resource snapshots") + }) + + It("should have re-created all the snapshots with the expected content", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "ClusterPlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicyName, + } + + for subIdx := 0; subIdx < 4; subIdx++ { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: "0", + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: fmt.Sprintf("%d", subIdx), + } + if subIdx == 0 { + // Only the primary snapshot carries the count label. + wantLabels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = "4" + } else { + snapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx) + } + + // Read the snapshot using the cached client; the cache might take a moment to catch up. + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot at sub-index %d", subIdx) + + if subIdx > 0 { + Expect(gotSnapshot.UID).ToNot(Equal(secondarySnapshotUIDs[subIdx-1]), + "The secondary placement resource snapshot at sub-index %d has not been re-created (UID unchanged)", subIdx) + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) labels diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.Annotations, wantAnnotations); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) annotations diff (-got +want):\n%s", subIdx, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) owner references diff (-got +want):\n%s", subIdx, diff)) + } + + // The resources are spread across the snapshots in groups of three, in the order of their names. + groupStart := subIdx * 3 + groupEnd := min(groupStart+3, len(configMapNames)) + wantIdentifiers := make([]placementv1alpha1.ObjectReference, 0, groupEnd-groupStart) + for _, name := range configMapNames[groupStart:groupEnd] { + wantIdentifiers = append(wantIdentifiers, placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: name, + APIVersion: "v1", + Kind: "ConfigMap", + }) + } + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(gotSnapshot.Spec.Resources)) + for idx := range gotSnapshot.Spec.Resources { + gotIdentifiers = append(gotIdentifiers, gotSnapshot.Spec.Resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) resource identifiers diff (-got +want):\n%s", subIdx, diff)) + } + + // Verify the content of each config map; only the last one has been updated. + for idx := range gotSnapshot.Spec.Resources { + configMapIdx := groupStart + idx + wantData := map[string]string{"key": fmt.Sprintf("value-%d", configMapIdx)} + if configMapIdx == len(configMapNames)-1 { + wantData["key"] = "updated-value" + } + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[idx].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + if diff := cmp.Diff(gotConfigMap.Data, wantData); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (sub-index %d) config map data (index %d) diff (-got +want):\n%s", subIdx, idx, diff)) + } + } + } + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; remove all the objects explicitly. + objs := []client.Object{ + &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0), + }}, + placementPolicy, + } + for subIdx := 1; subIdx <= 3; subIdx++ { + objs = append(objs, &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.SecondaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0, subIdx), + }}) + } + for _, name := range configMapNames { + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: appNamespaceName}}) + } + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) + + Context("GC", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + + var placementPolicy *placementv1alpha1.ClusterPlacementPolicy + + // updateConfigMapAndSnapshot updates the config map with the given data and calls the manager to snapshot + // the selected resources; it returns the snapshots created. + updateConfigMapAndSnapshot := func(data string) []placementv1alpha1.PlacementResourceSnapshotAccessor { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Data["key"] = data + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The new placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of created placement resource snapshots") + + // Wait until the cache has observed the new snapshot; otherwise the next call might not see it. + Eventually(func() error { + return hubClient.Get(ctx, client.ObjectKeyFromObject(gotSnapshots[0]), &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the new placement resource snapshot") + return gotSnapshots + } + + BeforeAll(func() { + // Create a config map in the app namespace. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + }, + Data: map[string]string{"key": "initial-value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + // Create a placement policy that selects the config map. + placementPolicy = &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementPolicyName, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + Name: configMapName, + }, + }, + }, + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create four primary placement resource snapshots", func() { + for idx := 0; idx < 4; idx++ { + gotSnapshots := updateConfigMapAndSnapshot(fmt.Sprintf("value-%d", idx)) + wantName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, idx) + Expect(gotSnapshots[0].GetName()).To(Equal(wantName), "Unexpected name of the placement resource snapshot at index %d", idx) + } + }) + + It("should have four primary placement resource snapshots at the expected indexes", func() { + wantOwnerRefs := []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: "ClusterPlacementPolicy", + Name: placementPolicyName, + UID: placementPolicy.UID, + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } + + seenHashes := map[string]bool{} + for idx := 0; idx < 4; idx++ { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, idx) + gotSnapshot := &placementv1alpha1.ClusterPlacementResourceSnapshot{} + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Name: snapshotName}, gotSnapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the placement resource snapshot at index %d", idx) + + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: fmt.Sprintf("%d", idx), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0", + placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey: "1", + } + if diff := cmp.Diff(gotSnapshot.Labels, wantLabels); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (index %d) labels diff (-got +want):\n%s", idx, diff)) + } + if diff := cmp.Diff(gotSnapshot.OwnerReferences, wantOwnerRefs); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (index %d) owner references diff (-got +want):\n%s", idx, diff)) + } + + // Each snapshot tracks a different state of the config map. + hash := gotSnapshot.Annotations[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] + Expect(hash).ToNot(BeEmpty(), "The placement resource snapshot (index %d) has no contents hash", idx) + Expect(seenHashes).ToNot(HaveKey(hash), "The placement resource snapshot (index %d) has the same contents hash as another snapshot", idx) + seenHashes[hash] = true + + Expect(gotSnapshot.Spec.Resources).To(HaveLen(1), "Unexpected number of resources in the placement resource snapshot (index %d)", idx) + gotConfigMap := &corev1.ConfigMap{} + Expect(json.Unmarshal(gotSnapshot.Spec.Resources[0].Manifest.Raw, gotConfigMap)).To(Succeed(), "Failed to unmarshal the config map manifest") + if diff := cmp.Diff(gotConfigMap.Data, map[string]string{"key": fmt.Sprintf("value-%d", idx)}); diff != "" { + Fail(fmt.Sprintf("The placement resource snapshot (index %d) config map data diff (-got +want):\n%s", idx, diff)) + } + } + + // There should be no other snapshots. + snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} + Expect(hubUncachedReader.List(ctx, snapshotList, client.MatchingLabels{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyName, + })).To(Succeed(), "Failed to list placement resource snapshots") + Expect(snapshotList.Items).To(HaveLen(4), "Unexpected number of placement resource snapshots") + }) + + It("should garbage collect the first primary placement resource snapshot", func() { + updateConfigMapAndSnapshot("value-4") + + firstSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 0) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Name: firstSnapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The first primary placement resource snapshot has not been garbage collected") + }) + + It("should garbage collect the second primary placement resource snapshot", func() { + updateConfigMapAndSnapshot("value-5") + + secondSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 1) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Name: secondSnapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The second primary placement resource snapshot has not been garbage collected") + }) + + It("can create a placement binding that uses the third primary placement resource snapshot", func() { + thirdSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 2) + placementBinding := &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementBindingName, + OwnerReferences: []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: placementv1alpha1.ClusterPlacementPolicyKind, + Name: placementPolicyName, + UID: placementPolicy.UID, + }, + }, + }, + Spec: placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: placementPolicyName, + ClusterName: "member-1", + ResourceSnapshotName: thirdSnapshotName, + }, + } + Expect(hubClient.Create(ctx, placementBinding)).To(Succeed(), "Failed to create the placement binding") + + // Wait until the cache (and its field index) has observed the placement binding; the garbage collection + // process relies on it to tell if a snapshot is still in use. + wantFieldVal := fmt.Sprintf( + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, placementPolicyName, thirdSnapshotName) + Eventually(func() (int, error) { + placementBindingList := &placementv1alpha1.ClusterPlacementBindingList{} + err := hubClient.List(ctx, placementBindingList, client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName: wantFieldVal, + }) + return len(placementBindingList.Items), err + }, eventuallyDuration, eventuallyInterval).Should(Equal(1), "Failed to find the placement binding in the cache") + }) + + It("should not garbage collect the third primary placement resource snapshot as it is in use", func() { + // The third snapshot is now the oldest one that is past the revision history limit; the garbage + // collection process should skip it as a placement binding is using it. + updateConfigMapAndSnapshot("value-6") + + thirdSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 2) + Consistently(func() error { + return hubUncachedReader.Get(ctx, types.NamespacedName{Name: thirdSnapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The third primary placement resource snapshot has been garbage collected unexpectedly") + }) + + It("should garbage collect the fourth primary placement resource snapshot but keep the third one", func() { + // The third and the fourth snapshots are now both past the revision history limit; the third one is + // still in use by a placement binding, so only the fourth one should be garbage collected. + updateConfigMapAndSnapshot("value-7") + + fourthSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 3) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Name: fourthSnapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The fourth primary placement resource snapshot has not been garbage collected") + + // The garbage collection requests are processed in order; the request for the third snapshot has been + // handled by now. + thirdSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 2) + Consistently(func() error { + return hubUncachedReader.Get(ctx, types.NamespacedName{Name: thirdSnapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The third primary placement resource snapshot has been garbage collected unexpectedly") + }) + + It("can delete the placement binding", func() { + placementBinding := &placementv1alpha1.ClusterPlacementBinding{ + ObjectMeta: metav1.ObjectMeta{ + Name: placementBindingName, + }, + } + Expect(hubClient.Delete(ctx, placementBinding)).To(Succeed(), "Failed to delete the placement binding") + + // Wait until the cache (and its field index) has observed the deletion; the garbage collection process + // relies on it to tell if a snapshot is still in use. + thirdSnapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, 2) + Eventually(func() (int, error) { + placementBindingList := &placementv1alpha1.ClusterPlacementBindingList{} + err := hubClient.List(ctx, placementBindingList, client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, placementPolicyName, thirdSnapshotName), + }) + return len(placementBindingList.Items), err + }, eventuallyDuration, eventuallyInterval).Should(Equal(0), "The placement binding is still found in the cache") + }) + + It("should garbage collect the third and the fifth primary placement resource snapshots", func() { + // The third snapshot is no longer in use; both the third and the fifth snapshots are now past the + // revision history limit. + updateConfigMapAndSnapshot("value-8") + + for _, idx := range []int{2, 4} { + snapshotName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, idx) + Eventually(func() bool { + err := hubUncachedReader.Get(ctx, types.NamespacedName{Name: snapshotName}, &placementv1alpha1.ClusterPlacementResourceSnapshot{}) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The primary placement resource snapshot at index %d has not been garbage collected", idx) + } + }) + + AfterAll(func() { + // The envtest environment does not run the garbage collector; remove all the objects explicitly. + objs := []client.Object{ + &placementv1alpha1.ClusterPlacementBinding{ObjectMeta: metav1.ObjectMeta{Name: placementBindingName}}, + placementPolicy, + } + for idx := 0; idx < 9; idx++ { + objs = append(objs, &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{ + Name: uniquename.PrimaryPlacementResourceSnapshotName(placementPolicyName, placementPolicy.UID, idx), + }}) + } + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed(), "Failed to delete object %T", obj) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T", obj) + } + }) + }) +}) + +var _ = Describe("GC variants", func() { + Context("revision history limit changes", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + placementPolicy.Spec.ResourceRevisionHistoryLimit = ptr.To(int32(5)) + + // primarySnapshotNames returns the names of the primary snapshots at indices [from, to]. + primarySnapshotNames := func(from, to int) []string { + names := make([]string, 0, to-from+1) + for idx := from; idx <= to; idx++ { + names = append(names, uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, idx)) + } + return names + } + // createSnapshot updates the selected config map, and calls the manager to create a snapshot at the given index. + createSnapshot := func(idx int) { + updateConfigMapData(appNamespaceName, configMapName, fmt.Sprintf("value-%d", idx)) + snapshotAndVerify(resourceSnapshotManager, placementPolicy, primarySnapshotNames(idx, idx)) + } + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + }, + Data: map[string]string{"key": "initial-value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create six snapshots", func() { + for idx := 0; idx < 6; idx++ { + createSnapshot(idx) + } + // No GC request has been enqueued yet, as each call has seen at most five snapshots. + Expect(checkOwnedSnapshots(placementPolicy, primarySnapshotNames(0, 5))).To(Succeed()) + }) + + It("should keep five snapshots after the next retrieval", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, primarySnapshotNames(5, 5)) + Eventually(func() error { + return checkOwnedSnapshots(placementPolicy, primarySnapshotNames(1, 5)) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect the oldest snapshot") + }) + + It("can lower the revision history limit to 1", func() { + updateRevisionHistoryLimit(placementPolicy, 1) + }) + + It("should garbage collect all but the latest snapshot at once", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, primarySnapshotNames(5, 5)) + Eventually(func() error { + return checkOwnedSnapshots(placementPolicy, primarySnapshotNames(5, 5)) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect all but the latest snapshot") + }) + + It("can raise the revision history limit to 3", func() { + updateRevisionHistoryLimit(placementPolicy, 3) + }) + + It("should keep the new snapshots up to the raised limit", func() { + for idx := 6; idx < 9; idx++ { + createSnapshot(idx) + } + // The limit is a soft one; right after a snapshot is created, there might be one more snapshot than + // the limit allows. + Expect(checkOwnedSnapshots(placementPolicy, primarySnapshotNames(5, 8))).To(Succeed()) + }) + + It("should trim the history to the raised limit at the next retrieval", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, primarySnapshotNames(8, 8)) + Eventually(func() error { + return checkOwnedSnapshots(placementPolicy, primarySnapshotNames(6, 8)) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to trim the history to the raised limit") + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 9) + objs = append(objs, placementPolicy, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + deleteObjectsAndWait(objs) + }) + }) + + Context("multi-group snapshots", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + resourceSelector := configMapLabelSelector(groupLabels) + resourceSelector.Namespace = appNamespaceName + placementPolicy := newClusterPlacementPolicy(resourceSelector) + placementPolicy.Spec.ResourceRevisionHistoryLimit = ptr.To(int32(1)) + + // The snapshot name depends on the UID of the placement policy; it is set once the policy is created. + var extraSecondarySnapshotName string + + BeforeAll(func() { + // Seven config maps make three groups, as each snapshot can hold at most three resources. + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 7, groupLabels) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + extraSecondarySnapshotName = uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 1, 2) + }) + + It("can create snapshots in three groups", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, snapshotSetNames(placementPolicy, 0, 3)) + }) + + It("can deselect three config maps", func() { + setConfigMapLabels(configMapNames[4:], nil) + }) + + It("should create snapshots in two groups", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, snapshotSetNames(placementPolicy, 1, 2)) + // No GC request has been enqueued, as the call has seen only one snapshot. + Expect(checkOwnedSnapshots(placementPolicy, slices.Concat( + snapshotSetNames(placementPolicy, 0, 3), + snapshotSetNames(placementPolicy, 1, 2), + ))).To(Succeed()) + }) + + It("can add an extra secondary snapshot at index 1", func() { + // The extra secondary snapshot mimics one left behind by an interrupted snapshot creation attempt; it has + // the same labels, annotations, and owner reference as the ones that the manager creates. + primarySnapshot := getPrimarySnapshotFromCache(placementPolicy, 1) + extraSecondarySnapshot, err := secondaryPlacementResourceSnapshot("", extraSecondarySnapshotName, placementPolicy, 1, 2, + nil, snapshotContentsHash(primarySnapshot), hubClient.Scheme()) + Expect(err).ToNot(HaveOccurred(), "Failed to build the extra secondary snapshot") + Expect(hubClient.Create(ctx, extraSecondarySnapshot)).To(Succeed(), "Failed to create the extra secondary snapshot") + getSnapshotFromCache("", extraSecondarySnapshotName) + }) + + It("can deselect three more config maps", func() { + setConfigMapLabels(configMapNames[1:4], nil) + }) + + It("should garbage collect the primary and both secondary snapshots at index 0", func() { + // This call enqueues the snapshots at index 0 for GC. + snapshotAndVerify(resourceSnapshotManager, placementPolicy, snapshotSetNames(placementPolicy, 2, 1)) + wantNames := slices.Concat( + snapshotSetNames(placementPolicy, 1, 2), + []string{extraSecondarySnapshotName}, + snapshotSetNames(placementPolicy, 2, 1), + ) + Eventually(func() error { + return checkOwnedSnapshots(placementPolicy, wantNames) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect the snapshots at index 0") + }) + + It("can select all config maps again", func() { + setConfigMapLabels(configMapNames[1:], groupLabels) + }) + + It("should garbage collect the snapshots at index 1, including the extra secondary snapshot", func() { + // This call enqueues the snapshots at index 1 for GC; all of them should be removed, regardless of the + // count label on the primary snapshot. + snapshotAndVerify(resourceSnapshotManager, placementPolicy, snapshotSetNames(placementPolicy, 3, 3)) + wantNames := slices.Concat( + snapshotSetNames(placementPolicy, 2, 1), + snapshotSetNames(placementPolicy, 3, 3), + ) + Eventually(func() error { + return checkOwnedSnapshots(placementPolicy, wantNames) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect the snapshots at index 1") + }) + + It("should garbage collect the single-group snapshot at index 2", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, snapshotSetNames(placementPolicy, 3, 3)) + Eventually(func() error { + return checkOwnedSnapshots(placementPolicy, snapshotSetNames(placementPolicy, 3, 3)) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect the snapshot at index 2") + }) + + AfterAll(func() { + var objs []client.Object + for idx := 0; idx < 4; idx++ { + for _, name := range snapshotSetNames(placementPolicy, idx, 3) { + objs = append(objs, placementResourceSnapshotObj("", name)) + } + } + objs = append(objs, placementResourceSnapshotObj("", extraSecondarySnapshotName), placementPolicy) + objs = append(objs, configMapObjs(configMapNames)...) + deleteObjectsAndWait(objs) + }) + }) + + Context("GC requests", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + // No revision history limit is set; the API server defaults it to 3. + placementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + + var pausedMgr *Manager + var primarySnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + + // The snapshot names depend on the UID of the placement policy; they are set once the policy is created. + var latestSnapshotNames []string + + // gcRequestsAt returns the GC requests for the primary snapshots at the given indices. + gcRequestsAt := func(idxs ...int) []snapshotGarbageCollectionRequest { + reqs := make([]snapshotGarbageCollectionRequest, 0, len(idxs)) + for _, idx := range idxs { + reqs = append(reqs, gcRequestFor(placementPolicy, primarySnapshots[idx])) + } + return reqs + } + // createSnapshot updates the selected config map, and calls the paused manager to create a snapshot at + // the given index. + createSnapshot := func(idx int) { + updateConfigMapData(appNamespaceName, configMapName, fmt.Sprintf("value-%d", idx)) + snapshotAndVerify(pausedMgr, placementPolicy, []string{uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, idx)}) + primarySnapshots = append(primarySnapshots, getPrimarySnapshotFromCache(placementPolicy, idx)) + } + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + }, + Data: map[string]string{"key": "initial-value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + latestSnapshotNames = []string{uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 4)} + pausedMgr = newPausedManager() + }) + + It("should default the revision history limit to 3", func() { + Expect(placementPolicy.Spec.ResourceRevisionHistoryLimit).To(Equal(ptr.To(int32(3))), "Unexpected default revision history limit") + }) + + It("should reject revision history limits out of range", func() { + for _, limit := range []int32{0, -1, 21} { + placementPolicyCopy := placementPolicy.DeepCopy() + placementPolicyCopy.Spec.ResourceRevisionHistoryLimit = ptr.To(limit) + err := hubClient.Update(ctx, placementPolicyCopy) + Expect(errors.IsInvalid(err)).To(BeTrue(), "Update with revision history limit %d = %v, want an invalid error", limit, err) + Expect(err.Error()).To(ContainSubstring(revisionHistoryLimitFieldPath), "Unexpected error message") + } + }) + + It("should not enqueue GC requests within the limit", func() { + for idx := 0; idx < 4; idx++ { + createSnapshot(idx) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + } + }) + + It("should enqueue the oldest snapshot once the limit is exceeded", func() { + // The GC requests are computed from the snapshots seen before the new snapshot is created. + createSnapshot(4) + verifyGCRequests(drainGCRequests(pausedMgr), gcRequestsAt(0)) + }) + + It("should not pile up duplicate GC requests", func() { + for idx := 0; idx < 3; idx++ { + snapshotAndVerify(pausedMgr, placementPolicy, latestSnapshotNames) + Expect(pausedMgr.gcwq.Len()).To(Equal(2), "Unexpected number of GC requests in the workqueue") + } + verifyGCRequests(drainGCRequests(pausedMgr), gcRequestsAt(0, 1)) + }) + + inMemoryLimitTestCases := []struct { + name string + limit *int32 + wantIdxs []int + }{ + { + name: "unset (the default of 3 applies)", + wantIdxs: []int{0, 1}, + }, + { + name: "0 (treated as 1)", + limit: ptr.To(int32(0)), + wantIdxs: []int{0, 1, 2, 3}, + }, + { + name: "-1 (treated as 1)", + limit: ptr.To(int32(-1)), + wantIdxs: []int{0, 1, 2, 3}, + }, + { + name: "1", + limit: ptr.To(int32(1)), + wantIdxs: []int{0, 1, 2, 3}, + }, + { + name: "4", + limit: ptr.To(int32(4)), + wantIdxs: []int{0}, + }, + { + name: "5", + limit: ptr.To(int32(5)), + }, + { + name: "20", + limit: ptr.To(int32(20)), + }, + } + for _, tc := range inMemoryLimitTestCases { + It(fmt.Sprintf("should honor an in-memory revision history limit of %s", tc.name), func() { + snapshotAndVerify(pausedMgr, withInMemoryRevisionHistoryLimit(placementPolicy, tc.limit), latestSnapshotNames) + verifyGCRequests(drainGCRequests(pausedMgr), gcRequestsAt(tc.wantIdxs...)) + }) + } + + It("should skip snapshots that are already gone", func() { + Expect(hubClient.Delete(ctx, primarySnapshots[0])).To(Succeed(), "Failed to delete the snapshot at index 0") + waitForSnapshotGoneFromCache(appNamespaceName, primarySnapshots[0].GetName()) + + Expect(pausedMgr.garbageCollect(ctx, gcRequestFor(placementPolicy, primarySnapshots[0]))).To(Succeed(), + "Failed to process the GC request for a snapshot that is already gone") + snapshotAndVerify(pausedMgr, placementPolicy, latestSnapshotNames) + verifyGCRequests(drainGCRequests(pausedMgr), gcRequestsAt(1)) + }) + + AfterAll(func() { + if pausedMgr != nil { + pausedMgr.gcwq.ShutDown() + } + objs := primarySnapshotObjs(placementPolicy, 5) + objs = append(objs, placementPolicy, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + deleteObjectsAndWait(objs) + }) + }) + + Context("snapshots pending deletion", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + var configMapNames []string + + placementPolicy := newPlacementPolicy(appNamespaceName, configMapLabelSelector(groupLabels)) + placementPolicy.Spec.ResourceRevisionHistoryLimit = ptr.To(int32(1)) + + var pausedMgr *Manager + var gcRequestForSnapshot0 snapshotGarbageCollectionRequest + + primarySnapshotName := func(idx int) string { + return uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, idx) + } + secondarySnapshotName := func(idx int) string { + return uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, idx, 1) + } + + BeforeAll(func() { + // Four config maps make two groups, as each snapshot can hold at most three resources. + configMapNames = createLabeledConfigMaps(configMapNamePrefix, 4, groupLabels) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + pausedMgr = newPausedManager() + }) + + It("can create snapshots at index 0 and 1", func() { + snapshotAndVerify(pausedMgr, placementPolicy, snapshotSetNames(placementPolicy, 0, 2)) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + + addSnapshotFinalizer(appNamespaceName, primarySnapshotName(0)) + gcRequestForSnapshot0 = gcRequestFor(placementPolicy, getPrimarySnapshotFromCache(placementPolicy, 0)) + + updateConfigMapData(appNamespaceName, configMapNames[0], "changed-1") + snapshotAndVerify(pausedMgr, placementPolicy, snapshotSetNames(placementPolicy, 1, 2)) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + It("can delete the primary snapshot at index 0 out of band", func() { + markSnapshotForDeletion(appNamespaceName, primarySnapshotName(0)) + }) + + It("should not enqueue a primary snapshot pending deletion", func() { + snapshotAndVerify(pausedMgr, placementPolicy, snapshotSetNames(placementPolicy, 1, 2)) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + It("should skip a primary snapshot pending deletion", func() { + Expect(pausedMgr.garbageCollect(ctx, gcRequestForSnapshot0)).To(Succeed(), "Failed to process the GC request") + + Expect(checkOwnedSnapshots(placementPolicy, slices.Concat( + snapshotSetNames(placementPolicy, 0, 2), + snapshotSetNames(placementPolicy, 1, 2), + ))).To(Succeed()) + Expect(verifySnapshotExists(appNamespaceName, primarySnapshotName(0)).GetDeletionTimestamp()).ToNot(BeNil(), + "The primary snapshot at index 0 is not pending deletion") + Expect(verifySnapshotExists(appNamespaceName, secondarySnapshotName(0)).GetDeletionTimestamp()).To(BeNil(), + "The secondary snapshot at index 0 has been deleted unexpectedly") + }) + + It("should leave the secondary snapshot at index 0 behind once its primary snapshot is gone", func() { + // This is expected: the GC process is keyed on primary snapshots, and the primary snapshot at index 0 + // has been deleted out of band, which does not happen in normal ops. + removeFinalizersAndDelete(placementResourceSnapshotObj(appNamespaceName, primarySnapshotName(0))) + waitForSnapshotGoneFromCache(appNamespaceName, primarySnapshotName(0)) + + snapshotAndVerify(pausedMgr, placementPolicy, snapshotSetNames(placementPolicy, 1, 2)) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + Expect(checkOwnedSnapshots(placementPolicy, slices.Concat( + []string{secondarySnapshotName(0)}, + snapshotSetNames(placementPolicy, 1, 2), + ))).To(Succeed()) + }) + + It("can create snapshots at index 2 while a secondary snapshot at index 1 has a finalizer", func() { + addSnapshotFinalizer(appNamespaceName, secondarySnapshotName(1)) + + updateConfigMapData(appNamespaceName, configMapNames[0], "changed-2") + snapshotAndVerify(pausedMgr, placementPolicy, snapshotSetNames(placementPolicy, 2, 2)) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + It("should delete the primary snapshot even when a secondary snapshot is pending deletion", func() { + snapshotAndVerify(pausedMgr, placementPolicy, snapshotSetNames(placementPolicy, 2, 2)) + gotReqs := drainGCRequests(pausedMgr) + verifyGCRequests(gotReqs, []snapshotGarbageCollectionRequest{ + gcRequestFor(placementPolicy, getPrimarySnapshotFromCache(placementPolicy, 1)), + }) + + Expect(pausedMgr.garbageCollect(ctx, gotReqs[0])).To(Succeed(), "Failed to process the GC request") + verifySnapshotNotFound(appNamespaceName, primarySnapshotName(1)) + Expect(verifySnapshotExists(appNamespaceName, secondarySnapshotName(1)).GetDeletionTimestamp()).ToNot(BeNil(), + "The secondary snapshot at index 1 is not pending deletion") + Expect(checkOwnedSnapshots(placementPolicy, slices.Concat( + []string{secondarySnapshotName(0), secondarySnapshotName(1)}, + snapshotSetNames(placementPolicy, 2, 2), + ))).To(Succeed()) + }) + + It("should finish the deletion once the finalizer is removed", func() { + removeFinalizersAndDelete(placementResourceSnapshotObj(appNamespaceName, secondarySnapshotName(1))) + waitForSnapshotGoneFromCache(appNamespaceName, primarySnapshotName(1)) + + snapshotAndVerify(pausedMgr, placementPolicy, snapshotSetNames(placementPolicy, 2, 2)) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + Expect(checkOwnedSnapshots(placementPolicy, slices.Concat( + []string{secondarySnapshotName(0)}, + snapshotSetNames(placementPolicy, 2, 2), + ))).To(Succeed()) + }) + + It("should not use a latest primary snapshot that is pending deletion", func() { + addSnapshotFinalizer(appNamespaceName, primarySnapshotName(2)) + markSnapshotForDeletion(appNamespaceName, primarySnapshotName(2)) + + gotSnapshots, isUpToDate, err := pausedMgr.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, latestSnapshotPendingDeletionErrMsg) + gotSnapshots, isUpToDate, err = pausedMgr.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, latestSnapshotPendingDeletionErrMsg) + + // A resource change does not lead to a new snapshot either. + updateConfigMapData(appNamespaceName, configMapNames[0], "changed-3") + gotSnapshots, isUpToDate, err = pausedMgr.SnapshotResourcesIfStale(ctx, placementPolicy) + verifySnapshotCallFailed(gotSnapshots, isUpToDate, err, kferrors.ErrCategoryTransient, latestSnapshotPendingDeletionErrMsg) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 3) + verifyGCRequests(drainGCRequests(pausedMgr), nil) + }) + + AfterAll(func() { + if pausedMgr != nil { + pausedMgr.gcwq.ShutDown() + } + for _, name := range []string{primarySnapshotName(0), primarySnapshotName(1), secondarySnapshotName(1), primarySnapshotName(2)} { + removeFinalizersAndDelete(placementResourceSnapshotObj(appNamespaceName, name)) + } + objs := []client.Object{ + placementResourceSnapshotObj(appNamespaceName, secondarySnapshotName(0)), + placementResourceSnapshotObj(appNamespaceName, secondarySnapshotName(2)), + placementPolicy, + } + objs = append(objs, configMapObjs(configMapNames)...) + deleteObjectsAndWait(objs) + }) + }) + + Context("snapshots in use by placement bindings", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + orphanedPlacementBindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + placementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + placementPolicy.Spec.ResourceRevisionHistoryLimit = ptr.To(int32(1)) + + var pausedMgr *Manager + + primarySnapshotName := func(idx int) string { + return uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, idx) + } + // verifySnapshotsKeptWithoutRetries verifies that the placement policy keeps the wanted snapshots, i.e., the + // manager under test does not garbage collect the snapshot of the given GC request, and that the GC request + // is dropped rather than retried. + // + // The GC request must have been enqueued before this helper is called. + verifySnapshotsKeptWithoutRetries := func(gcRequest snapshotGarbageCollectionRequest, wantNames []string) { + GinkgoHelper() + + // Wait until the GC worker has picked up the GC request; otherwise the checks below would also pass with + // a GC request that has not been processed at all. A retry, should one happen, comes after the request + // is picked up, and the Consistently block below catches it. + Eventually(func() int { + return resourceSnapshotManager.gcwq.Len() + }, eventuallyDuration, eventuallyInterval).Should(BeZero(), "The GC request has not been picked up by the GC worker") + + Consistently(func() error { + if err := checkOwnedSnapshots(placementPolicy, wantNames); err != nil { + return err + } + if requeueCnt := resourceSnapshotManager.gcwq.NumRequeues(gcRequest); requeueCnt != 0 { + return fmt.Errorf("the GC request has been retried %d time(s)", requeueCnt) + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Failed to keep the snapshot in use without retries") + } + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "initial-value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + pausedMgr = newPausedManager() + }) + + It("can create a snapshot at index 0, and a placement binding that uses it", func() { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{primarySnapshotName(0)}) + createPlacementBinding(placementPolicy, placementBindingName, primarySnapshotName(0), ptr.To(placementPolicy.UID)) + waitForPlacementBindingCntInCache(placementPolicy, primarySnapshotName(0), 1) + }) + + It("can create a snapshot at index 1", func() { + updateConfigMapData(appNamespaceName, configMapName, "changed-1") + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{primarySnapshotName(1)}) + }) + + It("should keep the snapshot at index 0 while the placement binding uses it", func() { + gcRequestForSnapshot0 := gcRequestFor(placementPolicy, getPrimarySnapshotFromCache(placementPolicy, 0)) + // This call enqueues the snapshot at index 0 for GC. + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{primarySnapshotName(1)}) + verifySnapshotsKeptWithoutRetries(gcRequestForSnapshot0, []string{primarySnapshotName(0), primarySnapshotName(1)}) + + // A GC attempt on a snapshot in use succeeds without deleting anything. + Expect(pausedMgr.garbageCollect(ctx, gcRequestForSnapshot0)).To(Succeed(), "Failed to process the GC request") + Expect(checkOwnedSnapshots(placementPolicy, []string{primarySnapshotName(0), primarySnapshotName(1)})).To(Succeed()) + }) + + It("can point the placement binding to the snapshot at index 1", func() { + updatePlacementBindingSnapshot(placementPolicy, placementBindingName, primarySnapshotName(1)) + waitForPlacementBindingCntInCache(placementPolicy, primarySnapshotName(0), 0) + }) + + It("should garbage collect the snapshot at index 0 at the next retrieval", func() { + // The GC request dropped earlier is enqueued again by this call. + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{primarySnapshotName(1)}) + Eventually(func() error { + return checkOwnedSnapshots(placementPolicy, []string{primarySnapshotName(1)}) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect the snapshot at index 0") + }) + + It("can create a placement binding without an owner reference that uses the snapshot at index 1", func() { + createPlacementBinding(placementPolicy, orphanedPlacementBindingName, primarySnapshotName(1), nil) + waitForPlacementBindingCntInCache(placementPolicy, primarySnapshotName(1), 2) + }) + + It("can create a snapshot at index 2, and point the owned placement binding to it", func() { + updateConfigMapData(appNamespaceName, configMapName, "changed-2") + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{primarySnapshotName(2)}) + updatePlacementBindingSnapshot(placementPolicy, placementBindingName, primarySnapshotName(2)) + waitForPlacementBindingCntInCache(placementPolicy, primarySnapshotName(1), 1) + }) + + It("should keep the snapshot at index 1 while the placement binding without an owner reference uses it", func() { + // The owners of placement bindings are not checked; any placement binding that names the placement policy + // and the snapshot keeps the snapshot in use. + gcRequestForSnapshot1 := gcRequestFor(placementPolicy, getPrimarySnapshotFromCache(placementPolicy, 1)) + // This call enqueues the snapshot at index 1 for GC. + snapshotAndVerify(resourceSnapshotManager, placementPolicy, []string{primarySnapshotName(2)}) + verifySnapshotsKeptWithoutRetries(gcRequestForSnapshot1, []string{primarySnapshotName(1), primarySnapshotName(2)}) + }) + + AfterAll(func() { + if pausedMgr != nil { + pausedMgr.gcwq.ShutDown() + } + objs := slices.Concat( + []client.Object{ + placementBindingObj(appNamespaceName, placementBindingName), + placementBindingObj(appNamespaceName, orphanedPlacementBindingName), + }, + primarySnapshotObjs(placementPolicy, 3), + []client.Object{ + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + }, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("same-named placement policies in different namespaces and scopes", Ordered, func() { + placementPolicyName := fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()) + otherNamespaceName := fmt.Sprintf(namespaceNameTemplate, utils.RandStr()) + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + placementBindingName := fmt.Sprintf(placementBindingNameTemplate, utils.RandStr()) + + appPlacementPolicy := newPlacementPolicy(appNamespaceName, configMapNameSelector(configMapName)) + appPlacementPolicy.Name = placementPolicyName + appPlacementPolicy.Spec.ResourceRevisionHistoryLimit = ptr.To(int32(1)) + + otherPlacementPolicy := newPlacementPolicy(otherNamespaceName, configMapNameSelector(configMapName)) + otherPlacementPolicy.Name = placementPolicyName + otherPlacementPolicy.Spec.ResourceRevisionHistoryLimit = ptr.To(int32(1)) + + resourceSelector := configMapNameSelector(configMapName) + resourceSelector.Namespace = otherNamespaceName + clusterPlacementPolicy := newClusterPlacementPolicy(resourceSelector) + clusterPlacementPolicy.Name = placementPolicyName + clusterPlacementPolicy.Spec.ResourceRevisionHistoryLimit = ptr.To(int32(1)) + + placementPolicies := []placementv1alpha1.PlacementPolicyAccessor{appPlacementPolicy, otherPlacementPolicy, clusterPlacementPolicy} + + // The placement policies share the same name, but their snapshot names differ, as the names factor in the + // UIDs of the placement policies. + primarySnapshotNames := func(placementPolicy placementv1alpha1.PlacementPolicyAccessor, idx int) []string { + return []string{uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), idx)} + } + + BeforeAll(func() { + createNamespace(otherNamespaceName) + for _, namespace := range []string{appNamespaceName, otherNamespaceName} { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: namespace, + }, + Data: map[string]string{"key": "initial-value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map in namespace %s", namespace) + } + for _, placementPolicy := range placementPolicies { + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create placement policy %s/%s", + placementPolicy.GetNamespace(), placementPolicy.GetName()) + } + }) + + It("can create snapshots for all three placement policies", func() { + for _, placementPolicy := range placementPolicies { + snapshotAndVerify(resourceSnapshotManager, placementPolicy, primarySnapshotNames(placementPolicy, 0)) + } + }) + + It("can create a placement binding in the other namespace that matches the in-use lookup of the app namespace snapshot", func() { + // The placement binding refers to the snapshot name of the placement policy in the app namespace, so + // that only the namespace sets it apart from a placement binding that uses that snapshot. + appSnapshotName := primarySnapshotNames(appPlacementPolicy, 0)[0] + createPlacementBinding(otherPlacementPolicy, placementBindingName, appSnapshotName, ptr.To(otherPlacementPolicy.UID)) + waitForPlacementBindingCntInCache(otherPlacementPolicy, appSnapshotName, 1) + // The same lookup in the app namespace finds no placement binding. + waitForPlacementBindingCntInCache(appPlacementPolicy, appSnapshotName, 0) + }) + + It("can update the config map in the app namespace", func() { + updateConfigMapData(appNamespaceName, configMapName, "changed") + }) + + It("should garbage collect only the old snapshot of the placement policy in the app namespace", func() { + snapshotAndVerify(resourceSnapshotManager, appPlacementPolicy, primarySnapshotNames(appPlacementPolicy, 1)) + // This call enqueues the snapshot at index 0 for GC. + snapshotAndVerify(resourceSnapshotManager, appPlacementPolicy, primarySnapshotNames(appPlacementPolicy, 1)) + + // The snapshot would be kept (with the GC request being retried) if the in-use check found the placement + // binding in the other namespace. + Eventually(func() error { + return checkOwnedSnapshots(appPlacementPolicy, primarySnapshotNames(appPlacementPolicy, 1)) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect the snapshot at index 0") + verifySnapshotExists(otherNamespaceName, primarySnapshotNames(otherPlacementPolicy, 0)[0]) + verifySnapshotExists("", primarySnapshotNames(clusterPlacementPolicy, 0)[0]) + }) + + It("can update the config map in the other namespace", func() { + updateConfigMapData(otherNamespaceName, configMapName, "changed") + }) + + It("should garbage collect only the old snapshot of the cluster placement policy", func() { + snapshotAndVerify(resourceSnapshotManager, clusterPlacementPolicy, primarySnapshotNames(clusterPlacementPolicy, 1)) + // This call enqueues the snapshot at index 0 for GC. + snapshotAndVerify(resourceSnapshotManager, clusterPlacementPolicy, primarySnapshotNames(clusterPlacementPolicy, 1)) + + Eventually(func() error { + return checkOwnedSnapshots(clusterPlacementPolicy, primarySnapshotNames(clusterPlacementPolicy, 1)) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to garbage collect the snapshot at index 0") + // The placement policy in the other namespace has not been called since its first snapshot. + Expect(checkOwnedSnapshots(otherPlacementPolicy, primarySnapshotNames(otherPlacementPolicy, 0))).To(Succeed()) + Expect(checkOwnedSnapshots(appPlacementPolicy, primarySnapshotNames(appPlacementPolicy, 1))).To(Succeed()) + }) + + AfterAll(func() { + var objs []client.Object + for _, placementPolicy := range placementPolicies { + objs = append(objs, primarySnapshotObjs(placementPolicy, 2)...) + } + objs = append(objs, placementBindingObj(otherNamespaceName, placementBindingName)) + for _, placementPolicy := range placementPolicies { + objs = append(objs, placementPolicy) + } + for _, namespace := range []string{appNamespaceName, otherNamespaceName} { + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: namespace}}) + } + deleteObjectsAndWait(objs) + }) + }) +}) diff --git a/pkg/v1/managers/placementresourcesnapshot/quorumread.go b/pkg/v1/managers/placementresourcesnapshot/quorumread.go index 5d8206637..bfda08580 100644 --- a/pkg/v1/managers/placementresourcesnapshot/quorumread.go +++ b/pkg/v1/managers/placementresourcesnapshot/quorumread.go @@ -25,13 +25,14 @@ import ( placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" errors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" ) func (m *Manager) primaryPlacementResourceSnapshotExistsAtIdx( ctx context.Context, placementPolicy placementv1alpha1.PlacementPolicyAccessor, idx int) (bool, error) { - name := uniqueNameForPrimaryPlacementResourceSnapshot(placementPolicy.GetName(), idx) + name := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), idx) namespace := placementPolicy.GetNamespace() kind := placementv1alpha1.ClusterPlacementResourceSnapshotKind diff --git a/pkg/v1/managers/placementresourcesnapshot/resources.go b/pkg/v1/managers/placementresourcesnapshot/resources.go index 1a0bd1fc6..3a3d9abc0 100644 --- a/pkg/v1/managers/placementresourcesnapshot/resources.go +++ b/pkg/v1/managers/placementresourcesnapshot/resources.go @@ -21,7 +21,9 @@ import ( "fmt" "sort" + batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" @@ -32,6 +34,7 @@ import ( "k8s.io/kubectl/pkg/util/deployment" placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" errors "go.goms.io/fleet/pkg/utils/errors" hasher "go.goms.io/fleet/pkg/utils/resource" ) @@ -44,8 +47,14 @@ const ( // resource snapshot to be ~1.2 MiB, or ~1.26 MB, which should be safe in most cases. Note that the padding // space is not just reserved for safety reasons, but also to accommodate the additional fields in // the placement resource snapshot object, such as metadata and labels. - maxPerSnapshotResourceDataSizeBytes = 1258291 // 1.2 MiB, or ~1.26 MB. - maxPerSnapshotResourceCnt = 50 + DefaultMaxPerSnapshotResourceDataSizeBytes = 1258291 // 1.2 MiB, or ~1.26 MB. + DefaultMaxPerSnapshotResourceCnt = 50 + + // The minimum and maximum limits for the size and count of resources in a single placement resource snapshot. + MinPerSnapshotResourceDataSizeBytes = 2000 // 2 KB. + MinPerSnapshotResourceCnt = 2 + MaxPerSnapshotResourceDataSizeBytes = 1468000 // 1.4 MiB, or ~1.47 MB. + MaxPerSnapshotResourceCnt = 100 ) const ( @@ -81,10 +90,44 @@ func (m *Manager) retrieveAndHashSelectedResources( // called at all if there are no resource selectors), hence the unexpected error returned here. return nil, "", errors.NewUnexpectedError(nil, "no resource selectors are present") } + if placementPolicyNSName := placementPolicyAccessor.GetNamespace(); placementPolicyNSName != "" { + // Check if the placement policy is in a namespace eligible for placement. + if m.resourceEligibilityChecker != nil && !m.resourceEligibilityChecker.AllowsNamespace(placementPolicyNSName) { + return nil, "", errors.NewUserError(nil, "placement policy is in a namespace that is ineligible for placement", + "placementPolicyNamespace", placementPolicyNSName) + } + } for idx := range placementPolicySpec.ResourceSelectors { selector := placementPolicySpec.ResourceSelectors[idx] + // Check if the resource selector selects object(s) from a namespace eligible for placement. + // + // Note that if the resource selector selects namespaces, the check will be done later at the retrieval step. + namespaceInSelector := selector.Namespace + if placementPolicyNamespace := placementPolicyAccessor.GetNamespace(); placementPolicyNamespace != "" { + if namespaceInSelector != "" && namespaceInSelector != placementPolicyNamespace { + return nil, "", errors.NewUserError(nil, "resource selector specifies an invalid namespace (a placement policy can only select resources within its own namespace)", + "resourceSelectorIndex", idx, "invalidNamespace", namespaceInSelector) + } + } else { + if m.resourceEligibilityChecker != nil && !m.resourceEligibilityChecker.AllowsNamespace(namespaceInSelector) { + return nil, "", errors.NewUserError(nil, "resource selector specifies a namespace that is ineligible for placement", + "resourceSelectorIndex", idx, "ineligibleNamespace", namespaceInSelector) + } + } + + // Check if the resource selector specifies a GVK eligible for placement. + gvkInSelector := schema.GroupVersionKind{ + Group: selector.APIGroup, + Version: selector.APIVersion, + Kind: selector.Kind, + } + if m.resourceEligibilityChecker != nil && !m.resourceEligibilityChecker.AllowsGVK(gvkInSelector) { + return nil, "", errors.NewUserError(nil, "resource selector specifies a GVK that is ineligible for placement", + "resourceSelectorIndex", idx, "ineligibleGVK", gvkInSelector) + } + resourcesAsSelected := make([]*placementv1alpha1.SnapshottedResource, 0, 5) switch { case len(selector.Name) != 0: @@ -158,15 +201,26 @@ func (m *Manager) retrieveResourceByName( } else { obj, err = m.hubDynamicInformerManager.Lister(gvr).ByNamespace(namespace).Get(resourceSelector.Name) } - if err != nil { - return placementv1alpha1.SnapshottedResource{}, errors.NewAPIServerError(err, "failed to get selected resource", true, + switch { + case apierrors.IsNotFound(err): + // The cache might be stale (e.g., the resource has just been created); confirm with a read from the + // API server directly before reporting the resource as absent. + klog.V(2).InfoS("The selected resource is not found in the cache; retrieving the resource directly from the API server", "gvr", gvr, "namespace", namespace, "name", resourceSelector.Name) - } - var ok bool - resource, ok = obj.(*unstructured.Unstructured) - if !ok { - return placementv1alpha1.SnapshottedResource{}, errors.NewUnexpectedError(nil, "failed to convert the retrieved resource to unstructured", + resource, err = m.retrieveResourceByNameWithNoCache(ctx, gvr, namespace, resourceSelector.Name) + if err != nil { + return placementv1alpha1.SnapshottedResource{}, err + } + case err != nil: + return placementv1alpha1.SnapshottedResource{}, errors.NewAPIServerError(err, "failed to get selected resource", true, "gvr", gvr, "namespace", namespace, "name", resourceSelector.Name) + default: + var ok bool + resource, ok = obj.(*unstructured.Unstructured) + if !ok { + return placementv1alpha1.SnapshottedResource{}, errors.NewUnexpectedError(nil, "failed to convert the retrieved resource to unstructured", + "gvr", gvr, "namespace", namespace, "name", resourceSelector.Name) + } } } else { // No informer is set up for the selected resource, or the informer has not been synced yet. @@ -174,13 +228,36 @@ func (m *Manager) retrieveResourceByName( // As a fallback, retrieve the resource directly from the API server. klog.V(2).InfoS("Informer for the selected resource is not set up or not synced; retrieving the resource directly from the API server", "gvr", gvr) - if namespace == "" { - resource, err = m.hubDynamicClient.Resource(gvr).Get(ctx, resourceSelector.Name, metav1.GetOptions{}) - } else { - resource, err = m.hubDynamicClient.Resource(gvr).Namespace(namespace).Get(ctx, resourceSelector.Name, metav1.GetOptions{}) + resource, err = m.retrieveResourceByNameWithNoCache(ctx, gvr, namespace, resourceSelector.Name) + if err != nil { + return placementv1alpha1.SnapshottedResource{}, err } + } + + // Check if the selected resource is eligible for placement. + // + // Note that here the resource snapshot manager will throw an error if the resource is not eligible for placement + // rather than silently ignoring it, as the selection is explicitly defined. + if m.resourceEligibilityChecker != nil { + // As a special case, if the selected resource is a namespace itself, check if the namespace is eligible for + // placement. + if gvk.Group == "" && gvk.Kind == utils.NamespaceKind { + if !m.resourceEligibilityChecker.AllowsNamespace(resource.GetName()) { + return placementv1alpha1.SnapshottedResource{}, errors.NewUserError(nil, + "selected namespace is not eligible for placement", + "gvr", gvr, "namespace", namespace, "name", resourceSelector.Name) + } + } + + // Check if the resource itself is eligible for placement. + isEligible, reason, err := m.resourceEligibilityChecker.AllowsObject(resource) if err != nil { - return placementv1alpha1.SnapshottedResource{}, errors.NewAPIServerError(err, "failed to get selected resource directly from the API server", false, + return placementv1alpha1.SnapshottedResource{}, errors.Wraps(err, "failed to check if the selected resource is eligible for placement", + "gvr", gvr, "namespace", namespace, "name", resourceSelector.Name) + } + if !isEligible { + return placementv1alpha1.SnapshottedResource{}, errors.NewUserError(nil, + fmt.Sprintf("selected resource is not eligible for placement (%s)", reason), "gvr", gvr, "namespace", namespace, "name", resourceSelector.Name) } } @@ -194,6 +271,31 @@ func (m *Manager) retrieveResourceByName( return snapshottedResource, nil } +// retrieveResourceByNameWithNoCache retrieves a resource selected by name directly from the API server, bypassing the +// cache. As the read is authoritative, a not found error is reported as a user error (wrapping the original error). +func (m *Manager) retrieveResourceByNameWithNoCache( + ctx context.Context, + gvr schema.GroupVersionResource, + namespace, name string, +) (*unstructured.Unstructured, error) { + var resource *unstructured.Unstructured + var err error + if namespace == "" { + resource, err = m.hubDynamicClient.Resource(gvr).Get(ctx, name, metav1.GetOptions{}) + } else { + resource, err = m.hubDynamicClient.Resource(gvr).Namespace(namespace).Get(ctx, name, metav1.GetOptions{}) + } + switch { + case apierrors.IsNotFound(err): + return nil, errors.NewUserError(err, "selected resource does not exist", + "gvr", gvr, "namespace", namespace, "name", name) + case err != nil: + return nil, errors.NewAPIServerError(err, "failed to get selected resource directly from the API server", false, + "gvr", gvr, "namespace", namespace, "name", name) + } + return resource, nil +} + func (m *Manager) retrieveResourcesByLabelSelector( ctx context.Context, placementPolicyAccessor placementv1alpha1.PlacementPolicyAccessor, @@ -260,14 +362,44 @@ func (m *Manager) retrieveResourcesByLabelSelector( } } - snapshottedResources := make([]placementv1alpha1.SnapshottedResource, len(resources)) + snapshottedResources := make([]placementv1alpha1.SnapshottedResource, 0, len(resources)) for idx := range resources { resource := resources[idx] - snapshottedResources[idx], err = snapshotResource(resource) + + // Check if the resource is eligible for placement. + // + // Note that here the resource snapshot manager will skip a resource if it is not eligible for placement + // rather than returning an error to the caller, as a label selector may match resources that the user + // did not explicitly pick (e.g., the built-in CA certificate config map in every namespace). + if m.resourceEligibilityChecker != nil { + // As a special case, if the selected resource is a namespace itself, check if the namespace is eligible for + // placement. + if gvk.Group == "" && gvk.Kind == utils.NamespaceKind { + if !m.resourceEligibilityChecker.AllowsNamespace(resource.GetName()) { + klog.V(2).InfoS("Skipping a namespace that is not eligible for placement", + "gvr", gvr, "resource", klog.KObj(resource)) + continue + } + } + + isEligible, reason, err := m.resourceEligibilityChecker.AllowsObject(resource) + if err != nil { + return nil, errors.Wraps(err, "failed to check if a selected resource is eligible for placement", + "gvr", gvr, "namespace", namespace, "name", resource.GetName()) + } + if !isEligible { + klog.V(2).InfoS("Skipping a selected resource that is not eligible for placement", + "gvr", gvr, "resource", klog.KObj(resource), "reason", reason) + continue + } + } + + snapshottedResource, err := snapshotResource(resource) if err != nil { return nil, errors.Wraps(err, "failed to snapshot selected resource", "gvr", gvr, "namespace", namespace, "name", resource.GetName()) } + snapshottedResources = append(snapshottedResources, snapshottedResource) } return snapshottedResources, nil } @@ -283,6 +415,9 @@ func (m *Manager) lookUpGVKGVRAndNamespace(resourceSelector placementv1alpha1.Re // Convert the GVK to a GVR using the REST mapper. mapping, err := m.restMapper.RESTMapping(gvk.GroupKind(), gvk.Version) if err != nil { + if meta.IsNoMatchError(err) { + return schema.GroupVersionKind{}, schema.GroupVersionResource{}, "", errors.NewUserError(err, "no match found for GVK; the API might not be available on the cluster", "gvk", gvk) + } return schema.GroupVersionKind{}, schema.GroupVersionResource{}, "", errors.NewUnexpectedError(err, "failed to map GVK to GVR", "gvk", gvk) } gvr := mapping.Resource @@ -304,6 +439,11 @@ func (m *Manager) lookUpGVKGVRAndNamespace(resourceSelector placementv1alpha1.Re errors.NewUserError(nil, "cluster-scoped resource cannot be selected by a placement policy; use cluster placement policy instead", "resourceSelector", resourceSelector) } + // When selecting a namespace-scoped resource, a namespace must be specified. + if scope == meta.RESTScopeNameNamespace && namespace == "" { + return schema.GroupVersionKind{}, schema.GroupVersionResource{}, "", + errors.NewUserError(nil, "a namespace must be specified for namespace-scoped resources", "resourceSelector", resourceSelector) + } // Check if the resolved scope matches with the resource selector, i.e., when selecting a cluster-scoped resource, no namespace can // be specified. if scope == meta.RESTScopeNameRoot && namespace != "" { @@ -347,10 +487,50 @@ func snapshotResource(resource *unstructured.Unstructured) (placementv1alpha1.Sn resourceCopy.SetResourceVersion("") resourceCopy.SetSelfLink("") resourceCopy.SetUID("") + resourceCopy.SetFinalizers(nil) // Remove the status field. unstructured.RemoveNestedField(resourceCopy.Object, "status") + // Apply resource-specific transformations as needed. + switch resourceCopy.GroupVersionKind() { + case corev1.SchemeGroupVersion.WithKind(utils.ServiceKind): + // Remove the cluster IP(s) allocated by the hub cluster, unless the Service is headless. + if clusterIP, exist, _ := unstructured.NestedString(resourceCopy.Object, "spec", "clusterIP"); exist && clusterIP != corev1.ClusterIPNone { + unstructured.RemoveNestedField(resourceCopy.Object, "spec", "clusterIP") + unstructured.RemoveNestedField(resourceCopy.Object, "spec", "clusterIPs") + } + // Remove all the node ports allocated by the hub cluster, if any. + unstructured.RemoveNestedField(resourceCopy.Object, "spec", "healthCheckNodePort") + + vals, found, err := unstructured.NestedFieldNoCopy(resourceCopy.Object, "spec", "ports") + if err != nil { + return placementv1alpha1.SnapshottedResource{}, errors.NewUnexpectedError(err, "failed to get the ports field in the Service object", + "resource", klog.KObj(resourceCopy)) + } + if found { + if ports, ok := vals.([]any); ok { + for i := range ports { + if port, ok := ports[i].(map[string]any); ok { + delete(port, "nodePort") + } + } + } + } + case batchv1.SchemeGroupVersion.WithKind(utils.JobKind): + if manualSelector, exist, _ := unstructured.NestedBool(resourceCopy.Object, "spec", "manualSelector"); !exist || !manualSelector { + // Remove the selector field and labels added by the API server if the Job is not created with a manual selector, + // as their values would conflict with the ones generated by the member cluster API server. + // https://github.com/kubernetes/kubernetes/blob/d4fde1e92a83cb533ae63b3abe9d49f08efb7a2f/pkg/registry/batch/job/strategy.go#L219 + // Kubernetes used to add a label called "controller-uid", but has switched to "batch.kubernetes.io/controller-uid" since 1.27. + unstructured.RemoveNestedField(resourceCopy.Object, "spec", "selector", "matchLabels", "controller-uid") + unstructured.RemoveNestedField(resourceCopy.Object, "spec", "selector", "matchLabels", "batch.kubernetes.io/controller-uid") + unstructured.RemoveNestedField(resourceCopy.Object, "spec", "template", "metadata", "creationTimestamp") + unstructured.RemoveNestedField(resourceCopy.Object, "spec", "template", "metadata", "labels", "controller-uid") + unstructured.RemoveNestedField(resourceCopy.Object, "spec", "template", "metadata", "labels", "batch.kubernetes.io/controller-uid") + } + } + resourceCopyRawData, err := resourceCopy.MarshalJSON() if err != nil { return placementv1alpha1.SnapshottedResource{}, errors.NewUnexpectedError(err, "failed to marshal the resource copy to JSON", @@ -380,7 +560,7 @@ func resourceUniqueId(resource *placementv1alpha1.SnapshottedResource) string { resource.Identifier.Name) } -func splitResourcesIntoSizeControlledGroups(resources []placementv1alpha1.SnapshottedResource) ([][]placementv1alpha1.SnapshottedResource, error) { +func (m *Manager) splitResourcesIntoSizeControlledGroups(resources []placementv1alpha1.SnapshottedResource) ([][]placementv1alpha1.SnapshottedResource, error) { if len(resources) == 0 { // Return one single empty group. return [][]placementv1alpha1.SnapshottedResource{{}}, nil @@ -398,16 +578,16 @@ func splitResourcesIntoSizeControlledGroups(resources []placementv1alpha1.Snapsh resourceSize += len(info) } - if resourceSize > maxPerSnapshotResourceDataSizeBytes { + if resourceSize > m.maxPerSnapshotResourceDataSizeBytes { // A single resource exceeds the per-snapshot size limit; it can never fit into any group. return nil, errors.NewUserError(nil, "a single selected resource is too large to fit in a placement resource snapshot", "resource", resource.Identifier, - "resourceSizeBytes", resourceSize, "maxPerSnapshotResourceDataSizeBytes", maxPerSnapshotResourceDataSizeBytes) + "resourceSizeBytes", resourceSize, "maxPerSnapshotResourceDataSizeBytes", m.maxPerSnapshotResourceDataSizeBytes) } // Start a new group if adding this resource would exceed either the size or the count limit. if len(currentGroup) > 0 && - (currentSize+resourceSize > maxPerSnapshotResourceDataSizeBytes || len(currentGroup) >= maxPerSnapshotResourceCnt) { + (currentSize+resourceSize > m.maxPerSnapshotResourceDataSizeBytes || len(currentGroup) >= m.maxPerSnapshotResourceCnt) { groups = append(groups, currentGroup) currentGroup = nil currentSize = 0 diff --git a/pkg/v1/managers/placementresourcesnapshot/resources_integration_test.go b/pkg/v1/managers/placementresourcesnapshot/resources_integration_test.go new file mode 100644 index 000000000..15688be8c --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/resources_integration_test.go @@ -0,0 +1,2417 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementresourcesnapshot + +import ( + "encoding/json" + "fmt" + "slices" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + autoscalingv1 "k8s.io/api/autoscaling/v1" + autoscalingv2 "k8s.io/api/autoscaling/v2" + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/types" + deploymentutil "k8s.io/kubectl/pkg/util/deployment" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" +) + +const ( + namespaceNameTemplate = "ns-%s" + fleetNamespaceNameTemplate = "fleet-%s" + serviceAccountNameTemplate = "serviceaccount-%s" + secretNameTemplate = "secret-%s" + hpaNameTemplate = "hpa-%s" + clusterRoleNameTemplate = "clusterrole-%s" + serviceNameTemplate = "service-%s" + jobNameTemplate = "job-%s" + + placementGroupLabelKey = "placement-group" + customAnnotationKey = "custom-annotation" + + // The legacy controller UID label, which the API server still adds to the pod template of a Job (without a + // manual selector) alongside batchv1.ControllerUidLabel. + legacyJobControllerUIDLabel = "controller-uid" + + // The default namespace and the Kubernetes API service within it are created by the API server itself, + // including the one that the envtest package runs. + defaultNamespaceName = "default" + kubernetesAPIServiceName = "kubernetes" + + testFinalizer = "test.kubefleet.dev/block-deletion" + anotherTestFinalizer = "test.kubefleet.dev/another" + + // The objects below are created automatically by the Kubernetes control plane in a real cluster; the + // environment prepared by the envtest package does not run the controllers that create them, so the + // test cases create them by hand. + kubeRootCACertConfigMapName = "kube-root-ca.crt" + defaultServiceAccountName = "default" + + ineligibleResourceErrMsg = "selected resource is not eligible for placement" + ineligibleNamespaceErrMsg = "selected namespace is not eligible for placement" + ineligibleSelectorNamespaceErrMsg = "resource selector specifies a namespace that is ineligible for placement" + ineligibleGVKErrMsg = "resource selector specifies a GVK that is ineligible for placement" + invalidLabelSelectorErrMsg = "invalid label selector" + noMatchForGVKErrMsg = "no match found for GVK" + resourceNotFoundErrMsg = "selected resource does not exist" +) + +var _ = Describe("resource selection (placement policies)", func() { + Context("invalid resource selectors", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + // The names of resources that do not exist; the errors are expected to occur before any retrieval. + resourceName := utils.RandStr() + + selfSelectingPlacementPolicy := newPlacementPolicy(appNamespaceName) + selfSelectingPlacementPolicy.Spec.ResourceSelectors = []placementv1alpha1.ResourceSelector{ + { + APIGroup: placementv1alpha1.GroupVersion.Group, + APIVersion: placementv1alpha1.GroupVersion.Version, + Kind: placementv1alpha1.PlacementPolicyKind, + Name: selfSelectingPlacementPolicy.Name, + }, + } + + testCases := []snapshotErrTestCase{ + { + desc: "the resource selector specifies a namespace other than that of the placement policy", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: fmt.Sprintf(namespaceNameTemplate, utils.RandStr()), + Name: configMapName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"resource selector specifies an invalid namespace"}, + }, + { + desc: "the placement policy is in a reserved Kubernetes namespace", + placementPolicy: newPlacementPolicy("kube-public", placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMapName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"placement policy is in a namespace that is ineligible for placement"}, + }, + { + desc: "the resource selector selects a cluster-scoped resource", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIGroup: rbacv1.GroupName, + APIVersion: "v1", + Kind: "ClusterRole", + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"cluster-scoped resource cannot be selected by a placement policy"}, + }, + { + desc: "the resource selector selects pods", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Pod", + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleGVKErrMsg}, + }, + { + desc: "the resource selector selects leases", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIGroup: "coordination.k8s.io", + APIVersion: "v1", + Kind: "Lease", + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleGVKErrMsg}, + }, + { + desc: "the resource selector selects the placement policy itself", + placementPolicy: selfSelectingPlacementPolicy, + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleGVKErrMsg}, + }, + { + desc: "a valid resource selector is followed by one that selects an ineligible resource type", + placementPolicy: newPlacementPolicy(appNamespaceName, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMapName, + }, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Pod", + Name: resourceName, + }, + ), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleGVKErrMsg}, + wantResourceSelectorIdx: ptr.To(1), + }, + { + // The CRD only enforces that the two fields are mutually exclusive. + desc: "the resource selector specifies neither a name nor a label selector", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"neither name nor label selector is specified"}, + }, + { + desc: "the label selector uses the In operator without any value", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{ + MatchExpressions: []metav1.LabelSelectorRequirement{ + {Key: placementGroupLabelKey, Operator: metav1.LabelSelectorOpIn}, + }, + }, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"invalid label selector", "values set can't be empty"}, + }, + { + desc: "the label selector uses an unknown operator", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{ + MatchExpressions: []metav1.LabelSelectorRequirement{ + {Key: placementGroupLabelKey, Operator: "Foo", Values: []string{"bar"}}, + }, + }, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"invalid label selector", "is not a valid label selector operator"}, + }, + { + desc: "the resource selector specifies an unknown kind", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMapp", + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{noMatchForGVKErrMsg, `no matches for kind "ConfigMapp" in version "v1"`}, + }, + { + desc: "the resource selector specifies an unknown API version", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v9", + Kind: "ConfigMap", + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{noMatchForGVKErrMsg, `no matches for kind "ConfigMap" in version "v9"`}, + }, + { + desc: "the resource selector specifies an unknown API group", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIGroup: "example.com", + APIVersion: "v1", + Kind: "Widget", + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{noMatchForGVKErrMsg, `no matches for kind "Widget" in version "example.com/v1"`}, + }, + { + desc: "the resource selector specifies an unknown kind with a label selector", + placementPolicy: newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMapp", + LabelSelector: &metav1.LabelSelector{}, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"failed to retrieve selected resources (label selector-based selector)", noMatchForGVKErrMsg}, + }, + } + + BeforeAll(func() { + // Create a config map that a valid resource selector can select. + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + }, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + + // Create the placement policies; this also verifies that the API server accepts them. + for _, tc := range testCases { + Expect(hubClient.Create(ctx, tc.placementPolicy)).To(Succeed(), "Failed to create the placement policy (%s)", tc.desc) + } + }) + + for _, tc := range testCases { + It(fmt.Sprintf("should return an error when %s", tc.desc), func() { + verifySnapshotErrTestCase(tc) + }) + } + + It("should return an unexpected error when no resource selectors are present", func() { + // The CRD requires at least one resource selector; use an in-memory placement policy instead. + placementPolicy := newPlacementPolicy(appNamespaceName) + err := snapshotResourcesAndExpectErr(placementPolicy) + verifyErrCategoryAndMsg(err, kferrors.ErrCategoryUnexpected, "no resource selectors are present") + }) + + It("should return an unexpected error when the placement policy accessor is nil", func() { + // Note that a literal nil is passed here; a typed nil pointer would yield a non-nil interface value. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, nil) + Expect(gotSnapshots).To(BeEmpty(), "Placement resource snapshots are returned unexpectedly") + Expect(isUpToDate).To(BeFalse(), "The up-to-date flag is set unexpectedly") + verifyErrCategoryAndMsg(err, kferrors.ErrCategoryUnexpected, "placement policy accessor is nil") + }) + + AfterAll(func() { + objs := make([]client.Object, 0, len(testCases)+1) + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}) + for _, tc := range testCases { + objs = append(objs, tc.placementPolicy) + } + deleteObjectsAndWait(objs) + }) + }) + + Context("named resource not found", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + placementPolicy := newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMapName, + }) + + BeforeAll(func() { + // Create a placement policy that selects a config map which does not exist yet. + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("should return a user error when the selected resource does not exist", func() { + err := snapshotResourcesAndExpectErr(placementPolicy) + verifyErrCategoryAndMsg(err, kferrors.ErrCategoryUser, resourceNotFoundErrMsg) + Expect(errors.IsNotFound(err)).To(BeTrue(), "The error does not wrap a not found error: %v", err) + }) + + It("can create the selected config map", func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + }, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + }) + + It("can create a placement resource snapshot once the selected resource exists", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: appNamespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + } + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 1) + objs = append(objs, + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("named resource not found (informer-backed)", Ordered, func() { + // Unlike config maps, Deployments are watched by an informer in the test environment; the snapshot manager + // reads them from the cache first, and reads from the API server directly when they are not found there. + deploymentName := fmt.Sprintf(deploymentNameTemplate, utils.RandStr()) + + placementPolicy := newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + Name: deploymentName, + }) + + BeforeAll(func() { + // Create a placement policy that selects a Deployment which does not exist yet. + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("should return a user error when the selected resource is in neither the cache nor the API server", func() { + err := snapshotResourcesAndExpectErr(placementPolicy) + verifyErrCategoryAndMsg(err, kferrors.ErrCategoryUser, resourceNotFoundErrMsg) + Expect(errors.IsNotFound(err)).To(BeTrue(), "The error does not wrap a not found error: %v", err) + }) + + It("can create the selected Deployment", func() { + // Do not wait for the Deployment to reach the cache. + Expect(hubClient.Create(ctx, newDeployment(deploymentName, appNamespaceName))).To(Succeed(), "Failed to create the Deployment") + }) + + It("can create a placement resource snapshot right away, whether or not the cache has caught up", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: appNamespaceName, + Name: deploymentName, + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + }, + } + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 1) + objs = append(objs, + placementPolicy, + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: deploymentName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("ineligible resources (name-based selection)", Ordered, func() { + // Some of the ineligible resources feature fixed names; use a fresh namespace. + namespaceName := fmt.Sprintf(namespaceNameTemplate, utils.RandStr()) + nameSuffix := utils.RandStr() + ownerConfigMapName := fmt.Sprintf(configMapNameTemplate, nameSuffix) + "-owner" + ownedConfigMapName := fmt.Sprintf(configMapNameTemplate, nameSuffix) + "-owned" + deletingConfigMapName := fmt.Sprintf(configMapNameTemplate, nameSuffix) + "-deleting" + tokenSecretName := fmt.Sprintf(secretNameTemplate, nameSuffix) + "-token" + opaqueSecretName := fmt.Sprintf(secretNameTemplate, nameSuffix) + "-opaque" + + testCases := []snapshotErrTestCase{ + { + desc: "the built-in CA certificate config map is selected by name", + placementPolicy: newPlacementPolicy(namespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: kubeRootCACertConfigMapName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleResourceErrMsg, + "the built-in CA certificate config map is created automatically in every namespace and cannot be placed"}, + }, + { + desc: "the default service account is selected by name", + placementPolicy: newPlacementPolicy(namespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ServiceAccount", + Name: defaultServiceAccountName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleResourceErrMsg, + "the default service account is created automatically in every namespace and cannot be placed"}, + }, + { + desc: "a service account token secret is selected by name", + placementPolicy: newPlacementPolicy(namespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Secret", + Name: tokenSecretName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleResourceErrMsg, + "service account token secrets are created automatically along with their service accounts and cannot be placed"}, + }, + { + desc: "a resource owned by a controller is selected by name", + placementPolicy: newPlacementPolicy(namespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: ownedConfigMapName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleResourceErrMsg, "the resource is under the management of a controller"}, + }, + { + desc: "a resource being deleted is selected by name", + placementPolicy: newPlacementPolicy(namespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: deletingConfigMapName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleResourceErrMsg, "the resource is being deleted and cannot be placed"}, + }, + } + + // The control placement policy selects eligible resources only. + controlPlacementPolicy := newPlacementPolicy(namespaceName, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: ownerConfigMapName, + }, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Secret", + Name: opaqueSecretName, + }, + ) + + BeforeAll(func() { + createNamespace(namespaceName) + + objs := []client.Object{ + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: kubeRootCACertConfigMapName, Namespace: namespaceName}, + Data: map[string]string{"ca.crt": "dummy"}, + }, + &corev1.ServiceAccount{ + ObjectMeta: metav1.ObjectMeta{Name: defaultServiceAccountName, Namespace: namespaceName}, + }, + &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: tokenSecretName, + Namespace: namespaceName, + // The API server requires this annotation for service account token secrets. + Annotations: map[string]string{corev1.ServiceAccountNameKey: defaultServiceAccountName}, + }, + Type: corev1.SecretTypeServiceAccountToken, + }, + &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: opaqueSecretName, Namespace: namespaceName}, + Type: corev1.SecretTypeOpaque, + Data: map[string][]byte{"key": []byte("value")}, + }, + } + for _, obj := range objs { + Expect(hubClient.Create(ctx, obj)).To(Succeed(), "Failed to create object %T %s", obj, obj.GetName()) + } + + ownerConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: ownerConfigMapName, Namespace: namespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, ownerConfigMap)).To(Succeed(), "Failed to create the owner config map") + ownedConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: ownedConfigMapName, + Namespace: namespaceName, + OwnerReferences: []metav1.OwnerReference{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Name: ownerConfigMapName, + UID: ownerConfigMap.UID, + Controller: ptr.To(true), + }, + }, + }, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, ownedConfigMap)).To(Succeed(), "Failed to create the owned config map") + + createDeletingConfigMap(deletingConfigMapName, namespaceName, nil) + + for _, tc := range testCases { + Expect(hubClient.Create(ctx, tc.placementPolicy)).To(Succeed(), "Failed to create the placement policy (%s)", tc.desc) + } + Expect(hubClient.Create(ctx, controlPlacementPolicy)).To(Succeed(), "Failed to create the control placement policy") + }) + + for _, tc := range testCases { + It(fmt.Sprintf("should return a user error when %s", tc.desc), func() { + verifySnapshotErrTestCase(tc) + }) + } + + It("can snapshot eligible resources selected by name", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, controlPlacementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, controlPlacementPolicy, 0) + + // The resources are sorted by their unique IDs, i.e., the config map comes first. + snapshot := getPrimarySnapshotFromCache(controlPlacementPolicy, 0) + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: namespaceName, + Name: ownerConfigMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Namespace: namespaceName, + Name: opaqueSecretName, + APIVersion: "v1", + Kind: "Secret", + }, + } + resources := snapshot.GetSpec().Resources + verifySnapshottedResourceIdentifiers(resources, wantIdentifiers) + + gotSecret := &corev1.Secret{} + Expect(json.Unmarshal(resources[1].Manifest.Raw, gotSecret)).To(Succeed(), "Failed to unmarshal the secret manifest") + wantSecret := &corev1.Secret{ + TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "Secret"}, + ObjectMeta: metav1.ObjectMeta{ + Name: opaqueSecretName, + Namespace: namespaceName, + }, + Type: corev1.SecretTypeOpaque, + Data: map[string][]byte{"key": []byte("value")}, + } + if diff := cmp.Diff(gotSecret, wantSecret); diff != "" { + Fail(fmt.Sprintf("Snapshotted secret diff (-got +want):\n%s", diff)) + } + }) + + AfterAll(func() { + removeFinalizersAndDelete(&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: deletingConfigMapName, Namespace: namespaceName}}) + + objs := primarySnapshotObjs(controlPlacementPolicy, 1) + objs = append(objs, controlPlacementPolicy) + for _, tc := range testCases { + objs = append(objs, tc.placementPolicy) + } + objs = append(objs, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: kubeRootCACertConfigMapName, Namespace: namespaceName}}, + &corev1.ServiceAccount{ObjectMeta: metav1.ObjectMeta{Name: defaultServiceAccountName, Namespace: namespaceName}}, + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: tokenSecretName, Namespace: namespaceName}}, + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: opaqueSecretName, Namespace: namespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ownedConfigMapName, Namespace: namespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ownerConfigMapName, Namespace: namespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("ineligible resources (label-based selection)", Ordered, func() { + // Some of the ineligible resources feature fixed names; use a fresh namespace. + namespaceName := fmt.Sprintf(namespaceNameTemplate, utils.RandStr()) + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + + nameSuffix := utils.RandStr() + configMapName := fmt.Sprintf(configMapNameTemplate, nameSuffix) + ownerConfigMapName := configMapName + "-owner" + ownedConfigMapName := configMapName + "-owned" + deletingConfigMapName := configMapName + "-deleting" + serviceAccountName := fmt.Sprintf(serviceAccountNameTemplate, nameSuffix) + secretName := fmt.Sprintf(secretNameTemplate, nameSuffix) + tokenSecretName := secretName + "-token" + + placementPolicy := newPlacementPolicy(namespaceName, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{MatchLabels: groupLabels}, + }, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ServiceAccount", + LabelSelector: &metav1.LabelSelector{MatchLabels: groupLabels}, + }, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Secret", + LabelSelector: &metav1.LabelSelector{MatchLabels: groupLabels}, + }, + ) + + // Only the eligible resources are selected; they are sorted by their unique IDs. + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: namespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Namespace: namespaceName, + Name: secretName, + APIVersion: "v1", + Kind: "Secret", + }, + { + Namespace: namespaceName, + Name: serviceAccountName, + APIVersion: "v1", + Kind: "ServiceAccount", + }, + } + + var wantHash string + + BeforeAll(func() { + createNamespace(namespaceName) + + // The owner config map is not labeled and thus not selected. + ownerConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: ownerConfigMapName, Namespace: namespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, ownerConfigMap)).To(Succeed(), "Failed to create the owner config map") + + objs := []client.Object{ + // Eligible resources. + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: namespaceName, Labels: groupLabels}, + Data: map[string]string{"key": "value"}, + }, + &corev1.ServiceAccount{ + ObjectMeta: metav1.ObjectMeta{Name: serviceAccountName, Namespace: namespaceName, Labels: groupLabels}, + }, + &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: secretName, Namespace: namespaceName, Labels: groupLabels}, + Type: corev1.SecretTypeOpaque, + Data: map[string][]byte{"key": []byte("value")}, + }, + // Ineligible resources. + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: kubeRootCACertConfigMapName, Namespace: namespaceName, Labels: groupLabels}, + Data: map[string]string{"ca.crt": "dummy"}, + }, + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: ownedConfigMapName, + Namespace: namespaceName, + Labels: groupLabels, + OwnerReferences: []metav1.OwnerReference{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Name: ownerConfigMapName, + UID: ownerConfigMap.UID, + Controller: ptr.To(true), + }, + }, + }, + Data: map[string]string{"key": "value"}, + }, + &corev1.ServiceAccount{ + ObjectMeta: metav1.ObjectMeta{Name: defaultServiceAccountName, Namespace: namespaceName, Labels: groupLabels}, + }, + &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Name: tokenSecretName, + Namespace: namespaceName, + Labels: groupLabels, + Annotations: map[string]string{corev1.ServiceAccountNameKey: defaultServiceAccountName}, + }, + Type: corev1.SecretTypeServiceAccountToken, + }, + } + for _, obj := range objs { + Expect(hubClient.Create(ctx, obj)).To(Succeed(), "Failed to create object %T %s", obj, obj.GetName()) + } + createDeletingConfigMap(deletingConfigMapName, namespaceName, groupLabels) + + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + verifySnapshottedResourceIdentifiers(resources, wantIdentifiers) + wantHash = hash + }) + + It("can create a placement resource snapshot with eligible resources only", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + verifySnapshotMetadata(snapshot, placementPolicy, 0, 0, 1, wantHash) + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + It("can make a selected config map ineligible by adding a controller owner reference", func() { + ownerConfigMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: namespaceName, Name: ownerConfigMapName}, ownerConfigMap)).To(Succeed(), + "Failed to get the owner config map") + + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: namespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.OwnerReferences = []metav1.OwnerReference{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Name: ownerConfigMapName, + UID: ownerConfigMap.UID, + Controller: ptr.To(true), + }, + } + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + }) + + It("should create a new snapshot that excludes the ineligible config map", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 1) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 1) + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers[1:]) + gotHash := snapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] + Expect(gotHash).ToNot(Equal(wantHash), "The hash of the selected resources has not changed") + }) + + It("can make the config map eligible again by removing the owner reference", func() { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: namespaceName, Name: configMapName}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.OwnerReferences = nil + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + }) + + It("should create another snapshot that includes the config map again", func() { + // The selected resources are the same as those in the snapshot at index 0; however, as the manager + // compares the selected resources with the latest snapshot (at index 1), a new snapshot is created. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 2) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 2) + verifySnapshotMetadata(snapshot, placementPolicy, 2, 0, 1, wantHash) + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + AfterAll(func() { + removeFinalizersAndDelete(&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: deletingConfigMapName, Namespace: namespaceName}}) + + objs := primarySnapshotObjs(placementPolicy, 3) + objs = append(objs, + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: namespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ownerConfigMapName, Namespace: namespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ownedConfigMapName, Namespace: namespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: kubeRootCACertConfigMapName, Namespace: namespaceName}}, + &corev1.ServiceAccount{ObjectMeta: metav1.ObjectMeta{Name: serviceAccountName, Namespace: namespaceName}}, + &corev1.ServiceAccount{ObjectMeta: metav1.ObjectMeta{Name: defaultServiceAccountName, Namespace: namespaceName}}, + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: secretName, Namespace: namespaceName}}, + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Name: tokenSecretName, Namespace: namespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("resources with non-controller owner references", Ordered, func() { + // Owner references that do not point to a controller do not make a resource ineligible for placement; they + // are stripped from the snapshotted manifests, as the owners do not exist on the member clusters. + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + ownerConfigMapName := configMapNamePrefix + "-owner" + namedConfigMapName := configMapNamePrefix + "-named" + labeledConfigMapName := configMapNamePrefix + "-labeled" + + placementPolicy := newPlacementPolicy(appNamespaceName, + configMapNameSelector(namedConfigMapName), + configMapLabelSelector(groupLabels), + ) + + BeforeAll(func() { + // The owner config map is not labeled and thus not selected. + ownerConfigMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: ownerConfigMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, ownerConfigMap)).To(Succeed(), "Failed to create the owner config map") + + ownerRefs := func(controller *bool) []metav1.OwnerReference { + return []metav1.OwnerReference{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Name: ownerConfigMapName, + UID: ownerConfigMap.UID, + Controller: controller, + }, + } + } + configMaps := []*corev1.ConfigMap{ + { + // A config map, selected by name, whose owner reference leaves the controller field unset. + ObjectMeta: metav1.ObjectMeta{ + Name: namedConfigMapName, + Namespace: appNamespaceName, + OwnerReferences: ownerRefs(nil), + }, + Data: map[string]string{"key": "value-named"}, + }, + { + // A config map, selected by labels, whose owner reference explicitly states that the owner + // is not a controller. + ObjectMeta: metav1.ObjectMeta{ + Name: labeledConfigMapName, + Namespace: appNamespaceName, + Labels: groupLabels, + OwnerReferences: ownerRefs(ptr.To(false)), + }, + Data: map[string]string{"key": "value-labeled"}, + }, + } + for _, configMap := range configMaps { + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create config map %s", configMap.Name) + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a placement resource snapshot with the owned config maps", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + }) + + It("should strip the owner references", func() { + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + resources := snapshot.GetSpec().Resources + + // The resources are sorted by their unique IDs, i.e., the labeled config map comes first. + // + // Compare the manifests as generic maps; this verifies that the owner references are absent. + wantManifests := []map[string]interface{}{ + { + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": map[string]interface{}{ + "name": labeledConfigMapName, + "namespace": appNamespaceName, + "labels": map[string]interface{}{placementGroupLabelKey: groupLabels[placementGroupLabelKey]}, + }, + "data": map[string]interface{}{"key": "value-labeled"}, + }, + { + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": map[string]interface{}{ + "name": namedConfigMapName, + "namespace": appNamespaceName, + }, + "data": map[string]interface{}{"key": "value-named"}, + }, + } + Expect(resources).To(HaveLen(len(wantManifests)), "Unexpected number of snapshotted resources") + for idx := range resources { + if diff := cmp.Diff(manifestAsMap(resources[idx].Manifest.Raw), wantManifests[idx]); diff != "" { + Fail(fmt.Sprintf("Snapshotted config map manifest (#%d) diff (-got +want):\n%s", idx, diff)) + } + } + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 1) + objs = append(objs, + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: namedConfigMapName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: labeledConfigMapName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ownerConfigMapName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("no resources selected", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + + placementPolicy := newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{MatchLabels: groupLabels}, + }) + + var emptyHash string + + BeforeAll(func() { + // Create a placement policy whose label selector matches no resources (yet). + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create an empty placement resource snapshot when the label selector matches no resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + Expect(resources).To(BeEmpty(), "Resources are selected unexpectedly") + emptyHash = hash + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + // Wait until the cache has observed the new snapshot. + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + verifySnapshotMetadata(snapshot, placementPolicy, 0, 0, 1, emptyHash) + Expect(snapshot.GetSpec().Resources).To(BeEmpty(), "The placement resource snapshot is not empty") + }) + + It("should return the existing empty snapshot as it is up-to-date", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + It("can create a config map that matches the label selector", func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + Labels: groupLabels, + }, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + }) + + It("should create a new placement resource snapshot once a resource matches", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 1) + + // Wait until the cache has observed the new snapshot. + snapshot := getPrimarySnapshotFromCache(placementPolicy, 1) + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: appNamespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + } + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + It("can delete the config map", func() { + deleteObjectsAndWait([]client.Object{ + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + }) + }) + + It("should create a new empty placement resource snapshot when no resources are selected any more", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 2) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 2) + verifySnapshotMetadata(snapshot, placementPolicy, 2, 0, 1, emptyHash) + Expect(snapshot.GetSpec().Resources).To(BeEmpty(), "The placement resource snapshot is not empty") + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 3) + objs = append(objs, + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("duplicate selections (name-based and label-based)", Ordered, func() { + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + configMapNames := []string{configMapNamePrefix + "-0", configMapNamePrefix + "-1"} + + placementPolicy := newPlacementPolicy(appNamespaceName, + // A placement policy can specify its own namespace in a resource selector. + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + Name: configMapNames[0], + }, + // The label selector selects both config maps, including the one selected above by name. + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{MatchLabels: groupLabels}, + }, + ) + + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: appNamespaceName, + Name: configMapNames[0], + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Namespace: appNamespaceName, + Name: configMapNames[1], + APIVersion: "v1", + Kind: "ConfigMap", + }, + } + + var wantHash string + + BeforeAll(func() { + for idx, name := range configMapNames { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: appNamespaceName, + Labels: groupLabels, + }, + Data: map[string]string{"key": fmt.Sprintf("value-%d", idx)}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + verifySnapshottedResourceIdentifiers(resources, wantIdentifiers) + wantHash = hash + }) + + It("can create a placement resource snapshot that includes each selected resource only once", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + verifySnapshotMetadata(snapshot, placementPolicy, 0, 0, 1, wantHash) + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 1) + objs = append(objs, placementPolicy) + for _, name := range configMapNames { + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: appNamespaceName}}) + } + deleteObjectsAndWait(objs) + }) + }) + + Context("same resource selected via different API versions", Ordered, func() { + hpaName := fmt.Sprintf(hpaNameTemplate, utils.RandStr()) + + hpaSelector := func(version string) placementv1alpha1.ResourceSelector { + return placementv1alpha1.ResourceSelector{ + APIGroup: autoscalingv1.GroupName, + APIVersion: version, + Kind: "HorizontalPodAutoscaler", + Name: hpaName, + } + } + hpaIdentifier := func(version string) placementv1alpha1.ObjectReference { + return placementv1alpha1.ObjectReference{ + Namespace: appNamespaceName, + Name: hpaName, + APIGroup: autoscalingv1.GroupName, + APIVersion: version, + Kind: "HorizontalPodAutoscaler", + } + } + + // The same HPA is selected twice; the resource selector that appears first wins. + placementPolicy := newPlacementPolicy(appNamespaceName, hpaSelector("v1"), hpaSelector("v2")) + + var firstHash string + + BeforeAll(func() { + hpa := &autoscalingv1.HorizontalPodAutoscaler{ + ObjectMeta: metav1.ObjectMeta{ + Name: hpaName, + Namespace: appNamespaceName, + }, + Spec: autoscalingv1.HorizontalPodAutoscalerSpec{ + // The scale target does not need to exist. + ScaleTargetRef: autoscalingv1.CrossVersionObjectReference{ + APIVersion: "apps/v1", + Kind: "Deployment", + Name: fmt.Sprintf(deploymentNameTemplate, utils.RandStr()), + }, + MinReplicas: ptr.To(int32(1)), + MaxReplicas: 3, + TargetCPUUtilizationPercentage: ptr.To(int32(50)), + }, + } + Expect(hubClient.Create(ctx, hpa)).To(Succeed(), "Failed to create the HPA") + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a placement resource snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + resources := snapshot.GetSpec().Resources + verifySnapshottedResourceIdentifiers(resources, []placementv1alpha1.ObjectReference{hpaIdentifier("v1")}) + firstHash = snapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] + + // The manifest should be that of the v1 API. + liveHPA := &autoscalingv1.HorizontalPodAutoscaler{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: hpaName}, liveHPA)).To(Succeed(), + "Failed to get the HPA") + gotHPA := &autoscalingv1.HorizontalPodAutoscaler{} + Expect(json.Unmarshal(resources[0].Manifest.Raw, gotHPA)).To(Succeed(), "Failed to unmarshal the HPA manifest") + wantHPA := &autoscalingv1.HorizontalPodAutoscaler{ + TypeMeta: metav1.TypeMeta{APIVersion: "autoscaling/v1", Kind: "HorizontalPodAutoscaler"}, + ObjectMeta: metav1.ObjectMeta{ + Name: hpaName, + Namespace: appNamespaceName, + }, + Spec: liveHPA.Spec, + } + if diff := cmp.Diff(gotHPA, wantHPA, cmpopts.EquateEmpty()); diff != "" { + Fail(fmt.Sprintf("Snapshotted HPA diff (-got +want):\n%s", diff)) + } + }) + + It("can swap the order of the resource selectors", func() { + updatedPlacementPolicy := &placementv1alpha1.PlacementPolicy{} + Expect(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(placementPolicy), updatedPlacementPolicy)).To(Succeed(), + "Failed to get the placement policy") + slices.Reverse(updatedPlacementPolicy.Spec.ResourceSelectors) + Expect(hubClient.Update(ctx, updatedPlacementPolicy)).To(Succeed(), "Failed to update the placement policy") + placementPolicy = updatedPlacementPolicy + }) + + It("should create a new snapshot that carries the resource as selected by the first resource selector", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 1) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 1) + resources := snapshot.GetSpec().Resources + verifySnapshottedResourceIdentifiers(resources, []placementv1alpha1.ObjectReference{hpaIdentifier("v2")}) + gotHash := snapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] + Expect(gotHash).ToNot(Equal(firstHash), "The hash of the selected resources has not changed") + + // The manifest should be that of the v2 API. + gotHPA := &autoscalingv2.HorizontalPodAutoscaler{} + Expect(json.Unmarshal(resources[0].Manifest.Raw, gotHPA)).To(Succeed(), "Failed to unmarshal the HPA manifest") + Expect(gotHPA.APIVersion).To(Equal("autoscaling/v2"), "Unexpected API version of the snapshotted HPA") + wantMetrics := []autoscalingv2.MetricSpec{ + { + Type: autoscalingv2.ResourceMetricSourceType, + Resource: &autoscalingv2.ResourceMetricSource{ + Name: corev1.ResourceCPU, + Target: autoscalingv2.MetricTarget{ + Type: autoscalingv2.UtilizationMetricType, + AverageUtilization: ptr.To(int32(50)), + }, + }, + }, + } + if diff := cmp.Diff(gotHPA.Spec.Metrics, wantMetrics); diff != "" { + Fail(fmt.Sprintf("Snapshotted HPA metrics diff (-got +want):\n%s", diff)) + } + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 2) + objs = append(objs, + placementPolicy, + &autoscalingv1.HorizontalPodAutoscaler{ObjectMeta: metav1.ObjectMeta{Name: hpaName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("reordered resource selectors", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + deploymentName := fmt.Sprintf(deploymentNameTemplate, utils.RandStr()) + + placementPolicy := newPlacementPolicy(appNamespaceName, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMapName, + }, + placementv1alpha1.ResourceSelector{ + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + Name: deploymentName, + }, + ) + + BeforeAll(func() { + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: configMapName, + Namespace: appNamespaceName, + }, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + deployment := newDeployment(deploymentName, appNamespaceName) + Expect(hubClient.Create(ctx, deployment)).To(Succeed(), "Failed to create the Deployment") + waitForDeploymentsInCache(deployment) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a placement resource snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + // Wait until the cache has observed the new snapshot. + _ = getPrimarySnapshotFromCache(placementPolicy, 0) + }) + + It("can reverse the order of the resource selectors", func() { + updatedPlacementPolicy := &placementv1alpha1.PlacementPolicy{} + Expect(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(placementPolicy), updatedPlacementPolicy)).To(Succeed(), + "Failed to get the placement policy") + slices.Reverse(updatedPlacementPolicy.Spec.ResourceSelectors) + Expect(hubClient.Update(ctx, updatedPlacementPolicy)).To(Succeed(), "Failed to update the placement policy") + placementPolicy = updatedPlacementPolicy + }) + + It("should return the existing snapshot as the selected resources have not changed", func() { + wantSnapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + if diff := cmp.Diff(gotSnapshots[0], wantSnapshot, cmpopts.IgnoreFields(metav1.TypeMeta{}, "APIVersion", "Kind")); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot diff (-got +want):\n%s", diff)) + } + + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 2) + objs = append(objs, + placementPolicy, + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: deploymentName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("manifest sanitization (config maps)", Ordered, func() { + configMapNamePrefix := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + // The names are in ascending order, which is also the order in which the selected resources are + // snapshotted. + configMapNames := []string{configMapNamePrefix + "-0", configMapNamePrefix + "-1", configMapNamePrefix + "-2"} + + lastAppliedConfig := `{"apiVersion":"v1","kind":"ConfigMap"}` + + placementPolicy := newPlacementPolicy(appNamespaceName) + for _, name := range configMapNames { + placementPolicy.Spec.ResourceSelectors = append(placementPolicy.Spec.ResourceSelectors, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: name, + }) + } + + BeforeAll(func() { + configMaps := []*corev1.ConfigMap{ + { + // A config map with the last applied configuration annotation and a custom annotation. + ObjectMeta: metav1.ObjectMeta{ + Name: configMapNames[0], + Namespace: appNamespaceName, + Annotations: map[string]string{ + corev1.LastAppliedConfigAnnotation: lastAppliedConfig, + customAnnotationKey: "keep", + }, + }, + Data: map[string]string{"key": "value-0"}, + }, + { + // A config map with the last applied configuration annotation only. + ObjectMeta: metav1.ObjectMeta{ + Name: configMapNames[1], + Namespace: appNamespaceName, + Annotations: map[string]string{ + corev1.LastAppliedConfigAnnotation: lastAppliedConfig, + }, + }, + Data: map[string]string{"key": "value-1"}, + }, + { + // A config map with a finalizer. + ObjectMeta: metav1.ObjectMeta{ + Name: configMapNames[2], + Namespace: appNamespaceName, + Finalizers: []string{testFinalizer}, + }, + Data: map[string]string{"key": "value-2"}, + }, + } + for _, configMap := range configMaps { + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a placement resource snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + }) + + It("should strip the last applied configuration annotation, the finalizers, and the read-only fields", func() { + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + resources := snapshot.GetSpec().Resources + Expect(resources).To(HaveLen(len(configMapNames)), "Unexpected number of snapshotted resources") + + // Compare the manifests as generic maps; this verifies that the stripped fields are absent. + wantManifests := []map[string]interface{}{ + { + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": map[string]interface{}{ + "name": configMapNames[0], + "namespace": appNamespaceName, + "annotations": map[string]interface{}{customAnnotationKey: "keep"}, + }, + "data": map[string]interface{}{"key": "value-0"}, + }, + { + "apiVersion": "v1", + "kind": "ConfigMap", + // The annotations field should be removed altogether, as no annotation is left. + "metadata": map[string]interface{}{ + "name": configMapNames[1], + "namespace": appNamespaceName, + }, + "data": map[string]interface{}{"key": "value-1"}, + }, + { + "apiVersion": "v1", + "kind": "ConfigMap", + // The finalizers are stripped, even though the live object still carries one. + "metadata": map[string]interface{}{ + "name": configMapNames[2], + "namespace": appNamespaceName, + }, + "data": map[string]interface{}{"key": "value-2"}, + }, + } + for idx := range resources { + if diff := cmp.Diff(manifestAsMap(resources[idx].Manifest.Raw), wantManifests[idx]); diff != "" { + Fail(fmt.Sprintf("Snapshotted config map manifest (#%d) diff (-got +want):\n%s", idx, diff)) + } + } + }) + + It("can update stripped fields only", func() { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapNames[0]}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Annotations[corev1.LastAppliedConfigAnnotation] = `{"apiVersion":"v1","kind":"ConfigMap","data":{"key":"value-0"}}` + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + + configMap = &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapNames[2]}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Finalizers = append(configMap.Finalizers, anotherTestFinalizer) + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + }) + + It("should return the existing snapshot as stripped fields are not snapshotted", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + It("can update a retained annotation", func() { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: configMapNames[0]}, configMap)).To(Succeed(), + "Failed to get the config map") + configMap.Annotations[customAnnotationKey] = "changed" + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update the config map") + }) + + It("should create a new snapshot when a retained annotation changes", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 1) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 1) + resources := snapshot.GetSpec().Resources + Expect(resources).To(HaveLen(len(configMapNames)), "Unexpected number of snapshotted resources") + gotMetadata := manifestAsMap(resources[0].Manifest.Raw)["metadata"] + wantMetadata := map[string]interface{}{ + "name": configMapNames[0], + "namespace": appNamespaceName, + "annotations": map[string]interface{}{customAnnotationKey: "changed"}, + } + if diff := cmp.Diff(gotMetadata, wantMetadata); diff != "" { + Fail(fmt.Sprintf("Snapshotted config map metadata diff (-got +want):\n%s", diff)) + } + }) + + AfterAll(func() { + removeFinalizersAndDelete(&corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapNames[2], Namespace: appNamespaceName}}) + + objs := primarySnapshotObjs(placementPolicy, 2) + objs = append(objs, + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapNames[0], Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapNames[1], Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("manifest sanitization (deployments)", Ordered, func() { + deploymentName := fmt.Sprintf(deploymentNameTemplate, utils.RandStr()) + + placementPolicy := newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + Name: deploymentName, + }) + + BeforeAll(func() { + deployment := newDeployment(deploymentName, appNamespaceName) + deployment.Annotations = map[string]string{ + deploymentutil.RevisionAnnotation: "1", + customAnnotationKey: "keep", + } + Expect(hubClient.Create(ctx, deployment)).To(Succeed(), "Failed to create the Deployment") + + // The envtest environment does not run the Deployment controller; set the status by hand. + deployment.Status = appsv1.DeploymentStatus{ + ObservedGeneration: 1, + Replicas: 2, + ReadyReplicas: 2, + } + Expect(hubClient.Status().Update(ctx, deployment)).To(Succeed(), "Failed to update the Deployment status") + waitForDeploymentsInCache(deployment) + + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a placement resource snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + }) + + It("should strip the revision annotation, the status, and the read-only fields", func() { + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + resources := snapshot.GetSpec().Resources + Expect(resources).To(HaveLen(1), "Unexpected number of snapshotted resources") + + // Only the top-level metadata is compared as a generic map; the pod template (part of the spec) + // carries its own metadata, which is not subject to sanitization. + gotManifest := manifestAsMap(resources[0].Manifest.Raw) + Expect(gotManifest).ToNot(HaveKey("status"), "The status of the Deployment has not been stripped") + wantMetadata := map[string]interface{}{ + "name": deploymentName, + "namespace": appNamespaceName, + "labels": map[string]interface{}{"app": "demo"}, + "annotations": map[string]interface{}{customAnnotationKey: "keep"}, + } + if diff := cmp.Diff(gotManifest["metadata"], wantMetadata); diff != "" { + Fail(fmt.Sprintf("Snapshotted Deployment metadata diff (-got +want):\n%s", diff)) + } + + // The spec should match that of the live object (which carries the defaulted fields). + liveDeployment := &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: deploymentName}, liveDeployment)).To(Succeed(), + "Failed to get the Deployment") + gotDeployment := &appsv1.Deployment{} + Expect(json.Unmarshal(resources[0].Manifest.Raw, gotDeployment)).To(Succeed(), "Failed to unmarshal the Deployment manifest") + if diff := cmp.Diff(gotDeployment.Spec, liveDeployment.Spec, cmpopts.EquateEmpty()); diff != "" { + Fail(fmt.Sprintf("Snapshotted Deployment spec diff (-got +want):\n%s", diff)) + } + }) + + It("can update the status and the revision annotation", func() { + deployment := &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: deploymentName}, deployment)).To(Succeed(), + "Failed to get the Deployment") + deployment.Status.ReadyReplicas = 1 + deployment.Status.UpdatedReplicas = 1 + Expect(hubClient.Status().Update(ctx, deployment)).To(Succeed(), "Failed to update the Deployment status") + + deployment = &appsv1.Deployment{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: deploymentName}, deployment)).To(Succeed(), + "Failed to get the Deployment") + deployment.Annotations[deploymentutil.RevisionAnnotation] = "2" + Expect(hubClient.Update(ctx, deployment)).To(Succeed(), "Failed to update the Deployment") + waitForDeploymentsInCache(deployment) + }) + + It("should return the existing snapshot as only stripped fields have changed", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 2) + objs = append(objs, + placementPolicy, + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: deploymentName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("resources in the default namespace", Ordered, func() { + // The default namespace itself cannot be placed, but the resources within it can. + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + labeledConfigMapName := configMapName + "-labeled" + + placementPolicy := newPlacementPolicy(defaultNamespaceName, + configMapNameSelector(configMapName), + configMapLabelSelector(groupLabels), + ) + kubernetesAPIServicePlacementPolicy := newPlacementPolicy(defaultNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Service", + Name: kubernetesAPIServiceName, + }) + + // The resources are sorted by their unique IDs; the labeled config map has a longer name with the same prefix. + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: defaultNamespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Namespace: defaultNamespaceName, + Name: labeledConfigMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + } + + BeforeAll(func() { + // Make sure that the Kubernetes API service has been created by the API server. + Eventually(func() error { + return hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: defaultNamespaceName, Name: kubernetesAPIServiceName}, &corev1.Service{}) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to find the Kubernetes API service") + + objs := []client.Object{ + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: defaultNamespaceName}, + Data: map[string]string{"key": "value"}, + }, + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: labeledConfigMapName, Namespace: defaultNamespaceName, Labels: groupLabels}, + Data: map[string]string{"key": "value"}, + }, + placementPolicy, + kubernetesAPIServicePlacementPolicy, + } + for _, obj := range objs { + Expect(hubClient.Create(ctx, obj)).To(Succeed(), "Failed to create object %T %s", obj, obj.GetName()) + } + }) + + It("can create a placement resource snapshot with resources in the default namespace", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + It("should return a user error when the Kubernetes API service is selected by name", func() { + err := snapshotResourcesAndExpectErr(kubernetesAPIServicePlacementPolicy) + verifyErrCategoryAndMsg(err, kferrors.ErrCategoryUser, ineligibleResourceErrMsg, + "the Kubernetes API service is created automatically in the default namespace and cannot be placed") + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 1) + objs = append(objs, + placementPolicy, + kubernetesAPIServicePlacementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: defaultNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: labeledConfigMapName, Namespace: defaultNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("manifest sanitization (services)", Ordered, func() { + serviceName := fmt.Sprintf(serviceNameTemplate, utils.RandStr()) + serviceKey := types.NamespacedName{Namespace: appNamespaceName, Name: serviceName} + + placementPolicy := newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Service", + Name: serviceName, + }) + + BeforeAll(func() { + Expect(hubClient.Create(ctx, newLoadBalancerService(serviceName, appNamespaceName))).To(Succeed(), "Failed to create the Service") + + // The fields to strip must be present on the live object; otherwise the specs below would pass trivially. + service := &corev1.Service{} + Expect(hubUncachedReader.Get(ctx, serviceKey, service)).To(Succeed(), "Failed to get the Service") + verifyServiceFieldsFilledIn(service) + + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a placement resource snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + }) + + It("should strip the cluster IPs and the node ports filled in by the API server", func() { + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + resources := snapshot.GetSpec().Resources + Expect(resources).To(HaveLen(1), "Unexpected number of snapshotted resources") + + // Check the raw manifest, so that the absence of the keys is verified, not just their zero values. + gotManifest := manifestAsMap(resources[0].Manifest.Raw) + Expect(gotManifest).ToNot(HaveKey("status"), "The status of the Service has not been stripped") + wantMetadata := map[string]interface{}{ + "name": serviceName, + "namespace": appNamespaceName, + "labels": map[string]interface{}{"app": "demo"}, + } + if diff := cmp.Diff(gotManifest["metadata"], wantMetadata); diff != "" { + Fail(fmt.Sprintf("Snapshotted Service metadata diff (-got +want):\n%s", diff)) + } + + gotSpec, found, err := unstructured.NestedMap(gotManifest, "spec") + Expect(err).ToNot(HaveOccurred(), "Failed to read the spec of the Service manifest") + Expect(found).To(BeTrue(), "The Service manifest has no spec") + for _, key := range []string{"clusterIP", "clusterIPs", "healthCheckNodePort"} { + Expect(gotSpec).ToNot(HaveKey(key), "Field spec.%s of the Service has not been stripped", key) + } + gotPorts, found, err := unstructured.NestedSlice(gotManifest, "spec", "ports") + Expect(err).ToNot(HaveOccurred(), "Failed to read the ports of the Service manifest") + Expect(found).To(BeTrue(), "The Service manifest has no ports") + Expect(gotPorts).To(HaveLen(2), "Unexpected number of ports in the Service manifest") + for idx := range gotPorts { + Expect(gotPorts[idx]).ToNot(HaveKey("nodePort"), "Field spec.ports[%d].nodePort of the Service has not been stripped", idx) + } + + // The rest of the spec should match that of the live object (which carries the defaulted fields). + liveService := &corev1.Service{} + Expect(hubUncachedReader.Get(ctx, serviceKey, liveService)).To(Succeed(), "Failed to get the Service") + gotService := &corev1.Service{} + Expect(json.Unmarshal(resources[0].Manifest.Raw, gotService)).To(Succeed(), "Failed to unmarshal the Service manifest") + if diff := cmp.Diff(gotService.Spec, serviceSpecWithoutFilledInFields(&liveService.Spec), cmpopts.EquateEmpty()); diff != "" { + Fail(fmt.Sprintf("Snapshotted Service spec diff (-got +want):\n%s", diff)) + } + }) + + It("can re-create the Service with the same spec", func() { + oldService := &corev1.Service{} + Expect(hubUncachedReader.Get(ctx, serviceKey, oldService)).To(Succeed(), "Failed to get the Service") + + deleteObjectsAndWait([]client.Object{&corev1.Service{ObjectMeta: metav1.ObjectMeta{Name: serviceName, Namespace: appNamespaceName}}}) + Expect(hubClient.Create(ctx, newLoadBalancerService(serviceName, appNamespaceName))).To(Succeed(), "Failed to re-create the Service") + + recreatedService := &corev1.Service{} + Expect(hubUncachedReader.Get(ctx, serviceKey, recreatedService)).To(Succeed(), "Failed to get the re-created Service") + verifyServiceFieldsFilledIn(recreatedService) + // The API server fills in new values; it is all but impossible for it to pick the same cluster IP and + // node ports again. + Expect(recreatedService.Spec).ToNot(Equal(oldService.Spec), "The API server has filled in the same values for the re-created Service") + }) + + It("should return the existing snapshot as only the stripped fields have changed", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 2) + objs = append(objs, + placementPolicy, + &corev1.Service{ObjectMeta: metav1.ObjectMeta{Name: serviceName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) + + Context("manifest sanitization (jobs)", Ordered, func() { + jobName := fmt.Sprintf(jobNameTemplate, utils.RandStr()) + jobKey := types.NamespacedName{Namespace: appNamespaceName, Name: jobName} + + placementPolicy := newPlacementPolicy(appNamespaceName, placementv1alpha1.ResourceSelector{ + APIGroup: batchv1.GroupName, + APIVersion: "v1", + Kind: "Job", + Name: jobName, + }) + + // Jobs in batch/v1 default to orphan propagation, which adds the orphan finalizer; the envtest environment + // runs no garbage collector to remove it. + backgroundPropagation := client.PropagationPolicy(metav1.DeletePropagationBackground) + + BeforeAll(func() { + Expect(hubClient.Create(ctx, newJob(jobName, appNamespaceName))).To(Succeed(), "Failed to create the Job") + + // The labels to strip must be present on the live object; otherwise the specs below would pass trivially. + job := &batchv1.Job{} + Expect(hubUncachedReader.Get(ctx, jobKey, job)).To(Succeed(), "Failed to get the Job") + verifyJobControllerUIDLabelsAdded(job) + + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the placement policy") + }) + + It("can create a placement resource snapshot", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + }) + + It("should strip the controller UID labels added by the API server", func() { + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + resources := snapshot.GetSpec().Resources + Expect(resources).To(HaveLen(1), "Unexpected number of snapshotted resources") + + gotManifest := manifestAsMap(resources[0].Manifest.Raw) + Expect(gotManifest).ToNot(HaveKey("status"), "The status of the Job has not been stripped") + wantMetadata := map[string]interface{}{ + "name": jobName, + "namespace": appNamespaceName, + "labels": map[string]interface{}{"app": "demo"}, + } + if diff := cmp.Diff(gotManifest["metadata"], wantMetadata); diff != "" { + Fail(fmt.Sprintf("Snapshotted Job metadata diff (-got +want):\n%s", diff)) + } + + // Check the raw manifest for the creation timestamp of the pod template, as an absent key and its zero + // value cannot be told apart once unmarshalled. + gotTemplateMetadata, found, err := unstructured.NestedMap(gotManifest, "spec", "template", "metadata") + Expect(err).ToNot(HaveOccurred(), "Failed to read the pod template metadata of the Job manifest") + Expect(found).To(BeTrue(), "The Job manifest has no pod template metadata") + Expect(gotTemplateMetadata).ToNot(HaveKey("creationTimestamp"), "The creation timestamp of the pod template has not been stripped") + + // The rest of the spec, including the job-name labels that the API server adds as well, should match + // that of the live object. + liveJob := &batchv1.Job{} + Expect(hubUncachedReader.Get(ctx, jobKey, liveJob)).To(Succeed(), "Failed to get the Job") + gotJob := &batchv1.Job{} + Expect(json.Unmarshal(resources[0].Manifest.Raw, gotJob)).To(Succeed(), "Failed to unmarshal the Job manifest") + if diff := cmp.Diff(gotJob.Spec, jobSpecWithoutControllerUIDLabels(&liveJob.Spec), cmpopts.EquateEmpty()); diff != "" { + Fail(fmt.Sprintf("Snapshotted Job spec diff (-got +want):\n%s", diff)) + } + }) + + It("can re-create the Job with the same spec", func() { + oldJob := &batchv1.Job{} + Expect(hubUncachedReader.Get(ctx, jobKey, oldJob)).To(Succeed(), "Failed to get the Job") + + deleteObjectsAndWait([]client.Object{&batchv1.Job{ObjectMeta: metav1.ObjectMeta{Name: jobName, Namespace: appNamespaceName}}}, backgroundPropagation) + Expect(hubClient.Create(ctx, newJob(jobName, appNamespaceName))).To(Succeed(), "Failed to re-create the Job") + + // The re-created Job has a new UID, and so new controller UID labels. + recreatedJob := &batchv1.Job{} + Expect(hubUncachedReader.Get(ctx, jobKey, recreatedJob)).To(Succeed(), "Failed to get the re-created Job") + Expect(recreatedJob.UID).ToNot(Equal(oldJob.UID), "The re-created Job has the same UID") + verifyJobControllerUIDLabelsAdded(recreatedJob) + }) + + It("should return the existing snapshot as only the stripped labels have changed", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + verifyNoPrimarySnapshotAtIdx(placementPolicy, 1) + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 2) + objs = append(objs, + placementPolicy, + &batchv1.Job{ObjectMeta: metav1.ObjectMeta{Name: jobName, Namespace: appNamespaceName}}, + ) + deleteObjectsAndWait(objs, backgroundPropagation) + }) + }) +}) + +var _ = Describe("resource selection (cluster placement policies)", func() { + Context("invalid resource selectors", Ordered, func() { + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + // The names of resources that do not exist; the errors are expected to occur before any retrieval. + resourceName := utils.RandStr() + + testCases := []snapshotErrTestCase{ + { + desc: "the resource selector specifies a reserved Kubernetes namespace", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: "kube-system", + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"resource selector specifies a namespace that is ineligible for placement"}, + }, + { + desc: "the resource selector specifies a reserved KubeFleet namespace", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + // The namespace does not need to exist. + Namespace: fmt.Sprintf(fleetNamespaceNameTemplate, utils.RandStr()), + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"resource selector specifies a namespace that is ineligible for placement"}, + }, + { + desc: "the resource selector selects a namespace-scoped resource without specifying a namespace", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"a namespace must be specified for namespace-scoped resources"}, + }, + { + desc: "the resource selector selects a cluster-scoped resource with a namespace specified", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIGroup: rbacv1.GroupName, + APIVersion: "v1", + Kind: "ClusterRole", + Namespace: appNamespaceName, + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"namespace must not be specified for cluster-scoped resources"}, + }, + { + desc: "the resource selector selects pods", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Pod", + Namespace: appNamespaceName, + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleGVKErrMsg}, + }, + { + desc: "the resource selector specifies neither a name nor a label selector", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"neither name nor label selector is specified"}, + }, + { + desc: "the label selector uses an unknown operator", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + LabelSelector: &metav1.LabelSelector{ + MatchExpressions: []metav1.LabelSelectorRequirement{ + {Key: placementGroupLabelKey, Operator: "Foo", Values: []string{"bar"}}, + }, + }, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{"invalid label selector", "is not a valid label selector operator"}, + }, + { + desc: "the resource selector specifies an unknown kind", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMapp", + Namespace: appNamespaceName, + Name: resourceName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{noMatchForGVKErrMsg, `no matches for kind "ConfigMapp" in version "v1"`}, + }, + { + desc: "the selected resource does not exist", + placementPolicy: newClusterPlacementPolicy(placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + Name: configMapName, + }), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{resourceNotFoundErrMsg}, + }, + } + + BeforeAll(func() { + // Create the cluster placement policies; this also verifies that the API server accepts them. + for _, tc := range testCases { + Expect(hubClient.Create(ctx, tc.placementPolicy)).To(Succeed(), "Failed to create the cluster placement policy (%s)", tc.desc) + } + }) + + for _, tc := range testCases { + It(fmt.Sprintf("should return an error when %s", tc.desc), func() { + verifySnapshotErrTestCase(tc) + }) + } + + It("should return an unexpected error when no resource selectors are present", func() { + // The CRD requires at least one resource selector; use an in-memory cluster placement policy instead. + placementPolicy := newClusterPlacementPolicy() + err := snapshotResourcesAndExpectErr(placementPolicy) + verifyErrCategoryAndMsg(err, kferrors.ErrCategoryUnexpected, "no resource selectors are present") + }) + + AfterAll(func() { + objs := make([]client.Object, 0, len(testCases)) + for _, tc := range testCases { + objs = append(objs, tc.placementPolicy) + } + deleteObjectsAndWait(objs) + }) + }) + + Context("namespaces (name-based selection)", Ordered, func() { + namespaceName := fmt.Sprintf(namespaceNameTemplate, utils.RandStr()) + fleetNamespaceName := fmt.Sprintf(fleetNamespaceNameTemplate, utils.RandStr()) + + namespaceSelector := func(name string) placementv1alpha1.ResourceSelector { + return placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Namespace", + Name: name, + } + } + + testCases := []snapshotErrTestCase{ + { + desc: "a reserved Kubernetes namespace is selected by name", + placementPolicy: newClusterPlacementPolicy(namespaceSelector("kube-system")), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleNamespaceErrMsg}, + }, + { + desc: "the default namespace is selected by name", + placementPolicy: newClusterPlacementPolicy(namespaceSelector("default")), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleResourceErrMsg, + "the default namespace itself cannot be placed"}, + }, + { + desc: "a reserved KubeFleet namespace is selected by name", + placementPolicy: newClusterPlacementPolicy(namespaceSelector(fleetNamespaceName)), + wantErrCategory: kferrors.ErrCategoryUser, + wantErrMsgSubstrs: []string{ineligibleNamespaceErrMsg}, + }, + } + + placementPolicy := newClusterPlacementPolicy(namespaceSelector(namespaceName)) + + BeforeAll(func() { + createNamespace(namespaceName) + createNamespace(fleetNamespaceName) + + for _, tc := range testCases { + Expect(hubClient.Create(ctx, tc.placementPolicy)).To(Succeed(), "Failed to create the cluster placement policy (%s)", tc.desc) + } + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the cluster placement policy") + }) + + for _, tc := range testCases { + It(fmt.Sprintf("should return a user error when %s", tc.desc), func() { + verifySnapshotErrTestCase(tc) + }) + } + + It("can create a placement resource snapshot for an eligible namespace", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + if diff := cmp.Diff(snapshot.GetOwnerReferences(), wantPlacementPolicyOwnerRefs(placementPolicy)); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot owner references diff (-got +want):\n%s", diff)) + } + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Name: namespaceName, + APIVersion: "v1", + Kind: "Namespace", + }, + } + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + It("should strip the status of the namespace", func() { + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + resources := snapshot.GetSpec().Resources + Expect(resources).To(HaveLen(1), "Unexpected number of snapshotted resources") + Expect(manifestAsMap(resources[0].Manifest.Raw)).ToNot(HaveKey("status"), "The status of the namespace has not been stripped") + + // Note that the finalizers in the spec of the namespace are kept; only those in the metadata are stripped. + gotNamespace := &corev1.Namespace{} + Expect(json.Unmarshal(resources[0].Manifest.Raw, gotNamespace)).To(Succeed(), "Failed to unmarshal the namespace manifest") + wantNamespace := &corev1.Namespace{ + TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "Namespace"}, + ObjectMeta: metav1.ObjectMeta{ + Name: namespaceName, + Labels: map[string]string{corev1.LabelMetadataName: namespaceName}, + }, + Spec: corev1.NamespaceSpec{ + Finalizers: []corev1.FinalizerName{corev1.FinalizerKubernetes}, + }, + } + if diff := cmp.Diff(gotNamespace, wantNamespace); diff != "" { + Fail(fmt.Sprintf("Snapshotted namespace diff (-got +want):\n%s", diff)) + } + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 1) + objs = append(objs, placementPolicy) + for _, tc := range testCases { + objs = append(objs, tc.placementPolicy) + } + deleteObjectsAndWait(objs) + }) + }) + + Context("namespaces (label-based selection)", Ordered, func() { + namespaceName := fmt.Sprintf(namespaceNameTemplate, utils.RandStr()) + fleetNamespaceName := fmt.Sprintf(fleetNamespaceNameTemplate, utils.RandStr()) + + // Namespaces are labeled with their names automatically. + namespaceSelectorByNames := func(names ...string) placementv1alpha1.ResourceSelector { + return placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Namespace", + LabelSelector: &metav1.LabelSelector{ + MatchExpressions: []metav1.LabelSelectorRequirement{ + { + Key: corev1.LabelMetadataName, + Operator: metav1.LabelSelectorOpIn, + Values: names, + }, + }, + }, + } + } + + placementPolicy := newClusterPlacementPolicy(namespaceSelectorByNames( + "default", "kube-system", "kube-public", "kube-node-lease", fleetNamespaceName, namespaceName)) + // This cluster placement policy selects ineligible namespaces only. + allIneligiblePlacementPolicy := newClusterPlacementPolicy(namespaceSelectorByNames( + "default", "kube-system", fleetNamespaceName)) + + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Name: namespaceName, + APIVersion: "v1", + Kind: "Namespace", + }, + } + + BeforeAll(func() { + createNamespace(namespaceName) + createNamespace(fleetNamespaceName) + + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the cluster placement policy") + Expect(hubClient.Create(ctx, allIneligiblePlacementPolicy)).To(Succeed(), "Failed to create the cluster placement policy") + }) + + It("can compute the hash of selected resources", func() { + // The ineligible namespaces are skipped silently. + resources, _, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + verifySnapshottedResourceIdentifiers(resources, wantIdentifiers) + }) + + It("can create a placement resource snapshot with the eligible namespace only", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + It("can create an empty placement resource snapshot when every matched namespace is ineligible", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, allIneligiblePlacementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, allIneligiblePlacementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(allIneligiblePlacementPolicy, 0) + Expect(snapshot.GetSpec().Resources).To(BeEmpty(), "The placement resource snapshot is not empty") + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 1) + objs = append(objs, primarySnapshotObjs(allIneligiblePlacementPolicy, 1)...) + objs = append(objs, placementPolicy, allIneligiblePlacementPolicy) + deleteObjectsAndWait(objs) + }) + }) + + Context("cluster-scoped and namespaced resources", Ordered, func() { + namespaceName := fmt.Sprintf(namespaceNameTemplate, utils.RandStr()) + // The config maps in the two namespaces share the same name. + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + deploymentName := fmt.Sprintf(deploymentNameTemplate, utils.RandStr()) + clusterRoleName := fmt.Sprintf(clusterRoleNameTemplate, utils.RandStr()) + + // The resource selectors are deliberately listed in the reverse order of the snapshotted resources. + placementPolicy := newClusterPlacementPolicy( + placementv1alpha1.ResourceSelector{ + APIGroup: rbacv1.GroupName, + APIVersion: "v1", + Kind: "ClusterRole", + Name: clusterRoleName, + }, + placementv1alpha1.ResourceSelector{ + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + Namespace: appNamespaceName, + Name: deploymentName, + }, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "Namespace", + Name: namespaceName, + }, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: namespaceName, + Name: configMapName, + }, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: appNamespaceName, + Name: configMapName, + }, + ) + + // The resources are sorted by their unique IDs, i.e., `[API-GROUP]/[KIND]/[NAMESPACE]/[NAME]`; the namespace + // `app` sorts before `ns-...`. + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: appNamespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Namespace: namespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Name: namespaceName, + APIVersion: "v1", + Kind: "Namespace", + }, + { + Namespace: appNamespaceName, + Name: deploymentName, + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + }, + { + Name: clusterRoleName, + APIGroup: rbacv1.GroupName, + APIVersion: "v1", + Kind: "ClusterRole", + }, + } + + var wantHash string + + BeforeAll(func() { + createNamespace(namespaceName) + + deployment := newDeployment(deploymentName, appNamespaceName) + objs := []client.Object{ + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}, + Data: map[string]string{"key": "value"}, + }, + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: namespaceName}, + Data: map[string]string{"key": "value"}, + }, + deployment, + &rbacv1.ClusterRole{ + ObjectMeta: metav1.ObjectMeta{Name: clusterRoleName}, + }, + } + for _, obj := range objs { + Expect(hubClient.Create(ctx, obj)).To(Succeed(), "Failed to create object %T %s", obj, obj.GetName()) + } + waitForDeploymentsInCache(deployment) + Expect(hubClient.Create(ctx, placementPolicy)).To(Succeed(), "Failed to create the cluster placement policy") + }) + + It("can compute the hash of selected resources", func() { + resources, hash, err := resourceSnapshotManager.retrieveAndHashSelectedResources(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to retrieve and hash the selected resources") + verifySnapshottedResourceIdentifiers(resources, wantIdentifiers) + wantHash = hash + }) + + It("can create a primary and a secondary placement resource snapshot", func() { + // Five resources, with at most three resources per snapshot: one primary and one secondary snapshot. + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The placement resource snapshots are not up-to-date") + + wantNames := []string{ + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0), + uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1), + } + gotNames := make([]string, 0, len(gotSnapshots)) + for _, snapshot := range gotSnapshots { + gotNames = append(gotNames, snapshot.GetName()) + } + if diff := cmp.Diff(gotNames, wantNames); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot names diff (-got +want):\n%s", diff)) + } + }) + + It("should persist the snapshots with resources in the sorted order", func() { + primarySnapshot := getSnapshotFromCache("", uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0)) + verifySnapshotMetadata(primarySnapshot, placementPolicy, 0, 0, 2, wantHash) + verifySnapshottedResourceIdentifiers(primarySnapshot.GetSpec().Resources, wantIdentifiers[:3]) + + secondarySnapshot := getSnapshotFromCache("", uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1)) + verifySnapshotMetadata(secondarySnapshot, placementPolicy, 0, 1, 2, wantHash) + secondaryResources := secondarySnapshot.GetSpec().Resources + verifySnapshottedResourceIdentifiers(secondaryResources, wantIdentifiers[3:]) + + // Verify the manifest of the cluster-scoped cluster role. + gotClusterRole := &rbacv1.ClusterRole{} + Expect(json.Unmarshal(secondaryResources[1].Manifest.Raw, gotClusterRole)).To(Succeed(), "Failed to unmarshal the cluster role manifest") + wantClusterRole := &rbacv1.ClusterRole{ + TypeMeta: metav1.TypeMeta{APIVersion: rbacv1.SchemeGroupVersion.String(), Kind: "ClusterRole"}, + ObjectMeta: metav1.ObjectMeta{ + Name: clusterRoleName, + }, + } + if diff := cmp.Diff(gotClusterRole, wantClusterRole); diff != "" { + Fail(fmt.Sprintf("Snapshotted cluster role diff (-got +want):\n%s", diff)) + } + }) + + AfterAll(func() { + objs := []client.Object{ + placementResourceSnapshotObj("", uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0)), + placementResourceSnapshotObj("", uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.Name, placementPolicy.UID, 0, 1)), + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: appNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: namespaceName}}, + &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: deploymentName, Namespace: appNamespaceName}}, + &rbacv1.ClusterRole{ObjectMeta: metav1.ObjectMeta{Name: clusterRoleName}}, + } + deleteObjectsAndWait(objs) + }) + }) + + Context("resources in the default namespace", Ordered, func() { + // The default namespace itself cannot be placed, but the resources within it can. + groupLabels := map[string]string{placementGroupLabelKey: utils.RandStr()} + configMapName := fmt.Sprintf(configMapNameTemplate, utils.RandStr()) + labeledConfigMapName := configMapName + "-labeled" + + placementPolicy := newClusterPlacementPolicy( + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: defaultNamespaceName, + Name: configMapName, + }, + placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Namespace: defaultNamespaceName, + LabelSelector: &metav1.LabelSelector{MatchLabels: groupLabels}, + }, + ) + + // The resources are sorted by their unique IDs; the labeled config map has a longer name with the same prefix. + wantIdentifiers := []placementv1alpha1.ObjectReference{ + { + Namespace: defaultNamespaceName, + Name: configMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + { + Namespace: defaultNamespaceName, + Name: labeledConfigMapName, + APIVersion: "v1", + Kind: "ConfigMap", + }, + } + + BeforeAll(func() { + objs := []client.Object{ + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: defaultNamespaceName}, + Data: map[string]string{"key": "value"}, + }, + &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: labeledConfigMapName, Namespace: defaultNamespaceName, Labels: groupLabels}, + Data: map[string]string{"key": "value"}, + }, + placementPolicy, + } + for _, obj := range objs { + Expect(hubClient.Create(ctx, obj)).To(Succeed(), "Failed to create object %T %s", obj, obj.GetName()) + } + }) + + It("can create a placement resource snapshot with resources in the default namespace", func() { + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + verifyReturnedPrimarySnapshot(gotSnapshots, isUpToDate, err, placementPolicy, 0) + + snapshot := getPrimarySnapshotFromCache(placementPolicy, 0) + verifySnapshottedResourceIdentifiers(snapshot.GetSpec().Resources, wantIdentifiers) + }) + + AfterAll(func() { + objs := primarySnapshotObjs(placementPolicy, 1) + objs = append(objs, + placementPolicy, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: configMapName, Namespace: defaultNamespaceName}}, + &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: labeledConfigMapName, Namespace: defaultNamespaceName}}, + ) + deleteObjectsAndWait(objs) + }) + }) +}) diff --git a/pkg/v1/managers/placementresourcesnapshot/resources_test.go b/pkg/v1/managers/placementresourcesnapshot/resources_test.go new file mode 100644 index 000000000..ec13ad93e --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/resources_test.go @@ -0,0 +1,550 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementresourcesnapshot + +import ( + "context" + "encoding/json" + "errors" + "testing" + + "github.com/google/go-cmp/cmp" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + dynamicfake "k8s.io/client-go/dynamic/fake" + "k8s.io/client-go/kubernetes/scheme" + clienttesting "k8s.io/client-go/testing" + "k8s.io/utils/ptr" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kferrors "go.goms.io/fleet/pkg/utils/errors" + testinformer "go.goms.io/fleet/test/utils/informer" +) + +// TestRetrieveResourceByName tests the retrieveResourceByName method, in particular how it falls back to reading +// from the API server directly when the selected resource is not found in the cache. +func TestRetrieveResourceByName(t *testing.T) { + const ( + namespaceName = "app" + configMapName = "app-config" + selectedNamespaceName = "selected-ns" + + sourceCache = "cache" + sourceAPIServer = "api-server" + ) + configMapGVR := corev1.SchemeGroupVersion.WithResource("configmaps") + namespaceGVR := corev1.SchemeGroupVersion.WithResource("namespaces") + + // The objects carry their source in their data (or labels), so that the tests can tell where they come from. + newConfigMap := func(source string) *unstructured.Unstructured { + return &unstructured.Unstructured{Object: map[string]interface{}{ + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": map[string]interface{}{"name": configMapName, "namespace": namespaceName}, + "data": map[string]interface{}{"source": source}, + }} + } + newNamespace := func(source string) *unstructured.Unstructured { + return &unstructured.Unstructured{Object: map[string]interface{}{ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": map[string]interface{}{ + "name": selectedNamespaceName, + "labels": map[string]interface{}{"source": source}, + }, + }} + } + + placementPolicy := &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Namespace: namespaceName, Name: "policy"}, + } + clusterPlacementPolicy := &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: "cluster-policy"}, + } + configMapSelector := placementv1alpha1.ResourceSelector{APIVersion: "v1", Kind: "ConfigMap", Name: configMapName} + namespaceSelector := placementv1alpha1.ResourceSelector{APIVersion: "v1", Kind: "Namespace", Name: selectedNamespaceName} + + restMapper := meta.NewDefaultRESTMapper([]schema.GroupVersion{corev1.SchemeGroupVersion}) + restMapper.Add(corev1.SchemeGroupVersion.WithKind("ConfigMap"), meta.RESTScopeNamespace) + restMapper.Add(corev1.SchemeGroupVersion.WithKind("Namespace"), meta.RESTScopeRoot) + + testCases := []struct { + name string + placementPolicy placementv1alpha1.PlacementPolicyAccessor + selector placementv1alpha1.ResourceSelector + gvr schema.GroupVersionResource + informerSynced bool + cachedObjs []runtime.Object + cacheErr error + apiServerObjs []runtime.Object + apiServerErr error + // The object that the snapshotted resource should derive from; nil if an error is expected. + wantResource *unstructured.Unstructured + wantErrCategory kferrors.ErrCategory + wantNotFound bool + wantAPIServerGets int + }{ + { + name: "found in the cache", + placementPolicy: placementPolicy, + selector: configMapSelector, + gvr: configMapGVR, + informerSynced: true, + cachedObjs: []runtime.Object{newConfigMap(sourceCache)}, + apiServerObjs: []runtime.Object{newConfigMap(sourceAPIServer)}, + wantResource: newConfigMap(sourceCache), + wantAPIServerGets: 0, + }, + { + name: "not found in the (stale) cache, found in the API server", + placementPolicy: placementPolicy, + selector: configMapSelector, + gvr: configMapGVR, + informerSynced: true, + apiServerObjs: []runtime.Object{newConfigMap(sourceAPIServer)}, + wantResource: newConfigMap(sourceAPIServer), + wantAPIServerGets: 1, + }, + { + name: "cluster-scoped, not found in the (stale) cache, found in the API server", + placementPolicy: clusterPlacementPolicy, + selector: namespaceSelector, + gvr: namespaceGVR, + informerSynced: true, + apiServerObjs: []runtime.Object{newNamespace(sourceAPIServer)}, + wantResource: newNamespace(sourceAPIServer), + wantAPIServerGets: 1, + }, + { + name: "found in neither the cache nor the API server", + placementPolicy: placementPolicy, + selector: configMapSelector, + gvr: configMapGVR, + informerSynced: true, + wantErrCategory: kferrors.ErrCategoryUser, + wantNotFound: true, + wantAPIServerGets: 1, + }, + { + name: "cluster-scoped, found in neither the cache nor the API server", + placementPolicy: clusterPlacementPolicy, + selector: namespaceSelector, + gvr: namespaceGVR, + informerSynced: true, + wantErrCategory: kferrors.ErrCategoryUser, + wantNotFound: true, + wantAPIServerGets: 1, + }, + { + name: "cache read fails with an error other than not found", + placementPolicy: placementPolicy, + selector: configMapSelector, + gvr: configMapGVR, + informerSynced: true, + cacheErr: errors.New("the cache is unavailable"), + apiServerObjs: []runtime.Object{newConfigMap(sourceAPIServer)}, + wantErrCategory: kferrors.ErrCategoryAPIServer, + wantAPIServerGets: 0, + }, + { + name: "not found in the cache, API server read fails with an error other than not found", + placementPolicy: placementPolicy, + selector: configMapSelector, + gvr: configMapGVR, + informerSynced: true, + apiServerErr: apierrors.NewServiceUnavailable("the API server is unavailable"), + wantErrCategory: kferrors.ErrCategoryAPIServer, + wantAPIServerGets: 1, + }, + { + name: "informer not synced, found in the API server", + placementPolicy: placementPolicy, + selector: configMapSelector, + gvr: configMapGVR, + informerSynced: false, + cachedObjs: []runtime.Object{newConfigMap(sourceCache)}, + apiServerObjs: []runtime.Object{newConfigMap(sourceAPIServer)}, + wantResource: newConfigMap(sourceAPIServer), + wantAPIServerGets: 1, + }, + { + name: "informer not synced, not found in the API server", + placementPolicy: placementPolicy, + selector: configMapSelector, + gvr: configMapGVR, + informerSynced: false, + wantErrCategory: kferrors.ErrCategoryUser, + wantNotFound: true, + wantAPIServerGets: 1, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + dynamicClient := dynamicfake.NewSimpleDynamicClient(scheme.Scheme, tc.apiServerObjs...) + if tc.apiServerErr != nil { + dynamicClient.PrependReactor("get", tc.gvr.Resource, func(_ clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, tc.apiServerErr + }) + } + m := &Manager{ + hubDynamicClient: dynamicClient, + hubDynamicInformerManager: &testinformer.FakeManager{ + Listers: map[schema.GroupVersionResource]*testinformer.FakeLister{ + tc.gvr: {Objects: tc.cachedObjs, Err: tc.cacheErr}, + }, + InformerSynced: ptr.To(tc.informerSynced), + }, + restMapper: restMapper, + } + + got, err := m.retrieveResourceByName(context.Background(), tc.placementPolicy, tc.selector) + + apiServerGets := 0 + for _, action := range dynamicClient.Actions() { + if action.GetVerb() == "get" { + apiServerGets++ + } + } + if apiServerGets != tc.wantAPIServerGets { + t.Errorf("retrieveResourceByName() made %d reads from the API server, want %d", apiServerGets, tc.wantAPIServerGets) + } + + if tc.wantResource == nil { + if err == nil { + t.Fatalf("retrieveResourceByName() = %v, want an error", got) + } + if gotCategory := kferrors.Category(err); gotCategory != tc.wantErrCategory { + t.Errorf("retrieveResourceByName() error category = %q, want %q (err: %v)", gotCategory, tc.wantErrCategory, err) + } + if gotNotFound := apierrors.IsNotFound(err); gotNotFound != tc.wantNotFound { + t.Errorf("retrieveResourceByName() apierrors.IsNotFound(err) = %t, want %t (err: %v)", gotNotFound, tc.wantNotFound, err) + } + return + } + + if err != nil { + t.Fatalf("retrieveResourceByName() = %v, want no error", err) + } + want, err := snapshotResource(tc.wantResource) + if err != nil { + t.Fatalf("snapshotResource() = %v, want no error", err) + } + if diff := cmp.Diff(got, want); diff != "" { + t.Errorf("retrieveResourceByName() mismatch (-got +want):\n%s", diff) + } + }) + } +} + +// TestSnapshotResource tests the snapshotResource function. +func TestSnapshotResource(t *testing.T) { + const ( + namespaceName = "app" + svcName = "svc" + jobName = "job" + controllerUID = "controller-uid-value" + ) + + svcIdentifier := placementv1alpha1.ObjectReference{ + Namespace: namespaceName, + Name: svcName, + APIGroup: "", + APIVersion: "v1", + Kind: "Service", + } + jobIdentifier := placementv1alpha1.ObjectReference{ + Namespace: namespaceName, + Name: jobName, + APIGroup: "batch", + APIVersion: "v1", + Kind: "Job", + } + + toRawExtension := func(t *testing.T, obj map[string]any) runtime.RawExtension { + t.Helper() + raw, err := json.Marshal(obj) + if err != nil { + t.Fatalf("json.Marshal() = %v, want no error", err) + } + return runtime.RawExtension{Raw: raw} + } + + // Compare manifests semantically rather than byte by byte for more readable diffs. + rawExtensionTransformer := cmp.Transformer("RawExtensionToMap", func(r runtime.RawExtension) map[string]any { + m := map[string]any{} + if err := json.Unmarshal(r.Raw, &m); err != nil { + return map[string]any{"unmarshalError": err.Error()} + } + return m + }) + + testCases := []struct { + name string + resource *unstructured.Unstructured + wantManifest map[string]any + wantID placementv1alpha1.ObjectReference + }{ + { + name: "service with hub-allocated cluster IPs and node ports", + resource: &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "v1", + "kind": "Service", + "metadata": map[string]any{ + "name": svcName, + "namespace": namespaceName, + }, + "spec": map[string]any{ + "type": "LoadBalancer", + "clusterIP": "10.0.0.1", + "clusterIPs": []any{"10.0.0.1"}, + "healthCheckNodePort": int64(30000), + "ports": []any{ + map[string]any{ + "name": "http", + "port": int64(80), + "protocol": "TCP", + "nodePort": int64(30080), + }, + map[string]any{ + "name": "https", + "port": int64(443), + "protocol": "TCP", + "nodePort": int64(30443), + }, + }, + "selector": map[string]any{ + "app": "nginx", + }, + }, + "status": map[string]any{ + "loadBalancer": map[string]any{}, + }, + }, + }, + wantManifest: map[string]any{ + "apiVersion": "v1", + "kind": "Service", + "metadata": map[string]any{ + "name": svcName, + "namespace": namespaceName, + }, + "spec": map[string]any{ + "type": "LoadBalancer", + "ports": []any{ + map[string]any{ + "name": "http", + "port": int64(80), + "protocol": "TCP", + }, + map[string]any{ + "name": "https", + "port": int64(443), + "protocol": "TCP", + }, + }, + "selector": map[string]any{ + "app": "nginx", + }, + }, + }, + wantID: svcIdentifier, + }, + { + name: "headless service", + resource: &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "v1", + "kind": "Service", + "metadata": map[string]any{ + "name": svcName, + "namespace": namespaceName, + }, + "spec": map[string]any{ + "clusterIP": "None", + "clusterIPs": []any{"None"}, + "ports": []any{ + map[string]any{ + "port": int64(80), + "protocol": "TCP", + }, + }, + }, + }, + }, + wantManifest: map[string]any{ + "apiVersion": "v1", + "kind": "Service", + "metadata": map[string]any{ + "name": svcName, + "namespace": namespaceName, + }, + "spec": map[string]any{ + "clusterIP": "None", + "clusterIPs": []any{"None"}, + "ports": []any{ + map[string]any{ + "port": int64(80), + "protocol": "TCP", + }, + }, + }, + }, + wantID: svcIdentifier, + }, + { + name: "job without manual selector", + resource: &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "batch/v1", + "kind": "Job", + "metadata": map[string]any{ + "name": jobName, + "namespace": namespaceName, + }, + "spec": map[string]any{ + "selector": map[string]any{ + "matchLabels": map[string]any{ + "controller-uid": controllerUID, + "batch.kubernetes.io/controller-uid": controllerUID, + }, + }, + "template": map[string]any{ + "metadata": map[string]any{ + "creationTimestamp": nil, + "labels": map[string]any{ + "app": "job", + "controller-uid": controllerUID, + "batch.kubernetes.io/controller-uid": controllerUID, + }, + }, + "spec": map[string]any{ + "restartPolicy": "Never", + }, + }, + }, + }, + }, + wantManifest: map[string]any{ + "apiVersion": "batch/v1", + "kind": "Job", + "metadata": map[string]any{ + "name": jobName, + "namespace": namespaceName, + }, + "spec": map[string]any{ + "selector": map[string]any{ + "matchLabels": map[string]any{}, + }, + "template": map[string]any{ + "metadata": map[string]any{ + "labels": map[string]any{ + "app": "job", + }, + }, + "spec": map[string]any{ + "restartPolicy": "Never", + }, + }, + }, + }, + wantID: jobIdentifier, + }, + { + name: "job with manual selector", + resource: &unstructured.Unstructured{ + Object: map[string]any{ + "apiVersion": "batch/v1", + "kind": "Job", + "metadata": map[string]any{ + "name": jobName, + "namespace": namespaceName, + }, + "spec": map[string]any{ + "manualSelector": true, + "selector": map[string]any{ + "matchLabels": map[string]any{ + "controller-uid": controllerUID, + }, + }, + "template": map[string]any{ + "metadata": map[string]any{ + "labels": map[string]any{ + "controller-uid": controllerUID, + }, + }, + }, + }, + }, + }, + wantManifest: map[string]any{ + "apiVersion": "batch/v1", + "kind": "Job", + "metadata": map[string]any{ + "name": jobName, + "namespace": namespaceName, + }, + "spec": map[string]any{ + "manualSelector": true, + "selector": map[string]any{ + "matchLabels": map[string]any{ + "controller-uid": controllerUID, + }, + }, + "template": map[string]any{ + "metadata": map[string]any{ + "labels": map[string]any{ + "controller-uid": controllerUID, + }, + }, + }, + }, + }, + wantID: jobIdentifier, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + original := tc.resource.DeepCopy() + + got, err := snapshotResource(tc.resource) + if err != nil { + t.Fatalf("snapshotResource() = %v, want no error", err) + } + + want := placementv1alpha1.SnapshottedResource{ + Identifier: tc.wantID, + Manifest: toRawExtension(t, tc.wantManifest), + } + if diff := cmp.Diff(got, want, rawExtensionTransformer); diff != "" { + t.Errorf("snapshotResource() mismatch (-got, +want):\n%s", diff) + } + + // The input resource must not be modified. + if diff := cmp.Diff(tc.resource, original); diff != "" { + t.Errorf("snapshotResource() modified the input resource (-got, +want):\n%s", diff) + } + }) + } +} diff --git a/pkg/v1/managers/placementresourcesnapshot/snapshots.go b/pkg/v1/managers/placementresourcesnapshot/snapshots.go index 20338c769..300c5ac90 100644 --- a/pkg/v1/managers/placementresourcesnapshot/snapshots.go +++ b/pkg/v1/managers/placementresourcesnapshot/snapshots.go @@ -17,6 +17,7 @@ limitations under the License. package placementresourcesnapshot import ( + "context" "crypto/sha256" "fmt" "strconv" @@ -26,6 +27,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" ) @@ -53,6 +55,7 @@ func primaryPlacementResourceSnapshot( } annotations := map[string]string{ placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: resourceHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: ownerPlacementPolicy.GetName(), } var primarySnapshot placementv1alpha1.PlacementResourceSnapshotAccessor @@ -103,6 +106,7 @@ func secondaryPlacementResourceSnapshot( } annotations := map[string]string{ placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: resourceHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: ownerPlacementPolicy.GetName(), } var secondarySnapshot placementv1alpha1.PlacementResourceSnapshotAccessor @@ -152,3 +156,32 @@ func placementPolicyOwnerLabelVal(placementPolicy placementv1alpha1.PlacementPol prefixLen := placementPolicyOwnerLabelValLenLimit - placementPolicyOwnerLabelHashLen - 1 return fmt.Sprintf("%s-%s", name[:prefixLen], hash) } + +// listPlacementResourceSnapshots lists cluster-scoped or namespace-scoped placement resource snapshots using the +// given list options, and returns them as accessors. +// +// Note that this method reads from the cache; field matchers (custom field indexes) can be used in the list options. +func (m *Manager) listPlacementResourceSnapshots(ctx context.Context, + isClusterScoped bool, listOpts []client.ListOption) ([]placementv1alpha1.PlacementResourceSnapshotAccessor, error) { + var snapshots []placementv1alpha1.PlacementResourceSnapshotAccessor + if isClusterScoped { + snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} + if err := m.hubClient.List(ctx, snapshotList, listOpts...); err != nil { + return nil, errors.NewAPIServerError(err, "failed to list cluster placement resource snapshots", true) + } + snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotList.Items)) + for i := range snapshotList.Items { + snapshots[i] = &snapshotList.Items[i] + } + } else { + snapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} + if err := m.hubClient.List(ctx, snapshotList, listOpts...); err != nil { + return nil, errors.NewAPIServerError(err, "failed to list placement resource snapshots", true) + } + snapshots = make([]placementv1alpha1.PlacementResourceSnapshotAccessor, len(snapshotList.Items)) + for i := range snapshotList.Items { + snapshots[i] = &snapshotList.Items[i] + } + } + return snapshots, nil +} diff --git a/pkg/v1/managers/placementresourcesnapshot/suite_test.go b/pkg/v1/managers/placementresourcesnapshot/suite_test.go new file mode 100644 index 000000000..04d27439e --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/suite_test.go @@ -0,0 +1,202 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementresourcesnapshot + +import ( + "context" + "flag" + "path/filepath" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/apiutil" + "sigs.k8s.io/controller-runtime/pkg/envtest" + "sigs.k8s.io/controller-runtime/pkg/handler" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + "sigs.k8s.io/controller-runtime/pkg/reconcile" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/informer" + "go.goms.io/fleet/pkg/utils/resourceeligibility" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +const ( + // The number of mutex slots used by the manager under test. + testMuSlotCnt = 256 + + appNamespaceName = "app" +) + +var ( + hubCfg *rest.Config + hubEnv *envtest.Environment + hubMgr ctrl.Manager + // hubClient is the client from the controller manager; reads are served from its cache, which + // matches how the manager under test reads objects (and is what makes the field indexes usable). + hubClient client.Client + // hubUncachedReader reads straight from the API server. + hubUncachedReader client.Reader + + resourceSnapshotManager *Manager + + ctx context.Context + cancel context.CancelFunc + // mgrStopped is closed once the controller manager has fully stopped. + mgrStopped chan struct{} +) + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Placement Resource Snapshot Manager Integration Test Suite") +} + +// noOpReconciler is a no-op controller whose only purpose is to have the controller manager set up +// informers (and thus populate the cache) for the objects the manager under test reads from the cache. +type noOpReconciler struct{} + +// Reconcile does nothing. +func (r *noOpReconciler) Reconcile(_ context.Context, _ reconcile.Request) (reconcile.Result, error) { + return reconcile.Result{}, nil +} + +func (r *noOpReconciler) SetupWithManager(mgr ctrl.Manager) error { + return ctrl.NewControllerManagedBy(mgr). + Named("placement-resource-snapshot-manager-test-noop"). + For(&placementv1alpha1.PlacementPolicy{}). + Watches(&placementv1alpha1.ClusterPlacementPolicy{}, &handler.EnqueueRequestForObject{}). + Watches(&placementv1alpha1.PlacementResourceSnapshot{}, &handler.EnqueueRequestForObject{}). + Watches(&placementv1alpha1.ClusterPlacementResourceSnapshot{}, &handler.EnqueueRequestForObject{}). + Watches(&placementv1alpha1.PlacementBinding{}, &handler.EnqueueRequestForObject{}). + Watches(&placementv1alpha1.ClusterPlacementBinding{}, &handler.EnqueueRequestForObject{}). + Complete(r) +} + +var _ = BeforeSuite(func() { + ctx, cancel = context.WithCancel(context.TODO()) + + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + logger := textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))) + klog.SetLogger(logger) + ctrl.SetLogger(logger) + + By("Bootstrapping the test environment") + hubEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{filepath.Join("../../../../", "config", "crd", "bases")}, + ErrorIfCRDPathMissing: true, + } + + var err error + hubCfg, err = hubEnv.Start() + Expect(err).ToNot(HaveOccurred()) + Expect(hubCfg).ToNot(BeNil()) + + By("Setting up the scheme") + Expect(placementv1alpha1.AddToScheme(scheme.Scheme)).To(Succeed()) + + By("Setting up the controller manager") + hubMgr, err = ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{ + BindAddress: "0", + }, + }) + Expect(err).ToNot(HaveOccurred()) + hubClient = hubMgr.GetClient() + hubUncachedReader = hubMgr.GetAPIReader() + + By("Creating the test namespace") + Expect(hubClient.Create(ctx, &corev1.Namespace{ + ObjectMeta: metav1.ObjectMeta{Name: appNamespaceName}, + })).To(Succeed()) + + By("Setting up the field indexes") + // The indexes must be set up before the manager starts. + Expect(fieldindexers.SetupWithHubAgentControllerManager(ctx, hubMgr)).To(Succeed()) + + By("Setting up the placement resource snapshot manager") + hubDynamicClient, err := dynamic.NewForConfig(hubCfg) + Expect(err).ToNot(HaveOccurred()) + hubDynamicInformerManager := informer.NewInformerManager(hubDynamicClient, 0, ctx.Done()) + hubDynamicInformerManager.CreateInformerForResource(informer.APIResourceMeta{ + GroupVersionKind: appsv1.SchemeGroupVersion.WithKind("Deployment"), + GroupVersionResource: appsv1.SchemeGroupVersion.WithResource("deployments"), + IsClusterScoped: false, + }) + hubDynamicInformerManager.Start() + hubDynamicInformerManager.WaitForCacheSync() + + By("Setting up the baseline resource eligibility checker") + httpClient, err := rest.HTTPClientFor(hubCfg) + Expect(err).ToNot(HaveOccurred()) + restMapper, err := apiutil.NewDynamicRESTMapper(hubCfg, httpClient) + Expect(err).ToNot(HaveOccurred()) + baselineResourceEligibilityChecker := resourceeligibility.Baseline(restMapper) + + resourceSnapshotManager, err = New(hubMgr, + hubDynamicClient, + hubDynamicInformerManager, + baselineResourceEligibilityChecker, + hubMgr.GetRESTMapper(), + testMuSlotCnt, + // Set maxPerSnapshotResourceDataSizeBytes and maxPerSnapshotResourceCnt to 10000 and 3 respectively to allow + // simpler test setup. + 10000, + 3, + ) + Expect(err).ToNot(HaveOccurred()) + Expect(hubMgr.Add(resourceSnapshotManager)).To(Succeed()) + + By("Setting up the no-op controller") + Expect((&noOpReconciler{}).SetupWithManager(hubMgr)).To(Succeed()) + + By("Starting the controller manager") + mgrStopped = make(chan struct{}) + go func() { + defer GinkgoRecover() + defer close(mgrStopped) + Expect(hubMgr.Start(ctx)).To(Succeed(), "Failed to run the controller manager") + }() +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + + cancel() + // Wait for the manager to stop first; otherwise the API server may not shut down in time as + // the manager's watches are still open. + Eventually(mgrStopped, "30s").Should(BeClosed()) + + By("Tearing down the test environment") + Expect(hubEnv.Stop()).To(Succeed()) +}) diff --git a/pkg/v1/managers/placementresourcesnapshot/uniquename.go b/pkg/v1/managers/placementresourcesnapshot/uniquename/uniquename.go similarity index 55% rename from pkg/v1/managers/placementresourcesnapshot/uniquename.go rename to pkg/v1/managers/placementresourcesnapshot/uniquename/uniquename.go index ecd752b68..d996fd6a4 100644 --- a/pkg/v1/managers/placementresourcesnapshot/uniquename.go +++ b/pkg/v1/managers/placementresourcesnapshot/uniquename/uniquename.go @@ -14,13 +14,17 @@ See the License for the specific language governing permissions and limitations under the License. */ -package placementresourcesnapshot +// Package uniquename generates the names of placement resource snapshots, which are unique across placement +// policies (including re-created ones of the same name) and always valid object names. +package uniquename import ( "crypto/sha256" "fmt" "strconv" "strings" + + "k8s.io/apimachinery/pkg/types" ) const ( @@ -29,59 +33,69 @@ const ( ) const ( - // The name format for primary placement resource snapshots. Typically, these snapshots are named using the - // format: + // The name format for primary placement resource snapshots is // - // `[PLACEMENT-POLICY-NAME]-resource-snapshot-[SNAPSHOT-INDEX]`, + // `[PLACEMENT-POLICY-NAME]-resource-snapshot-[SNAPSHOT-INDEX]-[HASH]`, // - // where `[PLACEMENT-POLICY-NAME]` is the name of the owner placement policy, and - // `[SNAPSHOT-INDEX]` is the monotonically increasing index of the snapshot. + // where: // - // If the name becomes too long (> 251 characters) or contains dots, KubeFleet will drop the dots and truncate the - // placement policy name segment and the snapshot index segment as appropriate and add a hash suffix, i.e., + // * `[PLACEMENT-POLICY-NAME]` is the name of the owner placement policy; + // * `[SNAPSHOT-INDEX]` is the monotonically increasing index of the snapshot; + // * `[HASH]` is the first few characters of the hash of the value + // `[PLACEMENT-POLICY-NAME]-resource-snapshot-[SNAPSHOT-INDEX]-[The UID of the placement policy]`. // - // `[PLACEMENT-POLICY-NAME-TRUNCATED]-resource-snapshot-[SNAPSHOT-INDEX-TRUNCATED]-[HASH]`, + // The UID part ensures that snapshot names are never reused across placement policies of the same name + // (e.g., when a placement policy is deleted and re-created, and the snapshot index restarts from 0). + // KubeFleet tracks resource rollouts by snapshot names (placement bindings, work objects, and placement + // policy statuses all refer to a primary snapshot by its name); a reused name with different contents + // could be mistaken as already synchronized. Note that snapshots left over from previous placement policies + // are detected via owner references, not via names. // - // where `[HASH]` is the first few characters of the hash of the value - // `[PLACEMENT-POLICY-NAME]-resource-snapshot-[SNAPSHOT-INDEX]`. - PrimaryPlacementResourceSnapshotNameFmt = "%s-resource-snapshot-%d" + // If the name becomes too long (> 251 characters) or contains dots, KubeFleet will drop the dots and truncate the + // placement policy name segment and the snapshot index segment as appropriate. PrimaryPlacementResourceSnapshotNameWithHashFmt = "%s-resource-snapshot-%s-%s" - // The name format for secondary placement resource snapshots. Typically, these snapshots are named using the - // format: + // The name format for secondary placement resource snapshots is // - // `[PLACEMENT-POLICY-NAME]-resource-snapshot-[SNAPSHOT-INDEX]-[SNAPSHOT-SUB-INDEX]`, + // `[PLACEMENT-POLICY-NAME]-resource-snapshot-[SNAPSHOT-INDEX]-[SNAPSHOT-SUB-INDEX]-[HASH]`, // - // where `[PLACEMENT-POLICY-NAME]` is the name of the owner placement policy, - // `[SNAPSHOT-INDEX]` is the monotonically increasing index of the snapshot, and - // `[SNAPSHOT-SUB-INDEX]` is the monotonically increasing sub-index of the snapshot. + // where: // - // If the name becomes too long (> 251 characters) or contains dots, KubeFleet will drop the dots and truncate the - // placement policy name segment and the `[SNAPSHOT-INDEX]-[SNAPSHOT-SUB-INDEX]` segment as appropriate and add - // a hash suffix, i.e., + // * `[PLACEMENT-POLICY-NAME]` is the name of the owner placement policy; + // * `[SNAPSHOT-INDEX]` is the monotonically increasing index of the snapshot; + // * `[SNAPSHOT-SUB-INDEX]` is the monotonically increasing sub-index of the snapshot; + // * `[HASH]` is the first few characters of the hash of the value + // `[PLACEMENT-POLICY-NAME]-resource-snapshot-[SNAPSHOT-INDEX]-[SNAPSHOT-SUB-INDEX]-[The UID of the placement policy]`. // - // `[PLACEMENT-POLICY-NAME-TRUNCATED]-resource-snapshot-[INDEX-TRUNCATED]-[HASH]`, + // As with primary snapshots, the UID part ensures that snapshot names are never reused across placement + // policies of the same name (e.g., when a placement policy is deleted and re-created), so that a secondary + // snapshot never shares its name with one left over from a previous placement policy. // - // where `[HASH]` is the first few characters of the hash of the value - // `[PLACEMENT-POLICY-NAME]-resource-snapshot-[SNAPSHOT-INDEX]-[SNAPSHOT-SUB-INDEX]`. - SecondaryPlacementResourceSnapshotNameFmt = "%s-resource-snapshot-%d-%d" + // If the name becomes too long (> 251 characters) or contains dots, KubeFleet will drop the dots and truncate the + // placement policy name segment and the `[SNAPSHOT-INDEX]-[SNAPSHOT-SUB-INDEX]` segment as appropriate. SecondaryPlacementResourceSnapshotNameWithHashFmt = "%s-resource-snapshot-%s-%s" ) -// uniqueNameForPrimaryPlacementResourceSnapshot generates a unique name for a primary placement resource snapshot. -func uniqueNameForPrimaryPlacementResourceSnapshot(placementPolicyName string, idx int) string { - name := fmt.Sprintf(PrimaryPlacementResourceSnapshotNameFmt, placementPolicyName, idx) - if len(name) <= nameLenLimit && !strings.Contains(name, ".") { - return name - } - - // The name is too long or contains dots; sanitize and truncate the placement policy name segment and append a hash suffix. - // The hash is computed over the full (untruncated) name. +// PrimaryPlacementResourceSnapshotName returns the name of the primary placement resource snapshot at the given +// index for a placement policy. +func PrimaryPlacementResourceSnapshotName(placementPolicyName string, placementPolicyUID types.UID, idx int) string { + // The hash is computed over the full (unsanitized and untruncated) placement policy name, the snapshot index, + // and the UID of the placement policy. // // Note that here only the first few (12) characters are kept. This does lead to increased risk of name // collisions, but the chances still remain extremely low. If such a collision does occur, manual intervention // is needed for resolution. - hash := fmt.Sprintf("%x", sha256.Sum256([]byte(name)))[:hashSegLen] + hashInput := fmt.Sprintf("%s-resource-snapshot-%d-%s", placementPolicyName, idx, placementPolicyUID) + hash := fmt.Sprintf("%x", sha256.Sum256([]byte(hashInput)))[:hashSegLen] + + idxStr := strconv.Itoa(idx) + name := fmt.Sprintf(PrimaryPlacementResourceSnapshotNameWithHashFmt, placementPolicyName, idxStr, hash) + if len(name) <= nameLenLimit && !strings.Contains(name, ".") { + return name + } + + // The name is too long or contains dots; sanitize and truncate the placement policy name segment and the + // snapshot index segment. // Compute how many characters are left for the two variable segments (the placement policy name and the // snapshot index); the index segment gets the space it needs, and the placement policy name segment takes @@ -102,7 +116,7 @@ func uniqueNameForPrimaryPlacementResourceSnapshot(placementPolicyName string, i truncatedPlacementPolicyName = truncatedPlacementPolicyName[:availableLenForNameSeg] } - truncatedIdxStr := strconv.Itoa(idx) + truncatedIdxStr := idxStr if len(truncatedIdxStr) > availableLenForIdxSeg { truncatedIdxStr = truncatedIdxStr[:availableLenForIdxSeg] } @@ -110,19 +124,26 @@ func uniqueNameForPrimaryPlacementResourceSnapshot(placementPolicyName string, i return fmt.Sprintf(PrimaryPlacementResourceSnapshotNameWithHashFmt, truncatedPlacementPolicyName, truncatedIdxStr, hash) } -func uniqueNameForSecondaryPlacementResourceSnapshot(placementPolicyName string, idx int, subIdx int) string { - name := fmt.Sprintf(SecondaryPlacementResourceSnapshotNameFmt, placementPolicyName, idx, subIdx) - if len(name) <= nameLenLimit && !strings.Contains(name, ".") { - return name - } - - // The name is too long or contains dots; sanitize and truncate the placement policy name segment and append a hash suffix. - // The hash is computed over the full (untruncated) name. +// SecondaryPlacementResourceSnapshotName returns the name of the secondary placement resource snapshot at the given +// index and sub-index for a placement policy. +func SecondaryPlacementResourceSnapshotName(placementPolicyName string, placementPolicyUID types.UID, idx, subIdx int) string { + // The hash is computed over the full (unsanitized and untruncated) placement policy name, the snapshot index, + // the snapshot sub-index, and the UID of the placement policy. // // Note that here only the first few (12) characters are kept. This does lead to increased risk of name // collisions, but the chances still remain extremely low. If such a collision does occur, manual intervention // is needed for resolution. - hash := fmt.Sprintf("%x", sha256.Sum256([]byte(name)))[:hashSegLen] + hashInput := fmt.Sprintf("%s-resource-snapshot-%d-%d-%s", placementPolicyName, idx, subIdx, placementPolicyUID) + hash := fmt.Sprintf("%x", sha256.Sum256([]byte(hashInput)))[:hashSegLen] + + idxStr := fmt.Sprintf("%d-%d", idx, subIdx) + name := fmt.Sprintf(SecondaryPlacementResourceSnapshotNameWithHashFmt, placementPolicyName, idxStr, hash) + if len(name) <= nameLenLimit && !strings.Contains(name, ".") { + return name + } + + // The name is too long or contains dots; sanitize and truncate the placement policy name segment and the + // combined snapshot index/sub-index segment. // Compute how many characters are left for the two variable segments (the placement policy name and the // combined snapshot index/sub-index segment); the index segment gets the space it needs, and the placement @@ -141,7 +162,7 @@ func uniqueNameForSecondaryPlacementResourceSnapshot(placementPolicyName string, truncatedPlacementPolicyName = truncatedPlacementPolicyName[:availableLenForNameSeg] } - truncatedIdxStr := fmt.Sprintf("%d-%d", idx, subIdx) + truncatedIdxStr := idxStr if len(truncatedIdxStr) > availableLenForIdxSeg { truncatedIdxStr = truncatedIdxStr[:availableLenForIdxSeg] } diff --git a/pkg/v1/managers/placementresourcesnapshot/uniquename/uniquename_test.go b/pkg/v1/managers/placementresourcesnapshot/uniquename/uniquename_test.go new file mode 100644 index 000000000..5b2416256 --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/uniquename/uniquename_test.go @@ -0,0 +1,324 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package uniquename + +import ( + "fmt" + "strings" + "testing" + + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/validation" +) + +const ( + shortPolicyName = "policy" + policyUID types.UID = "uid-1" + otherPolicyUID types.UID = "uid-2" +) + +// dottedName returns a name of the given length with a dot as every 10th character, starting from the given offset +// (0-9), except for the last character, e.g., dottedName(23, 9) = "abcdefghi.klmnopqrs.uvw". +func dottedName(length, dotOffset int) string { + var sb strings.Builder + for idx := 0; idx < length; idx++ { + if idx%10 == dotOffset && idx != length-1 { + sb.WriteByte('.') + continue + } + sb.WriteByte(byte('a' + idx%26)) + } + return sb.String() +} + +// verifyName verifies that a generated name is as wanted, within the length limit, and a valid DNS subdomain. +func verifyName(t *testing.T, call, gotName, wantName string) { + t.Helper() + + if gotName != wantName { + t.Errorf("%s = %q, want %q", call, gotName, wantName) + } + if len(gotName) > nameLenLimit { + t.Errorf("len(%s) = %d, want <= %d", call, len(gotName), nameLenLimit) + } + if errs := validation.IsDNS1123Subdomain(gotName); len(errs) > 0 { + t.Errorf("%s = %q, which is not a valid DNS subdomain: %v", call, gotName, errs) + } +} + +// TestPrimaryPlacementResourceSnapshotName tests the PrimaryPlacementResourceSnapshotName function. +// +// The hashes below are the first 12 hex characters of the SHA-256 digests of the hash inputs documented in +// uniquename.go (e.g., `policy-resource-snapshot-0-uid-1`), computed independently of the implementation. +func TestPrimaryPlacementResourceSnapshotName(t *testing.T) { + // The dot-free form of a dotted name, truncated to 209 characters (251 - len("-resource-snapshot-") - 10 - 1 - 12), + // where 10 is the room reserved for the snapshot index. + dottedPolicyName := dottedName(253, 5) + wantTruncatedDottedPolicyName := strings.ReplaceAll(dottedPolicyName, ".", "")[:209] + + testCases := []struct { + name string + policyName string + policyUID types.UID + idx int + wantName string + }{ + { + name: "short name", + policyName: shortPolicyName, + policyUID: policyUID, + idx: 0, + wantName: "policy-resource-snapshot-0-dcb49e6b8793", + }, + { + // A re-created placement policy of the same name gets different snapshot names. + name: "short name, different placement policy UID", + policyName: shortPolicyName, + policyUID: otherPolicyUID, + idx: 0, + wantName: "policy-resource-snapshot-0-50e127870132", + }, + { + name: "short name, multi-digit index", + policyName: shortPolicyName, + policyUID: policyUID, + idx: 12, + wantName: "policy-resource-snapshot-12-3f0897595cd2", + }, + { + // Dots are removed from the name segment, but the hash is computed over the original name. + name: "name with dots", + policyName: "my.policy.v1", + policyUID: policyUID, + idx: 0, + wantName: "mypolicyv1-resource-snapshot-0-df7214812235", + }, + { + name: "exactly at the length limit (not truncated)", + policyName: strings.Repeat("a", 218), + policyUID: policyUID, + idx: 0, + wantName: strings.Repeat("a", 218) + "-resource-snapshot-0-7e773e2d43c3", + }, + { + name: "one character over the length limit (truncated)", + policyName: strings.Repeat("a", 219), + policyUID: policyUID, + idx: 0, + wantName: strings.Repeat("a", 209) + "-resource-snapshot-0-cbb5025ff3ee", + }, + { + // The room reserved for the index fits the largest int32 value; the name is right at the length limit. + name: "longest valid name with the largest int32 index (truncated)", + policyName: strings.Repeat("a", 253), + policyUID: policyUID, + idx: 2147483647, + wantName: strings.Repeat("a", 209) + "-resource-snapshot-2147483647-3f30dad5f71c", + }, + { + // Truncation happens after the dots are removed, so the name never ends with a dot. + name: "dotted name over the length limit (truncated)", + policyName: dottedPolicyName, + policyUID: policyUID, + idx: 0, + wantName: wantTruncatedDottedPolicyName + "-resource-snapshot-0-0b77f00041b6", + }, + { + // When the name needs sanitization, an index longer than the reserved room is truncated as well; the + // hash (computed over the full index) keeps the name unique. + name: "index over 10 digits with a name over the length limit (both truncated)", + policyName: strings.Repeat("a", 219), + policyUID: policyUID, + idx: 12345678901, + wantName: strings.Repeat("a", 209) + "-resource-snapshot-1234567890-c9a32e12e9c1", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotName := PrimaryPlacementResourceSnapshotName(tc.policyName, tc.policyUID, tc.idx) + call := fmt.Sprintf("PrimaryPlacementResourceSnapshotName(%q, %q, %d)", tc.policyName, tc.policyUID, tc.idx) + verifyName(t, call, gotName, tc.wantName) + }) + } +} + +// TestSecondaryPlacementResourceSnapshotName tests the SecondaryPlacementResourceSnapshotName function. +// +// The hashes below are the first 12 hex characters of the SHA-256 digests of the hash inputs documented in +// uniquename.go (e.g., `policy-resource-snapshot-0-1-uid-1`), computed independently of the implementation. +func TestSecondaryPlacementResourceSnapshotName(t *testing.T) { + // The dot-free form of a dotted name, truncated to 207 characters (251 - len("-resource-snapshot-") - 12 - 1 - 12), + // where 12 is the room reserved for the snapshot index and sub-index. + dottedPolicyName := dottedName(253, 5) + wantTruncatedDottedPolicyName := strings.ReplaceAll(dottedPolicyName, ".", "")[:207] + + testCases := []struct { + name string + policyName string + policyUID types.UID + idx int + subIdx int + wantName string + }{ + { + name: "short name", + policyName: shortPolicyName, + policyUID: policyUID, + idx: 0, + subIdx: 1, + wantName: "policy-resource-snapshot-0-1-a314eee601b8", + }, + { + // A re-created placement policy of the same name gets different snapshot names. + name: "short name, different placement policy UID", + policyName: shortPolicyName, + policyUID: otherPolicyUID, + idx: 0, + subIdx: 1, + wantName: "policy-resource-snapshot-0-1-1ab4f5f5cf91", + }, + { + name: "short name, multi-digit sub-index", + policyName: shortPolicyName, + policyUID: policyUID, + idx: 0, + subIdx: 12, + wantName: "policy-resource-snapshot-0-12-c09358bdf1d3", + }, + { + // Dots are removed from the name segment, but the hash is computed over the original name. + name: "name with dots", + policyName: "my.policy.v1", + policyUID: policyUID, + idx: 0, + subIdx: 1, + wantName: "mypolicyv1-resource-snapshot-0-1-804481f6a22a", + }, + { + name: "exactly at the length limit (not truncated)", + policyName: strings.Repeat("a", 216), + policyUID: policyUID, + idx: 0, + subIdx: 1, + wantName: strings.Repeat("a", 216) + "-resource-snapshot-0-1-04c0e779ae04", + }, + { + name: "one character over the length limit (truncated)", + policyName: strings.Repeat("a", 217), + policyUID: policyUID, + idx: 0, + subIdx: 1, + wantName: strings.Repeat("a", 207) + "-resource-snapshot-0-1-2b4e249846f0", + }, + { + // The room reserved for the index segment fits the largest int32 index and a single-digit sub-index; + // the name is right at the length limit. + name: "longest valid name with the largest int32 index (truncated)", + policyName: strings.Repeat("a", 253), + policyUID: policyUID, + idx: 2147483647, + subIdx: 9, + wantName: strings.Repeat("a", 207) + "-resource-snapshot-2147483647-9-e61b33ff2355", + }, + { + // Truncation happens after the dots are removed, so the name never ends with a dot. + name: "dotted name over the length limit (truncated)", + policyName: dottedPolicyName, + policyUID: policyUID, + idx: 0, + subIdx: 1, + wantName: wantTruncatedDottedPolicyName + "-resource-snapshot-0-1-7e5b8d1134ab", + }, + { + // When the name needs sanitization, an index segment longer than the reserved room is truncated as + // well; the hash (computed over the full index and sub-index) keeps the name unique. + name: "index segment over 12 characters with a name over the length limit (both truncated)", + policyName: strings.Repeat("a", 253), + policyUID: policyUID, + idx: 2147483647, + subIdx: 15, + wantName: strings.Repeat("a", 207) + "-resource-snapshot-2147483647-1-8a62c7e54a25", + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + gotName := SecondaryPlacementResourceSnapshotName(tc.policyName, tc.policyUID, tc.idx, tc.subIdx) + call := fmt.Sprintf("SecondaryPlacementResourceSnapshotName(%q, %q, %d, %d)", tc.policyName, tc.policyUID, tc.idx, tc.subIdx) + verifyName(t, call, gotName, tc.wantName) + }) + } +} + +// TestPlacementResourceSnapshotNameUniqueness verifies that the generated names are deterministic and distinct +// across placement policies (including re-created ones of the same name), indices, and sub-indices. +func TestPlacementResourceSnapshotNameUniqueness(t *testing.T) { + type input struct { + policyName string + policyUID types.UID + idx int + // A negative sub-index stands for a primary snapshot. + subIdx int + } + longNameX := strings.Repeat("a", 240) + "x" + longNameY := strings.Repeat("a", 240) + "y" + inputs := []input{ + {policyName: shortPolicyName, policyUID: policyUID, idx: 0, subIdx: -1}, + {policyName: shortPolicyName, policyUID: policyUID, idx: 0, subIdx: 1}, + {policyName: shortPolicyName, policyUID: policyUID, idx: 0, subIdx: 2}, + {policyName: shortPolicyName, policyUID: policyUID, idx: 1, subIdx: -1}, + {policyName: shortPolicyName, policyUID: policyUID, idx: 1, subIdx: 1}, + {policyName: shortPolicyName, policyUID: policyUID, idx: 11, subIdx: -1}, + // A re-created placement policy of the same name. + {policyName: shortPolicyName, policyUID: otherPolicyUID, idx: 0, subIdx: -1}, + {policyName: shortPolicyName, policyUID: otherPolicyUID, idx: 0, subIdx: 1}, + // Names that collide after dot removal. + {policyName: "ab.c", policyUID: policyUID, idx: 0, subIdx: -1}, + {policyName: "a.bc", policyUID: policyUID, idx: 0, subIdx: -1}, + {policyName: "abc", policyUID: policyUID, idx: 0, subIdx: -1}, + {policyName: "ab.c", policyUID: policyUID, idx: 0, subIdx: 1}, + {policyName: "a.bc", policyUID: policyUID, idx: 0, subIdx: 1}, + // Names that differ only beyond the truncation point. + {policyName: longNameX, policyUID: policyUID, idx: 0, subIdx: -1}, + {policyName: longNameY, policyUID: policyUID, idx: 0, subIdx: -1}, + {policyName: longNameX, policyUID: policyUID, idx: 0, subIdx: 1}, + {policyName: longNameY, policyUID: policyUID, idx: 0, subIdx: 1}, + // Index segments that are the same after truncation. + {policyName: longNameX, policyUID: policyUID, idx: 2147483647, subIdx: 15}, + {policyName: longNameX, policyUID: policyUID, idx: 2147483647, subIdx: 16}, + } + + nameOf := func(in input) string { + if in.subIdx < 0 { + return PrimaryPlacementResourceSnapshotName(in.policyName, in.policyUID, in.idx) + } + return SecondaryPlacementResourceSnapshotName(in.policyName, in.policyUID, in.idx, in.subIdx) + } + + seen := make(map[string]input, len(inputs)) + for _, in := range inputs { + gotName := nameOf(in) + if gotAgain := nameOf(in); gotName != gotAgain { + t.Errorf("name for %+v = %q then %q, want a deterministic result", in, gotName, gotAgain) + } + if other, found := seen[gotName]; found { + t.Errorf("name for %+v = %q, which collides with the name for %+v", in, gotName, other) + } + seen[gotName] = in + } +} diff --git a/pkg/v1/managers/placementresourcesnapshot/utils_integration_test.go b/pkg/v1/managers/placementresourcesnapshot/utils_integration_test.go new file mode 100644 index 000000000..2f897cec4 --- /dev/null +++ b/pkg/v1/managers/placementresourcesnapshot/utils_integration_test.go @@ -0,0 +1,1209 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package placementresourcesnapshot + +import ( + "encoding/json" + "fmt" + "strconv" + "strings" + "sync" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils" + kferrors "go.goms.io/fleet/pkg/utils/errors" + "go.goms.io/fleet/pkg/v1/managers/placementresourcesnapshot/uniquename" + "go.goms.io/fleet/pkg/v1/utils/fieldindexers" +) + +// snapshotErrTestCase describes a placement policy for which snapshotting resources is expected to fail. +type snapshotErrTestCase struct { + desc string + placementPolicy placementv1alpha1.PlacementPolicyAccessor + wantErrCategory kferrors.ErrCategory + wantErrMsgSubstrs []string + // The index of the resource selector that should be reported in the error attributes (if set). + wantResourceSelectorIdx *int +} + +// newPlacementPolicy returns a placement policy (with a random name) in the given namespace that selects +// resources using the given resource selectors. +func newPlacementPolicy(namespace string, resourceSelectors ...placementv1alpha1.ResourceSelector) *placementv1alpha1.PlacementPolicy { + return &placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()), + Namespace: namespace, + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: resourceSelectors, + }, + } +} + +// newClusterPlacementPolicy returns a cluster placement policy (with a random name) that selects resources +// using the given resource selectors. +func newClusterPlacementPolicy(resourceSelectors ...placementv1alpha1.ResourceSelector) *placementv1alpha1.ClusterPlacementPolicy { + return &placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(placementPolicyNameTemplate, utils.RandStr()), + }, + Spec: placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: resourceSelectors, + }, + } +} + +// newDeployment returns a minimal Deployment with the given name in the given namespace. +func newDeployment(name, namespace string) *appsv1.Deployment { + return &appsv1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: namespace, + Labels: map[string]string{"app": "demo"}, + }, + Spec: appsv1.DeploymentSpec{ + Replicas: ptr.To(int32(2)), + Selector: &metav1.LabelSelector{ + MatchLabels: map[string]string{"app": "demo"}, + }, + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{"app": "demo"}, + }, + Spec: corev1.PodSpec{ + Containers: []corev1.Container{ + { + Name: "nginx", + Image: "nginx:1.27", + }, + }, + }, + }, + }, + } +} + +// waitForDeploymentsInCache waits until the informer cache, from which the manager under test reads Deployments, +// has observed each of the given Deployments at its current resource version (i.e., the version returned by the +// last create or update call on the object). +// +// As the informer applies watch events in order, once the cache has observed a write, it has also observed all +// earlier writes to Deployments, including those made during the clean-up of other test cases. +func waitForDeploymentsInCache(deployments ...*appsv1.Deployment) { + GinkgoHelper() + + lister := resourceSnapshotManager.hubDynamicInformerManager.Lister(appsv1.SchemeGroupVersion.WithResource("deployments")) + for _, deployment := range deployments { + Expect(deployment.ResourceVersion).ToNot(BeEmpty(), "Deployment %s has no resource version", client.ObjectKeyFromObject(deployment)) + Eventually(func() (string, error) { + obj, err := lister.ByNamespace(deployment.Namespace).Get(deployment.Name) + if err != nil { + return "", err + } + cachedDeployment, err := meta.Accessor(obj) + if err != nil { + return "", err + } + return cachedDeployment.GetResourceVersion(), nil + }, eventuallyDuration, eventuallyInterval).Should(Equal(deployment.ResourceVersion), + "The cache has not observed Deployment %s at resource version %s", client.ObjectKeyFromObject(deployment), deployment.ResourceVersion) + } +} + +// waitForDeploymentsRemovedFromCache waits until the informer cache, from which the manager under test reads +// Deployments, no longer holds any of the given Deployments. +func waitForDeploymentsRemovedFromCache(deployments ...*appsv1.Deployment) { + GinkgoHelper() + + lister := resourceSnapshotManager.hubDynamicInformerManager.Lister(appsv1.SchemeGroupVersion.WithResource("deployments")) + for _, deployment := range deployments { + Eventually(func() bool { + _, err := lister.ByNamespace(deployment.Namespace).Get(deployment.Name) + return errors.IsNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The cache still holds Deployment %s", client.ObjectKeyFromObject(deployment)) + } +} + +// newLoadBalancerService returns a LoadBalancer Service with two ports and the Local external traffic policy in +// the given namespace. For such a Service, the API server fills in the cluster IP(s), a node port for each port, +// and the health check node port. +func newLoadBalancerService(name, namespace string) *corev1.Service { + return &corev1.Service{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: namespace, + Labels: map[string]string{"app": "demo"}, + }, + Spec: corev1.ServiceSpec{ + Type: corev1.ServiceTypeLoadBalancer, + ExternalTrafficPolicy: corev1.ServiceExternalTrafficPolicyLocal, + Selector: map[string]string{"app": "demo"}, + Ports: []corev1.ServicePort{ + { + Name: "http", + Protocol: corev1.ProtocolTCP, + Port: 80, + TargetPort: intstr.FromInt32(8080), + }, + { + Name: "https", + Protocol: corev1.ProtocolTCP, + Port: 443, + TargetPort: intstr.FromInt32(8443), + }, + }, + }, + } +} + +// verifyServiceFieldsFilledIn verifies that the API server has filled in the cluster IP(s), the node ports, and +// the health check node port of the given Service. +func verifyServiceFieldsFilledIn(service *corev1.Service) { + GinkgoHelper() + + Expect(service.Spec.ClusterIP).ToNot(BeElementOf("", corev1.ClusterIPNone), "The API server has not assigned a cluster IP") + Expect(service.Spec.ClusterIPs).ToNot(BeEmpty(), "The API server has not assigned any cluster IPs") + Expect(service.Spec.HealthCheckNodePort).ToNot(BeZero(), "The API server has not assigned a health check node port") + for _, port := range service.Spec.Ports { + Expect(port.NodePort).ToNot(BeZero(), "The API server has not assigned a node port to port %s", port.Name) + } +} + +// serviceSpecWithoutFilledInFields returns a copy of the given Service spec without the fields that the API +// server fills in and the snapshot strips: the cluster IP(s), the node ports, and the health check node port. +func serviceSpecWithoutFilledInFields(spec *corev1.ServiceSpec) corev1.ServiceSpec { + specCopy := spec.DeepCopy() + specCopy.ClusterIP = "" + specCopy.ClusterIPs = nil + specCopy.HealthCheckNodePort = 0 + for idx := range specCopy.Ports { + specCopy.Ports[idx].NodePort = 0 + } + return *specCopy +} + +// newJob returns a minimal Job, without a manual selector, with the given name in the given namespace. +func newJob(name, namespace string) *batchv1.Job { + return &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: namespace, + Labels: map[string]string{"app": "demo"}, + }, + Spec: batchv1.JobSpec{ + Template: corev1.PodTemplateSpec{ + ObjectMeta: metav1.ObjectMeta{ + Labels: map[string]string{"app": "demo"}, + }, + Spec: corev1.PodSpec{ + RestartPolicy: corev1.RestartPolicyNever, + Containers: []corev1.Container{ + { + Name: "busybox", + Image: "busybox:1.36", + }, + }, + }, + }, + }, + } +} + +// verifyJobControllerUIDLabelsAdded verifies that the API server has added the controller UID labels, set to the +// UID of the given Job: the current label to both its selector and its pod template, and the legacy label to its +// pod template. +func verifyJobControllerUIDLabelsAdded(job *batchv1.Job) { + GinkgoHelper() + + Expect(job.Spec.Selector).ToNot(BeNil(), "The API server has not generated a selector for the Job") + Expect(job.Spec.Selector.MatchLabels).To(HaveKeyWithValue(batchv1.ControllerUidLabel, string(job.UID)), + "The API server has not added the controller UID label to the selector of the Job") + Expect(job.Spec.Template.Labels).To(HaveKeyWithValue(batchv1.ControllerUidLabel, string(job.UID)), + "The API server has not added the controller UID label to the pod template of the Job") + Expect(job.Spec.Template.Labels).To(HaveKeyWithValue(legacyJobControllerUIDLabel, string(job.UID)), + "The API server has not added the legacy controller UID label to the pod template of the Job") +} + +// jobSpecWithoutControllerUIDLabels returns a copy of the given Job spec without the controller UID labels (both +// the current and the legacy ones) that the API server adds and the snapshot strips. +func jobSpecWithoutControllerUIDLabels(spec *batchv1.JobSpec) batchv1.JobSpec { + specCopy := spec.DeepCopy() + for _, key := range []string{legacyJobControllerUIDLabel, batchv1.ControllerUidLabel} { + if specCopy.Selector != nil { + delete(specCopy.Selector.MatchLabels, key) + } + delete(specCopy.Template.Labels, key) + } + return *specCopy +} + +// placementResourceSnapshotObj returns an (empty) placement resource snapshot object of the matching scope with +// the given name; it is used for retrievals and clean-ups. +func placementResourceSnapshotObj(namespace, name string) placementv1alpha1.PlacementResourceSnapshotAccessor { + if namespace == "" { + return &placementv1alpha1.ClusterPlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{Name: name}} + } + return &placementv1alpha1.PlacementResourceSnapshot{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace}} +} + +// primarySnapshotObjs returns the primary placement resource snapshot objects at indices [0, cnt) of the given +// placement policy; it is used for clean-ups. +func primarySnapshotObjs(placementPolicy placementv1alpha1.PlacementPolicyAccessor, cnt int) []client.Object { + objs := make([]client.Object, 0, cnt) + for idx := 0; idx < cnt; idx++ { + objs = append(objs, placementResourceSnapshotObj(placementPolicy.GetNamespace(), + uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), idx))) + } + return objs +} + +// createNamespace creates a namespace with the given name. +// +// Note that the environment prepared by the envtest package does not support namespace deletion; the namespaces +// created by the test cases are left behind. +func createNamespace(name string) { + GinkgoHelper() + + Expect(hubClient.Create(ctx, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: name}})).To(Succeed(), + "Failed to create namespace %s", name) +} + +// createDeletingConfigMap creates a config map that has been marked for deletion, but cannot be deleted yet +// due to a finalizer. +func createDeletingConfigMap(name, namespace string, labels map[string]string) { + GinkgoHelper() + + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: namespace, + Labels: labels, + Finalizers: []string{testFinalizer}, + }, + Data: map[string]string{"key": "value"}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create the config map") + Expect(hubClient.Delete(ctx, configMap)).To(Succeed(), "Failed to delete the config map") + Eventually(func() error { + gotConfigMap := &corev1.ConfigMap{} + if err := hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(configMap), gotConfigMap); err != nil { + return err + } + if gotConfigMap.DeletionTimestamp == nil { + return fmt.Errorf("the config map has not been marked for deletion yet") + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark the config map for deletion") +} + +// deleteObjectsAndWait deletes the given objects (with the given delete options, if any) and waits until they are gone. +// +// The envtest environment does not run the garbage collector; objects must be removed explicitly. +func deleteObjectsAndWait(objs []client.Object, opts ...client.DeleteOption) { + GinkgoHelper() + + for _, obj := range objs { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj, opts...))).To(Succeed(), "Failed to delete object %T %s", obj, client.ObjectKeyFromObject(obj)) + Eventually(func() bool { + return errors.IsNotFound(hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj)) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "Failed to remove object %T %s", obj, client.ObjectKeyFromObject(obj)) + } +} + +// removeFinalizersAndDelete removes all finalizers from the given object and deletes it. +func removeFinalizersAndDelete(obj client.Object) { + GinkgoHelper() + + Eventually(func() error { + if err := hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(obj), obj); err != nil { + return client.IgnoreNotFound(err) + } + obj.SetFinalizers(nil) + return hubClient.Update(ctx, obj) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the finalizers from object %T %s", obj, client.ObjectKeyFromObject(obj)) + deleteObjectsAndWait([]client.Object{obj}) +} + +// verifyErrCategoryAndMsg verifies that the given error is a KubeFleet error of the wanted category, and that +// its message contains all the wanted substrings. +func verifyErrCategoryAndMsg(err error, wantErrCategory kferrors.ErrCategory, wantErrMsgSubstrs ...string) { + GinkgoHelper() + + Expect(err).To(HaveOccurred(), "No error is returned") + // The error category is the first key-value pair in the error attributes. + gotErrAttrs := kferrors.Args(err) + Expect(gotErrAttrs).ToNot(BeEmpty(), "The error is not a KubeFleet error: %v", err) + wantErrCategoryAttrs := []interface{}{"errCategory", wantErrCategory} + if diff := cmp.Diff(gotErrAttrs[:2], wantErrCategoryAttrs); diff != "" { + Fail(fmt.Sprintf("Error category attributes diff (-got +want):\n%s\nerror: %v", diff, err)) + } + for _, substr := range wantErrMsgSubstrs { + Expect(err.Error()).To(ContainSubstring(substr), "Unexpected error message") + } +} + +// errAttrValue returns the value of the given key in the attributes of the given (KubeFleet) error. +func errAttrValue(err error, key string) (interface{}, bool) { + attrs := kferrors.Args(err) + for idx := 0; idx+1 < len(attrs); idx += 2 { + if k, ok := attrs[idx].(string); ok && k == key { + return attrs[idx+1], true + } + } + return nil, false +} + +// verifyNoPlacementResourceSnapshots verifies that no placement resource snapshot has been created for the +// given placement policy. +func verifyNoPlacementResourceSnapshots(placementPolicy placementv1alpha1.PlacementPolicyAccessor) { + GinkgoHelper() + + matchingLabels := client.MatchingLabels{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyOwnerLabelVal(placementPolicy), + } + if placementPolicy.GetNamespace() == "" { + snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} + Expect(hubUncachedReader.List(ctx, snapshotList, matchingLabels)).To(Succeed(), "Failed to list cluster placement resource snapshots") + Expect(snapshotList.Items).To(BeEmpty(), "Cluster placement resource snapshots are created unexpectedly") + return + } + snapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} + Expect(hubUncachedReader.List(ctx, snapshotList, client.InNamespace(placementPolicy.GetNamespace()), matchingLabels)).To(Succeed(), + "Failed to list placement resource snapshots") + Expect(snapshotList.Items).To(BeEmpty(), "Placement resource snapshots are created unexpectedly") +} + +// snapshotResourcesAndExpectErr snapshots resources for the given placement policy, expecting a failure; it +// verifies that no placement resource snapshot is returned or created, and returns the error for further checks. +func snapshotResourcesAndExpectErr(placementPolicy placementv1alpha1.PlacementPolicyAccessor) error { + GinkgoHelper() + + gotSnapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfNoSnapshotExists(ctx, placementPolicy) + Expect(err).To(HaveOccurred(), "Snapshotting resources succeeded unexpectedly") + Expect(gotSnapshots).To(BeEmpty(), "Placement resource snapshots are returned unexpectedly") + Expect(isUpToDate).To(BeFalse(), "The up-to-date flag is set unexpectedly") + verifyNoPlacementResourceSnapshots(placementPolicy) + return err +} + +// verifySnapshotErrTestCase runs a snapshot error test case. +func verifySnapshotErrTestCase(tc snapshotErrTestCase) { + GinkgoHelper() + + err := snapshotResourcesAndExpectErr(tc.placementPolicy) + verifyErrCategoryAndMsg(err, tc.wantErrCategory, tc.wantErrMsgSubstrs...) + if tc.wantResourceSelectorIdx != nil { + gotIdx, found := errAttrValue(err, "resourceSelectorIndex") + Expect(found).To(BeTrue(), "The error does not report the index of the resource selector: %v", err) + Expect(gotIdx).To(Equal(*tc.wantResourceSelectorIdx), "Unexpected resource selector index in the error") + } +} + +// verifyReturnedPrimarySnapshot verifies that snapshotting resources has succeeded, and has returned exactly +// one up-to-date placement resource snapshot, i.e., the primary snapshot at the wanted index. +func verifyReturnedPrimarySnapshot( + gotSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, isUpToDate bool, err error, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, wantIdx int, +) { + GinkgoHelper() + + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The placement resource snapshot is not up-to-date") + Expect(gotSnapshots).To(HaveLen(1), "Unexpected number of placement resource snapshots") + wantName := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), wantIdx) + Expect(gotSnapshots[0].GetName()).To(Equal(wantName), "Unexpected name of the placement resource snapshot") +} + +// getSnapshotFromCache retrieves a placement resource snapshot using the cached client; the cache might take +// a moment to catch up. +func getSnapshotFromCache(namespace, name string) placementv1alpha1.PlacementResourceSnapshotAccessor { + GinkgoHelper() + + snapshot := placementResourceSnapshotObj(namespace, name) + Eventually(func() error { + return hubClient.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, snapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get placement resource snapshot %s", name) + return snapshot +} + +// getPrimarySnapshotFromCache retrieves the primary placement resource snapshot at the given index using the +// cached client. +func getPrimarySnapshotFromCache(placementPolicy placementv1alpha1.PlacementPolicyAccessor, idx int) placementv1alpha1.PlacementResourceSnapshotAccessor { + GinkgoHelper() + + return getSnapshotFromCache(placementPolicy.GetNamespace(), uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), idx)) +} + +// verifyNoPrimarySnapshotAtIdx verifies that no primary placement resource snapshot exists at the given index. +func verifyNoPrimarySnapshotAtIdx(placementPolicy placementv1alpha1.PlacementPolicyAccessor, idx int) { + GinkgoHelper() + + name := uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), idx) + err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: placementPolicy.GetNamespace(), Name: name}, + placementResourceSnapshotObj(placementPolicy.GetNamespace(), name)) + Expect(errors.IsNotFound(err)).To(BeTrue(), "Placement resource snapshot %s was created unexpectedly: %v", name, err) +} + +// wantPlacementPolicyOwnerRefs returns the owner references that a placement resource snapshot should carry. +func wantPlacementPolicyOwnerRefs(placementPolicy placementv1alpha1.PlacementPolicyAccessor) []metav1.OwnerReference { + kind := placementv1alpha1.PlacementPolicyKind + if placementPolicy.GetNamespace() == "" { + kind = placementv1alpha1.ClusterPlacementPolicyKind + } + return []metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: kind, + Name: placementPolicy.GetName(), + UID: placementPolicy.GetUID(), + Controller: ptr.To(true), + BlockOwnerDeletion: ptr.To(true), + }, + } +} + +// verifySnapshotMetadata verifies the labels, annotations, and owner references of a placement resource snapshot. +func verifySnapshotMetadata( + snapshot placementv1alpha1.PlacementResourceSnapshotAccessor, + placementPolicy placementv1alpha1.PlacementPolicyAccessor, + idx, subIdx, snapshotCnt int, + wantHash string, +) { + GinkgoHelper() + + wantLabels := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyOwnerLabelVal(placementPolicy), + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: strconv.Itoa(idx), + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: strconv.Itoa(subIdx), + } + if subIdx == 0 { + // Only the primary snapshot carries the count label. + wantLabels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = strconv.Itoa(snapshotCnt) + } + if diff := cmp.Diff(snapshot.GetLabels(), wantLabels); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot %s labels diff (-got +want):\n%s", snapshot.GetName(), diff)) + } + wantAnnotations := map[string]string{ + placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey: wantHash, + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicy.GetName(), + } + if diff := cmp.Diff(snapshot.GetAnnotations(), wantAnnotations); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot %s annotations diff (-got +want):\n%s", snapshot.GetName(), diff)) + } + if diff := cmp.Diff(snapshot.GetOwnerReferences(), wantPlacementPolicyOwnerRefs(placementPolicy)); diff != "" { + Fail(fmt.Sprintf("Placement resource snapshot %s owner references diff (-got +want):\n%s", snapshot.GetName(), diff)) + } +} + +// verifySnapshottedResourceIdentifiers verifies the identifiers of the given snapshotted resources. +func verifySnapshottedResourceIdentifiers(resources []placementv1alpha1.SnapshottedResource, wantIdentifiers []placementv1alpha1.ObjectReference) { + GinkgoHelper() + + gotIdentifiers := make([]placementv1alpha1.ObjectReference, 0, len(resources)) + for idx := range resources { + gotIdentifiers = append(gotIdentifiers, resources[idx].Identifier) + } + if diff := cmp.Diff(gotIdentifiers, wantIdentifiers); diff != "" { + Fail(fmt.Sprintf("Snapshotted resource identifiers diff (-got +want):\n%s", diff)) + } +} + +// manifestAsMap unmarshals a snapshotted manifest into a generic map. +// +// Comparing generic maps (rather than typed objects) verifies the absence of fields, which zero values in typed +// objects would otherwise hide. +func manifestAsMap(raw []byte) map[string]interface{} { + GinkgoHelper() + + m := map[string]interface{}{} + Expect(json.Unmarshal(raw, &m)).To(Succeed(), "Failed to unmarshal the manifest") + return m +} + +// snapshotCallResult is the outcome of a call to the placement resource snapshot manager. +type snapshotCallResult struct { + names []string + isUpToDate bool + err error +} + +// createDecoySnapshot creates a hand-crafted placement resource snapshot (a decoy) with an empty spec. +// +// A decoy without the owned-by annotation (or without the index/sub-index labels) is never indexed by the cache; +// it stays invisible to the field index-based lookups of the manager, yet is visible to quorum reads (uncached +// reads by name) and causes conflicts on creation. This deterministically simulates the situation where the +// API server has an object that the cache has yet to observe. +func createDecoySnapshot(namespace, name string, labels, annotations map[string]string, finalizers []string) placementv1alpha1.PlacementResourceSnapshotAccessor { + GinkgoHelper() + + decoy := placementResourceSnapshotObj(namespace, name) + decoy.SetLabels(labels) + decoy.SetAnnotations(annotations) + decoy.SetFinalizers(finalizers) + Expect(hubClient.Create(ctx, decoy)).To(Succeed(), "Failed to create the decoy placement resource snapshot %s", name) + return decoy +} + +// createOwnedDecoySnapshot creates a decoy (see createDecoySnapshot) in the namespace of the given placement policy, +// with a controller owner reference to the placement policy, the same way as the snapshots that the manager creates. +// +// The manager treats an indexed primary snapshot without owner references as a leftover; decoys that mimic the +// snapshots of a placement policy must be owned by the placement policy. +func createOwnedDecoySnapshot(placementPolicy placementv1alpha1.PlacementPolicyAccessor, name string, labels, annotations map[string]string) placementv1alpha1.PlacementResourceSnapshotAccessor { + GinkgoHelper() + + decoy := placementResourceSnapshotObj(placementPolicy.GetNamespace(), name) + decoy.SetLabels(labels) + decoy.SetAnnotations(annotations) + decoy.SetOwnerReferences(decoyOwnerRefs(placementPolicy)) + Expect(hubClient.Create(ctx, decoy)).To(Succeed(), "Failed to create the decoy placement resource snapshot %s", name) + return decoy +} + +// decoyOwnerRefs returns the owner references of a placement resource snapshot that the given placement policy +// controls. +func decoyOwnerRefs(placementPolicy placementv1alpha1.PlacementPolicyAccessor) []metav1.OwnerReference { + kind := placementv1alpha1.PlacementPolicyKind + if placementPolicy.GetNamespace() == "" { + kind = placementv1alpha1.ClusterPlacementPolicyKind + } + return []metav1.OwnerReference{*metav1.NewControllerRef(placementPolicy, placementv1alpha1.GroupVersion.WithKind(kind))} +} + +// decoyPrimaryLabels returns the labels of a primary placement resource snapshot; the values are strings so that +// malformed values can be injected. +func decoyPrimaryLabels(placementPolicy placementv1alpha1.PlacementPolicyAccessor, idx, cnt string) map[string]string { + labels := decoySecondaryLabels(placementPolicy, idx, "0") + labels[placementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey] = cnt + return labels +} + +// decoySecondaryLabels returns the labels of a secondary placement resource snapshot; the values are strings so +// that malformed values can be injected. +func decoySecondaryLabels(placementPolicy placementv1alpha1.PlacementPolicyAccessor, idx, subIdx string) map[string]string { + return map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey: placementPolicyOwnerLabelVal(placementPolicy), + placementv1alpha1.PlacementResourceSnapshotIndexLabelKey: idx, + placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: subIdx, + } +} + +// decoyOwnedByAnnotations returns the owned-by annotation, which a decoy must carry to be indexed by the cache. +func decoyOwnedByAnnotations(placementPolicy placementv1alpha1.PlacementPolicyAccessor) map[string]string { + return map[string]string{ + placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey: placementPolicy.GetName(), + } +} + +// waitForSnapshotGoneFromCache waits until the cache has observed the removal of a placement resource snapshot. +func waitForSnapshotGoneFromCache(namespace, name string) { + GinkgoHelper() + + Eventually(func() bool { + return errors.IsNotFound(hubClient.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, placementResourceSnapshotObj(namespace, name))) + }, eventuallyDuration, eventuallyInterval).Should(BeTrue(), "The cache still holds placement resource snapshot %s", name) +} + +// verifySnapshotExists verifies (via an uncached read) that a placement resource snapshot exists, and returns it. +func verifySnapshotExists(namespace, name string) placementv1alpha1.PlacementResourceSnapshotAccessor { + GinkgoHelper() + + snapshot := placementResourceSnapshotObj(namespace, name) + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, snapshot)).To(Succeed(), + "Failed to get placement resource snapshot %s", name) + return snapshot +} + +// verifySnapshotNotFound verifies (via an uncached read) that a placement resource snapshot does not exist. +func verifySnapshotNotFound(namespace, name string) { + GinkgoHelper() + + err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, placementResourceSnapshotObj(namespace, name)) + Expect(errors.IsNotFound(err)).To(BeTrue(), "Placement resource snapshot %s exists unexpectedly: %v", name, err) +} + +// snapshotNamesOf returns the names of the given placement resource snapshots. +func snapshotNamesOf(snapshots []placementv1alpha1.PlacementResourceSnapshotAccessor) []string { + names := make([]string, 0, len(snapshots)) + for _, snapshot := range snapshots { + names = append(names, snapshot.GetName()) + } + return names +} + +// snapshotContentsHash returns the contents hash annotation of a placement resource snapshot. +func snapshotContentsHash(snapshot placementv1alpha1.PlacementResourceSnapshotAccessor) string { + return snapshot.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotContentsHashAnnotationKey] +} + +// verifyReturnedSnapshots verifies that a call to the manager has succeeded, and has returned the wanted +// up-to-date placement resource snapshots. +func verifyReturnedSnapshots( + gotSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, isUpToDate bool, err error, + wantNames []string, +) { + GinkgoHelper() + + Expect(err).ToNot(HaveOccurred(), "Failed to snapshot resources") + Expect(isUpToDate).To(BeTrue(), "The placement resource snapshots are not up-to-date") + if diff := cmp.Diff(snapshotNamesOf(gotSnapshots), wantNames); diff != "" { + Fail(fmt.Sprintf("Returned placement resource snapshot names diff (-got +want):\n%s", diff)) + } +} + +// checkSnapshotCallFailed checks if a call to the manager has failed with an error of the wanted category and +// message; it returns an error describing the mismatch (if any), for use in Eventually and Consistently blocks. +func checkSnapshotCallFailed( + gotSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, isUpToDate bool, err error, + wantErrCategory kferrors.ErrCategory, wantErrMsgSubstr string, +) error { + switch { + case err == nil: + return fmt.Errorf("snapshotting resources succeeded unexpectedly (snapshots %v, up-to-date %t)", snapshotNamesOf(gotSnapshots), isUpToDate) + case len(gotSnapshots) != 0 || isUpToDate: + return fmt.Errorf("unexpected return values alongside the error (snapshots %v, up-to-date %t)", snapshotNamesOf(gotSnapshots), isUpToDate) + case kferrors.Category(err) != wantErrCategory: + return fmt.Errorf("error category = %s, want %s: %w", kferrors.Category(err), wantErrCategory, err) + case !strings.Contains(err.Error(), wantErrMsgSubstr): + return fmt.Errorf("error message does not contain %q: %w", wantErrMsgSubstr, err) + } + return nil +} + +// verifySnapshotCallFailed verifies that a call to the manager has failed with an error of the wanted category and +// message. +// +// Unlike snapshotResourcesAndExpectErr, it does not require that no placement resource snapshots exist. +func verifySnapshotCallFailed( + gotSnapshots []placementv1alpha1.PlacementResourceSnapshotAccessor, isUpToDate bool, err error, + wantErrCategory kferrors.ErrCategory, wantErrMsgSubstrs ...string, +) { + GinkgoHelper() + + Expect(gotSnapshots).To(BeEmpty(), "Placement resource snapshots are returned unexpectedly") + Expect(isUpToDate).To(BeFalse(), "The up-to-date flag is set unexpectedly") + verifyErrCategoryAndMsg(err, wantErrCategory, wantErrMsgSubstrs...) +} + +// verifyOwnedSnapshots verifies (via an uncached read) the names of all the placement resource snapshots owned by +// a placement policy. +// +// Ownership is determined by the owned-by annotation, which always carries the full name of the placement policy; +// the owned-by label might be truncated, and two placement policies might share the same label value. +func verifyOwnedSnapshots(placementPolicy placementv1alpha1.PlacementPolicyAccessor, wantNames []string) { + GinkgoHelper() + + var candidates []client.Object + if placementPolicy.GetNamespace() == "" { + snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} + Expect(hubUncachedReader.List(ctx, snapshotList)).To(Succeed(), "Failed to list cluster placement resource snapshots") + for idx := range snapshotList.Items { + candidates = append(candidates, &snapshotList.Items[idx]) + } + } else { + snapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} + Expect(hubUncachedReader.List(ctx, snapshotList, client.InNamespace(placementPolicy.GetNamespace()))).To(Succeed(), + "Failed to list placement resource snapshots") + for idx := range snapshotList.Items { + candidates = append(candidates, &snapshotList.Items[idx]) + } + } + + var gotNames []string + for _, candidate := range candidates { + if candidate.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] == placementPolicy.GetName() { + gotNames = append(gotNames, candidate.GetName()) + } + } + if diff := cmp.Diff(gotNames, wantNames, cmpopts.SortSlices(func(a, b string) bool { return a < b }), cmpopts.EquateEmpty()); diff != "" { + Fail(fmt.Sprintf("Owned placement resource snapshot names of %s diff (-got +want):\n%s", placementPolicy.GetName(), diff)) + } +} + +// configMapNameSelector returns a resource selector that selects a config map by name. +func configMapNameSelector(name string) placementv1alpha1.ResourceSelector { + return placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: name, + } +} + +// configMapLabelSelector returns a resource selector that selects config maps by labels. +func configMapLabelSelector(labels map[string]string) placementv1alpha1.ResourceSelector { + return placementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{MatchLabels: labels}, + } +} + +// createLabeledConfigMaps creates config maps, named `[PREFIX]-[INDEX]`, with the given labels in the app namespace, +// and returns their names. +func createLabeledConfigMaps(prefix string, cnt int, labels map[string]string) []string { + GinkgoHelper() + + names := make([]string, 0, cnt) + for idx := 0; idx < cnt; idx++ { + name := fmt.Sprintf("%s-%d", prefix, idx) + configMap := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: appNamespaceName, + Labels: labels, + }, + Data: map[string]string{"key": fmt.Sprintf("value-%d", idx)}, + } + Expect(hubClient.Create(ctx, configMap)).To(Succeed(), "Failed to create config map %s", name) + names = append(names, name) + } + return names +} + +// configMapObjs returns (empty) config map objects with the given names in the app namespace; it is used for +// clean-ups. +func configMapObjs(names []string) []client.Object { + objs := make([]client.Object, 0, len(names)) + for _, name := range names { + objs = append(objs, &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: appNamespaceName}}) + } + return objs +} + +// snapshotObjs returns (empty) placement resource snapshot objects of the matching scope with the given names; it +// is used for clean-ups. +func snapshotObjs(namespace string, names ...string) []client.Object { + objs := make([]client.Object, 0, len(names)) + for _, name := range names { + objs = append(objs, placementResourceSnapshotObj(namespace, name)) + } + return objs +} + +// updateConfigMapData updates the data of a config map in the given namespace. +func updateConfigMapData(namespace, name, value string) { + GinkgoHelper() + + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, configMap)).To(Succeed(), + "Failed to get config map %s", name) + configMap.Data = map[string]string{"key": value} + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update config map %s", name) +} + +// placementPolicyNameOfLen returns a valid placement policy name of the given length. +func placementPolicyNameOfLen(l int) string { + suffix := utils.RandStr() + return strings.Repeat(placementPolicyNameFillerLetter, l-len(suffix)-1) + "-" + suffix +} + +// verifySnapshotName verifies that a generated placement resource snapshot name is valid and ends with a hash, and +// that the placement policy name segment in it is sanitized (i.e., with dots dropped and/or truncated) only when +// wanted. unsanitizedNamePrefix is the wanted snapshot name minus the hash suffix, should no sanitization occur. +func verifySnapshotName(gotName, unsanitizedNamePrefix, hashedNamePattern string, wantSanitized bool) { + GinkgoHelper() + + Expect(validation.IsDNS1123Subdomain(gotName)).To(BeEmpty(), "Snapshot name %s is not a valid DNS subdomain", gotName) + Expect(len(gotName)).To(BeNumerically("<=", wantMaxSnapshotNameLen), "Snapshot name %s is too long", gotName) + Expect(gotName).ToNot(ContainSubstring("."), "Snapshot name %s contains dots", gotName) + Expect(gotName).To(MatchRegexp(hashedNamePattern), "Snapshot name %s does not end with a hash", gotName) + if !wantSanitized { + Expect(gotName).To(HavePrefix(unsanitizedNamePrefix), "Snapshot name %s is sanitized unexpectedly", gotName) + Expect(gotName[len(unsanitizedNamePrefix):]).To(MatchRegexp(wantHashSuffixPattern), "Unexpected hash suffix of snapshot name %s", gotName) + return + } + Expect(gotName).ToNot(HavePrefix(unsanitizedNamePrefix), "Snapshot name %s is not sanitized", gotName) +} + +// verifyOwnedByLabelVal verifies the value of the owned-by label for a placement policy. +func verifyOwnedByLabelVal(placementPolicy placementv1alpha1.PlacementPolicyAccessor, wantTruncated bool) { + GinkgoHelper() + + name := placementPolicy.GetName() + gotLabelVal := placementPolicyOwnerLabelVal(placementPolicy) + if !wantTruncated { + Expect(gotLabelVal).To(Equal(name), "The owned-by label value is truncated unexpectedly") + return + } + Expect(gotLabelVal).To(HaveLen(wantOwnedByLabelValLen), "Unexpected length of the truncated owned-by label value") + Expect(gotLabelVal).To(HavePrefix(name[:wantOwnedByLabelValPrefixLen]+"-"), "Unexpected prefix of the truncated owned-by label value") + Expect(gotLabelVal[wantOwnedByLabelValPrefixLen+1:]).To(MatchRegexp(wantHashSuffixPattern), "Unexpected hash suffix of the truncated owned-by label value") +} + +// snapshotConcurrently calls the manager to snapshot resources for a placement policy from multiple goroutines at +// the same time, and returns the results. +func snapshotConcurrently(n int, placementPolicy placementv1alpha1.PlacementPolicyAccessor) []snapshotCallResult { + results := make([]snapshotCallResult, n) + var wg sync.WaitGroup + for idx := 0; idx < n; idx++ { + wg.Add(1) + go func() { + defer GinkgoRecover() + defer wg.Done() + + snapshots, isUpToDate, err := resourceSnapshotManager.SnapshotResourcesIfStale(ctx, placementPolicy) + results[idx] = snapshotCallResult{names: snapshotNamesOf(snapshots), isUpToDate: isUpToDate, err: err} + }() + } + wg.Wait() + return results +} + +// verifyConcurrentSnapshotCalls verifies the results of concurrent calls to the manager: at least one call should +// succeed, every successful call should return the wanted up-to-date snapshots, and every failed call should be +// retryable. +func verifyConcurrentSnapshotCalls(results []snapshotCallResult, wantNames []string) { + GinkgoHelper() + + succeeded := 0 + for idx, res := range results { + if res.err == nil { + succeeded++ + Expect(res.isUpToDate).To(BeTrue(), "Call %d returned placement resource snapshots that are not up-to-date", idx) + if diff := cmp.Diff(res.names, wantNames); diff != "" { + Fail(fmt.Sprintf("Call %d returned placement resource snapshot names diff (-got +want):\n%s", idx, diff)) + } + continue + } + + // The cache might lag behind; the resulting errors must be retryable. + category := kferrors.Category(res.err) + isRetryable := category == kferrors.ErrCategoryTransient || (category == kferrors.ErrCategoryAPIServer && errors.IsAlreadyExists(res.err)) + Expect(isRetryable).To(BeTrue(), "Call %d returned a non-retryable error (category %s): %v", idx, category, res.err) + } + Expect(succeeded).To(BeNumerically(">=", 1), "No call has succeeded") +} + +// newPausedManager returns a placement resource snapshot manager that is never started; as no worker processes its +// GC workqueue, the test cases can inspect (and drain) the GC requests it enqueues. +// +// The paused manager shares the cache (and its field indexes) with the manager under test; a placement policy +// should only be handled by one of the two managers. +func newPausedManager() *Manager { + GinkgoHelper() + + pausedMgr, err := New(hubMgr, + resourceSnapshotManager.hubDynamicClient, + resourceSnapshotManager.hubDynamicInformerManager, + resourceSnapshotManager.resourceEligibilityChecker, + resourceSnapshotManager.restMapper, + testMuSlotCnt, + resourceSnapshotManager.maxPerSnapshotResourceDataSizeBytes, + resourceSnapshotManager.maxPerSnapshotResourceCnt, + ) + Expect(err).ToNot(HaveOccurred(), "Failed to create a paused placement resource snapshot manager") + return pausedMgr +} + +// drainGCRequests drains the GC workqueue of a paused manager, and returns the GC requests in the order they +// were enqueued. +func drainGCRequests(m *Manager) []snapshotGarbageCollectionRequest { + var reqs []snapshotGarbageCollectionRequest + for m.gcwq.Len() > 0 { + req, _ := m.gcwq.Get() + m.gcwq.Done(req) + m.gcwq.Forget(req) + reqs = append(reqs, req) + } + return reqs +} + +// gcRequestFor returns the GC request that the manager should enqueue for a primary placement resource snapshot. +func gcRequestFor( + placementPolicy placementv1alpha1.PlacementPolicyAccessor, + primarySnapshot placementv1alpha1.PlacementResourceSnapshotAccessor, +) snapshotGarbageCollectionRequest { + return snapshotGarbageCollectionRequest{ + snapshotNamespacedName: types.NamespacedName{ + Namespace: primarySnapshot.GetNamespace(), + Name: primarySnapshot.GetName(), + }, + snapshotUID: primarySnapshot.GetUID(), + placementPolicyNamespacedName: types.NamespacedName{ + Namespace: placementPolicy.GetNamespace(), + Name: placementPolicy.GetName(), + }, + } +} + +// verifyGCRequests verifies that the GC requests (in order) are as wanted. +func verifyGCRequests(gotReqs, wantReqs []snapshotGarbageCollectionRequest) { + GinkgoHelper() + + if diff := cmp.Diff(gotReqs, wantReqs, cmp.AllowUnexported(snapshotGarbageCollectionRequest{}), cmpopts.EquateEmpty()); diff != "" { + Fail(fmt.Sprintf("GC requests diff (-got +want):\n%s", diff)) + } +} + +// snapshotSetNames returns the names of the primary placement resource snapshot at the given index and its +// secondary snapshots, given the total number of snapshots at the index. +func snapshotSetNames(placementPolicy placementv1alpha1.PlacementPolicyAccessor, idx, cnt int) []string { + names := []string{uniquename.PrimaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), idx)} + for subIdx := 1; subIdx < cnt; subIdx++ { + names = append(names, uniquename.SecondaryPlacementResourceSnapshotName(placementPolicy.GetName(), placementPolicy.GetUID(), idx, subIdx)) + } + return names +} + +// snapshotAndVerify calls the given manager to snapshot the selected resources if the latest snapshots are stale, +// and verifies that the wanted up-to-date snapshots are returned; it then waits until the cache has observed them, +// so that the next call can see them. +func snapshotAndVerify(m *Manager, placementPolicy placementv1alpha1.PlacementPolicyAccessor, wantNames []string) { + GinkgoHelper() + + gotSnapshots, isUpToDate, err := m.SnapshotResourcesIfStale(ctx, placementPolicy) + verifyReturnedSnapshots(gotSnapshots, isUpToDate, err, wantNames) + for _, name := range wantNames { + getSnapshotFromCache(placementPolicy.GetNamespace(), name) + } +} + +// checkOwnedSnapshots checks (via an uncached read) the names of all the placement resource snapshots owned by a +// placement policy, as determined by the owned-by annotation; it returns an error describing the mismatch (if any), +// for use in Eventually blocks. +func checkOwnedSnapshots(placementPolicy placementv1alpha1.PlacementPolicyAccessor, wantNames []string) error { + var gotNames []string + if placementPolicy.GetNamespace() == "" { + snapshotList := &placementv1alpha1.ClusterPlacementResourceSnapshotList{} + if err := hubUncachedReader.List(ctx, snapshotList); err != nil { + return fmt.Errorf("failed to list cluster placement resource snapshots: %w", err) + } + for idx := range snapshotList.Items { + snapshot := &snapshotList.Items[idx] + if snapshot.Annotations[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] == placementPolicy.GetName() { + gotNames = append(gotNames, snapshot.Name) + } + } + } else { + snapshotList := &placementv1alpha1.PlacementResourceSnapshotList{} + if err := hubUncachedReader.List(ctx, snapshotList, client.InNamespace(placementPolicy.GetNamespace())); err != nil { + return fmt.Errorf("failed to list placement resource snapshots: %w", err) + } + for idx := range snapshotList.Items { + snapshot := &snapshotList.Items[idx] + if snapshot.Annotations[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] == placementPolicy.GetName() { + gotNames = append(gotNames, snapshot.Name) + } + } + } + + if diff := cmp.Diff(gotNames, wantNames, cmpopts.SortSlices(func(a, b string) bool { return a < b }), cmpopts.EquateEmpty()); diff != "" { + return fmt.Errorf("owned placement resource snapshot names of %s diff (-got +want):\n%s", placementPolicy.GetName(), diff) + } + return nil +} + +// addSnapshotFinalizer adds a finalizer to a placement resource snapshot, which blocks its deletion. +func addSnapshotFinalizer(namespace, name string) { + GinkgoHelper() + + Eventually(func() error { + snapshot := placementResourceSnapshotObj(namespace, name) + if err := hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, snapshot); err != nil { + return err + } + snapshot.SetFinalizers(append(snapshot.GetFinalizers(), testFinalizer)) + return hubClient.Update(ctx, snapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add a finalizer to placement resource snapshot %s", name) +} + +// markSnapshotForDeletion deletes a placement resource snapshot (whose deletion is blocked by a finalizer), and waits +// until the cache has observed its deletion timestamp. +func markSnapshotForDeletion(namespace, name string) { + GinkgoHelper() + + Expect(hubClient.Delete(ctx, placementResourceSnapshotObj(namespace, name))).To(Succeed(), "Failed to delete placement resource snapshot %s", name) + Eventually(func() error { + snapshot := placementResourceSnapshotObj(namespace, name) + if err := hubClient.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, snapshot); err != nil { + return err + } + if snapshot.GetDeletionTimestamp().IsZero() { + return fmt.Errorf("the cache has not observed the deletion timestamp yet") + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to mark placement resource snapshot %s for deletion", name) +} + +// setConfigMapLabels sets the labels of the given config maps in the app namespace; nil labels remove all labels. +func setConfigMapLabels(names []string, labels map[string]string) { + GinkgoHelper() + + for _, name := range names { + configMap := &corev1.ConfigMap{} + Expect(hubUncachedReader.Get(ctx, types.NamespacedName{Namespace: appNamespaceName, Name: name}, configMap)).To(Succeed(), + "Failed to get config map %s", name) + configMap.Labels = labels + Expect(hubClient.Update(ctx, configMap)).To(Succeed(), "Failed to update config map %s", name) + } +} + +// placementBindingObj returns an (empty) placement binding object of the matching scope with the given name. +func placementBindingObj(namespace, name string) placementv1alpha1.PlacementBindingAccessor { + if namespace == "" { + return &placementv1alpha1.ClusterPlacementBinding{ObjectMeta: metav1.ObjectMeta{Name: name}} + } + return &placementv1alpha1.PlacementBinding{ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace}} +} + +// createPlacementBinding creates a placement binding (of the same scope as the placement policy) that uses the given +// primary placement resource snapshot; the binding has an owner reference to the placement policy (with the +// given UID) only if the UID is not nil. +func createPlacementBinding(placementPolicy placementv1alpha1.PlacementPolicyAccessor, name, snapshotName string, ownerUID *types.UID) { + GinkgoHelper() + + placementBinding := placementBindingObj(placementPolicy.GetNamespace(), name) + if ownerUID != nil { + kind := placementv1alpha1.PlacementPolicyKind + if placementPolicy.GetNamespace() == "" { + kind = placementv1alpha1.ClusterPlacementPolicyKind + } + placementBinding.SetOwnerReferences([]metav1.OwnerReference{ + { + APIVersion: placementv1alpha1.GroupVersion.String(), + Kind: kind, + Name: placementPolicy.GetName(), + UID: *ownerUID, + }, + }) + } + placementBinding.SetSpec(placementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: placementPolicy.GetName(), + ClusterName: "member-1", + ResourceSnapshotName: snapshotName, + }) + Expect(hubClient.Create(ctx, placementBinding)).To(Succeed(), "Failed to create placement binding %s", name) +} + +// waitForPlacementBindingCntInCache waits until the cache (and its field index) reports the wanted number of +// placement bindings that are owned by the placement policy and use the given primary placement resource snapshot; +// the GC process relies on the field index to tell if a snapshot is still in use. +func waitForPlacementBindingCntInCache(placementPolicy placementv1alpha1.PlacementPolicyAccessor, snapshotName string, wantCnt int) { + GinkgoHelper() + + fieldMatchers := client.MatchingFields{ + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName: fmt.Sprintf( + fieldindexers.PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, placementPolicy.GetName(), snapshotName), + } + Eventually(func() (int, error) { + if placementPolicy.GetNamespace() == "" { + placementBindingList := &placementv1alpha1.ClusterPlacementBindingList{} + err := hubClient.List(ctx, placementBindingList, fieldMatchers) + return len(placementBindingList.Items), err + } + placementBindingList := &placementv1alpha1.PlacementBindingList{} + err := hubClient.List(ctx, placementBindingList, client.InNamespace(placementPolicy.GetNamespace()), fieldMatchers) + return len(placementBindingList.Items), err + }, eventuallyDuration, eventuallyInterval).Should(Equal(wantCnt), "Unexpected number of placement bindings in the cache that use snapshot %s", snapshotName) +} + +// withInMemoryRevisionHistoryLimit returns a copy of the placement policy with the given revision history limit; +// the change is not persisted, which allows the use of values that the API server would reject. +func withInMemoryRevisionHistoryLimit(placementPolicy placementv1alpha1.PlacementPolicyAccessor, limit *int32) placementv1alpha1.PlacementPolicyAccessor { + placementPolicyCopy := placementPolicy.DeepCopyObject().(placementv1alpha1.PlacementPolicyAccessor) + placementPolicyCopy.GetSpec().ResourceRevisionHistoryLimit = limit + return placementPolicyCopy +} + +// updateRevisionHistoryLimit updates the revision history limit of the placement policy; the object is refreshed +// in place. +func updateRevisionHistoryLimit(placementPolicy placementv1alpha1.PlacementPolicyAccessor, limit int32) { + GinkgoHelper() + + placementPolicy.GetSpec().ResourceRevisionHistoryLimit = ptr.To(limit) + Expect(hubClient.Update(ctx, placementPolicy)).To(Succeed(), "Failed to update the revision history limit of the placement policy") +} + +// recreatePlacementPolicy deletes the given placement policy, and creates a new one with the same name and spec; it +// returns the new placement policy, which has a different UID. +// +// The envtest environment does not run the Kubernetes GC; the placement resource snapshots of the deleted placement +// policy stay, along with their owner references (if any). +func recreatePlacementPolicy(placementPolicy placementv1alpha1.PlacementPolicyAccessor) placementv1alpha1.PlacementPolicyAccessor { + GinkgoHelper() + + previousUID := placementPolicy.GetUID() + deleteObjectsAndWait([]client.Object{placementPolicy}) + + objectMeta := metav1.ObjectMeta{Name: placementPolicy.GetName(), Namespace: placementPolicy.GetNamespace()} + var recreatedPlacementPolicy placementv1alpha1.PlacementPolicyAccessor + if placementPolicy.GetNamespace() == "" { + recreatedPlacementPolicy = &placementv1alpha1.ClusterPlacementPolicy{ObjectMeta: objectMeta, Spec: *placementPolicy.GetSpec().DeepCopy()} + } else { + recreatedPlacementPolicy = &placementv1alpha1.PlacementPolicy{ObjectMeta: objectMeta, Spec: *placementPolicy.GetSpec().DeepCopy()} + } + Expect(hubClient.Create(ctx, recreatedPlacementPolicy)).To(Succeed(), "Failed to re-create the placement policy") + Expect(recreatedPlacementPolicy.GetUID()).ToNot(Equal(previousUID), "The re-created placement policy has the same UID") + return recreatedPlacementPolicy +} + +// setSnapshotOwnerRefs sets the owner references of a placement resource snapshot (nil removes all of them), and +// waits until the cache has observed the change, as the manager reads placement resource snapshots from the cache. +func setSnapshotOwnerRefs(namespace, name string, ownerRefs []metav1.OwnerReference) { + GinkgoHelper() + + key := types.NamespacedName{Namespace: namespace, Name: name} + Eventually(func() error { + snapshot := placementResourceSnapshotObj(namespace, name) + if err := hubUncachedReader.Get(ctx, key, snapshot); err != nil { + return err + } + snapshot.SetOwnerReferences(ownerRefs) + return hubClient.Update(ctx, snapshot) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to set the owner references of placement resource snapshot %s", name) + Eventually(func() error { + snapshot := placementResourceSnapshotObj(namespace, name) + if err := hubClient.Get(ctx, key, snapshot); err != nil { + return err + } + if diff := cmp.Diff(snapshot.GetOwnerReferences(), ownerRefs, cmpopts.EquateEmpty()); diff != "" { + return fmt.Errorf("owner references in the cache diff (-got +want):\n%s", diff) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "The cache has not observed the owner references of placement resource snapshot %s", name) +} + +// updatePlacementBindingSnapshot points a placement binding (of the same scope as the placement policy) to the given +// primary placement resource snapshot. +func updatePlacementBindingSnapshot(placementPolicy placementv1alpha1.PlacementPolicyAccessor, name, snapshotName string) { + GinkgoHelper() + + Eventually(func() error { + placementBinding := placementBindingObj(placementPolicy.GetNamespace(), name) + if err := hubUncachedReader.Get(ctx, client.ObjectKeyFromObject(placementBinding), placementBinding); err != nil { + return err + } + placementBinding.GetSpec().ResourceSnapshotName = snapshotName + return hubClient.Update(ctx, placementBinding) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update placement binding %s", name) +} diff --git a/pkg/v1/utils/fieldindexers/hub.go b/pkg/v1/utils/fieldindexers/hub.go index 2e651cddb..29d6acab9 100644 --- a/pkg/v1/utils/fieldindexers/hub.go +++ b/pkg/v1/utils/fieldindexers/hub.go @@ -14,13 +14,13 @@ See the License for the specific language governing permissions and limitations under the License. */ +// Package fieldindexers provides utilities for setting up and using field-based indexes for KubeFleet API objects on the KubeFleet agent side. package fieldindexers import ( "context" "fmt" - "k8s.io/klog/v2" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" @@ -29,13 +29,13 @@ import ( ) const ( - // The field-based indexes set up for KubeFleet API objects. + // The field-based indexes set up for KubeFleet API objects (on the hub agent side). // - // Important: many KubeFleet components run under the assumption that proper custom fields + // Important: many KubeFleet components on the hub agent side run under the assumption that proper custom fields // have been added and indexed in the cache when running. Failure to complete such prior setup **before - // the manager starts** will result in unexpected behaviors. Make sure that all applicable components - // are properly set up using the client provided by the hub controller manager, and `SetupWithManager` is - // called before the manager starts. + // the manager starts** will result in unexpected behaviors/failures. Make sure that all applicable components + // are properly set up using the client provided by the hub controller manager, and + // `SetupWithHubAgentControllerManager` is called before the manager starts. // PlacementResourceSnapshotOwnedByAndSubIndexedCustomFieldName is the name of the custom field that indexes // placement resource snapshots by their owner placement policies and their sub-indices. @@ -50,6 +50,25 @@ const ( // This is added to help the placement resource snapshot manager retrieve all placement resource snapshots of a // specific index associated with a placement policy. PlacementResourceSnapshotOwnedByAndIndexedCustomFieldName = "ownedByWithIndex" + + // WorkOwnedByBindingCustomFieldName is the name of the custom field that indexes Work objects by their owner + // placement bindings. + WorkOwnedByBindingCustomFieldName = "workOwnedByBinding" + + // PlacementBindingOwnedByCustomFieldName is the name of the custom field that indexes placement binding objects + // by their owner placement policies. + PlacementBindingOwnedByCustomFieldName = "placementBindingOwnedBy" + + // PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName is the name of the custom field that indexes + // placement bindings by their owner placement policies and the (primary) resource snapshots they are using. + PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName = "ownedByAndInUseOfPrimaryResourceSnapshot" + + // ClusterClaimOwnedByPlacementPolicyCustomFieldName is the name of the custom field that indexes cluster + // claims by the placement policy that their immutable spec.placementPolicyRef names. + // + // This is added to help the placement policy controller retrieve the claims of a policy without scanning + // every claim in the fleet; the ownership labels are mutable, so the reference is what gets indexed. + ClusterClaimOwnedByPlacementPolicyCustomFieldName = "clusterClaimOwnedByPlacementPolicy" ) const ( @@ -66,55 +85,104 @@ const ( // // Note that slashes are used to avoid unexpected collisions. PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt = "%s/%s" -) -type fieldValueExtractor func(obj client.Object) ([]string, error) - -func indexCompositeField(ctx context.Context, - fieldIdxer client.FieldIndexer, - obj client.Object, - fieldName string, fieldValueExt fieldValueExtractor) error { - if err := fieldIdxer.IndexField(ctx, obj, fieldName, func(rawObj client.Object) []string { - fieldVals, extErr := fieldValueExt(rawObj) - if extErr != nil { - wrappedErr := errors.NewUnexpectedError(extErr, "failed to extract field value", "object", klog.KObj(rawObj)) - klog.ErrorS(wrappedErr, "failed to index field", errors.Args(wrappedErr)...) - return nil - } - return fieldVals - }); err != nil { - wrappedErr := errors.NewUnexpectedError(err, "", "fieldName", fieldName, "object", klog.KObj(obj)) - klog.ErrorS(wrappedErr, "failed to index field", errors.Args(wrappedErr)...) - return wrappedErr - } - return nil -} + // WorkOwnedByBindingCustomFieldValFmt is used to format the value for the custom field, + // `WorkOwnedByBindingCustomFieldName`. + // + // The first value is the owner binding namespace, and the second value is the owner binding name. + WorkOwnedByBindingCustomFieldValFmt = "%s/%s" + + // ClusterClaimOwnedByPlacementPolicyCustomFieldValFmt is used to format the value for the custom field, + // `ClusterClaimOwnedByPlacementPolicyCustomFieldName`. + // + // The values are the API group, kind, namespace, and name of the referenced placement policy; the API + // version is left out so that an API promotion does not orphan the claims of a policy. + ClusterClaimOwnedByPlacementPolicyCustomFieldValFmt = "%s/%s/%s/%s" + + // PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt is used to format the value for the custom field, + // `PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName`. + // + // The first value is the owner placement policy, and the second value is the primary resource snapshot being used. + PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt = "%s/%s" +) var ( placementResourceSnapshotOwnedByAndSubIdxedFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { - ownedBy := obj.GetLabels()[placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey] + ownedBy := obj.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] subIndex := obj.GetLabels()[placementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey] if ownedBy == "" || subIndex == "" { - wrappedErr := errors.NewUnexpectedError(nil, "placement resource snapshot is missing required labels") + wrappedErr := errors.NewUnexpectedError(nil, "placement resource snapshot is missing required labels or annotations") return nil, wrappedErr } return []string{fmt.Sprintf(PlacementResourceSnapshotOwnedByAndSubIndexedCustomFieldValFmt, ownedBy, subIndex)}, nil } placementResourceSnapshotOwnedByAndIdxedFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { - ownedBy := obj.GetLabels()[placementv1alpha1.PlacementResourceSnapshotOwnedByLabelKey] + ownedBy := obj.GetAnnotations()[placementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] index := obj.GetLabels()[placementv1alpha1.PlacementResourceSnapshotIndexLabelKey] if ownedBy == "" || index == "" { - wrappedErr := errors.NewUnexpectedError(nil, "placement resource snapshot is missing required labels") + wrappedErr := errors.NewUnexpectedError(nil, "placement resource snapshot is missing required labels or annotations") return nil, wrappedErr } return []string{fmt.Sprintf(PlacementResourceSnapshotOwnedByAndIndexedCustomFieldValFmt, ownedBy, index)}, nil } + + workOwnedByBindingFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + ownerNS, ownerNSFound := obj.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + ownerBinding := obj.GetLabels()[placementv1alpha1.WorkOwnedByPlacementBindingLabelKey] + if !ownerNSFound || ownerBinding == "" { + wrappedErr := errors.NewUnexpectedError(nil, "work is missing required owner binding metadata") + return nil, wrappedErr + } + return []string{fmt.Sprintf(WorkOwnedByBindingCustomFieldValFmt, ownerNS, ownerBinding)}, nil + } + + placementBindingOwnedByFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + binding, ok := obj.(placementv1alpha1.PlacementBindingAccessor) + if !ok { + return nil, errors.NewUnexpectedError(nil, "object is not a placement binding") + } + placementPolicyName := binding.GetSpec().PlacementPolicyName + if placementPolicyName == "" { + return nil, errors.NewUnexpectedError(nil, "placement binding is missing the placement policy name in its spec") + } + return []string{placementPolicyName}, nil + } + + placementBindingOwnedByAndInUseOfPrimaryResourceSnapshotFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + placementBinding, ok := obj.(placementv1alpha1.PlacementBindingAccessor) + if !ok { + wrappedErr := errors.NewUnexpectedError(nil, "object is not a placement binding", "type", fmt.Sprintf("%T", obj)) + return nil, wrappedErr + } + spec := placementBinding.GetSpec() + if spec.PlacementPolicyName == "" || spec.ResourceSnapshotName == "" { + wrappedErr := errors.NewUnexpectedError(nil, "placement binding is missing the owner placement policy name or the resource snapshot name") + return nil, wrappedErr + } + return []string{fmt.Sprintf(PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldValFmt, spec.PlacementPolicyName, spec.ResourceSnapshotName)}, nil + } + + clusterClaimOwnedByPlacementPolicyFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + claim, ok := obj.(*placementv1alpha1.ClusterClaim) + if !ok || claim.Spec.PlacementPolicyRef == nil { + wrappedErr := errors.NewUnexpectedError(nil, "object is not a cluster claim with a placement policy reference") + return nil, wrappedErr + } + return []string{ClusterClaimOwnedByPlacementPolicyCustomFieldVal(claim.Spec.PlacementPolicyRef)}, nil + } ) -// SetupWithHubControllerManager sets up the indices that controllers from the KubeFleet hub agent need to run properly. +// ClusterClaimOwnedByPlacementPolicyCustomFieldVal formats a placement policy reference as the value of the +// custom field, `ClusterClaimOwnedByPlacementPolicyCustomFieldName`. +func ClusterClaimOwnedByPlacementPolicyCustomFieldVal(ref *placementv1alpha1.ObjectReference) string { + return fmt.Sprintf(ClusterClaimOwnedByPlacementPolicyCustomFieldValFmt, ref.APIGroup, ref.Kind, ref.Namespace, ref.Name) +} + +// SetupWithHubAgentControllerManager sets up the indices that controllers from the KubeFleet hub agent need to run properly. +// // It must be called before the manager starts. -func SetupWithHubControllerManager(ctx context.Context, mgr ctrl.Manager) error { +func SetupWithHubAgentControllerManager(ctx context.Context, mgr ctrl.Manager) error { fieldIdxer := mgr.GetFieldIndexer() if err := indexCompositeField(ctx, fieldIdxer, @@ -145,5 +213,47 @@ func SetupWithHubControllerManager(ctx context.Context, mgr ctrl.Manager) error return errors.Wraps(err, "failed to set up cluster placement resource snapshot owner and index field index") } + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.Work{}, + WorkOwnedByBindingCustomFieldName, workOwnedByBindingFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up work owner binding field index") + } + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.PlacementBinding{}, + PlacementBindingOwnedByCustomFieldName, placementBindingOwnedByFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up placement binding owner field index") + } + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.PlacementBinding{}, + PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName, placementBindingOwnedByAndInUseOfPrimaryResourceSnapshotFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up placement binding owner and in-use primary resource snapshot field index") + } + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.ClusterPlacementBinding{}, + PlacementBindingOwnedByCustomFieldName, placementBindingOwnedByFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up cluster placement binding owner field index") + } + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.ClusterPlacementBinding{}, + PlacementBindingOwnedByAndInUseOfPrimaryResourceSnapshotCustomFieldName, placementBindingOwnedByAndInUseOfPrimaryResourceSnapshotFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up cluster placement binding owner and in-use primary resource snapshot field index") + } + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.ClusterClaim{}, + ClusterClaimOwnedByPlacementPolicyCustomFieldName, clusterClaimOwnedByPlacementPolicyFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up cluster claim owner placement policy field index") + } + return nil } diff --git a/pkg/v1/utils/fieldindexers/hub_test.go b/pkg/v1/utils/fieldindexers/hub_test.go new file mode 100644 index 000000000..b258125fd --- /dev/null +++ b/pkg/v1/utils/fieldindexers/hub_test.go @@ -0,0 +1,127 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package fieldindexers + +import ( + "testing" + + "github.com/google/go-cmp/cmp" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +// TestClusterClaimOwnedByPlacementPolicyFieldExtractor pins that cluster claims are indexed by every +// identifying part of the placement policy reference except its version. +func TestClusterClaimOwnedByPlacementPolicyFieldExtractor(t *testing.T) { + testCases := []struct { + name string + obj client.Object + want []string + wantErr bool + }{ + { + name: "namespaced policy", + obj: &placementv1alpha1.ClusterClaim{ + Spec: placementv1alpha1.ClusterClaimSpec{PlacementPolicyRef: &placementv1alpha1.ObjectReference{ + APIGroup: "placement.kubefleet.dev", APIVersion: "v1alpha1", Kind: "PlacementPolicy", Namespace: "work", Name: "app", + }}, + }, + want: []string{"placement.kubefleet.dev/PlacementPolicy/work/app"}, + }, + { + name: "cluster-scoped policy", + obj: &placementv1alpha1.ClusterClaim{ + Spec: placementv1alpha1.ClusterClaimSpec{PlacementPolicyRef: &placementv1alpha1.ObjectReference{ + APIGroup: "placement.kubefleet.dev", APIVersion: "v1beta1", Kind: "ClusterPlacementPolicy", Name: "app", + }}, + }, + want: []string{"placement.kubefleet.dev/ClusterPlacementPolicy//app"}, + }, + { + name: "no placement policy reference", + obj: &placementv1alpha1.ClusterClaim{ObjectMeta: metav1.ObjectMeta{Name: "orphan"}}, + wantErr: true, + }, + { + name: "not a cluster claim", + obj: &placementv1alpha1.Work{}, + wantErr: true, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := clusterClaimOwnedByPlacementPolicyFieldExtractor(tc.obj) + if (err != nil) != tc.wantErr { + t.Fatalf("clusterClaimOwnedByPlacementPolicyFieldExtractor() = %v, want error %t", err, tc.wantErr) + } + if diff := cmp.Diff(tc.want, got); diff != "" { + t.Errorf("clusterClaimOwnedByPlacementPolicyFieldExtractor() mismatch (-want +got):\n%s", diff) + } + // A lookup formats the policy's reference the same way the index does. + if claim, ok := tc.obj.(*placementv1alpha1.ClusterClaim); ok && !tc.wantErr { + if val := ClusterClaimOwnedByPlacementPolicyCustomFieldVal(claim.Spec.PlacementPolicyRef); val != got[0] { + t.Errorf("ClusterClaimOwnedByPlacementPolicyCustomFieldVal() = %q, want the indexed value %q", val, got[0]) + } + } + }) + } +} + +// TestPlacementBindingOwnedByFieldExtractor pins that both binding kinds are indexed +// by the name of their placement policy. +func TestPlacementBindingOwnedByFieldExtractor(t *testing.T) { + testCases := []struct { + name string + obj client.Object + want []string + wantErr bool + }{ + { + name: "placement binding", + obj: &placementv1alpha1.PlacementBinding{Spec: placementv1alpha1.PlacementBindingSpec{PlacementPolicyName: "app"}}, + want: []string{"app"}, + }, + { + name: "cluster placement binding", + obj: &placementv1alpha1.ClusterPlacementBinding{Spec: placementv1alpha1.PlacementBindingSpec{PlacementPolicyName: "app"}}, + want: []string{"app"}, + }, + { + name: "no placement policy name", + obj: &placementv1alpha1.PlacementBinding{}, + wantErr: true, + }, + { + name: "not a placement binding", + obj: &placementv1alpha1.Work{}, + wantErr: true, + }, + } + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + got, err := placementBindingOwnedByFieldExtractor(tc.obj) + if (err != nil) != tc.wantErr { + t.Fatalf("placementBindingOwnedByFieldExtractor() = %v, want error %t", err, tc.wantErr) + } + if diff := cmp.Diff(tc.want, got); diff != "" { + t.Errorf("placementBindingOwnedByFieldExtractor() mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/pkg/v1/utils/fieldindexers/member.go b/pkg/v1/utils/fieldindexers/member.go new file mode 100644 index 000000000..7a13ff447 --- /dev/null +++ b/pkg/v1/utils/fieldindexers/member.go @@ -0,0 +1,115 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package fieldindexers + +import ( + "context" + "fmt" + + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/utils/errors" +) + +const ( + // The field-based indexes set up for KubeFleet API objects (on the member agent side). + // + // Important: many KubeFleet components on the member agent side run under the assumption that proper custom fields + // have been added and indexed in the cache when running. Failure to complete such prior setup **before + // the manager starts** will result in unexpected behaviors/failures. Make sure that all applicable components + // are properly set up using the client provided by the member controller manager, and + // `SetupWithMemberAgentControllerManager` is called before the manager starts. + + // WorkOwnedByPlacementPolicyCustomFieldName is the name of the custom field that indexes + // work objects by their owner placement policies. + // + // This is added to help the work applier retrieve all work objects associated with a placement policy. + WorkOwnedByPlacementPolicyCustomFieldName = "ownedByPlacementPolicy" + + // WorkOwnedByPlacementBindingCustomFieldName is the name of the custom field that indexes + // work objects by their owner placement bindings. + // + // This is added to help the work applier retrieve all the work objects associated with a placement binding + // in one batch. + WorkOwnedByPlacementBindingCustomFieldName = "ownedByPlacementBinding" +) + +const ( + // The format of the custom field values for the field-based indexes defined above. + + // WorkOwnedByPlacementPolicyCustomFieldValFmt is used to format the value for the custom field, + // `WorkOwnedByPlacementPolicyCustomFieldName`, in the form of `[OWNER-NAMESPACE]/[OWNER-NAME]`. The owner + // namespace is empty for cluster-scoped placement policies. + // + // Note that slashes are used to avoid unexpected collisions. + WorkOwnedByPlacementPolicyCustomFieldValFmt = "%s/%s" + + // WorkOwnedByPlacementBindingCustomFieldValFmt is used to format the value for the custom field, + // `WorkOwnedByPlacementBindingCustomFieldName`, in the form of `[OWNER-NAMESPACE]/[OWNER-NAME]`. The owner + // namespace is empty for cluster-scoped placement bindings. + // + // Note that slashes are used to avoid unexpected collisions. + WorkOwnedByPlacementBindingCustomFieldValFmt = "%s/%s" +) + +var ( + workOwnedByPlacementPolicyFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + ownerNSName := obj.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + ownerName := obj.GetLabels()[placementv1alpha1.WorkOwnedByPlacementPolicyLabelKey] + if ownerName == "" { + wrappedErr := errors.NewUnexpectedError(nil, "work object is missing required labels") + return nil, wrappedErr + } + return []string{fmt.Sprintf(WorkOwnedByPlacementPolicyCustomFieldValFmt, ownerNSName, ownerName)}, nil + } + + workOwnedByPlacementBindingFieldExtractor fieldValueExtractor = func(obj client.Object) ([]string, error) { + ownerNSName := obj.GetLabels()[placementv1alpha1.WorkOwnerNamespaceLabelKey] + ownerName := obj.GetLabels()[placementv1alpha1.WorkOwnedByPlacementBindingLabelKey] + if ownerName == "" { + wrappedErr := errors.NewUnexpectedError(nil, "work object is missing required labels") + return nil, wrappedErr + } + return []string{fmt.Sprintf(WorkOwnedByPlacementBindingCustomFieldValFmt, ownerNSName, ownerName)}, nil + } +) + +// SetupWithMemberAgentControllerManager sets up the indices that controllers from the KubeFleet member agent need to +// run properly. +// +// It must be called before the manager starts. +func SetupWithMemberAgentControllerManager(ctx context.Context, mgr ctrl.Manager) error { + fieldIdxer := mgr.GetFieldIndexer() + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.Work{}, + WorkOwnedByPlacementPolicyCustomFieldName, workOwnedByPlacementPolicyFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up work owner and placement policy field index") + } + + if err := indexCompositeField(ctx, fieldIdxer, + &placementv1alpha1.Work{}, + WorkOwnedByPlacementBindingCustomFieldName, workOwnedByPlacementBindingFieldExtractor, + ); err != nil { + return errors.Wraps(err, "failed to set up work owner and placement binding field index") + } + + return nil +} diff --git a/pkg/v1/utils/fieldindexers/utils.go b/pkg/v1/utils/fieldindexers/utils.go new file mode 100644 index 000000000..2524467ce --- /dev/null +++ b/pkg/v1/utils/fieldindexers/utils.go @@ -0,0 +1,48 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package fieldindexers + +import ( + "context" + + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + + "go.goms.io/fleet/pkg/utils/errors" +) + +type fieldValueExtractor func(obj client.Object) ([]string, error) + +func indexCompositeField(ctx context.Context, + fieldIdxer client.FieldIndexer, + obj client.Object, + fieldName string, fieldValueExt fieldValueExtractor) error { + if err := fieldIdxer.IndexField(ctx, obj, fieldName, func(rawObj client.Object) []string { + fieldVals, extErr := fieldValueExt(rawObj) + if extErr != nil { + wrappedErr := errors.NewUnexpectedError(extErr, "failed to extract field value", "object", klog.KObj(rawObj)) + klog.ErrorS(wrappedErr, "failed to index field", errors.Args(wrappedErr)...) + return nil + } + return fieldVals + }); err != nil { + wrappedErr := errors.NewUnexpectedError(err, "", "fieldName", fieldName, "object", klog.KObj(obj)) + klog.ErrorS(wrappedErr, "failed to index field", errors.Args(wrappedErr)...) + return wrappedErr + } + return nil +} diff --git a/pkg/v1/utils/ownerreferences/refs.go b/pkg/v1/utils/ownerreferences/refs.go new file mode 100644 index 000000000..1a021e7d2 --- /dev/null +++ b/pkg/v1/utils/ownerreferences/refs.go @@ -0,0 +1,84 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package ownerreferences provides utilities for managing owner references of Kubernetes objects. +package ownerreferences + +import ( + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/klog/v2" + "sigs.k8s.io/controller-runtime/pkg/client" + + "go.goms.io/fleet/pkg/utils/errors" +) + +// Disown removes a specific owner reference from the given object. +// +// Note that this function will check for UID matches. To disown an object without checking for UIDs, +// call controllerutil.RemoveOwnerReference instead. +func Disown(obj *unstructured.Unstructured, owner *metav1.OwnerReference) { + ownerRefs := obj.GetOwnerReferences() + updatedOwnerRefs := make([]metav1.OwnerReference, 0, len(ownerRefs)) + + // Re-build the owner references; remove the given one from the list. + for idx := range ownerRefs { + if AreEqual(&ownerRefs[idx], owner) { + // Skip the expected owner reference. + continue + } + updatedOwnerRefs = append(updatedOwnerRefs, ownerRefs[idx]) + } + obj.SetOwnerReferences(updatedOwnerRefs) +} + +// AreEqual checks if two owner references are equal by comparing their UID, Name, Kind, and APIVersion fields. +func AreEqual(a, b *metav1.OwnerReference) bool { + return a.UID == b.UID && + a.Name == b.Name && + a.Kind == b.Kind && + a.APIVersion == b.APIVersion +} + +// Own adds a specific owner reference to the given object if it is not already present. +func Own(obj *unstructured.Unstructured, owner *metav1.OwnerReference) { + ownerRefs := obj.GetOwnerReferences() + for idx := range ownerRefs { + if AreEqual(&ownerRefs[idx], owner) { + // The object is already owned by the given owner. + return + } + } + ownerRefs = append(ownerRefs, *owner) + obj.SetOwnerReferences(ownerRefs) +} + +// IsOwnedByWithNoUIDCheck checks if the given object is owned by the specified owner without considering the UIDs. +func IsOwnedByWithNoUIDCheck(ctrlClient client.Client, obj metav1.Object, owner client.Object) (bool, error) { + ownerGVK, err := ctrlClient.GroupVersionKindFor(owner) + if err != nil { + return false, errors.NewUnexpectedError(err, "failed to resolve GVK for object", "ownerObj", klog.KObj(owner)) + } + ownerAPIVersion := ownerGVK.GroupVersion().String() + ownerKind := ownerGVK.Kind + + for _, ref := range obj.GetOwnerReferences() { + if ref.APIVersion == ownerAPIVersion && ref.Kind == ownerKind && ref.Name == owner.GetName() { + return true, nil + } + } + return false, nil +} diff --git a/pkg/v1/utils/sourcequeue/queue.go b/pkg/v1/utils/sourcequeue/queue.go new file mode 100644 index 000000000..a1052e001 --- /dev/null +++ b/pkg/v1/utils/sourcequeue/queue.go @@ -0,0 +1,76 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package sourcequeue feeds requests into a controller-runtime controller from outside the +// manager's watches. +package sourcequeue + +import ( + "context" + "sync" + + "k8s.io/client-go/util/workqueue" +) + +// Queue is a controller source that a producer outside the manager adds requests to. +// +// It is a source rather than a channel so that the producer is never blocked and repeated +// requests collapse: Add writes straight into the controller's own workqueue, which never blocks a +// writer and drops a request already queued. A channel would push both problems onto the +// producer, typically the resource watcher, which has a whole fleet's events to deliver and no +// business waiting on one controller. +// +// Requests added before the manager starts are held until it does, so the producer does not have +// to know when the controller became ready. That buffer collapses repeats, so it holds at most one +// entry per distinct request, but it has no other bound and is only ever drained by +// the controller calling Start, so a producer must not outlive the manager it was created for: in +// the hub agent both are started together under the same leader election. +// +// Register it with the controller builder's WatchesRawSource; the zero value is ready to use. +type Queue[T comparable] struct { + mu sync.Mutex + queue workqueue.TypedRateLimitingInterface[T] + pending map[T]struct{} +} + +// Add enqueues a request. It never blocks, and it is safe to call from any goroutine and before +// the manager has started. +func (q *Queue[T]) Add(req T) { + q.mu.Lock() + defer q.mu.Unlock() + if q.queue == nil { + if q.pending == nil { + q.pending = make(map[T]struct{}) + } + // Collapsed the same way the controller's queue collapses, so a request added many times + // before the manager started still costs one reconcile once it has. + q.pending[req] = struct{}{} + return + } + q.queue.Add(req) +} + +// Start hands the queue to the controller. It is called by the controller, never directly. +func (q *Queue[T]) Start(_ context.Context, queue workqueue.TypedRateLimitingInterface[T]) error { + q.mu.Lock() + defer q.mu.Unlock() + q.queue = queue + for req := range q.pending { + queue.Add(req) + } + q.pending = nil + return nil +} diff --git a/pkg/v1/utils/sourcequeue/queue_test.go b/pkg/v1/utils/sourcequeue/queue_test.go new file mode 100644 index 000000000..7b29e1f13 --- /dev/null +++ b/pkg/v1/utils/sourcequeue/queue_test.go @@ -0,0 +1,54 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package sourcequeue + +import ( + "context" + "slices" + "testing" + + "github.com/google/go-cmp/cmp" + "k8s.io/client-go/util/workqueue" +) + +// TestQueue pins that requests added before Start are held and collapsed, and that requests added +// after it go straight to the controller's queue, collapsed by the queue itself. +func TestQueue(t *testing.T) { + var q Queue[string] + q.Add("a") + q.Add("b") + q.Add("a") + + wq := workqueue.NewTypedRateLimitingQueue(workqueue.DefaultTypedControllerRateLimiter[string]()) + defer wq.ShutDown() + if err := q.Start(context.Background(), wq); err != nil { + t.Fatalf("Start() = %v, want no error", err) + } + q.Add("c") + q.Add("c") + + var got []string + for wq.Len() > 0 { + item, _ := wq.Get() + got = append(got, item) + wq.Done(item) + } + slices.Sort(got) + if diff := cmp.Diff([]string{"a", "b", "c"}, got); diff != "" { + t.Errorf("Queue handed the controller (-want +got):\n%s", diff) + } +} diff --git a/pkg/webhook/clusterresourceplacement/v1beta1_clusterresourceplacement_validating_webhook.go b/pkg/webhook/clusterresourceplacement/v1beta1_clusterresourceplacement_validating_webhook.go index 7b4b96e38..4947fd4eb 100644 --- a/pkg/webhook/clusterresourceplacement/v1beta1_clusterresourceplacement_validating_webhook.go +++ b/pkg/webhook/clusterresourceplacement/v1beta1_clusterresourceplacement_validating_webhook.go @@ -65,5 +65,5 @@ func (v *clusterResourcePlacementValidator) Handle(ctx context.Context, req admi // validateFunc func(obj placementv1beta1.PlacementObj) error { return validator.ValidateClusterResourcePlacement(obj.(*placementv1beta1.ClusterResourcePlacement)) - }) + }, nil) } diff --git a/pkg/webhook/fleetresourcehandler/fleetresourcehandler_webhook.go b/pkg/webhook/fleetresourcehandler/fleetresourcehandler_webhook.go index b9491a9de..53a688328 100644 --- a/pkg/webhook/fleetresourcehandler/fleetresourcehandler_webhook.go +++ b/pkg/webhook/fleetresourcehandler/fleetresourcehandler_webhook.go @@ -72,7 +72,13 @@ func (v *fleetResourceValidator) Handle(ctx context.Context, req admission.Reque case req.Kind == utils.NamespaceMetaGVK: klog.V(2).InfoS("handling namespace resource", "name", req.Name, "operation", req.Operation, "subResource", req.SubResource) response = v.handleNamespace(req) - case req.Kind == utils.IMCMetaGVK || req.Kind == utils.WorkMetaGVK || req.Kind == utils.EndpointSliceExportMetaGVK || req.Kind == utils.EndpointSliceImportMetaGVK || req.Kind == utils.InternalServiceExportMetaGVK || req.Kind == utils.InternalServiceImportMetaGVK: + case req.Kind == utils.IMCMetaGVK || + req.Kind == utils.WorkMetaGVK || + req.Kind == utils.PlacementPolicyWorkMetaGVK || + req.Kind == utils.EndpointSliceExportMetaGVK || + req.Kind == utils.EndpointSliceImportMetaGVK || + req.Kind == utils.InternalServiceExportMetaGVK || + req.Kind == utils.InternalServiceImportMetaGVK: klog.V(2).InfoS("handling fleet owned namespaced resource in fleet reserved namespaces", "GVK", req.RequestKind, "namespacedName", namespacedName, "operation", req.Operation, "subResource", req.SubResource) response = v.handleFleetReservedNamespacedResource(ctx, req) case req.Kind == utils.EventMetaGVK: diff --git a/pkg/webhook/fleetresourcehandler/fleetresourcehandler_webhook_test.go b/pkg/webhook/fleetresourcehandler/fleetresourcehandler_webhook_test.go index cbcb8effd..3edd1667c 100644 --- a/pkg/webhook/fleetresourcehandler/fleetresourcehandler_webhook_test.go +++ b/pkg/webhook/fleetresourcehandler/fleetresourcehandler_webhook_test.go @@ -8,6 +8,7 @@ import ( "testing" "github.com/crossplane/crossplane-runtime/v2/pkg/test" + "github.com/google/go-cmp/cmp" "github.com/stretchr/testify/assert" admissionv1 "k8s.io/api/admission/v1" authenticationv1 "k8s.io/api/authentication/v1" @@ -114,6 +115,34 @@ func TestHandleCRD(t *testing.T) { }, wantResponse: admission.Denied(fmt.Sprintf(validation.ResourceDeniedFormat, "test-user", utils.GenerateGroupString([]string{"test-group"}), admissionv1.Create, &utils.CRDMetaGVK, "", types.NamespacedName{Name: "memberclusters.cluster.kubernetes-fleet.io"})), }, + "deny non system user to modify placement policy API CRD": { + req: admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: "placementpolicies.placement.kubefleet.dev", + UserInfo: authenticationv1.UserInfo{ + Username: "test-user", + Groups: []string{"test-group"}, + }, + RequestKind: &utils.CRDMetaGVK, + Operation: admissionv1.Delete, + }, + }, + wantResponse: admission.Denied(fmt.Sprintf(validation.ResourceDeniedFormat, "test-user", utils.GenerateGroupString([]string{"test-group"}), admissionv1.Delete, &utils.CRDMetaGVK, "", types.NamespacedName{Name: "placementpolicies.placement.kubefleet.dev"})), + }, + "deny non system user to modify placement policy rollout API CRD": { + req: admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: "clusterstagedupdateruns.rollout.kubefleet.dev", + UserInfo: authenticationv1.UserInfo{ + Username: "test-user", + Groups: []string{"test-group"}, + }, + RequestKind: &utils.CRDMetaGVK, + Operation: admissionv1.Update, + }, + }, + wantResponse: admission.Denied(fmt.Sprintf(validation.ResourceDeniedFormat, "test-user", utils.GenerateGroupString([]string{"test-group"}), admissionv1.Update, &utils.CRDMetaGVK, "", types.NamespacedName{Name: "clusterstagedupdateruns.rollout.kubefleet.dev"})), + }, } for testName, testCase := range testCases { @@ -696,6 +725,110 @@ func TestHandleFleetReservedNamespacedResource(t *testing.T) { } } +// TestHandleWorksInFleetMemberNamespace verifies that Handle sends the Works of both the v1beta1 APIs and the +// placement policy APIs in a fleet member namespace through the member cluster identity check, so that a member +// agent which does not authenticate as a service account (e.g., one using an Azure identity) can update them. +func TestHandleWorksInFleetMemberNamespace(t *testing.T) { + memberNamespace := fmt.Sprintf(utils.NamespaceNameFormat, mcName) + mockClient := &test.MockClient{ + MockGet: func(ctx context.Context, key client.ObjectKey, obj client.Object) error { + if key.Name != mcName { + return errors.New("cannot find member cluster") + } + o := obj.(*clusterv1beta1.MemberCluster) + *o = clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: mcName, + }, + Spec: clusterv1beta1.MemberClusterSpec{ + Identity: rbacv1.Subject{ + Kind: rbacv1.UserKind, + Name: "test-identity", + }, + }, + } + return nil + }, + } + resourceValidator := fleetResourceValidator{ + client: mockClient, + whiteListedUsers: []string{"hub-agent"}, + } + + testCases := map[string]struct { + gvk metav1.GroupVersionKind + username string + operation admissionv1.Operation + subResource string + wantAllowed bool + }{ + "allow the member cluster identity to update the status of a placement policy Work": { + gvk: utils.PlacementPolicyWorkMetaGVK, + username: "test-identity", + operation: admissionv1.Update, + subResource: "status", + wantAllowed: true, + }, + "allow the member cluster identity to update a placement policy Work": { + gvk: utils.PlacementPolicyWorkMetaGVK, + username: "test-identity", + operation: admissionv1.Update, + wantAllowed: true, + }, + "deny other users to update the status of a placement policy Work": { + gvk: utils.PlacementPolicyWorkMetaGVK, + username: "test-user", + operation: admissionv1.Update, + subResource: "status", + wantAllowed: false, + }, + "allow white listed users to create a placement policy Work": { + gvk: utils.PlacementPolicyWorkMetaGVK, + username: "hub-agent", + operation: admissionv1.Create, + wantAllowed: true, + }, + "allow the member cluster identity to update the status of a v1beta1 Work": { + gvk: utils.WorkMetaGVK, + username: "test-identity", + operation: admissionv1.Update, + subResource: "status", + wantAllowed: true, + }, + } + + for testName, testCase := range testCases { + t.Run(testName, func(t *testing.T) { + gvk := testCase.gvk + groups := []string{"system:authenticated"} + namespacedName := types.NamespacedName{Name: "test-work", Namespace: memberNamespace} + req := admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: namespacedName.Name, + Namespace: namespacedName.Namespace, + Kind: gvk, + RequestKind: &gvk, + SubResource: testCase.subResource, + UserInfo: authenticationv1.UserInfo{ + Username: testCase.username, + Groups: groups, + }, + Operation: testCase.operation, + }, + } + + wantResponse := admission.Denied(fmt.Sprintf(validation.ResourceDeniedFormat, testCase.username, utils.GenerateGroupString(groups), testCase.operation, &gvk, testCase.subResource, namespacedName)) + if testCase.wantAllowed { + wantResponse = admission.Allowed(fmt.Sprintf(validation.ResourceAllowedFormat, testCase.username, utils.GenerateGroupString(groups), testCase.operation, &gvk, testCase.subResource, namespacedName)) + } + gotResponse := resourceValidator.Handle(context.Background(), req) + if diff := cmp.Diff(wantResponse, gotResponse); diff != "" { + t.Errorf("Handle() mismatch (-want +got):\n%s", diff) + } + }) + } +} + func TestHandleNamespace(t *testing.T) { testCases := map[string]struct { req admission.Request diff --git a/pkg/webhook/resourceplacement/v1beta1_resourceplacement_validating_webhook.go b/pkg/webhook/resourceplacement/v1beta1_resourceplacement_validating_webhook.go index b4dcc6ea8..254ff0800 100644 --- a/pkg/webhook/resourceplacement/v1beta1_resourceplacement_validating_webhook.go +++ b/pkg/webhook/resourceplacement/v1beta1_resourceplacement_validating_webhook.go @@ -36,13 +36,14 @@ var ( ) type resourcePlacementValidator struct { - decoder webhook.AdmissionDecoder + decoder webhook.AdmissionDecoder + authorizer validator.PlacementAuthorizer } // Add registers the webhook for K8s built-in object types. func Add(mgr manager.Manager) error { hookServer := mgr.GetWebhookServer() - hookServer.Register(ValidationPath, &webhook.Admission{Handler: &resourcePlacementValidator{admission.NewDecoder(mgr.GetScheme())}}) + hookServer.Register(ValidationPath, &webhook.Admission{Handler: &resourcePlacementValidator{decoder: admission.NewDecoder(mgr.GetScheme())}}) return nil } @@ -69,5 +70,6 @@ func (v *resourcePlacementValidator) Handle(ctx context.Context, req admission.R func(obj placementv1beta1.PlacementObj) error { return validator.ValidateResourcePlacement(obj.(*placementv1beta1.ResourcePlacement)) }, + v.authorizer, ) } diff --git a/pkg/webhook/validation/uservalidation.go b/pkg/webhook/validation/uservalidation.go index 3002570b9..441fb713e 100644 --- a/pkg/webhook/validation/uservalidation.go +++ b/pkg/webhook/validation/uservalidation.go @@ -10,6 +10,7 @@ import ( "strings" authenticationv1 "k8s.io/api/authentication/v1" + corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/equality" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" @@ -19,6 +20,8 @@ import ( "sigs.k8s.io/controller-runtime/pkg/webhook/admission" clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" "go.goms.io/fleet/pkg/utils" ) @@ -50,6 +53,8 @@ var ( clusterv1beta1.GroupVersion.Group, placementv1beta1.GroupVersion.Group, clusterinventory.GroupVersion.Group, + kfplacementv1alpha1.GroupVersion.Group, + kfrolloutv1alpha1.GroupVersion.Group, } ) @@ -95,8 +100,9 @@ func ValidateFleetMemberClusterUpdate(currentMC, oldMC clusterv1beta1.MemberClus isLabelUpdated := isMapFieldUpdated(currentMC.GetLabels(), oldMC.GetLabels()) if isLabelUpdated && !isUserInGroup(userInfo, mastersGroup) && shouldDenyLabelModification(currentMC.GetLabels(), oldMC.GetLabels(), denyModifyMemberClusterLabels, isUserAuthenticatedServiceAccount(userInfo)) { - // allow any user to modify kubernetes-fleet.io/* labels and service accounts to modify kubefleet.dev/* labels, - // but restricts other label modifications given denyModifyMemberClusterLabels is true. + // allow any user to modify kubernetes-fleet.io/* labels and service accounts to modify kubefleet.dev/* labels + // and the topology.kubernetes.io/region label, but restricts other label modifications given + // denyModifyMemberClusterLabels is true. klog.V(2).InfoS(DeniedModifyMemberClusterLabels, "user", userInfo.Username, "groups", userInfo.Groups, "operation", req.Operation, "GVK", req.RequestKind, "subResource", req.SubResource, "namespacedName", namespacedName) return admission.Denied(DeniedModifyMemberClusterLabels) } @@ -166,12 +172,16 @@ func isUserInGroup(userInfo authenticationv1.UserInfo, groupName string) bool { // every user; kubefleet.dev/* labels only for service accounts, so that the hub agent (which is not // in system:masters) can seed the cluster alias while a plain user under the guard cannot move an // alias, which is a scheduling label, from one cluster to another. +// +// Azure-specific change: the topology.kubernetes.io/region label is exempt for service accounts as well, +// for the same reason, so that the hub agent can seed it from the fleet.azure.com/location label. func shouldDenyLabelModification(currentLabels, oldLabels map[string]string, denyModifyMemberClusterLabels, isServiceAccount bool) bool { if !denyModifyMemberClusterLabels { return false } exempt := func(k string) bool { - return strings.HasPrefix(k, placementv1beta1.FleetPrefix) || (isServiceAccount && strings.HasPrefix(k, placementv1beta1.KubeFleetPrefix)) + return strings.HasPrefix(k, placementv1beta1.FleetPrefix) || + (isServiceAccount && (strings.HasPrefix(k, placementv1beta1.KubeFleetPrefix) || k == corev1.LabelTopologyRegion)) } for k, v := range currentLabels { oldV, exists := oldLabels[k] diff --git a/pkg/webhook/validation/uservalidation_test.go b/pkg/webhook/validation/uservalidation_test.go index b9a912710..663e020be 100644 --- a/pkg/webhook/validation/uservalidation_test.go +++ b/pkg/webhook/validation/uservalidation_test.go @@ -464,6 +464,114 @@ func TestValidateFleetMemberClusterUpdate(t *testing.T) { }, wantResponse: admission.Denied(DeniedModifyMemberClusterLabels), }, + // Azure-specific: the hub agent seeds topology.kubernetes.io/region from the fleet.azure.com/location + // label, together with the member name and cluster alias labels, in one update. + "allow label creation by service accounts for the topology.kubernetes.io/region label": { + denyModifyMemberClusterLabels: true, + oldMC: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-mc", + Labels: map[string]string{"fleet.azure.com/location": "eastus"}, + Annotations: map[string]string{ + "fleet.azure.com/cluster-resource-id": "test-cluster-resource-id", + }, + }, + }, + newMC: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-mc", + Labels: map[string]string{ + "fleet.azure.com/location": "eastus", + "kubernetes-fleet.io/member-name": "test-mc", + "kubefleet.dev/cluster-alias": "test-mc", + "topology.kubernetes.io/region": "eastus", + }, + Annotations: map[string]string{ + "fleet.azure.com/cluster-resource-id": "test-cluster-resource-id", + }, + }, + }, + req: admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: "test-mc", + UserInfo: authenticationv1.UserInfo{ + Username: "system:serviceaccount:fleet-system:hub-agent-sa", + Groups: []string{"system:serviceaccounts"}, + }, + RequestKind: &utils.MCMetaGVK, + Operation: admissionv1.Update, + }, + }, + wantResponse: admission.Allowed(fmt.Sprintf(ResourceAllowedFormat, "system:serviceaccount:fleet-system:hub-agent-sa", utils.GenerateGroupString([]string{"system:serviceaccounts"}), + admissionv1.Update, &utils.MCMetaGVK, "", types.NamespacedName{Name: "test-mc"})), + }, + "allow label modification by service accounts for the topology.kubernetes.io/region label": { + denyModifyMemberClusterLabels: true, + oldMC: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-mc", + Labels: map[string]string{"topology.kubernetes.io/region": "eastus"}, + Annotations: map[string]string{ + "fleet.azure.com/cluster-resource-id": "test-cluster-resource-id", + }, + }, + }, + newMC: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-mc", + Labels: map[string]string{"topology.kubernetes.io/region": "westus"}, + Annotations: map[string]string{ + "fleet.azure.com/cluster-resource-id": "test-cluster-resource-id", + }, + }, + }, + req: admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: "test-mc", + UserInfo: authenticationv1.UserInfo{ + Username: "system:serviceaccount:test-ns:test-sa", + Groups: []string{"system:serviceaccounts"}, + }, + RequestKind: &utils.MCMetaGVK, + Operation: admissionv1.Update, + }, + }, + wantResponse: admission.Allowed(fmt.Sprintf(ResourceAllowedFormat, "system:serviceaccount:test-ns:test-sa", utils.GenerateGroupString([]string{"system:serviceaccounts"}), + admissionv1.Update, &utils.MCMetaGVK, "", types.NamespacedName{Name: "test-mc"})), + }, + "deny label modification by non-system:masters user for the topology.kubernetes.io/region label": { + denyModifyMemberClusterLabels: true, + oldMC: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-mc", + Labels: map[string]string{"topology.kubernetes.io/region": "eastus"}, + Annotations: map[string]string{ + "fleet.azure.com/cluster-resource-id": "test-cluster-resource-id", + }, + }, + }, + newMC: &clusterv1beta1.MemberCluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "test-mc", + Labels: map[string]string{"topology.kubernetes.io/region": "westus"}, + Annotations: map[string]string{ + "fleet.azure.com/cluster-resource-id": "test-cluster-resource-id", + }, + }, + }, + req: admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: "test-mc", + UserInfo: authenticationv1.UserInfo{ + Username: "nonSystemMastersUser", + Groups: []string{"someGroup"}, + }, + RequestKind: &utils.MCMetaGVK, + Operation: admissionv1.Update, + }, + }, + wantResponse: admission.Denied(DeniedModifyMemberClusterLabels), + }, "allow label deletion by any user for kubernetes-fleet.io/* labels": { denyModifyMemberClusterLabels: true, oldMC: &clusterv1beta1.MemberCluster{ @@ -644,6 +752,56 @@ func TestValidateUserForFleetCRD(t *testing.T) { wantResponse: admission.Denied(fmt.Sprintf(ResourceDeniedFormat, "regular-user", utils.GenerateGroupString([]string{"regular-group"}), admissionv1.Create, &utils.CRDMetaGVK, "", types.NamespacedName{Name: "clusterprofiles.multicluster.x-k8s.io"})), }, + "deny non-admin user for placement policy API CRD": { + req: admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: "placementpolicies.placement.kubefleet.dev", + RequestKind: &utils.CRDMetaGVK, + UserInfo: authenticationv1.UserInfo{ + Username: "regular-user", + Groups: []string{"regular-group"}, + }, + Operation: admissionv1.Delete, + }, + }, + group: "placement.kubefleet.dev", + wantResponse: admission.Denied(fmt.Sprintf(ResourceDeniedFormat, "regular-user", utils.GenerateGroupString([]string{"regular-group"}), admissionv1.Delete, + &utils.CRDMetaGVK, "", types.NamespacedName{Name: "placementpolicies.placement.kubefleet.dev"})), + }, + "deny non-admin user for placement policy rollout API CRD": { + req: admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: "stagedupdateruns.rollout.kubefleet.dev", + RequestKind: &utils.CRDMetaGVK, + UserInfo: authenticationv1.UserInfo{ + Username: "regular-user", + Groups: []string{"regular-group"}, + }, + Operation: admissionv1.Update, + }, + }, + group: "rollout.kubefleet.dev", + wantResponse: admission.Denied(fmt.Sprintf(ResourceDeniedFormat, "regular-user", utils.GenerateGroupString([]string{"regular-group"}), admissionv1.Update, + &utils.CRDMetaGVK, "", types.NamespacedName{Name: "stagedupdateruns.rollout.kubefleet.dev"})), + }, + // The CRD installer runs with the hub agent's service account, which is white listed. + "allow white listed user for placement policy API CRD": { + req: admission.Request{ + AdmissionRequest: admissionv1.AdmissionRequest{ + Name: "works.placement.kubefleet.dev", + RequestKind: &utils.CRDMetaGVK, + UserInfo: authenticationv1.UserInfo{ + Username: "white-listed-user", + Groups: []string{"system:serviceaccounts"}, + }, + Operation: admissionv1.Update, + }, + }, + whiteListedUsers: []string{"white-listed-user"}, + group: "placement.kubefleet.dev", + wantResponse: admission.Allowed(fmt.Sprintf(ResourceAllowedFormat, "white-listed-user", utils.GenerateGroupString([]string{"system:serviceaccounts"}), admissionv1.Update, + &utils.CRDMetaGVK, "", types.NamespacedName{Name: "works.placement.kubefleet.dev"})), + }, "allow any user for non-fleet-managed CRD group": { req: admission.Request{ AdmissionRequest: admissionv1.AdmissionRequest{ diff --git a/pkg/webhook/webhook.go b/pkg/webhook/webhook.go index 449e3903c..8ff549737 100644 --- a/pkg/webhook/webhook.go +++ b/pkg/webhook/webhook.go @@ -499,10 +499,22 @@ func (w *Config) buildFleetValidatingWebhooks() []admv1.ValidatingWebhook { // When enableWorkload is true, skip pod and replicaset validating webhooks to allow workloads if !w.enableWorkload { + // The hub agent's own namespace is always allowed. Skipping it keeps these webhooks from blocking + // the hub agent's own pods and replica sets while the hub agent (and so the webhook) is down. + skipHubAgentNamespace := &metav1.LabelSelector{ + MatchExpressions: []metav1.LabelSelectorRequirement{ + { + Key: corev1.LabelMetadataName, + Operator: metav1.LabelSelectorOpNotIn, + Values: []string{w.serviceNamespace}, + }, + }, + } webHooks = append(webHooks, admv1.ValidatingWebhook{ Name: "fleet.pod.validating", ClientConfig: w.createClientConfig(pod.ValidationPath), FailurePolicy: &failFailurePolicy, + NamespaceSelector: skipHubAgentNamespace, SideEffects: &sideEffortsNone, AdmissionReviewVersions: admissionReviewVersions, Rules: []admv1.RuleWithOperations{ @@ -518,6 +530,7 @@ func (w *Config) buildFleetValidatingWebhooks() []admv1.ValidatingWebhook { Name: "fleet.replicaset.validating", ClientConfig: w.createClientConfig(replicaset.ValidationPath), FailurePolicy: &failFailurePolicy, + NamespaceSelector: skipHubAgentNamespace, SideEffects: &sideEffortsNone, AdmissionReviewVersions: admissionReviewVersions, Rules: []admv1.RuleWithOperations{ diff --git a/pkg/webhook/webhook_test.go b/pkg/webhook/webhook_test.go index d02d0d650..13bc95b0c 100644 --- a/pkg/webhook/webhook_test.go +++ b/pkg/webhook/webhook_test.go @@ -160,6 +160,37 @@ func TestBuildFleetJobWebhooks(t *testing.T) { } } +// TestBuildFleetValidatingWebhooks_SkipHubAgentNamespace checks that the pod and replica set webhooks skip the hub +// agent's own namespace, so that the hub agent's pods can be re-created while the hub agent is down. +func TestBuildFleetValidatingWebhooks_SkipHubAgentNamespace(t *testing.T) { + url := options.WebhookClientConnectionType("url") + config := Config{ + serviceNamespace: "hub-agent-namespace", + servicePort: 8080, + serviceURL: "test-url", + clientConnectionType: &url, + } + want := &metav1.LabelSelector{ + MatchExpressions: []metav1.LabelSelectorRequirement{ + { + Key: "kubernetes.io/metadata.name", + Operator: metav1.LabelSelectorOpNotIn, + Values: []string{"hub-agent-namespace"}, + }, + }, + } + + got := map[string]*metav1.LabelSelector{} + for _, wh := range config.buildFleetValidatingWebhooks() { + got[wh.Name] = wh.NamespaceSelector + } + for _, name := range []string{"fleet.pod.validating", "fleet.replicaset.validating"} { + if diff := cmp.Diff(want, got[name]); diff != "" { + t.Errorf("namespace selector of %s mismatch (-want +got):\n%s", name, diff) + } + } +} + func TestBuildFleetGuardRailValidatingWebhooks(t *testing.T) { url := options.WebhookClientConnectionType("url") testCases := map[string]struct { diff --git a/test/apis/kubefleet.dev/placement/v1alpha1/clusterclaim_validation_integration_test.go b/test/apis/kubefleet.dev/placement/v1alpha1/clusterclaim_validation_integration_test.go new file mode 100644 index 000000000..e91eebe6f --- /dev/null +++ b/test/apis/kubefleet.dev/placement/v1alpha1/clusterclaim_validation_integration_test.go @@ -0,0 +1,55 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +var _ = Describe("Test ClusterClaim API validation", func() { + It("should deny unsetting clusterSelectorTerms", func() { + clusterClaim := &placementv1alpha1.ClusterClaim{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cluster-claim-selector-terms-immutability", + }, + Spec: placementv1alpha1.ClusterClaimSpec{ + PlacementPolicyRef: &placementv1alpha1.ObjectReference{ + Name: "test-placement-policy", + APIVersion: placementv1alpha1.GroupVersion.Version, + Kind: "PlacementPolicy", + }, + ClusterSelectorTerms: []placementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + { + MatchLabels: map[string]string{"region": "west"}, + }, + }, + }, + } + Expect(hubClient.Create(ctx, clusterClaim)).Should(Succeed()) + DeferCleanup(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, clusterClaim))).Should(Succeed()) + }) + + clusterClaim.Spec.ClusterSelectorTerms = nil + Expect(hubClient.Update(ctx, clusterClaim)).Should(MatchError(ContainSubstring("the clusterSelectorTerms field cannot be added or removed after creation"))) + }) +}) diff --git a/test/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_validation_integration_test.go b/test/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_validation_integration_test.go new file mode 100644 index 000000000..269c30c89 --- /dev/null +++ b/test/apis/kubefleet.dev/placement/v1alpha1/placementpolicy_validation_integration_test.go @@ -0,0 +1,175 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + "errors" + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + k8sErrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "sigs.k8s.io/controller-runtime/pkg/client" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + placementPolicyNameTemplate = "test-placement-policy-%d" + + tolerationKey = "foo" + tolerationValue = "bar" + + operatorMustBeExistsMsg = "operator must be Exists when key is empty" + valueMustBeEmptyMsg = "value must be empty when operator is Exists" +) + +// testResourceSelectors is a resource selector list that is valid for both the +// PlacementPolicy and the ClusterPlacementPolicy APIs. +var testResourceSelectors = []placementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Name: "test-cm", + }, +} + +// createAndCheckValidation creates the given object and verifies that the API server accepts it +// (when wantMessage is empty) or rejects it with an error that contains wantMessage. Accepted +// objects are deleted when the current spec completes. +func createAndCheckValidation(obj client.Object, wantMessage string) { + GinkgoHelper() + + err := hubClient.Create(ctx, obj) + if wantMessage == "" { + Expect(err).To(Succeed(), "Expected the object to be accepted") + DeferCleanup(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed()) + }) + return + } + Expect(err).To(HaveOccurred(), "Expected the object to be rejected") + var statusErr *k8sErrors.StatusError + Expect(errors.As(err, &statusErr)).To(BeTrue(), "The returned error is not a StatusError") + Expect(statusErr.Status().Message).Should(ContainSubstring(wantMessage)) +} + +var _ = Describe("Test PlacementPolicy API validation", func() { + Context("Test toleration validation", func() { + // PlacementPolicy and ClusterPlacementPolicy share the same spec type, so each case + // runs against both kinds. + DescribeTable("the tolerations of a placement policy", + func(toleration placementv1alpha1.Toleration, wantMessage string) { + name := fmt.Sprintf(placementPolicyNameTemplate, GinkgoParallelProcess()) + spec := placementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: testResourceSelectors, + Tolerations: []placementv1alpha1.Toleration{toleration}, + } + + createAndCheckValidation(&placementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: metav1.NamespaceDefault, + }, + Spec: *spec.DeepCopy(), + }, wantMessage) + createAndCheckValidation(&placementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + }, + Spec: *spec.DeepCopy(), + }, wantMessage) + }, + Entry("an Exists toleration without a key or a value (tolerate all) is accepted", placementv1alpha1.Toleration{ + Operator: corev1.TolerationOpExists, + }, ""), + Entry("an Exists toleration with a key is accepted", placementv1alpha1.Toleration{ + Key: tolerationKey, + Operator: corev1.TolerationOpExists, + }, ""), + Entry("an Equal toleration with a key and a value is accepted", placementv1alpha1.Toleration{ + Key: tolerationKey, + Operator: corev1.TolerationOpEqual, + Value: tolerationValue, + }, ""), + Entry("a toleration with only a key (operator defaults to Equal) is accepted", placementv1alpha1.Toleration{ + Key: tolerationKey, + }, ""), + Entry("an empty toleration (operator defaults to Equal) is rejected", placementv1alpha1.Toleration{}, operatorMustBeExistsMsg), + Entry("an Equal toleration without a key is rejected", placementv1alpha1.Toleration{ + Operator: corev1.TolerationOpEqual, + Value: tolerationValue, + }, operatorMustBeExistsMsg), + Entry("an Exists toleration with a key and a value is rejected", placementv1alpha1.Toleration{ + Key: tolerationKey, + Operator: corev1.TolerationOpExists, + Value: tolerationValue, + }, valueMustBeEmptyMsg), + Entry("an Exists toleration with a value but no key is rejected", placementv1alpha1.Toleration{ + Operator: corev1.TolerationOpExists, + Value: tolerationValue, + }, valueMustBeEmptyMsg), + ) + + // The typed client drops empty keys and values (omitempty), so these cases use unstructured + // objects to send them explicitly, as a manifest applied with kubectl might. + DescribeTable("the explicitly empty fields of a toleration", + func(toleration map[string]interface{}, wantMessage string) { + name := fmt.Sprintf(placementPolicyNameTemplate, GinkgoParallelProcess()) + for _, kind := range []string{placementv1alpha1.PlacementPolicyKind, placementv1alpha1.ClusterPlacementPolicyKind} { + obj := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "spec": map[string]interface{}{ + "resourceSelectors": []interface{}{ + map[string]interface{}{ + "apiVersion": testResourceSelectors[0].APIVersion, + "kind": testResourceSelectors[0].Kind, + "name": testResourceSelectors[0].Name, + }, + }, + "tolerations": []interface{}{toleration}, + }, + }, + } + obj.SetGroupVersionKind(placementv1alpha1.GroupVersion.WithKind(kind)) + obj.SetName(name) + if kind == placementv1alpha1.PlacementPolicyKind { + obj.SetNamespace(metav1.NamespaceDefault) + } + createAndCheckValidation(obj, wantMessage) + } + }, + Entry("an Exists toleration with an empty key is accepted", map[string]interface{}{ + "key": "", + "operator": string(corev1.TolerationOpExists), + }, ""), + Entry("an Exists toleration with an empty value is accepted", map[string]interface{}{ + "key": tolerationKey, + "operator": string(corev1.TolerationOpExists), + "value": "", + }, ""), + Entry("an Equal toleration with an empty key is rejected", map[string]interface{}{ + "key": "", + "operator": string(corev1.TolerationOpEqual), + "value": tolerationValue, + }, operatorMustBeExistsMsg), + ) + }) +}) diff --git a/test/apis/kubefleet.dev/placement/v1alpha1/suite_test.go b/test/apis/kubefleet.dev/placement/v1alpha1/suite_test.go new file mode 100644 index 000000000..9a86fde3b --- /dev/null +++ b/test/apis/kubefleet.dev/placement/v1alpha1/suite_test.go @@ -0,0 +1,104 @@ +/* +Copyright 2025 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + "context" + "flag" + "path/filepath" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + placementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +var ( + hubTestEnv *envtest.Environment + hubClient client.Client + ctx context.Context + cancel context.CancelFunc +) + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Placement v1alpha1 API Validation Suite") +} + +var _ = BeforeSuite(func() { + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + ctx, cancel = context.WithCancel(context.TODO()) + + By("bootstrap the test environment") + // Start the cluster. + hubTestEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{ + filepath.Join("..", "..", "..", "..", "..", "config", "crd", "bases"), + }, + ErrorIfCRDPathMissing: true, + } + hubCfg, err := hubTestEnv.Start() + Expect(err).NotTo(HaveOccurred()) + Expect(hubCfg).NotTo(BeNil()) + + Expect(placementv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + klog.InitFlags(flag.CommandLine) + flag.Parse() + // Create the hub controller manager. + hubCtrlMgr, err := ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{ + BindAddress: "0", + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).NotTo(HaveOccurred()) + + // Set up the client. + // The client must be one with cache (i.e. configured by the controller manager) to make + // use of the cache indexes. + hubClient = hubCtrlMgr.GetClient() + Expect(hubClient).NotTo(BeNil()) + + go func() { + defer GinkgoRecover() + err = hubCtrlMgr.Start(ctx) + Expect(err).ToNot(HaveOccurred(), "failed to start manager for hub") + }() +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + cancel() + + By("tearing down the test environment") + Expect(hubTestEnv.Stop()).Should(Succeed()) +}) diff --git a/test/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_validation_integration_test.go b/test/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_validation_integration_test.go new file mode 100644 index 000000000..32050f2fb --- /dev/null +++ b/test/apis/kubefleet.dev/rollout/v1alpha1/stagedupdaterun_validation_integration_test.go @@ -0,0 +1,222 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + "errors" + "fmt" + "math" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + k8sErrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" +) + +const ( + updateStrategyNameTemplate = "test-update-strategy-%d" + updateRunNameTemplate = "test-update-run-%d" + stageName = "stage1" + placementPolicyName = "test-placement-policy" + updateStrategyName = "test-update-strategy" + + maxConcurrencyTooLowMsg = "maxConcurrency must be at least 1" + maxConcurrencyTooHighMsg = "maxConcurrency must be at most 1000" + waitTimeTooLongMsg = "waitTime must be at most 720h (30 days)" + waitTimeTooManyCharsMsg = "spec.stages[0].afterStageTasks[0].waitTime: Too long" + maxClusterFailuresTooHighMsg = "spec.failurePolicy.maxAllowedClusterFailures in body should be less than or equal to 1000" + maxClusterFailuresTooLowMsg = "spec.failurePolicy.maxAllowedClusterFailures in body should be greater than or equal to 0" + maxWaitTimePerClusterTooHighMsg = "spec.failurePolicy.maxWaitTimePerClusterMinutes in body should be less than or equal to 10080" +) + +// createAndCheckValidation creates the given object and verifies that the API server accepts it +// (when wantMessage is empty) or rejects it with an error that contains wantMessage. Accepted +// objects are deleted when the current spec completes. +func createAndCheckValidation(obj client.Object, wantMessage string) { + GinkgoHelper() + + err := hubClient.Create(ctx, obj) + if wantMessage == "" { + Expect(err).To(Succeed(), "Expected the object to be accepted") + DeferCleanup(func() { + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, obj))).To(Succeed()) + }) + return + } + Expect(err).To(HaveOccurred(), "Expected the object to be rejected") + var statusErr *k8sErrors.StatusError + Expect(errors.As(err, &statusErr)).To(BeTrue(), "The returned error is not a StatusError") + Expect(statusErr.Status().Message).Should(ContainSubstring(wantMessage)) +} + +// timedWaitAfterStageTasks returns an after-stage task list with a single TimedWait task. +func timedWaitAfterStageTasks(waitTime time.Duration) []rolloutv1alpha1.StageTask { + return []rolloutv1alpha1.StageTask{ + { + Type: rolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: waitTime}, + }, + } +} + +var _ = Describe("Test rollout v1alpha1 API validation", func() { + Context("Test StagedUpdateStrategy API validation", func() { + // StagedUpdateStrategy and ClusterStagedUpdateStrategy share the same spec type, so each + // case runs against both kinds. + DescribeTable("the bounds of a stage", + func(stage rolloutv1alpha1.Stage, wantMessage string) { + name := fmt.Sprintf(updateStrategyNameTemplate, GinkgoParallelProcess()) + stage.Name = stageName + spec := rolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []rolloutv1alpha1.Stage{stage}, + } + + createAndCheckValidation(&rolloutv1alpha1.StagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: metav1.NamespaceDefault, + }, + Spec: *spec.DeepCopy(), + }, wantMessage) + createAndCheckValidation(&rolloutv1alpha1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + }, + Spec: *spec.DeepCopy(), + }, wantMessage) + }, + Entry("an integer maxConcurrency of 1000 is accepted", rolloutv1alpha1.Stage{ + MaxConcurrency: ptr.To(intstr.FromInt32(1000)), + }, ""), + Entry("a percentage maxConcurrency of 100% is accepted", rolloutv1alpha1.Stage{ + MaxConcurrency: ptr.To(intstr.FromString("100%")), + }, ""), + Entry("an integer maxConcurrency of 0 is rejected", rolloutv1alpha1.Stage{ + MaxConcurrency: ptr.To(intstr.FromInt32(0)), + }, maxConcurrencyTooLowMsg), + Entry("an integer maxConcurrency of 1001 is rejected", rolloutv1alpha1.Stage{ + MaxConcurrency: ptr.To(intstr.FromInt32(1001)), + }, maxConcurrencyTooHighMsg), + Entry("a wait time of 720h is accepted", rolloutv1alpha1.Stage{ + AfterStageTasks: timedWaitAfterStageTasks(720 * time.Hour), + }, ""), + Entry("a wait time of 720h1s is rejected", rolloutv1alpha1.Stage{ + AfterStageTasks: timedWaitAfterStageTasks(720*time.Hour + time.Second), + }, waitTimeTooLongMsg), + Entry("the maximum wait time a client can encode is rejected", rolloutv1alpha1.Stage{ + AfterStageTasks: timedWaitAfterStageTasks(time.Duration(math.MaxInt64)), + }, waitTimeTooLongMsg), + ) + + // The typed client always encodes a wait time in its canonical form (e.g., 1h30m0s), so + // these cases use unstructured objects to send forms that only a raw manifest can carry. + DescribeTable("the raw wait time of a TimedWait task", + func(waitTime string, wantMessage string) { + name := fmt.Sprintf(updateStrategyNameTemplate, GinkgoParallelProcess()) + for _, kind := range []string{rolloutv1alpha1.StagedUpdateStrategyKind, rolloutv1alpha1.ClusterStagedUpdateStrategyKind} { + obj := &unstructured.Unstructured{ + Object: map[string]interface{}{ + "spec": map[string]interface{}{ + "stages": []interface{}{ + map[string]interface{}{ + "name": stageName, + "afterStageTasks": []interface{}{ + map[string]interface{}{ + "type": string(rolloutv1alpha1.StageTaskTypeTimedWait), + "waitTime": waitTime, + }, + }, + }, + }, + }, + }, + } + obj.SetGroupVersionKind(rolloutv1alpha1.GroupVersion.WithKind(kind)) + obj.SetName(name) + if kind == rolloutv1alpha1.StagedUpdateStrategyKind { + obj.SetNamespace(metav1.NamespaceDefault) + } + createAndCheckValidation(obj, wantMessage) + } + }, + Entry("a fractional wait time is accepted", "1.5h", ""), + Entry("720h expressed in minutes is accepted", "43200m", ""), + Entry("a wait time of 32 characters is accepted", strings.Repeat("1s", 16), ""), + Entry("a wait time over 720h expressed in minutes is rejected", "43201m", waitTimeTooLongMsg), + // The value overflows a duration; the API server cannot evaluate the rule on it, and + // reports the failure under the rule's message. + Entry("a wait time that overflows a duration is rejected", "9999999999999999999h", waitTimeTooLongMsg), + Entry("a wait time of 33 characters is rejected", "1"+strings.Repeat("0", 31)+"s", waitTimeTooManyCharsMsg), + ) + }) + + Context("Test StagedUpdateRun API validation", func() { + // StagedUpdateRun and ClusterStagedUpdateRun share the same spec type, so each case runs + // against both kinds. + DescribeTable("the bounds of a failure policy", + func(failurePolicy rolloutv1alpha1.StagedUpdateRunFailurePolicy, wantMessage string) { + name := fmt.Sprintf(updateRunNameTemplate, GinkgoParallelProcess()) + spec := rolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: placementPolicyName, + StagedUpdateStrategyName: updateStrategyName, + FailurePolicy: &failurePolicy, + } + + createAndCheckValidation(&rolloutv1alpha1.StagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + Namespace: metav1.NamespaceDefault, + }, + Spec: *spec.DeepCopy(), + }, wantMessage) + createAndCheckValidation(&rolloutv1alpha1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{ + Name: name, + }, + Spec: *spec.DeepCopy(), + }, wantMessage) + }, + Entry("the default failure policy is accepted", rolloutv1alpha1.StagedUpdateRunFailurePolicy{}, ""), + Entry("maxAllowedClusterFailures of 0 is accepted", rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(int32(0)), + }, ""), + Entry("maxAllowedClusterFailures of 1000 is accepted", rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(int32(1000)), + }, ""), + Entry("maxAllowedClusterFailures of -1 is rejected", rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(int32(-1)), + }, maxClusterFailuresTooLowMsg), + Entry("maxAllowedClusterFailures of 1001 is rejected", rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(int32(1001)), + }, maxClusterFailuresTooHighMsg), + Entry("maxWaitTimePerClusterMinutes of 10080 is accepted", rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxWaitTimePerClusterMinutes: 10080, + }, ""), + Entry("maxWaitTimePerClusterMinutes of 10081 is rejected", rolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxWaitTimePerClusterMinutes: 10081, + }, maxWaitTimePerClusterTooHighMsg), + ) + }) +}) diff --git a/test/apis/kubefleet.dev/rollout/v1alpha1/suite_test.go b/test/apis/kubefleet.dev/rollout/v1alpha1/suite_test.go new file mode 100644 index 000000000..a0c9ae98c --- /dev/null +++ b/test/apis/kubefleet.dev/rollout/v1alpha1/suite_test.go @@ -0,0 +1,104 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package v1alpha1 + +import ( + "context" + "flag" + "path/filepath" + "testing" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/klog/v2" + "k8s.io/klog/v2/textlogger" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/envtest" + metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server" + + rolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" +) + +var ( + hubTestEnv *envtest.Environment + hubClient client.Client + ctx context.Context + cancel context.CancelFunc +) + +func TestAPIs(t *testing.T) { + RegisterFailHandler(Fail) + + RunSpecs(t, "Rollout v1alpha1 API Validation Suite") +} + +var _ = BeforeSuite(func() { + By("Setup klog") + fs := flag.NewFlagSet("klog", flag.ContinueOnError) + klog.InitFlags(fs) + Expect(fs.Parse([]string{"--v", "5", "-add_dir_header", "true"})).Should(Succeed()) + + ctx, cancel = context.WithCancel(context.TODO()) + + By("bootstrap the test environment") + // Start the cluster. + hubTestEnv = &envtest.Environment{ + CRDDirectoryPaths: []string{ + filepath.Join("..", "..", "..", "..", "..", "config", "crd", "bases"), + }, + ErrorIfCRDPathMissing: true, + } + hubCfg, err := hubTestEnv.Start() + Expect(err).NotTo(HaveOccurred()) + Expect(hubCfg).NotTo(BeNil()) + + Expect(rolloutv1alpha1.AddToScheme(scheme.Scheme)).Should(Succeed()) + + klog.InitFlags(flag.CommandLine) + flag.Parse() + // Create the hub controller manager. + hubCtrlMgr, err := ctrl.NewManager(hubCfg, ctrl.Options{ + Scheme: scheme.Scheme, + Metrics: metricsserver.Options{ + BindAddress: "0", + }, + Logger: textlogger.NewLogger(textlogger.NewConfig(textlogger.Verbosity(4))), + }) + Expect(err).NotTo(HaveOccurred()) + + // Set up the client. + // The client must be one with cache (i.e. configured by the controller manager) to make + // use of the cache indexes. + hubClient = hubCtrlMgr.GetClient() + Expect(hubClient).NotTo(BeNil()) + + go func() { + defer GinkgoRecover() + err = hubCtrlMgr.Start(ctx) + Expect(err).ToNot(HaveOccurred(), "failed to start manager for hub") + }() +}) + +var _ = AfterSuite(func() { + defer klog.Flush() + cancel() + + By("tearing down the test environment") + Expect(hubTestEnv.Stop()).Should(Succeed()) +}) diff --git a/test/apis/placement/v1beta1/api_validation_integration_test.go b/test/apis/placement/v1beta1/api_validation_integration_test.go index 9e8c1f143..a50957840 100644 --- a/test/apis/placement/v1beta1/api_validation_integration_test.go +++ b/test/apis/placement/v1beta1/api_validation_integration_test.go @@ -2901,6 +2901,31 @@ var _ = Describe("Test placement v1beta1 API validation", func() { Expect(statusErr.ErrStatus.Message).Should(MatchRegexp("Too many: 3: must have at most 2 items")) }) + It("Should deny creation of ClusterStagedUpdateStrategy with BeforeStageTask of type Approval with waitTime specified", func() { + strategy := placementv1beta1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(updateRunStrategyNameTemplate, GinkgoParallelProcess()), + }, + Spec: placementv1beta1.UpdateStrategySpec{ + Stages: []placementv1beta1.StageConfig{ + { + Name: fmt.Sprintf(updateRunStageNameTemplate, GinkgoParallelProcess(), 1), + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + WaitTime: &metav1.Duration{Duration: time.Minute * 10}, + }, + }, + }, + }, + }, + } + err := hubClient.Create(ctx, &strategy) + var statusErr *k8sErrors.StatusError + Expect(errors.As(err, &statusErr)).To(BeTrue(), fmt.Sprintf("Create updateRunStrategy call produced error %s. Error type wanted is %s.", reflect.TypeOf(err), reflect.TypeOf(&k8sErrors.StatusError{}))) + Expect(statusErr.ErrStatus.Message).Should(MatchRegexp("BeforeStageTaskType is Approval, waitTime is not allowed")) + }) + It("Should deny creation of ClusterStagedUpdateStrategy with AfterStageTask of type Approval with waitTime specified", func() { strategy := placementv1beta1.ClusterStagedUpdateStrategy{ ObjectMeta: metav1.ObjectMeta{ @@ -3380,6 +3405,74 @@ var _ = Describe("Test placement v1beta1 API validation", func() { }) }) + // The delete stage tasks have the same rules in both scopes, as the cluster-scoped and the namespaced + // strategies share their spec; each CRD is generated separately, so both are checked. + Context("Test the delete stage of a StagedUpdateStrategy and a ClusterStagedUpdateStrategy", func() { + approval := placementv1beta1.StageTask{Type: placementv1beta1.StageTaskTypeApproval} + + deleteStageEntries := func(namespaced bool) []TableEntry { + scope := "ClusterStagedUpdateStrategy" + if namespaced { + scope = "StagedUpdateStrategy" + } + return []TableEntry{ + Entry(scope+": a delete stage without tasks is accepted", + namespaced, &placementv1beta1.DeleteStageConfig{}, ""), + Entry(scope+": a single Approval task is accepted", + namespaced, &placementv1beta1.DeleteStageConfig{BeforeStageTasks: []placementv1beta1.StageTask{approval}}, ""), + Entry(scope+": more than 1 task is rejected", + namespaced, &placementv1beta1.DeleteStageConfig{BeforeStageTasks: []placementv1beta1.StageTask{approval, approval}}, + "Too many: 2: must have at most 1 item"), + Entry(scope+": a TimedWait task is rejected", + namespaced, &placementv1beta1.DeleteStageConfig{BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeTimedWait, WaitTime: &metav1.Duration{Duration: time.Minute}}}}, + "BeforeStageTaskType cannot be TimedWait"), + Entry(scope+": an Approval task with a waitTime is rejected", + namespaced, &placementv1beta1.DeleteStageConfig{BeforeStageTasks: []placementv1beta1.StageTask{{Type: placementv1beta1.StageTaskTypeApproval, WaitTime: &metav1.Duration{Duration: 10 * time.Minute}}}}, + "BeforeStageTaskType is Approval, waitTime is not allowed"), + } + } + + DescribeTable("the before-stage tasks of the delete stage", + func(namespaced bool, deleteStage *placementv1beta1.DeleteStageConfig, wantMessage string) { + spec := placementv1beta1.UpdateStrategySpec{ + Stages: []placementv1beta1.StageConfig{ + { + Name: fmt.Sprintf(updateRunStageNameTemplate, GinkgoParallelProcess(), 1), + }, + }, + DeleteStage: deleteStage, + } + var strategy client.Object = &placementv1beta1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(updateRunStrategyNameTemplate, GinkgoParallelProcess()), + }, + Spec: spec, + } + if namespaced { + strategy = &placementv1beta1.StagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(updateRunStrategyNameTemplate, GinkgoParallelProcess()), + Namespace: testNamespace, + }, + Spec: spec, + } + } + + err := hubClient.Create(ctx, strategy) + if wantMessage == "" { + Expect(err).To(Succeed(), "Expected the strategy to be accepted") + Expect(hubClient.Delete(ctx, strategy)).To(Succeed()) + return + } + var statusErr *k8sErrors.StatusError + Expect(errors.As(err, &statusErr)).To(BeTrue(), fmt.Sprintf("Create updateRunStrategy call produced error %s. Error type wanted is %s.", reflect.TypeOf(err), reflect.TypeOf(&k8sErrors.StatusError{}))) + Expect(statusErr.ErrStatus.Message).Should(ContainSubstring(wantMessage)) + }, + deleteStageEntries(false), + deleteStageEntries(true), + ) + }) + Context("Test ClusterStagedUpdateRun State API validation - valid Initialize state transitions", func() { var updateRun *placementv1beta1.ClusterStagedUpdateRun updateRunName := fmt.Sprintf(validupdateRunNameTemplate, GinkgoParallelProcess()) diff --git a/test/e2e/annotation_placement_test.go b/test/e2e/annotation_placement_test.go new file mode 100644 index 000000000..d089df74b --- /dev/null +++ b/test/e2e/annotation_placement_test.go @@ -0,0 +1,395 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" +) + +// The event reason the hub agent records on a resource whose annotation cannot be parsed. It is +// spelled out here rather than imported so that a rename in the controller shows up as a failed +// spec, the same way it would show up for a user filtering events by reason. +const invalidClusterSelectorsAnnotationEventReason = "InvalidClusterSelectorsAnnotation" + +// generatedPolicyLabels returns the provenance labels the hub agent stamps on a policy it generates +// for the given resource; listing on them is how a user, and these specs, find the policy without +// knowing the generated name. +func generatedPolicyLabels(apiGroup, kind, name string) client.MatchingLabels { + return client.MatchingLabels{ + kfplacementv1alpha1.ParentAPIGroupLabel: apiGroup, + kfplacementv1alpha1.ParentKindLabel: kind, + kfplacementv1alpha1.ParentNameLabel: name, + } +} + +// setClusterSelectorsAnnotation sets, changes, or (with an empty value) removes the cluster +// selectors annotation on a hub resource, retrying on a conflict with the hub agent's own writes. +// Removing the annotation from a resource that is already gone counts as done, so that cleanup +// after a failed spec does not wait out the timeout on it. +func setClusterSelectorsAnnotation(object client.Object, value string) { + Eventually(func() error { + if err := hubClient.Get(ctx, client.ObjectKeyFromObject(object), object); err != nil { + if value == "" && apierrors.IsNotFound(err) { + return nil + } + return err + } + annotations := object.GetAnnotations() + if annotations == nil { + annotations = map[string]string{} + } + if value == "" { + delete(annotations, kfplacementv1alpha1.ClusterSelectorsAnnotation) + } else { + annotations[kfplacementv1alpha1.ClusterSelectorsAnnotation] = value + } + object.SetAnnotations(annotations) + return hubClient.Update(ctx, object) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to set the cluster selectors annotation on %s", object.GetName()) +} + +// annotationClusterSelector builds the cluster selector the hub agent generates from one segment of +// the annotation, with the fields the API defaults filled in as the API server stores them. +func annotationClusterSelector(matchLabels map[string]string, count intstr.IntOrString) kfplacementv1alpha1.ClusterSelector { + selector := kfplacementv1alpha1.ClusterSelector{ + Count: ptr.To(count), + WhenUnfulfilled: kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim, + } + if len(matchLabels) > 0 { + selector.Terms = []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{{MatchLabels: matchLabels}} + } + return selector +} + +// The FEP-0001 annotation-based placement experience is alpha and runs behind the +// enable-placement-policy-apis hub agent flag, which test/e2e/setup.sh turns on. These specs +// drive the feature the way a user does, through the annotation alone, and check the whole path +// that the controller's own envtest suite cannot: the resource watcher noticing the annotation, the +// generated policy watch noticing edits to the policy, and the event recorded on the resource. +var _ = Describe("annotation based placement", Label("placementpolicy"), Ordered, func() { + var ( + configMap corev1.ConfigMap + deployment appsv1.Deployment + namespace corev1.Namespace + configMapLabels client.MatchingLabels + ) + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + configMap = appConfigMap() + namespace = appNamespace() + configMapLabels = generatedPolicyLabels("", "ConfigMap", configMap.Name) + }) + + AfterAll(func() { + // A wait below that fails aborts the rest of this node; the work resources must go + // regardless, or they leak into every spec that follows in this process. + defer cleanupWorkResourcesOnHubAndMembers() + + // The annotations come off first so that the hub agent withdraws the generated policies + // itself, rather than leaving them to the garbage collector once the namespace goes; a + // policy that outlived its annotation would be the feature failing, not cleanup succeeding. + // Each object is un-annotated only if the node that created it got as far as naming it; + // otherwise the call would wait out its timeout on an empty name. + if configMap.Name != "" { + setClusterSelectorsAnnotation(&configMap, "") + } + if deployment.Name != "" { + setClusterSelectorsAnnotation(&deployment, "") + } + if namespace.Name != "" { + setClusterSelectorsAnnotation(&namespace, "") + } + Eventually(func() error { + policies := &kfplacementv1alpha1.ClusterPlacementPolicyList{} + if err := hubClient.List(ctx, policies, generatedPolicyLabels("", "Namespace", namespace.Name)); err != nil { + return err + } + if len(policies.Items) != 0 { + return fmt.Errorf("%d cluster placement policies generated for namespace %s are still present", len(policies.Items), namespace.Name) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to clean up the generated cluster placement policy") + + // The Deployment spec placed real pods; wait for them to leave so the namespace deletes + // that follow do not race pod termination. + var placed []client.Object + if deployment.Name != "" { + placed = append(placed, &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{Name: deployment.Name}}) + } + for _, c := range allMemberClusters { + Eventually(namespacedResourcesRemovedFromClusterActual(c, placed...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the placed resources from %s", c.ClusterName) + } + }) + + // generatedPolicy reads back the one policy generated for the ConfigMap, failing the caller's + // Eventually until exactly one exists. + generatedPolicy := func() (*kfplacementv1alpha1.PlacementPolicy, error) { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + if err := hubClient.List(ctx, policies, client.InNamespace(configMap.Namespace), configMapLabels); err != nil { + return nil, err + } + if len(policies.Items) != 1 { + return nil, fmt.Errorf("%d placement policies generated for the config map, want 1", len(policies.Items)) + } + return &policies.Items[0], nil + } + + noGeneratedPolicyActual := func() error { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + if err := hubClient.List(ctx, policies, client.InNamespace(configMap.Namespace), configMapLabels); err != nil { + return err + } + if len(policies.Items) != 0 { + return fmt.Errorf("%d placement policies generated for the config map, want none", len(policies.Items)) + } + return nil + } + + generatedPolicySpecActual := func(wantSelectors ...kfplacementv1alpha1.ClusterSelector) func() error { + wantSpec := kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: wantSelectors, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMap.Name, + }}, + ResourceRevisionHistoryLimit: ptr.To(int32(3)), + } + return func() error { + policy, err := generatedPolicy() + if err != nil { + return err + } + if diff := cmp.Diff(policy.Spec, wantSpec); diff != "" { + return fmt.Errorf("generated placement policy spec diff (-got, +want):\n%s", diff) + } + return nil + } + } + + It("should seed the cluster alias label on every member cluster", func() { + // The alias= shorthand of the annotation matches on this label; the member cluster + // reconciler seeds it from the cluster name, so that the shorthand works out of the box. + // + // The label is written by the member cluster reconciler, not at join time, so it is + // awaited rather than read once. + for _, name := range allMemberClusterNames { + Eventually(func() error { + mc := &clusterv1beta1.MemberCluster{} + if err := hubClient.Get(ctx, types.NamespacedName{Name: name}, mc); err != nil { + return err + } + if got := mc.Labels[placementv1beta1.ClusterAliasLabel]; got != name { + return fmt.Errorf("member cluster %s has alias label %q, want %q", name, got, name) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Member cluster %s does not carry its alias label", name) + } + }) + + It("should generate no policy for a resource that carries no annotation", func() { + Consistently(noGeneratedPolicyActual, consistentlyDuration, consistentlyInterval).Should(Succeed(), "A policy was generated for a resource without the annotation") + }) + + It("should generate a placement policy when the annotation is set", func() { + setClusterSelectorsAnnotation(&configMap, fmt.Sprintf("%s=%s,count=All", envLabelName, envCanary)) + + wantSelector := annotationClusterSelector(map[string]string{envLabelName: envCanary}, intstr.FromString("All")) + Eventually(generatedPolicySpecActual(wantSelector), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the placement policy") + + policy, err := generatedPolicy() + Expect(err).Should(Succeed()) + Expect(policy.Namespace).Should(Equal(configMap.Namespace), "A namespaced resource must generate a policy in its own namespace") + + Expect(hubClient.Get(ctx, client.ObjectKeyFromObject(&configMap), &configMap)).Should(Succeed()) + wantOwnerReferences := []metav1.OwnerReference{{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: configMap.Name, + UID: configMap.UID, + }} + diff := cmp.Diff(policy.OwnerReferences, wantOwnerReferences) + Expect(diff).To(BeEmpty(), "generated placement policy owner references diff (-got, +want):\n%s", diff) + + By("checking that the ConfigMap is applied on the canary cluster only") + Eventually(appConfigMapPlacedOnClustersActual(memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to apply the ConfigMap on the canary cluster") + Consistently(appConfigMapRemovedFromClustersActual(memberCluster1EastProd, memberCluster3WestProd), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The ConfigMap was applied on a cluster that was not selected") + }) + + It("should update the policy when the annotation changes", func() { + // The region shorthand expands to the well-known topology label rather than being matched + // literally. No e2e member carries that label, so the policy stays unscheduled; these specs + // check the generated spec only. + setClusterSelectorsAnnotation(&configMap, fmt.Sprintf("region=%s,count=2", regionEast)) + + wantSelector := annotationClusterSelector(map[string]string{corev1.LabelTopologyRegion: regionEast}, intstr.FromInt32(2)) + Eventually(generatedPolicySpecActual(wantSelector), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the placement policy") + + By("checking that the ConfigMap is removed from the canary cluster, which no longer matches") + Eventually(appConfigMapRemovedFromClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from the cluster that no longer matches") + }) + + It("should restore the policy when it is edited", func() { + // Retried: the placement policy controllers write the policy too. + Eventually(func() error { + policy, err := generatedPolicy() + if err != nil { + return err + } + policy.Spec.ClusterSelectors = nil + return hubClient.Update(ctx, policy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to edit the generated placement policy") + + // Nothing happened to the ConfigMap; only the watch on generated policies can bring the + // hub agent back to this resource. + wantSelector := annotationClusterSelector(map[string]string{corev1.LabelTopologyRegion: regionEast}, intstr.FromInt32(2)) + Eventually(generatedPolicySpecActual(wantSelector), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to restore the edited placement policy") + }) + + It("should recreate the policy when it is deleted", func() { + policy, err := generatedPolicy() + Expect(err).Should(Succeed()) + deletedUID := policy.UID + Expect(hubClient.Delete(ctx, policy)).Should(Succeed(), "Failed to delete the generated placement policy") + + Eventually(func() error { + policy, err := generatedPolicy() + if err != nil { + return err + } + if policy.UID == deletedUID { + return fmt.Errorf("the deleted placement policy is still present") + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to recreate the deleted placement policy") + wantSelector := annotationClusterSelector(map[string]string{corev1.LabelTopologyRegion: regionEast}, intstr.FromInt32(2)) + Eventually(generatedPolicySpecActual(wantSelector), eventuallyDuration, eventuallyInterval).Should(Succeed(), "The recreated placement policy does not match the annotation") + }) + + It("should keep the policy and record an event when the annotation becomes invalid", func() { + setClusterSelectorsAnnotation(&configMap, envLabelName) + + Eventually(func() error { + events := &corev1.EventList{} + if err := hubClient.List(ctx, events, client.InNamespace(configMap.Namespace), client.MatchingFields{ + "involvedObject.name": configMap.Name, + "reason": invalidClusterSelectorsAnnotationEventReason, + }); err != nil { + return err + } + if len(events.Items) == 0 { + return fmt.Errorf("no %s event recorded on the config map", invalidClusterSelectorsAnnotationEventReason) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to record the invalid annotation event") + + wantSelector := annotationClusterSelector(map[string]string{corev1.LabelTopologyRegion: regionEast}, intstr.FromInt32(2)) + Consistently(generatedPolicySpecActual(wantSelector), consistentlyDuration, consistentlyInterval).Should(Succeed(), "The policy from the last valid annotation was not left in place") + }) + + It("should delete the policy when the annotation is removed", func() { + setClusterSelectorsAnnotation(&configMap, "") + Eventually(noGeneratedPolicyActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to delete the placement policy") + }) + + It("should carry the API group of a non-core resource into the generated policy", func() { + // The ConfigMap above sits in the core group, whose empty name would also match a policy + // that forgot the group altogether; a Deployment pins the field to a real value. + deployment = appDeployment() + Expect(hubClient.Create(ctx, &deployment)).Should(Succeed(), "Failed to create the deployment") + setClusterSelectorsAnnotation(&deployment, fmt.Sprintf("%s=%s", envLabelName, envProd)) + + deploymentLabels := generatedPolicyLabels("apps", "Deployment", deployment.Name) + wantSpec := kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + annotationClusterSelector(map[string]string{envLabelName: envProd}, intstr.FromInt32(1)), + }, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIGroup: "apps", + APIVersion: "v1", + Kind: "Deployment", + Name: deployment.Name, + }}, + ResourceRevisionHistoryLimit: ptr.To(int32(3)), + } + Eventually(func() error { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + if err := hubClient.List(ctx, policies, client.InNamespace(deployment.Namespace), deploymentLabels); err != nil { + return err + } + if len(policies.Items) != 1 { + return fmt.Errorf("%d placement policies generated for the deployment, want 1", len(policies.Items)) + } + if diff := cmp.Diff(policies.Items[0].Spec, wantSpec); diff != "" { + return fmt.Errorf("generated placement policy spec diff (-got, +want):\n%s", diff) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the placement policy for the deployment") + }) + + It("should generate a cluster placement policy for a cluster scoped resource", func() { + // The alias shorthand expands to the cluster alias label the hub agent seeds above. + setClusterSelectorsAnnotation(&namespace, fmt.Sprintf("alias=%s", memberCluster1EastProdName)) + + wantSpec := kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + annotationClusterSelector(map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName}, intstr.FromInt32(1)), + }, + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{{ + APIVersion: "v1", + Kind: "Namespace", + Name: namespace.Name, + }}, + ResourceRevisionHistoryLimit: ptr.To(int32(3)), + } + Eventually(func() error { + policies := &kfplacementv1alpha1.ClusterPlacementPolicyList{} + if err := hubClient.List(ctx, policies, generatedPolicyLabels("", "Namespace", namespace.Name)); err != nil { + return err + } + if len(policies.Items) != 1 { + return fmt.Errorf("%d cluster placement policies generated for the namespace, want 1", len(policies.Items)) + } + if diff := cmp.Diff(policies.Items[0].Spec, wantSpec); diff != "" { + return fmt.Errorf("generated cluster placement policy spec diff (-got, +want):\n%s", diff) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the cluster placement policy") + }) +}) diff --git a/test/e2e/cluster_staged_updaterun_test.go b/test/e2e/cluster_staged_updaterun_test.go index ed778d4ce..197baab14 100644 --- a/test/e2e/cluster_staged_updaterun_test.go +++ b/test/e2e/cluster_staged_updaterun_test.go @@ -691,6 +691,123 @@ var _ = Describe("test CRP rollout with staged update run", func() { }) }) + Context("Test cluster shrink with before-stage tasks for the delete stage", Ordered, func() { + var strategy *placementv1beta1.ClusterStagedUpdateStrategy + updateRunNames := []string{} + + BeforeAll(func() { + // Create a test namespace and a configMap inside it on the hub cluster. + createWorkResources() + + // Create the CRP with external rollout strategy and pick fixed policy. + crp := &placementv1beta1.ClusterResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{ + Name: crpName, + // Add a custom finalizer; this would allow us to better observe + // the behavior of the controllers. + Finalizers: []string{customDeletionBlockerFinalizer}, + }, + Spec: placementv1beta1.PlacementSpec{ + ResourceSelectors: workResourceSelector(), + Policy: &placementv1beta1.PlacementPolicy{ + PlacementType: placementv1beta1.PickFixedPlacementType, + ClusterNames: []string{allMemberClusterNames[0], allMemberClusterNames[1]}, // member-cluster-1 and member-cluster-2 + }, + Strategy: placementv1beta1.RolloutStrategy{ + Type: placementv1beta1.ExternalRolloutStrategyType, + }, + }, + } + Expect(hubClient.Create(ctx, crp)).To(Succeed(), "Failed to create CRP") + + // Create the clusterStagedUpdateStrategy with a single stage and a gated delete stage. + strategy = &placementv1beta1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: strategyName, + }, + Spec: updateStrategySpecWithDeleteStageTasks(), + } + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create ClusterStagedUpdateStrategy") + + for i := 0; i < 2; i++ { + updateRunNames = append(updateRunNames, fmt.Sprintf(clusterStagedUpdateRunNameWithSubIndexTemplate, GinkgoParallelProcess(), i)) + } + }) + + AfterAll(func() { + // Remove the custom deletion blocker finalizer from the CRP. + ensureCRPAndRelatedResourcesDeleted(crpName, allMemberClusters) + + // Remove all the clusterStagedUpdateRuns. + for _, name := range updateRunNames { + ensureClusterStagedUpdateRunDeletion(name) + } + + // Delete the clusterStagedUpdateStrategy. + ensureClusterUpdateRunStrategyDeletion(strategyName) + }) + + It("Should successfully schedule the crp", func() { + validateLatestClusterSchedulingPolicySnapshot(crpName, policySnapshotIndex1st, 2) + }) + + It("Should create a cluster staged update run successfully", func() { + createClusterStagedUpdateRunWithAutoCreatedSnapshot(updateRunNames[0], crpName, strategyName, placementv1beta1.StateRun) + }) + + It("Should rollout resources to member-cluster-1 and member-cluster-2 and complete the cluster staged update run without the delete stage tasks as there is no cluster to delete", func() { + csurSucceededActual := testutilsupdaterun.ClusterStagedUpdateRunStatusSucceededActual(ctx, hubClient, updateRunNames[0], resourceSnapshotIndex1st, policySnapshotIndex1st, 2, defaultApplyStrategy, &strategy.Spec, [][]string{{allMemberClusterNames[0], allMemberClusterNames[1]}}, nil, nil, nil, true) + Eventually(csurSucceededActual, updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to validate updateRun %s succeeded", updateRunNames[0]) + checkIfPlacedWorkResourcesOnMemberClustersInUpdateRun([]*framework.Cluster{allMemberClusters[0], allMemberClusters[1]}) + }) + + It("Update the crp to only keep member-cluster-3", func() { + Eventually(func() error { + crp := &placementv1beta1.ClusterResourcePlacement{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: crpName}, crp); err != nil { + return fmt.Errorf("failed to get the crp: %w", err) + } + crp.Spec.Policy.ClusterNames = []string{allMemberClusterNames[2]} + return hubClient.Update(ctx, crp) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the crp to only keep member-cluster-3") + }) + + It("Should successfully schedule the crp", func() { + validateLatestClusterSchedulingPolicySnapshot(crpName, policySnapshotIndex2nd, 1) + }) + + It("Should create a cluster staged update run successfully", func() { + createClusterStagedUpdateRunSucceed(updateRunNames[1], crpName, resourceSnapshotIndex1st, strategyName, placementv1beta1.StateRun) + }) + + It("Should rollout resources to member-cluster-3 and keep them on member-cluster-1 and member-cluster-2 until the delete stage tasks complete", func() { + checkIfPlacedWorkResourcesOnMemberClustersInUpdateRun([]*framework.Cluster{allMemberClusters[2]}) + + By("Validating the approval request of the delete stage is created") + Eventually(func() error { + appReq := &placementv1beta1.ClusterApprovalRequest{} + return hubClient.Get(ctx, client.ObjectKey{Name: fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunNames[1], placementv1beta1.UpdateRunDeleteStageTaskName)}, appReq) + }, updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the approval request of the delete stage") + + By("Validating the resources are kept while the delete stage waits for the approval") + checkIfPlacedWorkResourcesOnMemberClustersConsistently(allMemberClusters) + }) + + It("Should remove resources on member-cluster-1 and member-cluster-2 after approval and complete the cluster staged update run successfully", func() { + validateAndApproveClusterApprovalRequests(updateRunNames[1], placementv1beta1.UpdateRunDeleteStageTaskName, placementv1beta1.BeforeStageApprovalTaskNameFmt, placementv1beta1.BeforeStageTaskLabelValue) + + csurSucceededActual := testutilsupdaterun.ClusterStagedUpdateRunStatusSucceededActual(ctx, hubClient, updateRunNames[1], resourceSnapshotIndex1st, policySnapshotIndex2nd, 1, defaultApplyStrategy, &strategy.Spec, [][]string{{allMemberClusterNames[2]}}, []string{allMemberClusterNames[0], allMemberClusterNames[1]}, nil, nil, true) + Eventually(csurSucceededActual, 2*updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to validate updateRun %s succeeded", updateRunNames[1]) + checkIfRemovedWorkResourcesFromMemberClusters([]*framework.Cluster{allMemberClusters[0], allMemberClusters[1]}) + checkIfPlacedWorkResourcesOnMemberClustersConsistently([]*framework.Cluster{allMemberClusters[2]}) + }) + + It("Should update crp status as completed with member-cluster-3 only", func() { + crpStatusUpdatedActual := crpStatusWithExternalStrategyActual(workResourceIdentifiers(), resourceSnapshotIndex1st, true, []string{allMemberClusterNames[2]}, []string{resourceSnapshotIndex1st}, []bool{true}, nil, nil) + Eventually(crpStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to keep CRP %s status as expected", crpName) + }) + }) + Context("Test cluster scale out and shrink using pickN policy with staged update run", Ordered, func() { var strategy *placementv1beta1.ClusterStagedUpdateStrategy updateRunNames := []string{} @@ -1996,6 +2113,27 @@ var _ = Describe("Test member cluster join and leave flow with updateRun", Label }) }) +// updateStrategySpecWithDeleteStageTasks returns an update strategy that updates all the clusters in one stage +// and waits for an approval before the delete stage. +func updateStrategySpecWithDeleteStageTasks() placementv1beta1.UpdateStrategySpec { + return placementv1beta1.UpdateStrategySpec{ + Stages: []placementv1beta1.StageConfig{ + { + Name: "all", + LabelSelector: &metav1.LabelSelector{}, + MaxConcurrency: ptr.To(intstr.FromString("100%")), + }, + }, + DeleteStage: &placementv1beta1.DeleteStageConfig{ + BeforeStageTasks: []placementv1beta1.StageTask{ + { + Type: placementv1beta1.StageTaskTypeApproval, + }, + }, + }, + } +} + func createClusterStagedUpdateStrategySucceed(strategyName string) *placementv1beta1.ClusterStagedUpdateStrategy { strategy := &placementv1beta1.ClusterStagedUpdateStrategy{ ObjectMeta: metav1.ObjectMeta{ diff --git a/test/e2e/placement_policy_actuals_test.go b/test/e2e/placement_policy_actuals_test.go new file mode 100644 index 000000000..be917b388 --- /dev/null +++ b/test/e2e/placement_policy_actuals_test.go @@ -0,0 +1,1247 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + "slices" + "strings" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/test/e2e/framework" +) + +// placementPolicyStatusActual compares the status of a placement policy with the wanted one in one shot. +// +// Unless the wanted status names a latest resource revision, the actual fills in the name of the policy's latest +// primary resource snapshot; it also sets the observed generation of every wanted condition that names none to the +// policy's generation. +func placementPolicyStatusActual(key types.NamespacedName, wantStatus kfplacementv1alpha1.PlacementPolicyStatus) func() error { + return comparePlacementPolicyStatusActual(key, wantStatus, true) +} + +// placementPolicyStatusWithoutResourceRevisionActual is placementPolicyStatusActual for a policy that has no +// resource snapshot yet, so its status names no latest resource revision either. +func placementPolicyStatusWithoutResourceRevisionActual(key types.NamespacedName, wantStatus kfplacementv1alpha1.PlacementPolicyStatus) func() error { + return comparePlacementPolicyStatusActual(key, wantStatus, false) +} + +func comparePlacementPolicyStatusActual(key types.NamespacedName, wantStatus kfplacementv1alpha1.PlacementPolicyStatus, fillInLatestRevision bool) func() error { + return func() error { + policy, err := retrievePlacementPolicy(key) + if err != nil { + return err + } + + want := wantStatus.DeepCopy() + if fillInLatestRevision && want.LatestResourceRevisionName == nil { + latest, err := latestPrimarySnapshotOfPlacementPolicy(policy) + if err != nil { + return err + } + want.LatestResourceRevisionName = ptr.To(latest.GetName()) + } + for i := range want.Conditions { + if want.Conditions[i].ObservedGeneration == 0 { + want.Conditions[i].ObservedGeneration = policy.GetGeneration() + } + } + + if diff := cmp.Diff(*policy.GetStatus(), *want, placementPolicyCmpOptions); diff != "" { + return fmt.Errorf("placement policy %s status diff (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// placementBindingView is the part of a placement binding that the specs compare in one shot. +type placementBindingView struct { + ControlledByPolicy bool + Deleting bool + Spec kfplacementv1alpha1.PlacementBindingSpec + Status kfplacementv1alpha1.PlacementBindingStatus +} + +// wantPlacementBinding describes the binding that a spec expects for one cluster. +type wantPlacementBinding struct { + selectors []kfplacementv1alpha1.ClusterSelectorWithTermsOnly + // snapshotName is the resource snapshot that the binding points to and has processed; empty means the policy's + // latest primary snapshot. + snapshotName string + // status is the wanted binding status. The actual fills in the last processed snapshot name and the observed + // generations of the binding-level conditions. + status kfplacementv1alpha1.PlacementBindingStatus +} + +// placementPolicyBindingsActual compares, keyed by cluster, the bindings that name the policy with the wanted ones +// in one shot. Each wanted binding is controlled by the policy, not being deleted, and carries the policy's sync +// strategy. +func placementPolicyBindingsActual(key types.NamespacedName, wantBindings map[string]wantPlacementBinding) func() error { + return comparePlacementPolicyBindingsActual(key, wantBindings, false) +} + +// placementPolicyBindingsIgnoringOrphansActual is placementPolicyBindingsActual for a policy re-created under the name +// of one deleted with orphan propagation: the orphaned bindings that it has not adopted, i.e., those that no object +// controls, are left out of the comparison. +func placementPolicyBindingsIgnoringOrphansActual(key types.NamespacedName, wantBindings map[string]wantPlacementBinding) func() error { + return comparePlacementPolicyBindingsActual(key, wantBindings, true) +} + +func comparePlacementPolicyBindingsActual(key types.NamespacedName, wantBindings map[string]wantPlacementBinding, ignoreOrphans bool) func() error { + return func() error { + policy, err := retrievePlacementPolicy(key) + if err != nil { + return err + } + latest, err := latestPrimarySnapshotOfPlacementPolicy(policy) + if err != nil { + return err + } + bindings, err := placementPolicyBindings(policy) + if err != nil { + return err + } + + got := make(map[string]placementBindingView, len(bindings)) + generations := make(map[string]int64, len(bindings)) + for _, binding := range bindings { + if ignoreOrphans && metav1.GetControllerOf(binding) == nil { + continue + } + clusterName := binding.GetSpec().ClusterName + if _, found := got[clusterName]; found { + return fmt.Errorf("placement policy %s has more than one binding for cluster %s", key, clusterName) + } + got[clusterName] = placementBindingView{ + ControlledByPolicy: metav1.IsControlledBy(binding, policy), + Deleting: !binding.GetDeletionTimestamp().IsZero(), + Spec: *binding.GetSpec(), + Status: *binding.GetStatus(), + } + generations[clusterName] = binding.GetGeneration() + } + + want := make(map[string]placementBindingView, len(wantBindings)) + for clusterName, wantBinding := range wantBindings { + snapshotName := wantBinding.snapshotName + if snapshotName == "" { + snapshotName = latest.GetName() + } + wantStatus := wantBinding.status.DeepCopy() + wantStatus.LastProcessedResourceSnapshotName = ptr.To(snapshotName) + for i := range wantStatus.Conditions { + wantStatus.Conditions[i].ObservedGeneration = generations[clusterName] + } + want[clusterName] = placementBindingView{ + ControlledByPolicy: true, + Spec: kfplacementv1alpha1.PlacementBindingSpec{ + PlacementPolicyName: key.Name, + ClusterSelectors: wantBinding.selectors, + ClusterName: clusterName, + ResourceSnapshotName: snapshotName, + SyncStrategy: policy.GetSpec().SyncStrategy, + }, + Status: *wantStatus, + } + } + + // Time values (e.g., when a diff was first observed) carry no contract either. + if diff := cmp.Diff(got, want, placementPolicyCmpOptions, ignoreTimeTypeFields); diff != "" { + return fmt.Errorf("placement policy %s bindings diff, keyed by cluster (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// allSynchronizedPlacementPolicyBindingsActual checks that the policy has exactly one binding per wanted cluster, +// each pointing to the policy's latest primary resource snapshot, recording the given cluster selectors, and +// reporting all of its wantResourceCount resources as synchronized and available. +func allSynchronizedPlacementPolicyBindingsActual( + key types.NamespacedName, + wantSelectorsByCluster map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, + wantResourceCount int32, +) func() error { + return placementPolicyBindingsActual(key, allSynchronizedPlacementBindings(wantSelectorsByCluster, wantResourceCount)) +} + +// allSynchronizedAdoptedPlacementPolicyBindingsActual is allSynchronizedPlacementPolicyBindingsActual for a policy +// re-created under the name of one deleted with orphan propagation; see placementPolicyBindingsIgnoringOrphansActual. +func allSynchronizedAdoptedPlacementPolicyBindingsActual( + key types.NamespacedName, + wantSelectorsByCluster map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, + wantResourceCount int32, +) func() error { + return placementPolicyBindingsIgnoringOrphansActual(key, allSynchronizedPlacementBindings(wantSelectorsByCluster, wantResourceCount)) +} + +// allSynchronizedPlacementBindings returns the wanted bindings, keyed by cluster, that record the given cluster +// selectors and report all of their wantResourceCount resources as synchronized and available. +func allSynchronizedPlacementBindings( + wantSelectorsByCluster map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, + wantResourceCount int32, +) map[string]wantPlacementBinding { + wantBindings := make(map[string]wantPlacementBinding, len(wantSelectorsByCluster)) + for clusterName, selectors := range wantSelectorsByCluster { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: selectors, + status: synchronizedPlacementBindingStatus(wantResourceCount), + } + } + return wantBindings +} + +// configMapDataOnClustersActual checks that the ConfigMap exists on each given cluster with exactly the given data. +func configMapDataOnClustersActual(key types.NamespacedName, wantData map[string]string, clusters ...*framework.Cluster) func() error { + return func() error { + for _, c := range clusters { + cm := &corev1.ConfigMap{} + if err := c.KubeClient.Get(ctx, key, cm); err != nil { + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", c.ClusterName, key, err) + } + if diff := cmp.Diff(cm.Data, wantData); diff != "" { + return fmt.Errorf("cluster %s: ConfigMap %s data diff (-got, +want):\n%s", c.ClusterName, key, diff) + } + } + return nil + } +} + +// configMapAbsentOnClustersActual checks that none of the given clusters has the ConfigMap. +func configMapAbsentOnClustersActual(key types.NamespacedName, clusters ...*framework.Cluster) func() error { + return func() error { + for _, c := range clusters { + err := c.KubeClient.Get(ctx, key, &corev1.ConfigMap{}) + switch { + case err == nil: + return fmt.Errorf("cluster %s: ConfigMap %s exists", c.ClusterName, key) + case !apierrors.IsNotFound(err): + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", c.ClusterName, key, err) + } + } + return nil + } +} + +// configMapTakenOverOnClusterActual checks that the ConfigMap on the cluster matches its hub copy and is owned by an +// AppliedWork of the placement policy APIs, i.e., the placement policy work applier has taken it over. +func configMapTakenOverOnClusterActual(cluster *framework.Cluster, key types.NamespacedName) func() error { + return func() error { + if err := validateConfigMapOnCluster(cluster, key); err != nil { + return fmt.Errorf("cluster %s: %w", cluster.ClusterName, err) + } + cm := &corev1.ConfigMap{} + if err := cluster.KubeClient.Get(ctx, key, cm); err != nil { + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", cluster.ClusterName, key, err) + } + if !ownedByPlacementPolicyAppliedWork(cm) { + return fmt.Errorf("cluster %s: ConfigMap %s is not owned by an AppliedWork of the placement policy APIs (owner references: %+v)", + cluster.ClusterName, key, cm.OwnerReferences) + } + return nil + } +} + +// noPlacementPolicyWorksActual checks that no Work on the hub was generated for the policy's bindings. +func noPlacementPolicyWorksActual(key types.NamespacedName) func() error { + return func() error { + works, err := placementPolicyWorks(key) + if err != nil { + return err + } + if len(works) > 0 { + return fmt.Errorf("work %s of placement policy %s still exists", client.ObjectKeyFromObject(&works[0]), key) + } + return nil + } +} + +// configMapOrphanedOnClustersActual checks that the ConfigMap stays on each given cluster with the given data, but is +// no longer owned by any AppliedWork, i.e., KubeFleet has let go of it. +func configMapOrphanedOnClustersActual(key types.NamespacedName, wantData map[string]string, clusters ...*framework.Cluster) func() error { + dataActual := configMapDataOnClustersActual(key, wantData, clusters...) + return func() error { + if err := dataActual(); err != nil { + return err + } + for _, c := range clusters { + cm := &corev1.ConfigMap{} + if err := c.KubeClient.Get(ctx, key, cm); err != nil { + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", c.ClusterName, key, err) + } + for _, ownerRef := range cm.OwnerReferences { + if ownerRef.Kind == kfplacementv1alpha1.AppliedWorkKind { + return fmt.Errorf("cluster %s: ConfigMap %s is still owned by AppliedWork %s (%s)", c.ClusterName, key, ownerRef.Name, ownerRef.APIVersion) + } + } + } + return nil + } +} + +// deploymentAvailableOnClustersActual checks that the Deployment on each given cluster has rolled out its current +// generation and has all of its replicas available. +func deploymentAvailableOnClustersActual(key types.NamespacedName, clusters ...*framework.Cluster) func() error { + return func() error { + for _, c := range clusters { + deploy := &appsv1.Deployment{} + if err := c.KubeClient.Get(ctx, key, deploy); err != nil { + return fmt.Errorf("cluster %s: failed to get Deployment %s: %w", c.ClusterName, key, err) + } + wantReplicas := ptr.Deref(deploy.Spec.Replicas, 1) + if deploy.Status.ObservedGeneration != deploy.Generation || deploy.Status.AvailableReplicas != wantReplicas { + return fmt.Errorf("cluster %s: Deployment %s has %d of %d replicas available (observed generation %d, generation %d)", + c.ClusterName, key, deploy.Status.AvailableReplicas, wantReplicas, deploy.Status.ObservedGeneration, deploy.Generation) + } + } + return nil + } +} + +// stagedUpdateRunSucceededActual checks that a staged update run (or a cluster staged update run) has completed +// successfully. +func stagedUpdateRunSucceededActual(key types.NamespacedName) func() error { + return func() error { + run, err := retrieveStagedUpdateRun(key) + if err != nil { + return err + } + cond := meta.FindStatusCondition(run.GetStatus().Conditions, kfrolloutv1alpha1.StagedUpdateRunCondTypeCompleted) + if cond == nil || cond.Status != metav1.ConditionTrue || cond.Reason != kfrolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded || + cond.ObservedGeneration != run.GetGeneration() { + return fmt.Errorf("staged update run %s has not completed successfully yet (Completed condition: %+v)", key, cond) + } + return nil + } +} + +// onlyCanaryUpdatedActual checks the state of a policy over all three member clusters midway through a rollout whose +// first stage is the canary cluster: the canary cluster's binding points to the new resource snapshot and the cluster +// has the new data, while the prod clusters' bindings still point to the initial snapshot and the clusters keep the +// initial data. +func onlyCanaryUpdatedActual( + policyKey, cmKey types.NamespacedName, + initialSnapshotName, newSnapshotName string, + initialData, newData map[string]string, +) func() error { + snapshotsActual := placementPolicyBindingSnapshotsActual(policyKey, map[string]string{ + memberCluster2EastCanaryName: newSnapshotName, + memberCluster1EastProdName: initialSnapshotName, + memberCluster3WestProdName: initialSnapshotName, + }) + canaryActual := configMapDataOnClustersActual(cmKey, newData, memberCluster2EastCanary) + prodActual := configMapDataOnClustersActual(cmKey, initialData, memberCluster1EastProd, memberCluster3WestProd) + return func() error { + if err := snapshotsActual(); err != nil { + return err + } + if err := canaryActual(); err != nil { + return err + } + return prodActual() + } +} + +// stagedUpdateRunConditionActual checks that a staged update run (or a cluster staged update run) reports the given +// condition with the given status and reason, for its current generation. +func stagedUpdateRunConditionActual(key types.NamespacedName, condType string, wantStatus metav1.ConditionStatus, wantReason string) func() error { + return func() error { + run, err := retrieveStagedUpdateRun(key) + if err != nil { + return err + } + cond := meta.FindStatusCondition(run.GetStatus().Conditions, condType) + if cond == nil || cond.Status != wantStatus || cond.Reason != wantReason || cond.ObservedGeneration != run.GetGeneration() { + return fmt.Errorf("staged update run %s reports %s condition %+v, want status %s and reason %s for generation %d", + key, condType, cond, wantStatus, wantReason, run.GetGeneration()) + } + return nil + } +} + +// stagedUpdateRunClusterResultsActual compares, keyed by cluster, how a staged update run (or a cluster staged update +// run) has ended the update of each cluster in any of its stages: the reason of the cluster's Completed condition, or +// an empty string for a cluster whose update has not completed (or not started). +func stagedUpdateRunClusterResultsActual(key types.NamespacedName, wantResults map[string]string) func() error { + return func() error { + run, err := retrieveStagedUpdateRun(key) + if err != nil { + return err + } + got := map[string]string{} + for _, stage := range run.GetStatus().Stages { + for _, cluster := range stage.Clusters { + result := "" + if cond := meta.FindStatusCondition(cluster.Conditions, kfrolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted); cond != nil && cond.Status == metav1.ConditionTrue { + result = cond.Reason + } + got[cluster.ClusterName] = result + } + } + if diff := cmp.Diff(got, wantResults); diff != "" { + return fmt.Errorf("staged update run %s cluster results diff, keyed by cluster (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// placementPolicyBindingManagerActual checks which process, if any (a nil wantManager), holds the binding manager role +// of the policy. +func placementPolicyBindingManagerActual(key types.NamespacedName, wantManager *kfplacementv1alpha1.BindingManager) func() error { + return func() error { + policy, err := retrievePlacementPolicy(key) + if err != nil { + return err + } + if diff := cmp.Diff(policy.GetStatus().BindingManager, wantManager); diff != "" { + return fmt.Errorf("placement policy %s binding manager diff (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// placementPolicyBindingSnapshotsActual compares, keyed by cluster, the resource snapshots that the bindings naming +// the policy point to. +func placementPolicyBindingSnapshotsActual(key types.NamespacedName, wantSnapshotNames map[string]string) func() error { + return func() error { + bindings, err := placementPolicyBindingsByCluster(key) + if err != nil { + return err + } + got := make(map[string]string, len(bindings)) + for clusterName, binding := range bindings { + got[clusterName] = binding.GetSpec().ResourceSnapshotName + } + if diff := cmp.Diff(got, wantSnapshotNames); diff != "" { + return fmt.Errorf("placement policy %s binding resource snapshots diff, keyed by cluster (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// approvalRequestAbsentActual checks that the approval request (or cluster approval request) is gone. +func approvalRequestAbsentActual(key types.NamespacedName) func() error { + return func() error { + _, err := retrieveApprovalRequest(key) + switch { + case err == nil: + return fmt.Errorf("approval request %s still exists", key) + case !apierrors.IsNotFound(err): + return err + } + return nil + } +} + +// appliedWorksAbsentOnClustersActual checks that none of the given AppliedWorks, keyed by the name of the member +// cluster they would be on, exists. An AppliedWork carries the name of its Work. +func appliedWorksAbsentOnClustersActual(namesByCluster map[string][]string) func() error { + return func() error { + for clusterName, names := range namesByCluster { + cluster := memberClusterNamed(clusterName) + for _, name := range names { + err := cluster.KubeClient.Get(ctx, types.NamespacedName{Name: name}, &kfplacementv1alpha1.AppliedWork{}) + switch { + case err == nil: + return fmt.Errorf("cluster %s: AppliedWork %s still exists", clusterName, name) + case !apierrors.IsNotFound(err): + return fmt.Errorf("cluster %s: failed to get AppliedWork %s: %w", clusterName, name, err) + } + } + } + return nil + } +} + +// namespaceLeftUnmanagedOnClustersActual checks that the namespace exists on each given cluster, not being deleted +// and not owned by any AppliedWork: the work applier created it for a namespaced resource and leaves it behind. +func namespaceLeftUnmanagedOnClustersActual(name string, clusters ...*framework.Cluster) func() error { + return func() error { + for _, c := range clusters { + ns := &corev1.Namespace{} + if err := c.KubeClient.Get(ctx, types.NamespacedName{Name: name}, ns); err != nil { + return fmt.Errorf("cluster %s: failed to get namespace %s: %w", c.ClusterName, name, err) + } + if !ns.DeletionTimestamp.IsZero() { + return fmt.Errorf("cluster %s: namespace %s is being deleted", c.ClusterName, name) + } + for _, ownerRef := range ns.OwnerReferences { + if ownerRef.Kind == kfplacementv1alpha1.AppliedWorkKind { + return fmt.Errorf("cluster %s: namespace %s is owned by AppliedWork %s (%s)", c.ClusterName, name, ownerRef.Name, ownerRef.APIVersion) + } + } + } + return nil + } +} + +// noPlacementPolicyBindingsActual checks that no binding names the policy. +func noPlacementPolicyBindingsActual(key types.NamespacedName) func() error { + return func() error { + bindings, err := placementPolicyBindingsByCluster(key) + if err != nil { + return err + } + if len(bindings) > 0 { + return fmt.Errorf("placement policy %s has bindings for clusters %v, want none", key, slices.Sorted(maps.Keys(bindings))) + } + return nil + } +} + +// placementPolicyLeftoversRemovedActual checks that neither a binding nor a resource snapshot of a deleted policy +// is left on the hub. The garbage collector removes them only after the policy is gone, and a policy re-created +// under the same name before then would find them; specs that re-create a policy wait for this first. +func placementPolicyLeftoversRemovedActual(key types.NamespacedName) func() error { + return func() error { + if err := noPlacementPolicyBindingsActual(key)(); err != nil { + return err + } + + snapshots, err := placementPolicySnapshots(key) + if err != nil { + return err + } + if len(snapshots) > 0 { + return fmt.Errorf("resource snapshot %s of placement policy %s still exists", snapshots[0].GetName(), key) + } + return nil + } +} + +// placementPolicyPrimarySnapshotNamesActual checks that the primary resource snapshots that name the policy are +// exactly the given ones, i.e., that the older ones have been garbage-collected and the given ones kept. +func placementPolicyPrimarySnapshotNamesActual(key types.NamespacedName, wantNames ...string) func() error { + return func() error { + snapshots, err := placementPolicySnapshots(key) + if err != nil { + return err + } + var gotNames []string + for _, snapshot := range snapshots { + if snapshot.GetLabels()[kfplacementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey] == "0" { + gotNames = append(gotNames, snapshot.GetName()) + } + } + if diff := cmp.Diff(gotNames, wantNames, cmpopts.SortSlices(func(a, b string) bool { return a < b })); diff != "" { + return fmt.Errorf("placement policy %s primary resource snapshots diff (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// placementPolicyConditionMessageActual checks that the policy reports the condition of the given type with a +// message that contains the given text. Specs use it where several causes share a condition reason, to tell which +// one the policy reports. +func placementPolicyConditionMessageActual(key types.NamespacedName, condType, wantText string) func() error { + return func() error { + policy, err := retrievePlacementPolicy(key) + if err != nil { + return err + } + cond := meta.FindStatusCondition(policy.GetStatus().Conditions, condType) + if cond == nil { + return fmt.Errorf("placement policy %s has no %s condition", key, condType) + } + if !strings.Contains(cond.Message, wantText) { + return fmt.Errorf("placement policy %s condition %s has message %q, want it to contain %q", key, condType, cond.Message, wantText) + } + return nil + } +} + +// configMapManagedOnClustersActual checks that the ConfigMap on each given cluster has exactly the given data and is +// owned by an AppliedWork of the placement policy APIs. Unlike configMapTakenOverOnClusterActual, it allows data that +// the hub copy does not have, such as keys that only the member cluster set. +func configMapManagedOnClustersActual(key types.NamespacedName, wantData map[string]string, clusters ...*framework.Cluster) func() error { + dataActual := configMapDataOnClustersActual(key, wantData, clusters...) + return func() error { + if err := dataActual(); err != nil { + return err + } + for _, c := range clusters { + cm := &corev1.ConfigMap{} + if err := c.KubeClient.Get(ctx, key, cm); err != nil { + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", c.ClusterName, key, err) + } + if !ownedByPlacementPolicyAppliedWork(cm) { + return fmt.Errorf("cluster %s: ConfigMap %s is not owned by an AppliedWork of the placement policy APIs (owner references: %+v)", + c.ClusterName, key, cm.OwnerReferences) + } + } + return nil + } +} + +// placementPolicyClaimActual checks that the policy has exactly one cluster claim outstanding, referencing the +// policy and carrying the given selector terms. +func placementPolicyClaimActual(key types.NamespacedName, wantTerms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm) func() error { + kind := kfplacementv1alpha1.PlacementPolicyKind + if key.Namespace == "" { + kind = kfplacementv1alpha1.ClusterPlacementPolicyKind + } + wantSpec := kfplacementv1alpha1.ClusterClaimSpec{ + PlacementPolicyRef: &kfplacementv1alpha1.ObjectReference{ + APIGroup: kfplacementv1alpha1.GroupVersion.Group, + APIVersion: kfplacementv1alpha1.GroupVersion.Version, + Kind: kind, + Namespace: key.Namespace, + Name: key.Name, + }, + ClusterSelectorTerms: wantTerms, + } + return func() error { + claims, err := claimsOfPolicy(key.Name, key.Namespace) + if err != nil { + return err + } + if len(claims.Items) != 1 { + return fmt.Errorf("placement policy %s has %d cluster claims, want 1", key, len(claims.Items)) + } + claim := &claims.Items[0] + if !claim.DeletionTimestamp.IsZero() { + return fmt.Errorf("cluster claim %s of placement policy %s is being withdrawn", claim.Name, key) + } + if diff := cmp.Diff(claim.Spec, wantSpec); diff != "" { + return fmt.Errorf("cluster claim %s of placement policy %s spec diff (-got, +want):\n%s", claim.Name, key, diff) + } + return nil + } +} + +// noPlacementPolicyClaimsActual checks that the policy has no cluster claim, not even one being withdrawn. +func noPlacementPolicyClaimsActual(key types.NamespacedName) func() error { + return func() error { + claims, err := claimsOfPolicy(key.Name, key.Namespace) + if err != nil { + return err + } + if len(claims.Items) > 0 { + names := make([]string, 0, len(claims.Items)) + for i := range claims.Items { + names = append(names, claims.Items[i].Name) + } + return fmt.Errorf("placement policy %s has cluster claims %v, want none", key, names) + } + return nil + } +} + +// placementPolicyBindingsKeptActual checks that, for each given cluster, the policy still has the binding of the +// given UID, i.e., the binding was neither deleted nor replaced. +func placementPolicyBindingsKeptActual(key types.NamespacedName, wantUIDsByCluster map[string]types.UID) func() error { + return func() error { + bindings, err := placementPolicyBindingsByCluster(key) + if err != nil { + return err + } + for clusterName, wantUID := range wantUIDsByCluster { + binding, found := bindings[clusterName] + if !found { + return fmt.Errorf("placement policy %s has no binding for cluster %s", key, clusterName) + } + if binding.GetUID() != wantUID { + return fmt.Errorf("the binding of placement policy %s for cluster %s was replaced (UID %s, want %s)", key, clusterName, binding.GetUID(), wantUID) + } + } + return nil + } +} + +// namespacePlacedOnClustersActual checks that the namespace exists on each given cluster and is managed by the +// placement policy work applier. The applier also creates the namespace of a namespaced resource it applies, but +// leaves such a namespace unmanaged; only a selected Namespace object is owned by its AppliedWork. +func namespacePlacedOnClustersActual(name string, clusters ...*framework.Cluster) func() error { + return func() error { + for _, c := range clusters { + ns := &corev1.Namespace{} + if err := c.KubeClient.Get(ctx, types.NamespacedName{Name: name}, ns); err != nil { + return fmt.Errorf("cluster %s: failed to get namespace %s: %w", c.ClusterName, name, err) + } + if !ns.DeletionTimestamp.IsZero() { + return fmt.Errorf("cluster %s: namespace %s is being deleted", c.ClusterName, name) + } + if !ownedByPlacementPolicyAppliedWork(ns) { + return fmt.Errorf("cluster %s: namespace %s is not owned by an AppliedWork of the placement policy APIs (owner references: %+v)", + c.ClusterName, name, ns.OwnerReferences) + } + } + return nil + } +} + +// namespaceAbsentOnClustersActual checks that none of the given clusters has the namespace, terminating or not. +func namespaceAbsentOnClustersActual(name string, clusters ...*framework.Cluster) func() error { + return func() error { + for _, c := range clusters { + err := c.KubeClient.Get(ctx, types.NamespacedName{Name: name}, &corev1.Namespace{}) + switch { + case err == nil: + return fmt.Errorf("cluster %s: namespace %s exists", c.ClusterName, name) + case !apierrors.IsNotFound(err): + return fmt.Errorf("cluster %s: failed to get namespace %s: %w", c.ClusterName, name, err) + } + } + return nil + } +} + +// clusterRolePlacedOnClustersActual checks that the ClusterRole on each given cluster has the rules of its hub copy. +func clusterRolePlacedOnClustersActual(name string, clusters ...*framework.Cluster) func() error { + return func() error { + want := &rbacv1.ClusterRole{} + if err := hubClient.Get(ctx, types.NamespacedName{Name: name}, want); err != nil { + return fmt.Errorf("failed to get ClusterRole %s on the hub cluster: %w", name, err) + } + for _, c := range clusters { + got := &rbacv1.ClusterRole{} + if err := c.KubeClient.Get(ctx, types.NamespacedName{Name: name}, got); err != nil { + return fmt.Errorf("cluster %s: failed to get ClusterRole %s: %w", c.ClusterName, name, err) + } + if diff := cmp.Diff(got.Rules, want.Rules); diff != "" { + return fmt.Errorf("cluster %s: ClusterRole %s rules diff (-got, +want):\n%s", c.ClusterName, name, diff) + } + } + return nil + } +} + +// clusterRoleAbsentOnClustersActual checks that none of the given clusters has the ClusterRole. +func clusterRoleAbsentOnClustersActual(name string, clusters ...*framework.Cluster) func() error { + return func() error { + for _, c := range clusters { + err := c.KubeClient.Get(ctx, types.NamespacedName{Name: name}, &rbacv1.ClusterRole{}) + switch { + case err == nil: + return fmt.Errorf("cluster %s: ClusterRole %s exists", c.ClusterName, name) + case !apierrors.IsNotFound(err): + return fmt.Errorf("cluster %s: failed to get ClusterRole %s: %w", c.ClusterName, name, err) + } + } + return nil + } +} + +// compareTimePointersByPresence treats two *metav1.Time values as equal when both or neither are set, so that a +// wanted status can check that a timestamp is set without pinning its value. cmpopts.IgnoreTypes(metav1.Time{}) +// does not cover pointers: cmp compares them with (*metav1.Time).Equal before it ever dereferences them. +var compareTimePointersByPresence = cmp.Comparer(func(a, b *metav1.Time) bool { + return (a == nil) == (b == nil) +}) + +// stagedUpdateRunStatusActual compares the status of a staged update run (or a cluster staged update run) with the +// wanted one in one shot, ignoring condition messages and timestamp values; a wanted timestamp that is set (to any +// value) checks only that the actual one is set too. The observed generation of every wanted condition, at any +// level, is set to the run's generation, as the run controller stamps them all with it. +func stagedUpdateRunStatusActual(key types.NamespacedName, wantStatus kfrolloutv1alpha1.StagedUpdateRunStatus) func() error { + return func() error { + run, err := retrieveStagedUpdateRun(key) + if err != nil { + return err + } + + want := wantStatus.DeepCopy() + generation := run.GetGeneration() + setObservedGeneration := func(conds []metav1.Condition) { + for i := range conds { + conds[i].ObservedGeneration = generation + } + } + setObservedGeneration(want.Conditions) + for i := range want.Stages { + stage := &want.Stages[i] + setObservedGeneration(stage.Conditions) + for j := range stage.Clusters { + setObservedGeneration(stage.Clusters[j].Conditions) + } + for j := range stage.BeforeStageTasks { + setObservedGeneration(stage.BeforeStageTasks[j].Conditions) + } + for j := range stage.AfterStageTasks { + setObservedGeneration(stage.AfterStageTasks[j].Conditions) + } + } + + if diff := cmp.Diff(*run.GetStatus(), *want, placementPolicyCmpOptions, ignoreTimeTypeFields, compareTimePointersByPresence); diff != "" { + return fmt.Errorf("staged update run %s status diff (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// approvalRequestActual checks that the approval request (or cluster approval request) exists for the given stage of +// the run, is controlled by the run, and, if wantAccepted is set, that the run has accepted its approval. +func approvalRequestActual(key, runKey types.NamespacedName, wantStageName string, wantAccepted bool) func() error { + return func() error { + run, err := retrieveStagedUpdateRun(runKey) + if err != nil { + return err + } + request, err := retrieveApprovalRequest(key) + if err != nil { + return err + } + + if !metav1.IsControlledBy(request, run) { + return fmt.Errorf("approval request %s is not controlled by staged update run %s (owner references: %+v)", + key, runKey, request.GetOwnerReferences()) + } + wantSpec := kfrolloutv1alpha1.ApprovalRequestSpec{ + StagedUpdateRunName: runKey.Name, + StageName: wantStageName, + } + if diff := cmp.Diff(*request.GetSpec(), wantSpec); diff != "" { + return fmt.Errorf("approval request %s spec diff (-got, +want):\n%s", key, diff) + } + if wantAccepted && !meta.IsStatusConditionTrue(request.GetStatus().Conditions, kfrolloutv1alpha1.ApprovalRequestCondTypeApprovalAccepted) { + return fmt.Errorf("approval request %s has not been accepted yet (conditions: %+v)", key, request.GetStatus().Conditions) + } + return nil + } +} + +// placementPolicyWorkClustersActual checks that the Works generated for the policy's bindings are for exactly the given +// member clusters, i.e., that the work generator has removed the Works for every other cluster. +func placementPolicyWorkClustersActual(key types.NamespacedName, wantClusterNames ...string) func() error { + return func() error { + namesByCluster, err := placementPolicyWorkNamesByCluster(key) + if err != nil { + return err + } + gotClusterNames := slices.Collect(maps.Keys(namesByCluster)) + if diff := cmp.Diff(gotClusterNames, wantClusterNames, cmpopts.SortSlices(func(a, b string) bool { return a < b }), cmpopts.EquateEmpty()); diff != "" { + return fmt.Errorf("placement policy %s clusters that have Works diff (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// suspendedPlacementBindingView is the part of a suspended placement binding that the specs compare in one shot. +type suspendedPlacementBindingView struct { + UID types.UID + ControlledByPolicy bool + Deleting bool + Suspended bool + ResourceSnapshotName string + LastProcessedResourceSnapshotName *string +} + +// suspendedPlacementPolicyBindingActual checks that the policy's binding for the cluster is still the one of the given +// UID, controlled by the policy and pointing to the given resource snapshot, but suspended and with no resource snapshot +// processed: the work generator has removed its Works. The binding's conditions are left as they were before the +// suspension, for an older generation, and carry no contract. +func suspendedPlacementPolicyBindingActual(key types.NamespacedName, clusterName string, wantUID types.UID, wantSnapshotName string) func() error { + return func() error { + policy, err := retrievePlacementPolicy(key) + if err != nil { + return err + } + bindings, err := placementPolicyBindingsByCluster(key) + if err != nil { + return err + } + binding, found := bindings[clusterName] + if !found { + return fmt.Errorf("placement policy %s has no binding for cluster %s", key, clusterName) + } + + got := suspendedPlacementBindingView{ + UID: binding.GetUID(), + ControlledByPolicy: metav1.IsControlledBy(binding, policy), + Deleting: !binding.GetDeletionTimestamp().IsZero(), + Suspended: binding.GetSpec().Suspended, + ResourceSnapshotName: binding.GetSpec().ResourceSnapshotName, + LastProcessedResourceSnapshotName: binding.GetStatus().LastProcessedResourceSnapshotName, + } + want := suspendedPlacementBindingView{ + UID: wantUID, + ControlledByPolicy: true, + Suspended: true, + ResourceSnapshotName: wantSnapshotName, + } + if diff := cmp.Diff(got, want); diff != "" { + return fmt.Errorf("placement policy %s suspended binding for cluster %s diff (-got, +want):\n%s", key, clusterName, diff) + } + return nil + } +} + +// placementBindingOwnershipView is the part of a placement binding that tells which object, if any, owns it. +type placementBindingOwnershipView struct { + UID types.UID + HasController bool + ControlledByPolicy bool + Deleting bool +} + +// placementPolicyBindingOwnershipActual compares, keyed by cluster, the bindings that name the policy with the given +// UIDs in one shot. None of them is being deleted, and each is either controlled by the policy as it exists now +// (wantControlled) or by no object at all, i.e., orphaned. +func placementPolicyBindingOwnershipActual(key types.NamespacedName, wantUIDsByCluster map[string]types.UID, wantControlled bool) func() error { + var controlledClusterNames []string + if wantControlled { + controlledClusterNames = slices.Collect(maps.Keys(wantUIDsByCluster)) + } + return placementPolicyBindingControllersActual(key, wantUIDsByCluster, controlledClusterNames...) +} + +// placementPolicyBindingControllersActual compares, keyed by cluster, the bindings that name the policy with the given +// UIDs in one shot. None of them is being deleted; those for the given clusters are controlled by the policy as it +// exists now, and the others by no object at all, i.e., they are orphans that the policy has not adopted. +func placementPolicyBindingControllersActual(key types.NamespacedName, wantUIDsByCluster map[string]types.UID, controlledClusterNames ...string) func() error { + return func() error { + var policy kfplacementv1alpha1.PlacementPolicyAccessor + if len(controlledClusterNames) > 0 { + var err error + if policy, err = retrievePlacementPolicy(key); err != nil { + return err + } + } + bindings, err := placementPolicyBindingsByCluster(key) + if err != nil { + return err + } + + got := make(map[string]placementBindingOwnershipView, len(bindings)) + for clusterName, binding := range bindings { + got[clusterName] = placementBindingOwnershipView{ + UID: binding.GetUID(), + HasController: metav1.GetControllerOf(binding) != nil, + ControlledByPolicy: policy != nil && metav1.IsControlledBy(binding, policy), + Deleting: !binding.GetDeletionTimestamp().IsZero(), + } + } + want := make(map[string]placementBindingOwnershipView, len(wantUIDsByCluster)) + for clusterName, uid := range wantUIDsByCluster { + controlled := slices.Contains(controlledClusterNames, clusterName) + want[clusterName] = placementBindingOwnershipView{ + UID: uid, + HasController: controlled, + ControlledByPolicy: controlled, + } + } + if diff := cmp.Diff(got, want); diff != "" { + return fmt.Errorf("placement policy %s binding ownership diff, keyed by cluster (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// orphanedPlacementBindingUnchangedActual checks that the binding naming the policy for the cluster is still the +// orphan of the given UID, controlled by no object, and still at the given resource version, i.e., that nothing has +// written to it since. +func orphanedPlacementBindingUnchangedActual(key types.NamespacedName, clusterName string, wantUID types.UID, wantResourceVersion string) func() error { + return func() error { + bindings, err := placementPolicyBindingsByCluster(key) + if err != nil { + return err + } + binding, found := bindings[clusterName] + switch { + case !found: + return fmt.Errorf("placement policy %s has no binding for cluster %s", key, clusterName) + case binding.GetUID() != wantUID: + return fmt.Errorf("binding %s for cluster %s was replaced (UID %s, want %s)", binding.GetName(), clusterName, binding.GetUID(), wantUID) + case metav1.GetControllerOf(binding) != nil: + return fmt.Errorf("binding %s for cluster %s is controlled by %+v, want no controller", binding.GetName(), clusterName, metav1.GetControllerOf(binding)) + case binding.GetResourceVersion() != wantResourceVersion: + return fmt.Errorf("binding %s for cluster %s changed (resource version %s, want %s; spec: %+v; status: %+v)", + binding.GetName(), clusterName, binding.GetResourceVersion(), wantResourceVersion, binding.GetSpec(), binding.GetStatus()) + } + return nil + } +} + +// placementPolicySnapshotsOrphanedActual checks that the policy, now gone, has left resource snapshots behind, none of +// them owned by any object or being deleted. +func placementPolicySnapshotsOrphanedActual(key types.NamespacedName) func() error { + return func() error { + snapshots, err := placementPolicySnapshots(key) + if err != nil { + return err + } + if len(snapshots) == 0 { + return fmt.Errorf("placement policy %s has left no resource snapshot behind", key) + } + for _, snapshot := range snapshots { + if len(snapshot.GetOwnerReferences()) > 0 { + return fmt.Errorf("resource snapshot %s of placement policy %s is still owned (owner references: %+v)", snapshot.GetName(), key, snapshot.GetOwnerReferences()) + } + if !snapshot.GetDeletionTimestamp().IsZero() { + return fmt.Errorf("resource snapshot %s of placement policy %s is being deleted", snapshot.GetName(), key) + } + } + return nil + } +} + +// placementPolicyWorksLinkedToSnapshotActual checks that every Work generated for the policy's bindings is linked to +// the given primary resource snapshot and is not being deleted, i.e., that no Work of an earlier policy of the same +// name is left. +func placementPolicyWorksLinkedToSnapshotActual(key types.NamespacedName, wantSnapshotName string) func() error { + return func() error { + works, err := placementPolicyWorks(key) + if err != nil { + return err + } + if len(works) == 0 { + return fmt.Errorf("placement policy %s has no Works", key) + } + for i := range works { + work := &works[i] + if !work.DeletionTimestamp.IsZero() { + return fmt.Errorf("work %s of placement policy %s is being deleted", client.ObjectKeyFromObject(work), key) + } + if got := work.GetAnnotations()[kfplacementv1alpha1.WorkLinkedToPrimaryPlacementResourceSnapshotAnnotationKey]; got != wantSnapshotName { + return fmt.Errorf("work %s of placement policy %s is linked to resource snapshot %q, want %q", client.ObjectKeyFromObject(work), key, got, wantSnapshotName) + } + } + return nil + } +} + +// placementPolicyWorkUIDs returns, keyed by namespace/name, the UIDs of the Works on the hub that were generated for +// the policy's bindings. +func placementPolicyWorkUIDs(key types.NamespacedName) (map[string]types.UID, error) { + works, err := placementPolicyWorks(key) + if err != nil { + return nil, err + } + uids := make(map[string]types.UID, len(works)) + for i := range works { + uids[client.ObjectKeyFromObject(&works[i]).String()] = works[i].UID + } + return uids, nil +} + +// placementPolicyWorksKeptActual checks that the Works generated for the policy's bindings are exactly the given ones, +// keyed by namespace/name, with the given UIDs, i.e., that none was removed, re-created, or added. +func placementPolicyWorksKeptActual(key types.NamespacedName, wantUIDsByWork map[string]types.UID) func() error { + return func() error { + got, err := placementPolicyWorkUIDs(key) + if err != nil { + return err + } + if diff := cmp.Diff(got, wantUIDsByWork); diff != "" { + return fmt.Errorf("placement policy %s Work UIDs diff, keyed by Work (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// configMapResourceVersionsByCluster returns, keyed by cluster, the resource versions of the ConfigMap on the given +// clusters. +func configMapResourceVersionsByCluster(key types.NamespacedName, clusters ...*framework.Cluster) (map[string]string, error) { + versions := make(map[string]string, len(clusters)) + for _, c := range clusters { + cm := &corev1.ConfigMap{} + if err := c.KubeClient.Get(ctx, key, cm); err != nil { + return nil, fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", c.ClusterName, key, err) + } + versions[c.ClusterName] = cm.ResourceVersion + } + return versions, nil +} + +// configMapUnchangedOnClustersActual checks that the ConfigMap on each member cluster named in the map is still at the +// given resource version, i.e., that nothing has written to it since. +func configMapUnchangedOnClustersActual(key types.NamespacedName, wantResourceVersionsByCluster map[string]string) func() error { + var clusters []*framework.Cluster + for _, c := range allMemberClusters { + if _, found := wantResourceVersionsByCluster[c.ClusterName]; found { + clusters = append(clusters, c) + } + } + return func() error { + got, err := configMapResourceVersionsByCluster(key, clusters...) + if err != nil { + return err + } + if diff := cmp.Diff(got, wantResourceVersionsByCluster); diff != "" { + return fmt.Errorf("ConfigMap %s resource versions diff, keyed by cluster (-got, +want):\n%s", key, diff) + } + return nil + } +} + +// configMapPlacedByPlacementPolicyActual checks that the ConfigMap on the cluster matches its hub copy and that, of the +// AppliedWorks of the placement policy APIs, exactly those of the given policy's Works for the cluster own it, i.e., +// that the policy, and no other, has placed it. +func configMapPlacedByPlacementPolicyActual(cluster *framework.Cluster, cmKey, policyKey types.NamespacedName) func() error { + return func() error { + if err := validateConfigMapOnCluster(cluster, cmKey); err != nil { + return fmt.Errorf("cluster %s: %w", cluster.ClusterName, err) + } + cm := &corev1.ConfigMap{} + if err := cluster.KubeClient.Get(ctx, cmKey, cm); err != nil { + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", cluster.ClusterName, cmKey, err) + } + var gotOwners []string + for _, ownerRef := range cm.OwnerReferences { + if ownerRef.APIVersion == kfplacementv1alpha1.GroupVersion.String() && ownerRef.Kind == kfplacementv1alpha1.AppliedWorkKind { + gotOwners = append(gotOwners, ownerRef.Name) + } + } + + // An AppliedWork carries the name of its Work. + workNamesByCluster, err := placementPolicyWorkNamesByCluster(policyKey) + if err != nil { + return err + } + wantOwners := workNamesByCluster[cluster.ClusterName] + if len(wantOwners) == 0 { + return fmt.Errorf("placement policy %s has no Work for cluster %s", policyKey, cluster.ClusterName) + } + if diff := cmp.Diff(gotOwners, wantOwners, cmpopts.SortSlices(func(a, b string) bool { return a < b })); diff != "" { + return fmt.Errorf("cluster %s: ConfigMap %s AppliedWork owners diff (-got, +want):\n%s", cluster.ClusterName, cmKey, diff) + } + return nil + } +} + +// placementPolicyClaimKeptActual checks that the policy's only cluster claim is still the one of the given UID and is +// not being withdrawn, i.e., that it was neither withdrawn nor reissued. +func placementPolicyClaimKeptActual(key types.NamespacedName, wantUID types.UID) func() error { + return func() error { + claims, err := claimsOfPolicy(key.Name, key.Namespace) + if err != nil { + return err + } + if len(claims.Items) != 1 { + return fmt.Errorf("placement policy %s has %d cluster claims, want 1", key, len(claims.Items)) + } + claim := &claims.Items[0] + if claim.UID != wantUID { + return fmt.Errorf("cluster claim %s of placement policy %s was reissued (UID %s, want %s)", claim.Name, key, claim.UID, wantUID) + } + if !claim.DeletionTimestamp.IsZero() { + return fmt.Errorf("cluster claim %s of placement policy %s is being withdrawn", claim.Name, key) + } + return nil + } +} + +// placementPolicyClaimObservedClusterCreationActual checks that the policy's only cluster claim records, as the most +// recent member cluster creation it has observed, a time no earlier than the given one; i.e., that the policy has +// reconciled the claim since a member cluster was created at that time. +func placementPolicyClaimObservedClusterCreationActual(key types.NamespacedName, wantObservedNoEarlierThan metav1.Time) func() error { + return func() error { + claims, err := claimsOfPolicy(key.Name, key.Namespace) + if err != nil { + return err + } + if len(claims.Items) != 1 { + return fmt.Errorf("placement policy %s has %d cluster claims, want 1", key, len(claims.Items)) + } + claim := &claims.Items[0] + observed := claim.Status.LastObservedMostRecentClusterCreationTimestamp + if observed == nil || observed.Before(&wantObservedNoEarlierThan) { + return fmt.Errorf("cluster claim %s of placement policy %s has observed the most recent member cluster creation at %v, want no earlier than %v", + claim.Name, key, observed, wantObservedNoEarlierThan) + } + return nil + } +} + +// configMapUIDsByCluster returns, keyed by cluster, the UIDs of the ConfigMap on the given clusters. +func configMapUIDsByCluster(key types.NamespacedName, clusters ...*framework.Cluster) (map[string]types.UID, error) { + uids := make(map[string]types.UID, len(clusters)) + for _, c := range clusters { + cm := &corev1.ConfigMap{} + if err := c.KubeClient.Get(ctx, key, cm); err != nil { + return nil, fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", c.ClusterName, key, err) + } + uids[c.ClusterName] = cm.UID + } + return uids, nil +} + +// placementPolicyLatestSnapshotChangedActual checks that the policy's latest primary resource snapshot is no longer +// the given one, i.e., that a new resource snapshot has been taken, and stores the new snapshot's name in newName. +func placementPolicyLatestSnapshotChangedActual(key types.NamespacedName, oldName string, newName *string) func() error { + return func() error { + policy, err := retrievePlacementPolicy(key) + if err != nil { + return err + } + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + if err != nil { + return err + } + if snapshot.GetName() == oldName { + return fmt.Errorf("placement policy %s has taken no new resource snapshot since %s", key, oldName) + } + *newName = snapshot.GetName() + return nil + } +} + +// stagedUpdateRunApprovalTaskActual checks that a staged update run (or a cluster staged update run) reports, for the +// approval task before or after the stage at the given index, the given approval request name and the given condition +// with the given status and reason, for the run's current generation. +func stagedUpdateRunApprovalTaskActual( + runKey types.NamespacedName, + stageIdx int, + beforeStage bool, + wantRequestName, condType string, + wantStatus metav1.ConditionStatus, + wantReason string, +) func() error { + return func() error { + run, err := retrieveStagedUpdateRun(runKey) + if err != nil { + return err + } + stages := run.GetStatus().Stages + if len(stages) <= stageIdx { + return fmt.Errorf("staged update run %s has no status for stage %d yet", runKey, stageIdx) + } + tasks := stages[stageIdx].AfterStageTasks + if beforeStage { + tasks = stages[stageIdx].BeforeStageTasks + } + taskIdx := slices.IndexFunc(tasks, func(task kfrolloutv1alpha1.PerStageTaskStatus) bool { + return task.Type == kfrolloutv1alpha1.StageTaskTypeApproval + }) + if taskIdx < 0 { + return fmt.Errorf("staged update run %s has no approval task status for stage %d (before stage: %t)", runKey, stageIdx, beforeStage) + } + task := tasks[taskIdx] + if task.ApprovalRequestName != wantRequestName { + return fmt.Errorf("staged update run %s names approval request %q for stage %d, want %q", runKey, task.ApprovalRequestName, stageIdx, wantRequestName) + } + cond := meta.FindStatusCondition(task.Conditions, condType) + if cond == nil || cond.Status != wantStatus || cond.Reason != wantReason || cond.ObservedGeneration != run.GetGeneration() { + return fmt.Errorf("staged update run %s reports approval task condition %s %+v for stage %d, want status %s and reason %s for generation %d", + runKey, condType, cond, stageIdx, wantStatus, wantReason, run.GetGeneration()) + } + return nil + } +} diff --git a/test/e2e/placement_policy_auto_rollout_test.go b/test/e2e/placement_policy_auto_rollout_test.go new file mode 100644 index 000000000..98c598d35 --- /dev/null +++ b/test/e2e/placement_policy_auto_rollout_test.go @@ -0,0 +1,674 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" +) + +const ( + placementPolicyAutoRolloutNameTemplate = "pp-auto-rollout-%d" + placementPolicyAutoRolloutDisabledNameTemplate = "pp-auto-rollout-disabled-%d" + placementPolicyAutoRolloutCoexistenceNameTemplate = "pp-auto-rollout-coexist-%d" + placementPolicyAutoRolloutCoexistenceStrategyNameTemplate = "pp-auto-rollout-coexist-strategy-%d" + placementPolicyAutoRolloutCoexistenceRunNameTemplate = "pp-auto-rollout-coexist-run-%d" + placementPolicyAutoRolloutWaitPeriodNameTemplate = "pp-auto-rollout-wait-period-%d" + + // placementPolicyAutoRolloutMinimumWaitPeriod is the minimum wait period that the specs set on their policies. Only + // a second auto-rollout waits on it, and the period starts with the first one, before the specs have checked that + // the first one settled; twice the shortest period allowed keeps enough of it left to observe the hold on a slow + // runner. + placementPolicyAutoRolloutMinimumWaitPeriod = 2 * time.Minute + + // placementPolicyAutoRolloutMinimumHoldObserved is the least time that must be left of the minimum wait period once + // a second change shows as pending, for the specs to observe that auto-rollout holds it back. + placementPolicyAutoRolloutMinimumHoldObserved = 15 * time.Second + + // placementPolicyAutoRolloutDisabledConsistentlyDuration is how long the specs watch a pending change that + // auto-rollout must leave alone. + placementPolicyAutoRolloutDisabledConsistentlyDuration = 30 * time.Second + + // placementPolicyAutoRolloutRetryEventuallyDuration bounds the wait on an auto-rollout that a staged update run has + // held back. Auto-rollout does not react to the run giving up the binding manager role; it retries its claim on a + // backoff that starts at 5s and doubles, up to 180s, so the retry comes at most one backoff interval after the run + // lets go. The steps before the wait bound how long the run holds the role (about 160s at most, with each of them + // running out its own timeout), so that interval never exceeds 160s; the bound leaves room for it, plus the rollout + // itself. + placementPolicyAutoRolloutRetryEventuallyDuration = 4 * time.Minute +) + +// autoRolloutAnnotations returns the annotations that enable auto-rollout on a policy with the specs' minimum wait +// period. +func autoRolloutAnnotations() map[string]string { + return map[string]string{ + kfrolloutv1alpha1.AutoRolloutEnabledAnnotationKey: kfrolloutv1alpha1.AutoRolloutEnabledAnnotationValue, + kfrolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey: placementPolicyAutoRolloutMinimumWaitPeriod.String(), + } +} + +// lastAutoRolloutAttemptOf returns when auto-rollout last attempted a rollout of the policy, as recorded on it. +func lastAutoRolloutAttemptOf(key types.NamespacedName) time.Time { + GinkgoHelper() + + policy, err := retrievePlacementPolicy(key) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", key) + val, found := policy.GetAnnotations()[kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] + Expect(found).To(BeTrue(), "Placement policy %s records no auto-rollout attempt", key) + attemptAt, err := time.Parse(time.RFC3339Nano, val) + Expect(err).NotTo(HaveOccurred(), "Placement policy %s records an invalid auto-rollout attempt timestamp %q", key, val) + return attemptAt +} + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy auto-rollout", Label("placementpolicy"), func() { + Context("placement policy with auto-rollout enabled", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyAutoRolloutNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + v2Data := map[string]string{"data": "test-v2"} + + var initialSnapshotName, newSnapshotName string + var lastAttemptAt time.Time + + allClustersSelectors := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + wantSelectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantSelectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + return wantSelectorsByCluster + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy with auto-rollout enabled") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + // The minimum wait period does not hold back the first auto-rollout, which follows no earlier one. + policy.SetAnnotations(autoRolloutAnnotations()) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, allClustersSelectors(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of placement policy %s", policyKey) + initialSnapshotName = snapshot.GetName() + }) + + It("should not have attempted an auto-rollout for the initial placement", func() { + // The initial placement needs no rollout: new bindings get the latest resource snapshot directly. + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + Expect(policy.GetAnnotations()).NotTo(HaveKey(kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey), + "An auto-rollout was attempted for the initial placement") + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("should roll out the new version to every member cluster without a staged update run", func() { + Eventually(func() error { + policy, err := retrievePlacementPolicy(policyKey) + if err != nil { + return err + } + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + if err != nil { + return err + } + if snapshot.GetName() == initialSnapshotName { + return fmt.Errorf("no new resource snapshot has been taken yet") + } + newSnapshotName = snapshot.GetName() + return nil + }, placementPolicyRolloutEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take a new resource snapshot") + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, allClustersSelectors(), 1), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll out the new resource snapshot to every binding") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + }) + + It("should record the auto-rollout attempt on the policy", func() { + Eventually(func() error { + policy, err := retrievePlacementPolicy(policyKey) + if err != nil { + return err + } + val, found := policy.GetAnnotations()[kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] + if !found { + return fmt.Errorf("placement policy %s has no last rollout attempt timestamp", policyKey) + } + if _, err := time.Parse(time.RFC3339, val); err != nil { + return fmt.Errorf("placement policy %s has an invalid last rollout attempt timestamp %q: %w", policyKey, val, err) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to record the auto-rollout attempt") + }) + + It("should report the policy as synchronized on every member cluster, with the binding manager role released", func() { + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("can change the ConfigMap on the hub again, right after the auto-rollout", func() { + lastAttemptAt = lastAutoRolloutAttemptOf(policyKey) + updateConfigMapDataOnHub(cmKey, v2Data) + }) + + It("should hold the second change back until the minimum wait period since the last attempt has passed", func() { + wantStatus := resourcesChangedPendingRolloutPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + dataActual := configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...) + heldActual := func() error { + if err := statusActual(); err != nil { + return err + } + return dataActual() + } + Eventually(heldActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the second change as pending a rollout") + // Hold until shortly before the period ends, leaving a margin for the controller to act on time. Too little + // time left means that the first auto-rollout took too long to settle for the hold to be observed. + holdFor := time.Until(lastAttemptAt.Add(placementPolicyAutoRolloutMinimumWaitPeriod - 5*time.Second)) + Expect(holdFor).To(BeNumerically(">=", placementPolicyAutoRolloutMinimumHoldObserved), + "Only %v of the minimum wait period is left to observe the hold; the first auto-rollout took too long to settle", holdFor) + Consistently(heldActual, holdFor, consistentlyInterval). + Should(Succeed(), "The second change was rolled out before the minimum wait period passed") + }) + + It("should roll the second change out once the minimum wait period has passed", func() { + Eventually(configMapDataOnClustersActual(cmKey, v2Data, allMemberClusters...), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the second change on every member cluster") + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, allClustersSelectors(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the latest resource snapshot out to every binding") + Expect(latestPrimarySnapshotNameOfPlacementPolicy(policyKey)).NotTo(Equal(newSnapshotName), "No new resource snapshot was taken for the second change") + + secondAttemptAt := lastAutoRolloutAttemptOf(policyKey) + Expect(secondAttemptAt.Sub(lastAttemptAt)).To(BeNumerically(">=", placementPolicyAutoRolloutMinimumWaitPeriod), + "The second auto-rollout attempt at %v came less than the minimum wait period after the first one at %v", secondAttemptAt, lastAttemptAt) + }) + }) + + Context("placement policy with auto-rollout not enabled", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyAutoRolloutDisabledNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName string + + // pendingActual checks that the change waits for a rollout: no new snapshot, every cluster still on the initial + // version, and no auto-rollout attempt recorded. + pendingActual := func() func() error { + wantStatus := resourcesChangedPendingRolloutPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + wantBindings := make(map[string]wantPlacementBinding, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: initialSnapshotName, + status: synchronizedPlacementBindingStatus(1), + } + } + bindingsActual := placementPolicyBindingsActual(policyKey, wantBindings) + dataActual := configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...) + return func() error { + if err := statusActual(); err != nil { + return err + } + if err := bindingsActual(); err != nil { + return err + } + if err := dataActual(); err != nil { + return err + } + policy, err := retrievePlacementPolicy(policyKey) + if err != nil { + return err + } + if val, found := policy.GetAnnotations()[kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey]; found { + return fmt.Errorf("placement policy %s records an auto-rollout attempt at %s", policyKey, val) + } + return nil + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy without the auto-rollout annotations") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of placement policy %s", policyKey) + initialSnapshotName = snapshot.GetName() + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("should keep the change pending a rollout while the policy has no auto-rollout annotation", func() { + Eventually(pendingActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the change as pending a rollout") + // An enabled auto-rollout acts within seconds of a change that follows no earlier attempt. + Consistently(pendingActual(), placementPolicyAutoRolloutDisabledConsistentlyDuration, consistentlyInterval). + Should(Succeed(), "The change was rolled out without the auto-rollout annotation") + }) + + It("can set the auto-rollout annotation to false", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + annotations := policy.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string) + } + annotations[kfrolloutv1alpha1.AutoRolloutEnabledAnnotationKey] = "false" + policy.SetAnnotations(annotations) + }) + }) + + It("should keep the change pending a rollout with the annotation set to false", func() { + Consistently(pendingActual(), placementPolicyAutoRolloutDisabledConsistentlyDuration, consistentlyInterval). + Should(Succeed(), "The change was rolled out with auto-rollout set to false") + }) + }) + + Context("placement policy with auto-rollout enabled, while a staged update run holds the binding manager role", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyAutoRolloutCoexistenceNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyAutoRolloutCoexistenceStrategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyAutoRolloutCoexistenceRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName string + var approvalRequestKey types.NamespacedName + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy with auto-rollout enabled") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + policy.SetAnnotations(autoRolloutAnnotations()) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + + By("creating a strategy with a canary stage, gated by an approval, before a prod stage") + strategy := newStagedUpdateStrategyObject(strategyKey, canaryApprovalStrategySpec()) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can create a staged update run, which holds the binding manager role while it waits for an approval", func() { + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", runKey) + // Nothing has changed on the hub yet, so the run rolls out the initial snapshot, which every cluster already + // has; its canary stage goes straight to the approval. + approvalRequestKey = waitForStageApprovalRequest(runKey, 0, false) + Expect(resourceSnapshotNameToRolloutOf(runKey)).To(Equal(initialSnapshotName), "Staged update run %s rolls out another resource snapshot", runKey) + Eventually(placementPolicyBindingManagerActual(policyKey, &kfplacementv1alpha1.BindingManager{ + ControllerName: stagedUpdateRunBindingManagerName, + ObjectRefs: []kfplacementv1alpha1.ObjectReference{stagedUpdateRunObjectReference(runKey)}, + }), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to hold the binding manager role") + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("should not auto-roll the change out while the run holds the binding manager role", func() { + wantStatus := resourcesChangedPendingRolloutPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + wantStatus.BindingManager = &kfplacementv1alpha1.BindingManager{ + ControllerName: stagedUpdateRunBindingManagerName, + ObjectRefs: []kfplacementv1alpha1.ObjectReference{stagedUpdateRunObjectReference(runKey)}, + } + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + dataActual := configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...) + heldActual := func() error { + if err := statusActual(); err != nil { + return err + } + if err := dataActual(); err != nil { + return err + } + // Auto-rollout records an attempt only once it has claimed the role. + policy, err := retrievePlacementPolicy(policyKey) + if err != nil { + return err + } + if val, found := policy.GetAnnotations()[kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey]; found { + return fmt.Errorf("placement policy %s records an auto-rollout attempt at %s", policyKey, val) + } + return nil + } + Eventually(heldActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the change as pending a rollout") + // The window is kept short: every failed claim doubles auto-rollout's backoff. + Consistently(heldActual, consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The change was auto-rolled out while the run held the binding manager role") + }) + + It("can approve the canary stage", func() { + approvePlacementPolicyApprovalRequest(approvalRequestKey) + }) + + It("should complete the run", func() { + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s", runKey) + }) + + It("should auto-roll the change out once the run has given up the binding manager role", func() { + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), placementPolicyAutoRolloutRetryEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to auto-roll the change out to every member cluster") + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the latest resource snapshot out to every binding") + Expect(latestPrimarySnapshotNameOfPlacementPolicy(policyKey)).NotTo(Equal(initialSnapshotName), "No new resource snapshot was taken for the change") + Expect(lastAutoRolloutAttemptOf(policyKey)).NotTo(BeZero(), "Placement policy %s records no auto-rollout attempt", policyKey) + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("annotation-generated placement policy with auto-rollout set up on it", Ordered, func() { + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + canaryLabels := map[string]string{envLabelName: envCanary} + canarySelectors := map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster2EastCanaryName: {clusterSelectorTermsMatchingLabels(canaryLabels)}, + } + + var configMap corev1.ConfigMap + var policyKey types.NamespacedName + var initialSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("annotating the ConfigMap so that the hub agent generates a placement policy for the canary cluster") + configMap = appConfigMap() + setClusterSelectorsAnnotation(&configMap, fmt.Sprintf("%s=%s", envLabelName, envCanary)) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // Removing the annotation makes the hub agent delete the generated policy. + if configMap.Name != "" { + setClusterSelectorsAnnotation(&configMap, "") + } + if policyKey.Name != "" { + ensurePlacementPolicyRemoved(policyKey) + } + }) + + It("should generate a placement policy that places the ConfigMap on the canary cluster", func() { + Eventually(func() error { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + if err := hubClient.List(ctx, policies, client.InNamespace(cmKey.Namespace), generatedPolicyLabels("", "ConfigMap", cmKey.Name)); err != nil { + return err + } + if len(policies.Items) != 1 { + return fmt.Errorf("%d placement policies are generated for ConfigMap %s, want 1", len(policies.Items), cmKey) + } + policyKey = types.NamespacedName{Namespace: policies.Items[0].Namespace, Name: policies.Items[0].Name} + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate a placement policy for ConfigMap %s", cmKey) + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, canarySelectors, 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on the canary cluster") + }) + + It("can set up auto-rollout on the generated policy", func() { + // The hub agent owns the spec of a generated policy, but leaves the annotations it does not generate alone. + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + annotations := policy.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string) + } + maps.Copy(annotations, autoRolloutAnnotations()) + policy.SetAnnotations(annotations) + }) + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("should auto-roll the change out to the canary cluster", func() { + Eventually(configMapDataOnClustersActual(cmKey, v1Data, memberCluster2EastCanary), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to auto-roll the change out to the canary cluster") + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, canarySelectors, 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the latest resource snapshot out to the canary binding") + Expect(latestPrimarySnapshotNameOfPlacementPolicy(policyKey)).NotTo(Equal(initialSnapshotName), "No new resource snapshot was taken for the change") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + Expect(policy.GetAnnotations()).To(HaveKeyWithValue(kfrolloutv1alpha1.AutoRolloutEnabledAnnotationKey, kfrolloutv1alpha1.AutoRolloutEnabledAnnotationValue), + "The auto-rollout annotation is gone from the generated policy") + Expect(lastAutoRolloutAttemptOf(policyKey)).NotTo(BeZero(), "Placement policy %s records no auto-rollout attempt", policyKey) + }) + }) + + // The auto-rollout ITs cover the bounds and the default of the minimum wait period in full; these specs check them + // on the hub agent the cheap way, as the ITs do: by backdating the last rollout attempt, so that each case costs + // seconds of waiting rather than minutes. + Context("placement policy with a minimum wait period out of the allowed range, or malformed", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyAutoRolloutWaitPeriodNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + v2Data := map[string]string{"data": "test-v2"} + v3Data := map[string]string{"data": "test-v3"} + + var changedAt, lastAttemptAt time.Time + + // setWaitPeriodSinceAttempt sets the minimum wait period and backdates the last rollout attempt, in one update: + // a change to the timestamp alone does not wake auto-rollout up, while one to the wait period does. + setWaitPeriodSinceAttempt := func(period string, attemptAt time.Time) { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + annotations := policy.GetAnnotations() + if annotations == nil { + annotations = make(map[string]string) + } + annotations[kfrolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey] = period + annotations[kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] = attemptAt.UTC().Format(time.RFC3339Nano) + policy.SetAnnotations(annotations) + }) + } + // lastAttemptNoEarlierThanActual checks that auto-rollout has recorded a rollout attempt no earlier than the + // given time on the policy. + lastAttemptNoEarlierThanActual := func(t time.Time) func() error { + return func() error { + policy, err := retrievePlacementPolicy(policyKey) + if err != nil { + return err + } + val := policy.GetAnnotations()[kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey] + attemptAt, err := time.Parse(time.RFC3339Nano, val) + if err != nil { + return fmt.Errorf("placement policy %s records an invalid auto-rollout attempt timestamp %q: %w", policyKey, val, err) + } + if attemptAt.Before(t) { + return fmt.Errorf("placement policy %s last recorded an auto-rollout attempt at %v, want one no earlier than %v", policyKey, attemptAt, t) + } + return nil + } + } + // rolledOutOnlyAfter checks that the member clusters keep the old version until shortly before releaseAt, and + // get the new one after that. + rolledOutOnlyAfter := func(oldData, newData map[string]string, releaseAt time.Time) { + GinkgoHelper() + + // The specs leave about 30s between the change and the release; a hold much shorter than that means that + // the spec ran too slowly to tell an early rollout apart. + hold := time.Until(releaseAt) - 5*time.Second + Expect(hold).To(BeNumerically(">", 10*time.Second), "Too little time is left before the release to check that the change is held back") + Consistently(configMapDataOnClustersActual(cmKey, oldData, allMemberClusters...), hold, consistentlyInterval). + Should(Succeed(), "The change was rolled out before %v", releaseAt) + Eventually(configMapDataOnClustersActual(cmKey, newData, allMemberClusters...), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the change out after %v", releaseAt) + // The attempt is recorded once the bindings are refreshed; wait for it, so that the next spec's backdated + // timestamp is not overwritten. + Eventually(lastAttemptNoEarlierThanActual(releaseAt), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to record the auto-rollout attempt") + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy with auto-rollout enabled, a 5h minimum wait period, and the last rollout attempt 61m ago") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + policy.SetAnnotations(map[string]string{ + kfrolloutv1alpha1.AutoRolloutEnabledAnnotationKey: kfrolloutv1alpha1.AutoRolloutEnabledAnnotationValue, + kfrolloutv1alpha1.AutoRolloutMinimumWaitPeriodAnnotationKey: "5h", + kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey: time.Now().Add(-61 * time.Minute).UTC().Format(time.RFC3339Nano), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can change the ConfigMap on the hub", func() { + changedAt = time.Now() + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("should roll the change out at once, as a wait period above 1h is clamped down to 1h, which has passed", func() { + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the change out") + Eventually(lastAttemptNoEarlierThanActual(changedAt), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to record the auto-rollout attempt") + }) + + It("can set a wait period below 1m, with the last rollout attempt 30s ago, and change the ConfigMap", func() { + lastAttemptAt = time.Now().Add(-30 * time.Second) + setWaitPeriodSinceAttempt("1s", lastAttemptAt) + updateConfigMapDataOnHub(cmKey, v2Data) + }) + + It("should hold the change back until 1m after the last attempt, as a wait period below 1m is clamped up to 1m", func() { + rolledOutOnlyAfter(v1Data, v2Data, lastAttemptAt.Add(time.Minute)) + }) + + It("can set a malformed wait period, with the last rollout attempt 2m30s ago, and change the ConfigMap", func() { + lastAttemptAt = time.Now().Add(-150 * time.Second) + setWaitPeriodSinceAttempt("soon", lastAttemptAt) + updateConfigMapDataOnHub(cmKey, v3Data) + }) + + It("should hold the change back until 3m after the last attempt, as a malformed wait period falls back to the 3m default", func() { + rolledOutOnlyAfter(v2Data, v3Data, lastAttemptAt.Add(3*time.Minute)) + }) + }) +}) diff --git a/test/e2e/placement_policy_binding_suspension_test.go b/test/e2e/placement_policy_binding_suspension_test.go new file mode 100644 index 000000000..064396e1c --- /dev/null +++ b/test/e2e/placement_policy_binding_suspension_test.go @@ -0,0 +1,396 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/test/e2e/framework" +) + +const ( + placementPolicySuspendedBindingNameTemplate = "pp-suspended-binding-%d" + clusterPlacementPolicySuspendedBindingNameTemplate = "cpp-suspended-binding-%d" + placementPolicySuspendedAutoRolloutNameTemplate = "pp-suspended-auto-rollout-%d" +) + +// Users do not normally edit a binding's spec.suspended; upcoming APIs build on it, and these specs set it directly as a +// stand-in for them. On suspension the work generator removes the binding's Works and clears the binding's last +// processed resource snapshot; the work applier then handles the placed resources as whenPlacementDeleted asks. The +// placement policy controller leaves the field alone. +var _ = Describe("placement policy binding suspension", Label("placementpolicy"), func() { + Context("placement policy with a binding suspended, then resumed", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicySuspendedBindingNameTemplate, GinkgoParallelProcess()), + } + suspendedClusterName := memberCluster2EastCanaryName + otherClusterNames := []string{memberCluster1EastProdName, memberCluster3WestProdName} + // The member clusters are known only once the suite is set up, after the spec tree is built. + clusterCount := func() int32 { return int32(len(allMemberClusterNames)) } + + var snapshotName string + var bindingUIDsByCluster map[string]types.UID + var workNamesByCluster map[string][]string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the ConfigMap on every member cluster", func() { + snapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + + bindingUIDsByCluster = placementPolicyBindingUIDsByCluster(policyKey) + var err error + workNamesByCluster, err = placementPolicyWorkNamesByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of placement policy %s", policyKey) + }) + + It("can suspend the binding for member cluster 2", func() { + setPlacementPolicyBindingSuspended(policyKey, suspendedClusterName, true) + }) + + It("should remove the Works and the ConfigMap of the suspended binding only", func() { + Eventually(suspendedPlacementPolicyBindingActual(policyKey, suspendedClusterName, bindingUIDsByCluster[suspendedClusterName], snapshotName), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to process the suspension of the binding") + Eventually(placementPolicyWorkClustersActual(policyKey, otherClusterNames...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the Works of the suspended binding") + Eventually(appliedWorksAbsentOnClustersActual(map[string][]string{suspendedClusterName: workNamesByCluster[suspendedClusterName]}), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the AppliedWorks from member cluster 2") + Eventually(appConfigMapRemovedFromClustersActual(memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from member cluster 2") + }) + + It("should report the policy as synchronized on the other member clusters only", func() { + // The suspended binding has no resource snapshot processed, so it counts as neither synchronized nor on an + // older resource revision. + wantStatus := someClustersNotSynchronizedPlacementPolicyStatus(clusterCount(), clusterCount()-1) + wantStatus.LatestResourceRevisionName = ptr.To(snapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should keep the binding suspended, with no replacement, and leave the other member clusters as they were", func() { + wantStatus := someClustersNotSynchronizedPlacementPolicyStatus(clusterCount(), clusterCount()-1) + wantStatus.LatestResourceRevisionName = ptr.To(snapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + suspendedActual := suspendedPlacementPolicyBindingActual(policyKey, suspendedClusterName, bindingUIDsByCluster[suspendedClusterName], snapshotName) + // Every binding is kept, and the policy has no other. + bindingsActual := placementPolicyBindingOwnershipActual(policyKey, bindingUIDsByCluster, true) + worksActual := placementPolicyWorkClustersActual(policyKey, otherClusterNames...) + placedActual := appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster3WestProd) + removedActual := appConfigMapRemovedFromClustersActual(memberCluster2EastCanary) + Consistently(func() error { + for _, actual := range []func() error{statusActual, suspendedActual, bindingsActual, worksActual, placedActual, removedActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The suspended binding or the other member clusters changed") + }) + + It("can resume the binding for member cluster 2", func() { + setPlacementPolicyBindingSuspended(policyKey, suspendedClusterName, false) + }) + + It("should place the ConfigMap on member cluster 2 again, through the same binding", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to synchronize every binding") + Expect(placementPolicyBindingsKeptActual(policyKey, bindingUIDsByCluster)()).To(Succeed(), "A binding was replaced") + Eventually(placementPolicyWorkClustersActual(policyKey, allMemberClusterNames...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to re-create the Works of the resumed binding") + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + + wantStatus := allSynchronizedPlacementPolicyStatus(clusterCount()) + wantStatus.LatestResourceRevisionName = ptr.To(snapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("cluster placement policy orphaning its resources, with a binding suspended, then resumed", Ordered, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicySuspendedBindingNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + suspendedClusterName := memberCluster1EastProdName + otherClusterNames := []string{memberCluster2EastCanaryName, memberCluster3WestProdName} + // The member clusters are known only once the suite is set up, after the spec tree is built. + clusterCount := func() int32 { return int32(len(allMemberClusterNames)) } + + var snapshotName string + var bindingUIDsByCluster map[string]types.UID + var workNamesByCluster map[string][]string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects every member cluster and orphans its resources when they are no longer placed") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + SyncStrategy: &kfplacementv1alpha1.SyncStrategy{ + WhenPlacementDeleted: kfplacementv1alpha1.WhenPlacementDeletedOptionOrphanResources, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The orphaned ConfigMaps go with the member namespaces, which the cleanup removes even if deleting the + // policy fails. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should place the ConfigMap on every member cluster, managed by KubeFleet", func() { + snapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + for _, cluster := range allMemberClusters { + Eventually(configMapTakenOverOnClusterActual(cluster, cmKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on cluster %s", cluster.ClusterName) + } + + bindingUIDsByCluster = placementPolicyBindingUIDsByCluster(policyKey) + var err error + workNamesByCluster, err = placementPolicyWorkNamesByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of cluster placement policy %s", policyKey) + }) + + It("can suspend the binding for member cluster 1", func() { + setPlacementPolicyBindingSuspended(policyKey, suspendedClusterName, true) + }) + + It("should remove the Works of the suspended binding, leaving the ConfigMap on member cluster 1, orphaned", func() { + Eventually(suspendedPlacementPolicyBindingActual(policyKey, suspendedClusterName, bindingUIDsByCluster[suspendedClusterName], snapshotName), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to process the suspension of the binding") + Eventually(placementPolicyWorkClustersActual(policyKey, otherClusterNames...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the Works of the suspended binding") + // The resumed binding's Work, and with it its AppliedWork, gets the same name; wait for the old one to go + // before resuming. + Eventually(appliedWorksAbsentOnClustersActual(map[string][]string{suspendedClusterName: workNamesByCluster[suspendedClusterName]}), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the AppliedWorks from member cluster 1") + + orphanedActual := configMapOrphanedOnClustersActual(cmKey, appConfigMap().Data, memberCluster1EastProd) + Eventually(orphanedActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to orphan the ConfigMap on member cluster 1") + Consistently(orphanedActual, consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The orphaned ConfigMap was removed, or KubeFleet took it back") + }) + + It("should report the policy as synchronized on the other member clusters only", func() { + wantStatus := someClustersNotSynchronizedPlacementPolicyStatus(clusterCount(), clusterCount()-1) + wantStatus.LatestResourceRevisionName = ptr.To(snapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + + It("can resume the binding for member cluster 1", func() { + setPlacementPolicyBindingSuspended(policyKey, suspendedClusterName, false) + }) + + It("should report the orphaned ConfigMap as not taken over", func() { + // The default whenAlreadyExists=ReportError leaves the ConfigMap alone, now that KubeFleet no longer owns it. + wantBindings := map[string]wantPlacementBinding{ + suspendedClusterName: { + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: snapshotName, + status: notTakenOverPlacementBindingStatus(cmKey), + }, + } + for _, clusterName := range otherClusterNames { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: snapshotName, + status: synchronizedPlacementBindingStatus(1), + } + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as not taken over") + Expect(placementPolicyBindingsKeptActual(policyKey, bindingUIDsByCluster)()).To(Succeed(), "A binding was replaced") + + wantStatus := someClustersNotSynchronizedPlacementPolicyStatus(clusterCount(), clusterCount()-1) + wantStatus.LatestResourceRevisionName = ptr.To(snapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + Consistently(configMapOrphanedOnClustersActual(cmKey, appConfigMap().Data, memberCluster1EastProd), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The orphaned ConfigMap was taken over") + }) + + It("can switch the policy to take over existing resources that have no differences", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().SyncStrategy = &kfplacementv1alpha1.SyncStrategy{ + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff, + WhenPlacementDeleted: kfplacementv1alpha1.WhenPlacementDeletedOptionOrphanResources, + } + }) + }) + + It("should take the ConfigMap on member cluster 1 over again, without a rollout", func() { + // The orphaned ConfigMap still matches its hub copy. + wantBindings := make(map[string]wantPlacementBinding, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: snapshotName, + status: synchronizedPlacementBindingStatus(1), + } + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to synchronize every binding") + Eventually(configMapTakenOverOnClusterActual(memberCluster1EastProd, cmKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to take the ConfigMap over on member cluster 1") + + wantStatus := allSynchronizedPlacementPolicyStatus(clusterCount()) + wantStatus.LatestResourceRevisionName = ptr.To(snapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) + + Context("placement policy auto-rolling out a change while a binding is suspended", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicySuspendedAutoRolloutNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + suspendedClusterName := memberCluster2EastCanaryName + otherClusterNames := []string{memberCluster1EastProdName, memberCluster3WestProdName} + otherClusters := func() []*framework.Cluster { + return []*framework.Cluster{memberCluster1EastProd, memberCluster3WestProd} + } + // The member clusters are known only once the suite is set up, after the spec tree is built. + clusterCount := func() int32 { return int32(len(allMemberClusterNames)) } + + var initialSnapshotName, newSnapshotName string + var bindingUIDsByCluster map[string]types.UID + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects every member cluster, with auto-rollout enabled") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + policy.SetAnnotations(autoRolloutAnnotations()) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + bindingUIDsByCluster = placementPolicyBindingUIDsByCluster(policyKey) + }) + + It("can suspend the binding for member cluster 2", func() { + setPlacementPolicyBindingSuspended(policyKey, suspendedClusterName, true) + }) + + It("should remove the Works and the ConfigMap of the suspended binding", func() { + Eventually(suspendedPlacementPolicyBindingActual(policyKey, suspendedClusterName, bindingUIDsByCluster[suspendedClusterName], initialSnapshotName), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to process the suspension of the binding") + Eventually(placementPolicyWorkClustersActual(policyKey, otherClusterNames...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the Works of the suspended binding") + Eventually(appConfigMapRemovedFromClustersActual(memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from member cluster 2") + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("should roll the change out to every binding, but apply it only on the clusters whose binding is not suspended", func() { + // Auto-rollout moves every binding of the policy to the new resource snapshot, the suspended one included; + // the work generator does not act on the suspended one, though. + Eventually(placementPolicyLatestSnapshotChangedActual(policyKey, initialSnapshotName, &newSnapshotName), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to take a new resource snapshot") + suspendedActual := suspendedPlacementPolicyBindingActual(policyKey, suspendedClusterName, bindingUIDsByCluster[suspendedClusterName], newSnapshotName) + Eventually(suspendedActual, placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to move the suspended binding to the new resource snapshot") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, otherClusters()...), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on the other member clusters") + + // The suspended binding names the latest revision but has processed none, so it counts as neither + // synchronized nor on an older revision. + wantStatus := someClustersNotSynchronizedPlacementPolicyStatus(clusterCount(), clusterCount()-1) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + Eventually(statusActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + + worksActual := placementPolicyWorkClustersActual(policyKey, otherClusterNames...) + removedActual := appConfigMapRemovedFromClustersActual(memberCluster2EastCanary) + Consistently(func() error { + for _, actual := range []func() error{suspendedActual, worksActual, removedActual, statusActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The change reached member cluster 2, or the suspended binding changed") + }) + + It("can resume the binding for member cluster 2", func() { + setPlacementPolicyBindingSuspended(policyKey, suspendedClusterName, false) + }) + + It("should place the latest version on member cluster 2", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to synchronize every binding on the latest resource snapshot") + Expect(placementPolicyBindingsKeptActual(policyKey, bindingUIDsByCluster)()).To(Succeed(), "A binding was replaced") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the latest version on every member cluster") + + wantStatus := allSynchronizedPlacementPolicyStatus(clusterCount()) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) +}) diff --git a/test/e2e/placement_policy_interop_test.go b/test/e2e/placement_policy_interop_test.go new file mode 100644 index 000000000..d094e97b3 --- /dev/null +++ b/test/e2e/placement_policy_interop_test.go @@ -0,0 +1,589 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/controllers/workapplier" + "go.goms.io/fleet/test/e2e/framework" +) + +const ( + placementPolicyInteropCRPNameTemplate = "pp-interop-crp-%d" + placementPolicyAfterCRPNameTemplate = "pp-after-crp-%d" + placementPolicyInteropLaterCRPNameTemplate = "pp-interop-later-crp-%d" + placementPolicyBeforeCRPNameTemplate = "pp-before-crp-%d" + placementPolicySharedFirstNameTemplate = "pp-shared-first-%d" + placementPolicySharedSecondNameTemplate = "pp-shared-second-%d" +) + +// placementPolicyOwnedByOtherKubeFleetAPIObjectsReason is the apply result type, copied into a binding's failed +// resources, of a manifest whose object an AppliedWork of the older placement APIs owns. +const placementPolicyOwnedByOtherKubeFleetAPIObjectsReason = "OwnedByOtherKubeFleetAPIObjects" + +// isOlderPlacementAPIsAppliedWorkOwnerRef reports whether the owner reference points to an AppliedWork of the older +// placement APIs, in any version of their API group. +func isOlderPlacementAPIsAppliedWorkOwnerRef(ownerRef metav1.OwnerReference) bool { + gv, err := schema.ParseGroupVersion(ownerRef.APIVersion) + return err == nil && gv.Group == placementv1beta1.GroupVersion.Group && ownerRef.Kind == placementv1beta1.AppliedWorkKind +} + +// configMapPlacedByPlacementPolicyOnlyActual checks that the given placement policy, and no other KubeFleet +// placement API object of either experience, has placed the ConfigMap on the cluster; see +// configMapPlacedByPlacementPolicyActual. +func configMapPlacedByPlacementPolicyOnlyActual(cluster *framework.Cluster, cmKey, policyKey types.NamespacedName) func() error { + placedActual := configMapPlacedByPlacementPolicyActual(cluster, cmKey, policyKey) + return func() error { + if err := placedActual(); err != nil { + return err + } + cm := &corev1.ConfigMap{} + if err := cluster.KubeClient.Get(ctx, cmKey, cm); err != nil { + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", cluster.ClusterName, cmKey, err) + } + for _, ownerRef := range cm.OwnerReferences { + if isOlderPlacementAPIsAppliedWorkOwnerRef(ownerRef) { + return fmt.Errorf("cluster %s: ConfigMap %s is also owned by AppliedWork %s of the older placement APIs", + cluster.ClusterName, cmKey, ownerRef.Name) + } + } + return nil + } +} + +// namespaceTakenOverByOlderPlacementAPIsActual checks that the namespace on the cluster, which the placement policy +// work applier created without owning it, is now owned by an AppliedWork of the older placement APIs alone, i.e., that +// a CRP has taken it over. +func namespaceTakenOverByOlderPlacementAPIsActual(cluster *framework.Cluster, name string) func() error { + return func() error { + ns := &corev1.Namespace{} + if err := cluster.KubeClient.Get(ctx, types.NamespacedName{Name: name}, ns); err != nil { + return fmt.Errorf("cluster %s: failed to get namespace %s: %w", cluster.ClusterName, name, err) + } + if !ns.DeletionTimestamp.IsZero() { + return fmt.Errorf("cluster %s: namespace %s is being deleted", cluster.ClusterName, name) + } + if ownedByPlacementPolicyAppliedWork(ns) { + return fmt.Errorf("cluster %s: namespace %s is owned by an AppliedWork of the placement policy APIs (owner references: %+v)", + cluster.ClusterName, name, ns.OwnerReferences) + } + for _, ownerRef := range ns.OwnerReferences { + if isOlderPlacementAPIsAppliedWorkOwnerRef(ownerRef) { + return nil + } + } + return fmt.Errorf("cluster %s: namespace %s is not owned by an AppliedWork of the older placement APIs (owner references: %+v)", + cluster.ClusterName, name, ns.OwnerReferences) + } +} + +// crpRefusedConfigMapOnClusterActual checks that the CRP, which places the work resources on the given cluster alone, +// reports, for its current generation, that the namespace is applied but the ConfigMap is not, as a placement policy +// already owns it. +func crpRefusedConfigMapOnClusterActual(crpKey, cmKey types.NamespacedName, clusterName string) func() error { + return func() error { + crp := &placementv1beta1.ClusterResourcePlacement{} + if err := hubClient.Get(ctx, crpKey, crp); err != nil { + return err + } + wantStatus := placementv1beta1.PlacementStatus{ + Conditions: crpAppliedFailedConditions(crp.Generation), + SelectedResources: workResourceIdentifiers(), + PerClusterPlacementStatuses: []placementv1beta1.PerClusterPlacementStatus{ + { + ClusterName: clusterName, + ObservedResourceIndex: "0", + Conditions: perClusterApplyFailedConditions(crp.Generation), + FailedPlacements: []placementv1beta1.FailedResourcePlacement{ + { + ResourceIdentifier: placementv1beta1.ResourceIdentifier{ + Version: "v1", + Kind: "ConfigMap", + Name: cmKey.Name, + Namespace: cmKey.Namespace, + }, + Condition: metav1.Condition{ + Type: string(placementv1beta1.PerClusterAppliedConditionType), + Status: metav1.ConditionFalse, + Reason: string(workapplier.ApplyOrReportDiffResTypeOwnedByOtherKubeFleetAPIObjects), + }, + }, + }, + }, + }, + ObservedResourceIndex: "0", + } + if diff := cmp.Diff(crp.Status, wantStatus, placementStatusCmpOptions...); diff != "" { + return fmt.Errorf("CRP %s status diff (-got, +want):\n%s", crpKey.Name, diff) + } + return nil + } +} + +// configMapPlacedByOlderPlacementAPIsOnlyActual checks that the ConfigMap on the cluster matches its hub copy and is +// owned by an AppliedWork of the older placement APIs (e.g., that of a CRP), and by none of the placement policy APIs. +func configMapPlacedByOlderPlacementAPIsOnlyActual(cluster *framework.Cluster, key types.NamespacedName) func() error { + return func() error { + if err := validateConfigMapOnCluster(cluster, key); err != nil { + return fmt.Errorf("cluster %s: %w", cluster.ClusterName, err) + } + cm := &corev1.ConfigMap{} + if err := cluster.KubeClient.Get(ctx, key, cm); err != nil { + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", cluster.ClusterName, key, err) + } + if ownedByPlacementPolicyAppliedWork(cm) { + return fmt.Errorf("cluster %s: ConfigMap %s is owned by an AppliedWork of the placement policy APIs (owner references: %+v)", + cluster.ClusterName, key, cm.OwnerReferences) + } + for _, ownerRef := range cm.OwnerReferences { + if ownerRef.APIVersion == placementv1beta1.GroupVersion.String() && ownerRef.Kind == placementv1beta1.AppliedWorkKind { + return nil + } + } + return fmt.Errorf("cluster %s: ConfigMap %s is not owned by an AppliedWork of the older placement APIs (owner references: %+v)", + cluster.ClusterName, key, cm.OwnerReferences) + } +} + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy interoperability", Label("placementpolicy"), func() { + Context("placement policy selecting a ConfigMap that a CRP has placed", Ordered, func() { + crpKey := types.NamespacedName{Name: fmt.Sprintf(placementPolicyInteropCRPNameTemplate, GinkgoParallelProcess())} + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyAfterCRPNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + // Both the CRP and the policy place on cluster-1 alone. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + crpStatusActual := func() func() error { + return customizedPlacementStatusUpdatedActual(crpKey, workResourceIdentifiers(), []string{memberCluster1EastProdName}, nil, "0", true) + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a CRP that places the app namespace, and the ConfigMap in it, on member cluster 1") + crp := &placementv1beta1.ClusterResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{ + Name: crpKey.Name, + // The custom finalizer lets ensureCRPAndRelatedResourcesDeleted observe the CRP's own cleanup. + Finalizers: []string{customDeletionBlockerFinalizer}, + }, + Spec: placementv1beta1.PlacementSpec{ + ResourceSelectors: workResourceSelector(), + Strategy: placementv1beta1.RolloutStrategy{ + Type: placementv1beta1.RollingUpdateRolloutStrategyType, + RollingUpdate: &placementv1beta1.RollingUpdateConfig{ + UnavailablePeriodSeconds: ptr.To(2), + }, + }, + Policy: &placementv1beta1.PlacementPolicy{ + PlacementType: placementv1beta1.PickFixedPlacementType, + ClusterNames: []string{memberCluster1EastProdName}, + }, + }, + } + Expect(hubClient.Create(ctx, crp)).To(Succeed(), "Failed to create CRP %s", crpKey.Name) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy or the CRP fails; otherwise they leak into later + // specs. + defer cleanupWorkResourcesOnHubAndMembers() + // The policy goes first: it never placed the ConfigMap, so removing it leaves the CRP's placement alone. + deletePlacementPolicyAndWait(policyKey) + ensureCRPAndRelatedResourcesDeleted(crpKey.Name, []*framework.Cluster{memberCluster1EastProd}) + }) + + It("should place the namespace and the ConfigMap on member cluster 1 through the CRP", func() { + Eventually(crpStatusActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected CRP status") + Eventually(workNamespaceAndConfigMapPlacedOnClusterActual(memberCluster1EastProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the namespace and the ConfigMap on member cluster 1") + Expect(configMapPlacedByOlderPlacementAPIsOnlyActual(memberCluster1EastProd, cmKey)()). + To(Succeed(), "The ConfigMap on member cluster 1 is not managed by the CRP alone") + }) + + It("can create a placement policy that selects the same ConfigMap for member cluster 1, sharing ownership", func() { + // Even the most permissive sync strategy, which takes over any existing object and shares its ownership, + // must not take over an object that another KubeFleet placement API manages. + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + SyncStrategy: &kfplacementv1alpha1.SyncStrategy{ + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: kfplacementv1alpha1.WhenOwnedByOthersOptionShareOwnership, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + It("should report the ConfigMap as owned by another KubeFleet placement API", func() { + wantBindings := map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{clusterSelectorTermsMatchingLabels(aliasLabels)}, + status: configMapNotAppliedPlacementBindingStatus(cmKey, placementPolicyOwnedByOtherKubeFleetAPIObjectsReason, nil), + }, + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as owned by another KubeFleet placement API") + Eventually(placementPolicyStatusActual(policyKey, someClustersNotSynchronizedPlacementPolicyStatus(1, 0)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should leave the CRP and its ConfigMap on member cluster 1 as they were", func() { + statusActual := crpStatusActual() + cmActual := configMapPlacedByOlderPlacementAPIsOnlyActual(memberCluster1EastProd, cmKey) + Consistently(func() error { + if err := statusActual(); err != nil { + return err + } + return cmActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The CRP or its ConfigMap on member cluster 1 changed") + }) + + It("can delete the placement policy", func() { + deletePlacementPolicyAndWait(policyKey) + }) + + It("should still leave the CRP and its ConfigMap on member cluster 1 as they were", func() { + // The policy never took the ConfigMap over, so removing its AppliedWork leaves the ConfigMap alone. + statusActual := crpStatusActual() + cmActual := configMapPlacedByOlderPlacementAPIsOnlyActual(memberCluster1EastProd, cmKey) + Consistently(func() error { + if err := statusActual(); err != nil { + return err + } + return cmActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The CRP or its ConfigMap on member cluster 1 changed") + }) + }) + + Context("CRP selecting a namespace whose ConfigMap a placement policy has placed", Ordered, func() { + nsName := appNamespace().Name + crpKey := types.NamespacedName{Name: fmt.Sprintf(placementPolicyInteropLaterCRPNameTemplate, GinkgoParallelProcess())} + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyBeforeCRPNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + // Both the policy and the CRP place on cluster-1 alone. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + wantSelectorsByCluster := map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster1EastProdName: {clusterSelectorTermsMatchingLabels(aliasLabels)}, + } + + var policyCMUID types.UID + + // policyUndisturbedActual checks that the policy still has the ConfigMap that it created placed on cluster-1, + // synchronized, and owned by its AppliedWork alone. + policyUndisturbedActual := func() func() error { + bindingsActual := allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster, 1) + statusActual := placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(1)) + cmActual := configMapPlacedByPlacementPolicyOnlyActual(memberCluster1EastProd, cmKey, policyKey) + return func() error { + for _, actual := range []func() error{bindingsActual, statusActual, cmActual} { + if err := actual(); err != nil { + return err + } + } + uids, err := configMapUIDsByCluster(cmKey, memberCluster1EastProd) + if err != nil { + return err + } + if uids[memberCluster1EastProdName] != policyCMUID { + return fmt.Errorf("ConfigMap %s on member cluster 1 was re-created (UID %s, want %s)", cmKey, uids[memberCluster1EastProdName], policyCMUID) + } + return nil + } + } + // crpRefusedActual checks that the CRP has taken the namespace over but refuses the ConfigMap, leaving the policy + // alone. + crpRefusedActual := func() func() error { + statusActual := crpRefusedConfigMapOnClusterActual(crpKey, cmKey, memberCluster1EastProdName) + nsActual := namespaceTakenOverByOlderPlacementAPIsActual(memberCluster1EastProd, nsName) + policyActual := policyUndisturbedActual() + return func() error { + for _, actual := range []func() error{statusActual, nsActual, policyActual} { + if err := actual(); err != nil { + return err + } + } + return nil + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that places the ConfigMap on member cluster 1") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy or the CRP fails; otherwise they leak into later + // specs. + defer cleanupWorkResourcesOnHubAndMembers() + // The policy goes first. The CRP owns the namespace that it took over, so deleting the CRP while the policy + // still places the ConfigMap would delete the namespace, and the ConfigMap in it, under the policy. + deletePlacementPolicyAndWait(policyKey) + ensureCRPAndRelatedResourcesDeleted(crpKey.Name, []*framework.Cluster{memberCluster1EastProd}) + }) + + It("should place the ConfigMap on member cluster 1 through the placement policy, in a namespace that no one owns", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster, 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to synchronize the binding of the placement policy") + Eventually(configMapPlacedByPlacementPolicyOnlyActual(memberCluster1EastProd, cmKey, policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap through the placement policy") + // The work applier creates the namespace of the ConfigMap but does not manage it. + Expect(namespaceLeftUnmanagedOnClustersActual(nsName, memberCluster1EastProd)()). + To(Succeed(), "The namespace of the ConfigMap on member cluster 1 is managed by KubeFleet") + + uids, err := configMapUIDsByCluster(cmKey, memberCluster1EastProd) + Expect(err).NotTo(HaveOccurred(), "Failed to read the ConfigMap on member cluster 1") + policyCMUID = uids[memberCluster1EastProdName] + Eventually(policyUndisturbedActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the placement policy as synchronized") + }) + + It("can create a CRP that places the app namespace, and the ConfigMap in it, on member cluster 1", func() { + crp := &placementv1beta1.ClusterResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{ + Name: crpKey.Name, + // The custom finalizer lets ensureCRPAndRelatedResourcesDeleted observe the CRP's own cleanup. + Finalizers: []string{customDeletionBlockerFinalizer}, + }, + Spec: placementv1beta1.PlacementSpec{ + ResourceSelectors: workResourceSelector(), + Strategy: placementv1beta1.RolloutStrategy{ + Type: placementv1beta1.RollingUpdateRolloutStrategyType, + RollingUpdate: &placementv1beta1.RollingUpdateConfig{ + UnavailablePeriodSeconds: ptr.To(2), + }, + }, + Policy: &placementv1beta1.PlacementPolicy{ + PlacementType: placementv1beta1.PickFixedPlacementType, + ClusterNames: []string{memberCluster1EastProdName}, + }, + }, + } + Expect(hubClient.Create(ctx, crp)).To(Succeed(), "Failed to create CRP %s", crpKey.Name) + }) + + It("should refuse the ConfigMap for the CRP, taking over only the namespace, and leave the placement policy alone", func() { + // The namespace has no owner, so the CRP takes it over (the default whenToTakeOver is Always); the ConfigMap + // is owned by the placement policy's AppliedWork, which the older work applier refuses before any takeover. + Eventually(crpRefusedActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to refuse the ConfigMap for the CRP") + Consistently(crpRefusedActual(), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The CRP took the ConfigMap over, or the placement policy changed") + }) + + It("can allow the CRP to share the ownership of the resources it places", func() { + Eventually(func() error { + crp := &placementv1beta1.ClusterResourcePlacement{} + if err := hubClient.Get(ctx, crpKey, crp); err != nil { + return err + } + if crp.Spec.Strategy.ApplyStrategy == nil { + crp.Spec.Strategy.ApplyStrategy = &placementv1beta1.ApplyStrategy{} + } + crp.Spec.Strategy.ApplyStrategy.AllowCoOwnership = true + return hubClient.Update(ctx, crp) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to allow co-ownership on CRP %s", crpKey.Name) + }) + + It("should still refuse the ConfigMap for the CRP, and leave the placement policy alone", func() { + // The safeguard runs whatever the apply strategy, so co-ownership does not let the two placement experiences + // manage the same object. The CRP status is checked for the CRP's new generation. + Eventually(crpRefusedActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to refuse the ConfigMap for the CRP with co-ownership allowed") + Consistently(crpRefusedActual(), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The CRP took the ConfigMap over, or the placement policy changed") + }) + + It("can delete the placement policy", func() { + deletePlacementPolicyAndWait(policyKey) + }) + + It("should let the CRP create the ConfigMap anew once the placement policy has removed it", func() { + Eventually(customizedPlacementStatusUpdatedActual(crpKey, workResourceIdentifiers(), []string{memberCluster1EastProdName}, nil, "0", true), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the CRP as applied") + Eventually(configMapPlacedByOlderPlacementAPIsOnlyActual(memberCluster1EastProd, cmKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap through the CRP") + + uids, err := configMapUIDsByCluster(cmKey, memberCluster1EastProd) + Expect(err).NotTo(HaveOccurred(), "Failed to read the ConfigMap on member cluster 1") + Expect(uids[memberCluster1EastProdName]).NotTo(Equal(policyCMUID), "The CRP took over the placement policy's ConfigMap instead of creating it anew") + }) + }) + + Context("placement policies selecting the same ConfigMap for the same member cluster", Ordered, func() { + nsName := appNamespace().Name + firstPolicyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicySharedFirstNameTemplate, GinkgoParallelProcess())} + secondPolicyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicySharedSecondNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + // Both policies place on cluster-1 alone. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + newPolicy := func(key types.NamespacedName, syncStrategy *kfplacementv1alpha1.SyncStrategy) kfplacementv1alpha1.PlacementPolicyAccessor { + return newPlacementPolicyObject(key, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + SyncStrategy: syncStrategy, + }) + } + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + return map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster1EastProdName: {clusterSelectorTermsMatchingLabels(aliasLabels)}, + } + } + // secondPolicyFailedActual checks that the second policy's binding reports the ConfigMap as not applied, for the + // given reason, and that the policy reports the cluster as not synchronized. + secondPolicyFailedActual := func(reason string) func() error { + bindingsActual := placementPolicyBindingsActual(secondPolicyKey, map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: wantSelectorsByCluster()[memberCluster1EastProdName], + status: configMapNotAppliedPlacementBindingStatus(cmKey, reason, nil), + }, + }) + statusActual := placementPolicyStatusActual(secondPolicyKey, someClustersNotSynchronizedPlacementPolicyStatus(1, 0)) + return func() error { + if err := bindingsActual(); err != nil { + return err + } + return statusActual() + } + } + // firstPolicyUndisturbedActual checks that the first policy still has the ConfigMap placed on cluster-1 and + // synchronized, with no other policy owning it. + firstPolicyUndisturbedActual := func() func() error { + bindingsActual := allSynchronizedPlacementPolicyBindingsActual(firstPolicyKey, wantSelectorsByCluster(), 1) + statusActual := placementPolicyStatusActual(firstPolicyKey, allSynchronizedPlacementPolicyStatus(1)) + cmActual := configMapPlacedByPlacementPolicyActual(memberCluster1EastProd, cmKey, firstPolicyKey) + return func() error { + for _, actual := range []func() error{bindingsActual, statusActual, cmActual} { + if err := actual(); err != nil { + return err + } + } + return nil + } + } + + var firstCMUID types.UID + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating the first placement policy, with the default sync strategy") + Expect(hubClient.Create(ctx, newPolicy(firstPolicyKey, nil))).To(Succeed(), "Failed to create placement policy %s", firstPolicyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policies fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // The first policy goes first; the ConfigMap can leave the member cluster only once neither policy places it. + deletePlacementPolicyAndWait(firstPolicyKey) + ensurePlacementPolicyRemoved(secondPolicyKey) + }) + + It("should place the ConfigMap on member cluster 1 through the first policy", func() { + Eventually(firstPolicyUndisturbedActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap through the first placement policy") + + cm := &corev1.ConfigMap{} + Expect(memberCluster1EastProd.KubeClient.Get(ctx, cmKey, cm)).To(Succeed(), "Failed to get ConfigMap %s on member cluster 1", cmKey) + firstCMUID = cm.UID + }) + + It("can create the second placement policy, which always takes over existing resources and shares their ownership", func() { + // Even the most permissive sync strategy must not take over an object that another placement policy + // manages. + Expect(hubClient.Create(ctx, newPolicy(secondPolicyKey, &kfplacementv1alpha1.SyncStrategy{ + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: kfplacementv1alpha1.WhenOwnedByOthersOptionShareOwnership, + }))).To(Succeed(), "Failed to create placement policy %s", secondPolicyKey) + }) + + It("should fail to take the ConfigMap over for the second policy, leaving the first policy alone", func() { + // KubeFleet refuses to place the same object twice, co-ownership or not. + Eventually(secondPolicyFailedActual("FailedToTakeOver"), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as not taken over for the second placement policy") + Consistently(firstPolicyUndisturbedActual(), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The first placement policy or its ConfigMap changed") + }) + + It("can switch the second placement policy to the default sync strategy", func() { + // With a takeover option, the second policy's work applier would drop the ConfigMap's owner reference to the + // first policy's AppliedWork as soon as the first policy's Work is gone, and could take the ConfigMap over + // before the garbage collector removes it. Under the default whenAlreadyExists=ReportError, the ConfigMap + // must go first, so the second policy creates it anew. + updatePlacementPolicy(secondPolicyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().SyncStrategy = nil + }) + }) + + It("should report the ConfigMap as not taken over for the second policy, leaving the first policy alone", func() { + Eventually(secondPolicyFailedActual("NotTakenOver"), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as not taken over for the second placement policy") + Consistently(firstPolicyUndisturbedActual(), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The first placement policy or its ConfigMap changed") + }) + + It("can delete the first placement policy", func() { + deletePlacementPolicyAndWait(firstPolicyKey) + }) + + It("should create the ConfigMap on member cluster 1 anew through the second policy", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(secondPolicyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to synchronize the binding of the second placement policy") + Eventually(configMapPlacedByPlacementPolicyActual(memberCluster1EastProd, cmKey, secondPolicyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap through the second placement policy") + Eventually(placementPolicyStatusActual(secondPolicyKey, allSynchronizedPlacementPolicyStatus(1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + + cm := &corev1.ConfigMap{} + Expect(memberCluster1EastProd.KubeClient.Get(ctx, cmKey, cm)).To(Succeed(), "Failed to get ConfigMap %s on member cluster 1", cmKey) + Expect(cm.UID).NotTo(Equal(firstCMUID), "The second placement policy took over the first policy's ConfigMap instead of creating it anew") + }) + }) +}) diff --git a/test/e2e/placement_policy_join_leave_test.go b/test/e2e/placement_policy_join_leave_test.go new file mode 100644 index 000000000..fbe72f698 --- /dev/null +++ b/test/e2e/placement_policy_join_leave_test.go @@ -0,0 +1,483 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/test/e2e/framework" +) + +const ( + clusterPlacementPolicyNeverJoinedNameTemplate = "cpp-never-joined-%d" + neverJoinedMemberClusterNameTemplate = "pp-never-joined-cluster-%d" + + clusterPlacementPolicyAgentStoppedNameTemplate = "cpp-agent-stopped-%d" + + clusterPlacementPolicyNewClusterNameTemplate = "cpp-new-cluster-%d" + + // memberAgentDeploymentName is the Deployment that runs the member agent on a member cluster, in the fleet-system + // namespace. + memberAgentDeploymentName = "member-agent" + // memberAgentStoppedEventuallyDuration bounds the wait for the placement policy controller to treat a cluster + // whose member agent stopped as no longer eligible: the eligibility checker's heartbeat timeout is 5m, and the + // controller re-evaluates a fulfilled policy at least every 150s, with a margin on top. + memberAgentStoppedEventuallyDuration = 9 * time.Minute + + // neverJoinedLabelName labels the member cluster that the specs register without ever letting it join; no member + // cluster of the fleet carries it. + neverJoinedLabelName = "fleet-e2e/never-joined" + // newClusterLabelName labels a member cluster as it joins the fleet anew; no member cluster of the fleet carries it + // otherwise. + newClusterLabelName = "fleet-e2e/new-cluster" +) + +// noPlacementBindingTargetsClusterActual checks that no placement binding, in any namespace, and no cluster placement +// binding targets the member cluster. +func noPlacementBindingTargetsClusterActual(clusterName string) func() error { + return func() error { + bindings := &kfplacementv1alpha1.PlacementBindingList{} + if err := hubClient.List(ctx, bindings); err != nil { + return err + } + for i := range bindings.Items { + if bindings.Items[i].Spec.ClusterName == clusterName { + return fmt.Errorf("placement binding %s targets member cluster %s", client.ObjectKeyFromObject(&bindings.Items[i]), clusterName) + } + } + clusterBindings := &kfplacementv1alpha1.ClusterPlacementBindingList{} + if err := hubClient.List(ctx, clusterBindings); err != nil { + return err + } + for i := range clusterBindings.Items { + if clusterBindings.Items[i].Spec.ClusterName == clusterName { + return fmt.Errorf("cluster placement binding %s targets member cluster %s", clusterBindings.Items[i].Name, clusterName) + } + } + return nil + } +} + +// placementPolicyClaimUnfulfilledActual checks that the policy's only cluster claim has not been fulfilled: no +// cluster has been provisioned for it, and it does not report completion. +func placementPolicyClaimUnfulfilledActual(key types.NamespacedName) func() error { + return func() error { + claims, err := claimsOfPolicy(key.Name, key.Namespace) + if err != nil { + return err + } + if len(claims.Items) != 1 { + return fmt.Errorf("placement policy %s has %d cluster claims, want 1", key, len(claims.Items)) + } + claim := &claims.Items[0] + if claim.Status.ProvisionedClusterName != nil { + return fmt.Errorf("cluster claim %s of placement policy %s names provisioned cluster %s", claim.Name, key, *claim.Status.ProvisionedClusterName) + } + if cond := meta.FindStatusCondition(claim.Status.Conditions, kfplacementv1alpha1.ClusterClaimCondTypeCompleted); cond != nil { + return fmt.Errorf("cluster claim %s of placement policy %s reports a %s condition: %+v", claim.Name, key, kfplacementv1alpha1.ClusterClaimCondTypeCompleted, cond) + } + return nil + } +} + +// rejoinMemberClusterWithExtraLabels lets a member cluster that has left the fleet join it again, carrying its usual +// labels plus the given ones, and waits until it has joined. +func rejoinMemberClusterWithExtraLabels(cluster *framework.Cluster, extraLabels map[string]string) { + GinkgoHelper() + + labels := maps.Clone(labelsByClusterName[cluster.ClusterName]) + maps.Copy(labels, extraLabels) + createMemberCluster(cluster.ClusterName, cluster.PresentingServiceAccountInHubClusterName, labels, annotationsByClusterName[cluster.ClusterName]) + checkIfMemberClusterHasJoined(cluster) +} + +// restoreMemberClusterInFleet makes sure that the member cluster is in the fleet again, without the given labels, and +// waits until it has joined. It lets a member cluster that has left join again with its usual labels, after any +// leave in progress has completed. +func restoreMemberClusterInFleet(cluster *framework.Cluster, labelKeysToRemove ...string) { + GinkgoHelper() + + var registered bool + Eventually(func() error { + mc := &clusterv1beta1.MemberCluster{} + err := hubClient.Get(ctx, types.NamespacedName{Name: cluster.ClusterName}, mc) + switch { + case apierrors.IsNotFound(err): + registered = false + return nil + case err != nil: + return err + case !mc.DeletionTimestamp.IsZero(): + return fmt.Errorf("member cluster %s is still leaving the fleet", cluster.ClusterName) + } + registered = true + return nil + }, longEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to wait for member cluster %s to settle", cluster.ClusterName) + + if !registered { + createMemberCluster(cluster.ClusterName, cluster.PresentingServiceAccountInHubClusterName, + labelsByClusterName[cluster.ClusterName], annotationsByClusterName[cluster.ClusterName]) + } else { + Eventually(func() error { + mc := &clusterv1beta1.MemberCluster{} + if err := hubClient.Get(ctx, types.NamespacedName{Name: cluster.ClusterName}, mc); err != nil { + return err + } + changed := false + for _, key := range labelKeysToRemove { + if _, found := mc.Labels[key]; found { + delete(mc.Labels, key) + changed = true + } + } + if !changed { + return nil + } + return hubClient.Update(ctx, mc) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove labels %v from member cluster %s", labelKeysToRemove, cluster.ClusterName) + } + checkIfMemberClusterHasJoined(cluster) +} + +// setMemberAgentReplicas scales the member agent Deployment on the member cluster, retrying on conflicts. +func setMemberAgentReplicas(cluster *framework.Cluster, replicas int32) { + GinkgoHelper() + + Eventually(func() error { + deploy := &appsv1.Deployment{} + if err := cluster.KubeClient.Get(ctx, types.NamespacedName{Namespace: fleetSystemNS, Name: memberAgentDeploymentName}, deploy); err != nil { + return err + } + deploy.Spec.Replicas = ptr.To(replicas) + return cluster.KubeClient.Update(ctx, deploy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to scale the member agent on cluster %s to %d replicas", cluster.ClusterName, replicas) +} + +// stopMemberAgent scales the member agent on the member cluster down, and waits until none of its pods is left, so +// that the cluster sends no more heartbeats to the hub. +func stopMemberAgent(cluster *framework.Cluster) { + GinkgoHelper() + + setMemberAgentReplicas(cluster, 0) + Eventually(func() error { + deploy := &appsv1.Deployment{} + if err := cluster.KubeClient.Get(ctx, types.NamespacedName{Namespace: fleetSystemNS, Name: memberAgentDeploymentName}, deploy); err != nil { + return err + } + selector, err := metav1.LabelSelectorAsSelector(deploy.Spec.Selector) + if err != nil { + return err + } + pods := &corev1.PodList{} + if err := cluster.KubeClient.List(ctx, pods, client.InNamespace(fleetSystemNS), client.MatchingLabelsSelector{Selector: selector}); err != nil { + return err + } + if len(pods.Items) > 0 { + return fmt.Errorf("cluster %s still runs %d member agent pods", cluster.ClusterName, len(pods.Items)) + } + return nil + }, longEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to stop the member agent on cluster %s", cluster.ClusterName) +} + +// startMemberAgent scales the member agent on the member cluster back up, and waits until the cluster has sent a +// heartbeat since and reports itself as joined and healthy again. +func startMemberAgent(cluster *framework.Cluster) { + GinkgoHelper() + + // Heartbeat timestamps have a resolution of one second. + startedAt := time.Now().Truncate(time.Second) + setMemberAgentReplicas(cluster, 1) + Eventually(func() error { + mc := &clusterv1beta1.MemberCluster{} + if err := hubClient.Get(ctx, types.NamespacedName{Name: cluster.ClusterName}, mc); err != nil { + return err + } + agentStatus := mc.GetAgentStatus(clusterv1beta1.MemberAgent) + if agentStatus == nil || agentStatus.LastReceivedHeartbeat.Time.Before(startedAt) { + return fmt.Errorf("member cluster %s has received no heartbeat from its member agent since %v (agent status: %+v)", cluster.ClusterName, startedAt, agentStatus) + } + return nil + }, longEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to restart the member agent on cluster %s", cluster.ClusterName) + checkIfMemberClusterHasJoined(cluster) +} + +// The specs below change the member clusters of the fleet, adding one, stopping the member agent of one, or letting +// one leave and join again, so they are Serial: they run only after the parallel specs finish, and undo every change +// they make, so that the fleet is whole for whatever runs next. +var _ = Describe("placement policy with member clusters joining and leaving the fleet", Label("placementpolicy"), func() { + Context("cluster placement policy whose selector a registered member cluster matches without ever joining", Ordered, Serial, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyNeverJoinedNameTemplate, GinkgoParallelProcess()), + } + memberClusterName := fmt.Sprintf(neverJoinedMemberClusterNameTemplate, GinkgoParallelProcess()) + neverJoinedLabels := map[string]string{neverJoinedLabelName: "true"} + neverJoinedTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: neverJoinedLabels}, + } + + var claimUID types.UID + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects one cluster carrying a label that no member cluster has") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(neverJoinedTerms, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources and the registered member cluster must go even if removing the policy fails; otherwise + // they leak into later specs. The cleanup strips the member cluster's finalizers, as no agent ever reports + // that it has left. + defer cleanupWorkResourcesOnHubAndMembers() + defer ensureMemberClusterAndRelatedResourcesDeletion(memberClusterName) + // Deleting the policy also withdraws its cluster claim. + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should add a cluster claim, binding no cluster", func() { + Eventually(placementPolicyClaimActual(policyKey, neverJoinedTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + Expect(noPlacementPolicyBindingsActual(policyKey)()).To(Succeed(), "A cluster was bound") + + claims, err := claimsOfPolicy(policyKey.Name, policyKey.Namespace) + Expect(err).NotTo(HaveOccurred(), "Failed to list the cluster claims of cluster placement policy %s", policyKey) + Expect(claims.Items).To(HaveLen(1), "Cluster placement policy %s does not have exactly one cluster claim", policyKey) + claimUID = claims.Items[0].UID + }) + + It("can register a member cluster that carries the label, but never let it join", func() { + // No member agent runs for the cluster, so it never joins the fleet, and it is never marked as healthy. + createMemberCluster(memberClusterName, hubClusterSAName, neverJoinedLabels, nil) + }) + + It("should observe the registered member cluster when reconciling the cluster claim", func() { + // Every policy reconciles on a member cluster creation, and records the newest creation time it has seen on + // its outstanding claims; this shows that the policy has evaluated its selector with the cluster in place. + mc := &clusterv1beta1.MemberCluster{} + Expect(hubClient.Get(ctx, types.NamespacedName{Name: memberClusterName}, mc)).To(Succeed(), "Failed to get member cluster %s", memberClusterName) + Eventually(placementPolicyClaimObservedClusterCreationActual(policyKey, mc.CreationTimestamp), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to observe the registered member cluster on the cluster claim") + }) + + It("should keep the cluster claim outstanding, binding no cluster, as the registered member cluster is not eligible", func() { + // A claim is withdrawn only once eligible clusters fulfill its selector, and a cluster that has never joined + // is not eligible. + claimKeptActual := placementPolicyClaimKeptActual(policyKey, claimUID) + claimActual := placementPolicyClaimActual(policyKey, neverJoinedTerms) + bindingsActual := noPlacementPolicyBindingsActual(policyKey) + statusActual := placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)) + Consistently(func() error { + for _, actual := range []func() error{claimKeptActual, claimActual, bindingsActual, statusActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster claim was withdrawn or changed, or a cluster was bound") + }) + }) + + Context("cluster placement policy whose bound cluster loses its member agent for a while", Ordered, Serial, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyAgentStoppedNameTemplate, GinkgoParallelProcess()), + } + prodTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{envLabelName: envProd}}, + } + + // Which prod cluster ranks first depends on live capacity, so the specs learn it rather than assume it. + var stoppedClusterName, otherClusterName string + var stoppedBindingUID types.UID + var agentStopped bool + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects one prod cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(prodTerms, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // The member agent must run again before the policy goes, so that it removes the ConfigMap it placed; the + // specs that run next need the whole fleet as well. + if agentStopped { + startMemberAgent(memberClusterNamed(stoppedClusterName)) + } + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind one prod cluster", func() { + binding := waitForSinglePlacementPolicyBinding(policyKey) + stoppedClusterName, stoppedBindingUID = binding.GetSpec().ClusterName, binding.GetUID() + otherClusterName = otherProdClusterName(stoppedClusterName) + + waitForPlacementPolicyPlacedOnlyOn(policyKey, prodTerms, stoppedClusterName) + }) + + It("can stop the member agent on the bound cluster", func() { + agentStopped = true + stopMemberAgent(memberClusterNamed(stoppedClusterName)) + }) + + It("should keep the cluster bound, and bind the other prod cluster in its place, once the cluster's heartbeat goes stale", func() { + // The cluster keeps its binding and its resources, so it is still a target of the resource status, and its + // binding still reports what the member agent last reported; it no longer counts toward the selector, + // though, so the other prod cluster fills the gap. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, sameSelectorsForClusters(prodTerms, stoppedClusterName, otherClusterName), 1), + memberAgentStoppedEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind the other prod cluster") + Expect(placementPolicyBindingsKeptActual(policyKey, map[string]types.UID{stoppedClusterName: stoppedBindingUID})()). + To(Succeed(), "Failed to keep the binding of the cluster whose member agent stopped") + Eventually(appConfigMapPlacedOnClustersActual(memberClusterNamed(stoppedClusterName), memberClusterNamed(otherClusterName)), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to place the ConfigMap on both prod clusters") + + wantStatus := allSynchronizedPlacementPolicyStatus(2) + wantStatus.DesiredClusters = ptr.To(int32(1)) + wantStatus.ScheduledClusters = ptr.To(int32(1)) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + + It("can restart the member agent", func() { + startMemberAgent(memberClusterNamed(stoppedClusterName)) + agentStopped = false + }) + + It("should unbind the other prod cluster, and keep the originally bound one", func() { + // Back in the running, the originally bound cluster has the older binding, so the count trims the newer one. + waitForPlacementPolicyPlacedOnlyOn(policyKey, prodTerms, stoppedClusterName) + Expect(placementPolicyBindingsKeptActual(policyKey, map[string]types.UID{stoppedClusterName: stoppedBindingUID})()). + To(Succeed(), "Failed to keep the binding of the originally bound cluster") + }) + }) + + Context("cluster placement policy whose selector a member cluster fulfills as it joins the fleet", Ordered, Serial, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyNewClusterNameTemplate, GinkgoParallelProcess()), + } + newClusterLabels := map[string]string{newClusterLabelName: "true"} + newClusterTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: newClusterLabels}, + } + + var claimUID types.UID + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + }) + + AfterAll(func() { + // The work resources must go, and cluster-3 must be back in the fleet with its usual labels, even if the + // specs failed midway; otherwise they leak into later specs. The policy goes first, so that it removes the + // ConfigMap it placed on cluster-3 while cluster-3 still carries the label. + defer cleanupWorkResourcesOnHubAndMembers() + defer restoreMemberClusterInFleet(memberCluster3WestProd, newClusterLabelName) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should have no placement binding targeting cluster-3", func() { + // While a known issue stands, the placement policy controller releases the binding of a cluster as soon as it + // starts leaving, before it has left; with no binding on cluster-3, the leave below cannot run into it. + Eventually(noPlacementBindingTargetsClusterActual(memberCluster3WestProdName), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "A placement binding still targets cluster-3") + }) + + It("can let cluster-3 leave the fleet", func() { + setMemberClusterToLeave(memberCluster3WestProd) + checkIfMemberClusterHasLeft(memberCluster3WestProd) + }) + + It("can create a cluster placement policy that selects one cluster carrying a label that no member cluster has", func() { + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(newClusterTerms, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + It("should add a cluster claim that stays unfulfilled, binding no cluster", func() { + Eventually(placementPolicyClaimActual(policyKey, newClusterTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + + claims, err := claimsOfPolicy(policyKey.Name, policyKey.Namespace) + Expect(err).NotTo(HaveOccurred(), "Failed to list the cluster claims of cluster placement policy %s", policyKey) + Expect(claims.Items).To(HaveLen(1), "Cluster placement policy %s does not have exactly one cluster claim", policyKey) + claimUID = claims.Items[0].UID + + // Nothing in the environment provisions clusters for claims. + claimKeptActual := placementPolicyClaimKeptActual(policyKey, claimUID) + claimUnfulfilledActual := placementPolicyClaimUnfulfilledActual(policyKey) + bindingsActual := noPlacementPolicyBindingsActual(policyKey) + Consistently(func() error { + for _, actual := range []func() error{claimKeptActual, claimUnfulfilledActual, bindingsActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The cluster claim was withdrawn or fulfilled, or a cluster was bound") + }) + + It("can let cluster-3 join the fleet again, carrying the label", func() { + rejoinMemberClusterWithExtraLabels(memberCluster3WestProd, newClusterLabels) + }) + + It("should bind cluster-3, place the ConfigMap on it, and withdraw the cluster claim", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, newClusterTerms, memberCluster3WestProdName) + Eventually(noPlacementPolicyClaimsActual(policyKey), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to withdraw the cluster claim") + }) + }) +}) diff --git a/test/e2e/placement_policy_lifecycle_test.go b/test/e2e/placement_policy_lifecycle_test.go new file mode 100644 index 000000000..490e20850 --- /dev/null +++ b/test/e2e/placement_policy_lifecycle_test.go @@ -0,0 +1,583 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + placementPolicyCleanedUpNameTemplate = "pp-cleaned-up-%d" + placementPolicyRecreatedNameTemplate = "pp-recreated-%d" + placementPolicyRecreatedNarrowerNameTemplate = "pp-recreated-narrower-%d" + clusterPlacementPolicyQuickRecreateNameTemplate = "cpp-quick-recreate-%d" + placementPolicyNamespaceDeletedNameTemplate = "pp-namespace-deleted-%d" +) + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy deletion", Label("placementpolicy"), func() { + Context("placement policy deleted with the default sync strategy", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyCleanedUpNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + + // The Works of the policy, keyed by the member cluster they are for; an AppliedWork carries the name of its Work. + var workNamesByCluster map[string][]string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy with the default sync strategy") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the ConfigMap on every member cluster, managed by KubeFleet", func() { + waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + for _, cluster := range allMemberClusters { + Eventually(configMapTakenOverOnClusterActual(cluster, cmKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on cluster %s", cluster.ClusterName) + } + + var err error + workNamesByCluster, err = placementPolicyWorkNamesByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of placement policy %s", policyKey) + for _, cluster := range allMemberClusters { + Expect(workNamesByCluster[cluster.ClusterName]).NotTo(BeEmpty(), "Placement policy %s has no Work for cluster %s", policyKey, cluster.ClusterName) + for _, name := range workNamesByCluster[cluster.ClusterName] { + Expect(cluster.KubeClient.Get(ctx, types.NamespacedName{Name: name}, &kfplacementv1alpha1.AppliedWork{})).To(Succeed(), + "Failed to find AppliedWork %s on cluster %s", name, cluster.ClusterName) + } + } + }) + + It("can delete the placement policy", func() { + deletePlacementPolicyAndWait(policyKey) + }) + + It("should remove the bindings, Works, and resource snapshots of the policy from the hub", func() { + Eventually(placementPolicyLeftoversRemovedActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the bindings and resource snapshots of the policy") + Eventually(noPlacementPolicyWorksActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the Works of the policy") + }) + + It("should remove the AppliedWorks and the ConfigMap from every member cluster", func() { + Eventually(appliedWorksAbsentOnClustersActual(workNamesByCluster), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the AppliedWorks from the member clusters") + Eventually(appConfigMapRemovedFromClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from the member clusters") + }) + + It("should leave behind the namespace that the work applier created on every member cluster", func() { + // The applier creates the namespace of a namespaced resource it applies, but does not manage it, so it + // outlives the policy. + Consistently(namespaceLeftUnmanagedOnClustersActual(nsName, allMemberClusters...), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The namespace that the work applier created was removed, or is managed by KubeFleet") + }) + }) + + Context("placement policy deleted while orphaning its dependents, then re-created under the same name", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyRecreatedNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + // The policy selects every member cluster. + newPolicy := func() kfplacementv1alpha1.PlacementPolicyAccessor { + return newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + } + + var bindingUIDsByCluster map[string]types.UID + var workUIDsByWork map[string]types.UID + var workNamesByCluster map[string][]string + var cmResourceVersionsByCluster map[string]string + // The resource snapshot of the original policy, and the one the re-created policy takes in its place. + var oldSnapshotName, newSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy") + Expect(hubClient.Create(ctx, newPolicy())).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + // Deleting the hub namespace also removes the bindings and the resource snapshot, should the spec fail with + // them still orphaned. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the ConfigMap on every member cluster", func() { + waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + + bindingUIDsByCluster = placementPolicyBindingUIDsByCluster(policyKey) + oldSnapshotName = latestPrimarySnapshotNameOfPlacementPolicy(policyKey) + var err error + workUIDsByWork, err = placementPolicyWorkUIDs(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of placement policy %s", policyKey) + Expect(workUIDsByWork).NotTo(BeEmpty(), "Placement policy %s has no Works", policyKey) + workNamesByCluster, err = placementPolicyWorkNamesByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of placement policy %s", policyKey) + cmResourceVersionsByCluster, err = configMapResourceVersionsByCluster(cmKey, allMemberClusters...) + Expect(err).NotTo(HaveOccurred(), "Failed to read the ConfigMap on the member clusters") + }) + + It("can delete the placement policy, orphaning its dependents", func() { + deletePlacementPolicyAndWait(policyKey, client.PropagationPolicy(metav1.DeletePropagationOrphan)) + }) + + It("should leave the bindings and the resource snapshot behind, orphaned, with the Works and the ConfigMaps as they were", func() { + // The garbage collector drops the dependents' owner references before it lets the policy go. Nothing collects + // the orphaned resource snapshot until a policy of the same name comes back. + orphanedActual := placementPolicyBindingOwnershipActual(policyKey, bindingUIDsByCluster, false) + Eventually(orphanedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to orphan the bindings") + snapshotsActual := placementPolicyPrimarySnapshotNamesActual(policyKey, oldSnapshotName) + snapshotsOrphanedActual := placementPolicySnapshotsOrphanedActual(policyKey) + Eventually(snapshotsOrphanedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to orphan the resource snapshot") + + worksActual := placementPolicyWorksKeptActual(policyKey, workUIDsByWork) + cmActual := configMapUnchangedOnClustersActual(cmKey, cmResourceVersionsByCluster) + Consistently(func() error { + for _, actual := range []func() error{orphanedActual, snapshotsActual, snapshotsOrphanedActual, worksActual, cmActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The orphaned bindings, the resource snapshot, the Works, or the ConfigMaps changed") + }) + + It("can re-create the placement policy under the same name", func() { + Expect(hubClient.Create(ctx, newPolicy())).To(Succeed(), "Failed to re-create placement policy %s", policyKey) + }) + + It("should collect the orphaned resource snapshot, and take a new one under a different name", func() { + // The re-created policy never reuses a snapshot it does not own: it has the orphaned one collected, even though + // the orphaned bindings still name it, and then starts over at index 0. Snapshot names hash the policy UID. + Eventually(func() error { + if err := placementPolicyLatestSnapshotChangedActual(policyKey, oldSnapshotName, &newSnapshotName)(); err != nil { + return err + } + return placementPolicyPrimarySnapshotNamesActual(policyKey, newSnapshotName)() + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to replace the orphaned resource snapshot") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of placement policy %s", policyKey) + Expect(snapshot.GetLabels()[kfplacementv1alpha1.PlacementResourceSnapshotIndexLabelKey]).To(Equal("0"), + "The re-created placement policy did not start its resource snapshots over at index 0") + }) + + It("should adopt the orphaned bindings, moving them to the new resource snapshot, and keep the Works and the ConfigMaps as they were", func() { + adoptedActual := placementPolicyBindingOwnershipActual(policyKey, bindingUIDsByCluster, true) + Eventually(adoptedActual, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to adopt the orphaned bindings") + // The bindings point to, and have processed, the policy's latest resource snapshot, i.e., the new one. + synchronizedActual := allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(synchronizedActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to synchronize the adopted bindings on the new resource snapshot") + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + Eventually(statusActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + + // The Works keep their names, which come from the policy's namespaced name, and re-linking them to a resource + // snapshot of the same contents changes only their metadata, so the member clusters have nothing to apply. The + // resource versions recorded before the deletion show that the ConfigMaps never changed. + worksActual := placementPolicyWorksKeptActual(policyKey, workUIDsByWork) + cmActual := configMapUnchangedOnClustersActual(cmKey, cmResourceVersionsByCluster) + Consistently(func() error { + for _, actual := range []func() error{adoptedActual, synchronizedActual, statusActual, worksActual, cmActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The adopted bindings, the policy, the Works, or the ConfigMaps changed") + }) + + It("can delete the re-created placement policy", func() { + deletePlacementPolicyAndWait(policyKey) + }) + + It("should remove the bindings, Works, and resource snapshots of the policy from the hub", func() { + // No resource snapshot of either policy is left. + Eventually(placementPolicyLeftoversRemovedActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the bindings and resource snapshots of the policy") + Eventually(noPlacementPolicyWorksActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the Works of the policy") + }) + + It("should remove the AppliedWorks and the ConfigMap from every member cluster", func() { + Eventually(appliedWorksAbsentOnClustersActual(workNamesByCluster), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the AppliedWorks from the member clusters") + Eventually(appConfigMapRemovedFromClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from the member clusters") + }) + }) + + Context("placement policy deleted while orphaning its dependents, then re-created for fewer clusters", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyRecreatedNarrowerNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + eastTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: regionEast}}, + } + eastClusterNames := []string{memberCluster1EastProdName, memberCluster2EastCanaryName} + + var bindingUIDsByCluster map[string]types.UID + var workUIDsByWork map[string]types.UID + var workNamesByCluster map[string][]string + var cmResourceVersionsByCluster map[string]string + // The resource version of cluster-3's binding once it has been orphaned. + var orphanResourceVersion string + var oldSnapshotName, newSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + // Deleting the hub namespace also removes cluster-3's binding, should the spec fail with it still orphaned. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the ConfigMap on every member cluster", func() { + waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + + bindingUIDsByCluster = placementPolicyBindingUIDsByCluster(policyKey) + oldSnapshotName = latestPrimarySnapshotNameOfPlacementPolicy(policyKey) + var err error + workUIDsByWork, err = placementPolicyWorkUIDs(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of placement policy %s", policyKey) + workNamesByCluster, err = placementPolicyWorkNamesByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of placement policy %s", policyKey) + Expect(workNamesByCluster[memberCluster3WestProdName]).NotTo(BeEmpty(), "Placement policy %s has no Work for cluster-3", policyKey) + cmResourceVersionsByCluster, err = configMapResourceVersionsByCluster(cmKey, allMemberClusters...) + Expect(err).NotTo(HaveOccurred(), "Failed to read the ConfigMap on the member clusters") + }) + + It("can delete the placement policy, orphaning its dependents", func() { + deletePlacementPolicyAndWait(policyKey, client.PropagationPolicy(metav1.DeletePropagationOrphan)) + Eventually(placementPolicyBindingOwnershipActual(policyKey, bindingUIDsByCluster, false), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to orphan the bindings") + + bindings, err := placementPolicyBindingsByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the bindings of placement policy %s", policyKey) + orphanResourceVersion = bindings[memberCluster3WestProdName].GetResourceVersion() + }) + + It("can re-create the placement policy under the same name, selecting the east clusters only", func() { + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(eastTerms, intstr.FromString(clusterSelectorCountAll)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to re-create placement policy %s", policyKey) + }) + + It("should replace the orphaned resource snapshot, and adopt the bindings of the east clusters only", func() { + Eventually(func() error { + if err := placementPolicyLatestSnapshotChangedActual(policyKey, oldSnapshotName, &newSnapshotName)(); err != nil { + return err + } + return placementPolicyPrimarySnapshotNamesActual(policyKey, newSnapshotName)() + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to replace the orphaned resource snapshot") + + Eventually(placementPolicyBindingControllersActual(policyKey, bindingUIDsByCluster, eastClusterNames...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to adopt the bindings of the east clusters alone") + // The adopted bindings carry the new cluster selectors and the new resource snapshot. + Eventually(allSynchronizedAdoptedPlacementPolicyBindingsActual(policyKey, sameSelectorsForClusters(eastTerms, eastClusterNames...), 1), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to synchronize the adopted bindings on the new resource snapshot") + // cluster-3's orphan does not count toward the policy status. + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(eastClusterNames))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should leave cluster-3's orphaned binding alone, with its Works and ConfigMap as they were", func() { + // The orphan still names the collected resource snapshot, but it is in sync, and the work generator reads + // snapshots only to (re)process a binding, so nothing happens to it. + orphanActual := orphanedPlacementBindingUnchangedActual(policyKey, memberCluster3WestProdName, bindingUIDsByCluster[memberCluster3WestProdName], orphanResourceVersion) + adoptedActual := placementPolicyBindingControllersActual(policyKey, bindingUIDsByCluster, eastClusterNames...) + worksActual := placementPolicyWorksKeptActual(policyKey, workUIDsByWork) + cmActual := configMapUnchangedOnClustersActual(cmKey, cmResourceVersionsByCluster) + Consistently(func() error { + for _, actual := range []func() error{orphanActual, adoptedActual, worksActual, cmActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "Cluster-3's orphaned binding, the Works, or the ConfigMaps changed") + }) + + It("can delete cluster-3's orphaned binding by hand", func() { + bindings, err := placementPolicyBindingsByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the bindings of placement policy %s", policyKey) + orphan := bindings[memberCluster3WestProdName] + Expect(orphan).NotTo(BeNil(), "Placement policy %s has no binding for cluster-3", policyKey) + uid := orphan.GetUID() + Expect(hubClient.Delete(ctx, orphan, client.Preconditions{UID: &uid})).To(Succeed(), "Failed to delete binding %s", orphan.GetName()) + }) + + It("should remove the orphan's Works and cluster-3's ConfigMap, without needing the collected resource snapshot", func() { + eastUIDs := map[string]types.UID{ + memberCluster1EastProdName: bindingUIDsByCluster[memberCluster1EastProdName], + memberCluster2EastCanaryName: bindingUIDsByCluster[memberCluster2EastCanaryName], + } + Eventually(placementPolicyBindingOwnershipActual(policyKey, eastUIDs, true), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove cluster-3's binding") + Eventually(placementPolicyWorkClustersActual(policyKey, eastClusterNames...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove cluster-3's Works") + Eventually(appliedWorksAbsentOnClustersActual(map[string][]string{memberCluster3WestProdName: workNamesByCluster[memberCluster3WestProdName]}), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the AppliedWorks from cluster-3") + Eventually(appConfigMapRemovedFromClustersActual(memberCluster3WestProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from cluster-3") + + // The policy never controlled the orphan, so it is left as it was. + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(eastClusterNames))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + cmActual := configMapUnchangedOnClustersActual(cmKey, map[string]string{ + memberCluster1EastProdName: cmResourceVersionsByCluster[memberCluster1EastProdName], + memberCluster2EastCanaryName: cmResourceVersionsByCluster[memberCluster2EastCanaryName], + }) + Consistently(func() error { + if err := statusActual(); err != nil { + return err + } + return cmActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The placement policy or its ConfigMaps on the east clusters changed") + }) + }) + + Context("cluster placement policy deleted and re-created under the same name right away", Ordered, func() { + policyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyQuickRecreateNameTemplate, GinkgoParallelProcess())} + newPolicy := func() kfplacementv1alpha1.PlacementPolicyAccessor { + return newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + } + + var oldBindingUIDsByCluster map[string]types.UID + var oldSnapshotName, newSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects every member cluster") + Expect(hubClient.Create(ctx, newPolicy())).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the ConfigMap on every member cluster", func() { + waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + oldBindingUIDsByCluster = placementPolicyBindingUIDsByCluster(policyKey) + oldSnapshotName = latestPrimarySnapshotNameOfPlacementPolicy(policyKey) + }) + + It("can delete the cluster placement policy and re-create it right away", func() { + // The deletion waits only for the policy itself to go; the garbage collector removes its bindings and + // resource snapshots afterwards. + deletePlacementPolicyAndWait(policyKey) + Expect(hubClient.Create(ctx, newPolicy())).To(Succeed(), "Failed to re-create cluster placement policy %s", policyKey) + + // Whether any dependent of the deleted policy is still around depends on timing; record it, as it tells + // whether the run went through the path that waits for them. + bindings, bindingsErr := placementPolicyBindingsByCluster(policyKey) + snapshots, snapshotsErr := placementPolicySnapshots(policyKey) + GinkgoWriter.Printf("At re-creation: %d bindings (err: %v) and %d resource snapshots (err: %v) named the policy\n", + len(bindings), bindingsErr, len(snapshots), snapshotsErr) + }) + + It("should converge on new bindings and a new resource snapshot, leaving nothing of the deleted policy behind", func() { + // The re-created policy never reuses the deleted policy's resource snapshots: it waits for the garbage + // collector, then starts over under a new name. + Eventually(func() error { + if err := placementPolicyLatestSnapshotChangedActual(policyKey, oldSnapshotName, &newSnapshotName)(); err != nil { + return err + } + return placementPolicyPrimarySnapshotNamesActual(policyKey, newSnapshotName)() + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take a new resource snapshot") + + // Only bindings that the re-created policy controls are left. + bindingsActual := allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(bindingsActual, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind every member cluster anew") + newBindingUIDsByCluster := placementPolicyBindingUIDsByCluster(policyKey) + for clusterName, uid := range newBindingUIDsByCluster { + Expect(uid).NotTo(Equal(oldBindingUIDsByCluster[clusterName]), "The binding for cluster %s was kept from the deleted policy", clusterName) + } + + // The Works share their names with those of the deleted policy; none of the old ones is left. + worksActual := placementPolicyWorksLinkedToSnapshotActual(policyKey, newSnapshotName) + Eventually(worksActual, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to replace the Works of the deleted policy") + Eventually(placementPolicyWorkClustersActual(policyKey, allMemberClusterNames...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to generate Works for every member cluster") + + // The ConfigMap may have been removed and created anew in between, so its UID is not pinned. + cmActual := appConfigMapPlacedOnClustersActual(allMemberClusters...) + Eventually(cmActual, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + Eventually(statusActual, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the expected cluster placement policy status") + + Consistently(func() error { + for _, actual := range []func() error{bindingsActual, worksActual, cmActual, statusActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The re-created cluster placement policy did not stay settled") + }) + }) + + Context("placement policy whose namespace is deleted on the hub", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyNamespaceDeletedNameTemplate, GinkgoParallelProcess())} + eastTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: regionEast}}, + } + // No member cluster is in this region, so the policy keeps a cluster claim outstanding. + nowhereTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: "nowhere"}}, + } + + var workNamesByCluster map[string][]string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects every east cluster, and one cluster in a region that has none") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(eastTerms, intstr.FromString(clusterSelectorCountAll)), + clusterSelectorWithTerms(nowhereTerms, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. The + // policy goes with its namespace; deleting it here matters only should the spec fail before that. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should place the ConfigMap on both east clusters, and add a cluster claim for the other selector", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, sameSelectorsForClusters(eastTerms, memberCluster1EastProdName, memberCluster2EastCanaryName), 1), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind both east clusters") + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on both east clusters") + Eventually(placementPolicyClaimActual(policyKey, nowhereTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + // Both east clusters count toward the first selector, and the second one wants a cluster it cannot find. + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(3, 2, 1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + + Expect(placementPolicyWorkClustersActual(policyKey, memberCluster1EastProdName, memberCluster2EastCanaryName)()). + To(Succeed(), "Placement policy %s does not have Works for exactly the east clusters", policyKey) + var err error + workNamesByCluster, err = placementPolicyWorkNamesByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of placement policy %s", policyKey) + }) + + It("can delete the namespace of the placement policy on the hub", func() { + ns := appNamespace() + Expect(hubClient.Delete(ctx, &ns)).To(Succeed(), "Failed to delete namespace %s on the hub cluster", ns.Name) + }) + + It("should remove the placement policy, withdrawing its cluster claim", func() { + Eventually(func() error { + _, err := retrievePlacementPolicy(policyKey) + if err == nil { + return fmt.Errorf("placement policy %s still exists", policyKey) + } + return client.IgnoreNotFound(err) + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove placement policy %s", policyKey) + Eventually(noPlacementPolicyClaimsActual(policyKey), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to withdraw the cluster claim") + }) + + It("should remove the bindings, Works, and resource snapshots of the policy, and the namespace, from the hub", func() { + Eventually(placementPolicyLeftoversRemovedActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the bindings and resource snapshots of the policy") + Eventually(noPlacementPolicyWorksActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the Works of the policy") + Eventually(workNamespaceRemovedFromClusterActual(hubCluster), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove namespace %s from the hub cluster", nsName) + }) + + It("should remove the AppliedWorks and the ConfigMap from both east clusters", func() { + Eventually(appliedWorksAbsentOnClustersActual(workNamesByCluster), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the AppliedWorks from the east clusters") + Eventually(appConfigMapRemovedFromClustersActual(memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from the east clusters") + }) + }) +}) diff --git a/test/e2e/placement_policy_resources_eligibility_test.go b/test/e2e/placement_policy_resources_eligibility_test.go new file mode 100644 index 000000000..1a3349b55 --- /dev/null +++ b/test/e2e/placement_policy_resources_eligibility_test.go @@ -0,0 +1,266 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + placementPolicyReservedNamespaceNameTemplate = "pp-reserved-ns-%d" + clusterPlacementPolicyIneligibleNameTemplate = "cpp-ineligible-%d" + placementPolicyDefaultNamespaceNameTemplate = "pp-default-ns-%d" + placementPolicyControllerOwnedNameTemplate = "pp-controller-owned-%d" + + // placementPolicyDefaultNamespaceConfigMapNameTemplate names the ConfigMap that a policy in the default namespace + // places; the namespace is shared by every process, hence the process number. + placementPolicyDefaultNamespaceConfigMapNameTemplate = "pp-e2e-default-ns-%d" +) + +var _ = Describe("placement policy resource eligibility", Label("placementpolicy"), func() { + Context("placement policy in a namespace reserved for the system", Ordered, func() { + // Namespaces prefixed with kube- or fleet- are reserved; kube-public exists in every cluster. + policyKey := types.NamespacedName{ + Namespace: "kube-public", + Name: fmt.Sprintf(placementPolicyReservedNamespaceNameTemplate, GinkgoParallelProcess()), + } + + BeforeAll(func() { + By("creating a placement policy in kube-public") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Name: "cluster-info"}, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + deletePlacementPolicyAndWait(policyKey) + }) + + It("should report the selected resources as not collected, as the namespace is ineligible", func() { + Eventually(placementPolicyStatusWithoutResourceRevisionActual(policyKey, resourcesNotCollectedPlacementPolicyStatus(int32(len(allMemberClusters)))), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the expected placement policy status") + Expect(placementPolicyConditionMessageActual(policyKey, kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, + "placement policy is in a namespace that is ineligible for placement")()).To(Succeed(), "The policy reports another cause") + }) + + It("should not bind any cluster", func() { + Consistently(noPlacementPolicyBindingsActual(policyKey), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "A cluster was bound") + }) + }) + + Context("cluster placement policy naming resources that cannot be placed", Ordered, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyIneligibleNameTemplate, GinkgoParallelProcess()), + } + + notCollectedActual := func(wantText string) func() error { + statusActual := placementPolicyStatusWithoutResourceRevisionActual(policyKey, resourcesNotCollectedPlacementPolicyStatus(int32(len(allMemberClusters)))) + messageActual := placementPolicyConditionMessageActual(policyKey, kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, wantText) + return func() error { + if err := statusActual(); err != nil { + return err + } + return messageActual() + } + } + + BeforeAll(func() { + By("creating the work resources") + // Kubernetes adds the kube-root-ca.crt ConfigMap to every namespace, the app namespace included. + createWorkResources() + + By("creating a cluster placement policy that names the kube-root-ca.crt ConfigMap") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Name: "kube-root-ca.crt", Namespace: appNamespace().Name}, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should report the kube-root-ca.crt ConfigMap as ineligible", func() { + // A resource named in a selector is the user's explicit pick, so an ineligible one fails the selection + // rather than being skipped. Kubernetes publishes the ConfigMap shortly after the namespace is created; + // until then, the policy reports it as missing instead. + Eventually(notCollectedActual("the built-in CA certificate config map is created automatically"), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as ineligible") + }) + + It("can switch to naming the default namespace", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ResourceSelectors = []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "Namespace", Name: "default"}, + } + }) + }) + + It("should report the default namespace as ineligible", func() { + // Resources in the default namespace may be placed, but not the namespace itself. + Eventually(notCollectedActual("the default namespace itself cannot be placed"), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the default namespace as ineligible") + }) + + It("should not bind any cluster", func() { + Consistently(noPlacementPolicyBindingsActual(policyKey), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "A cluster was bound") + }) + }) + + Context("placement policy in the default namespace", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: metav1.NamespaceDefault, + Name: fmt.Sprintf(placementPolicyDefaultNamespaceNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{ + Namespace: metav1.NamespaceDefault, + Name: fmt.Sprintf(placementPolicyDefaultNamespaceConfigMapNameTemplate, GinkgoParallelProcess()), + } + cmData := map[string]string{"data": "in-default"} + + BeforeAll(func() { + By("creating a ConfigMap in the default namespace") + cm := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: cmKey.Name, Namespace: cmKey.Namespace}, + Data: cmData, + } + Expect(hubClient.Create(ctx, cm)).To(Succeed(), "Failed to create ConfigMap %s", cmKey) + + By("creating a placement policy in the default namespace that selects the ConfigMap") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Name: cmKey.Name}, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The hub ConfigMap must go even if removing the policy fails; the default namespace outlives the specs. + defer func() { + cm := &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: cmKey.Name, Namespace: cmKey.Namespace}} + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, cm))).To(Succeed(), "Failed to delete ConfigMap %s", cmKey) + }() + deletePlacementPolicyAndWait(policyKey) + Eventually(configMapAbsentOnClustersActual(cmKey, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from the member clusters") + }) + + It("should place the ConfigMap on every member cluster", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(configMapManagedOnClustersActual(cmKey, cmData, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + }) + + It("should report the policy as synchronized on every member cluster", func() { + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("placement policy whose label selector also matches a controller-owned ConfigMap", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyControllerOwnedNameTemplate, GinkgoParallelProcess()), + } + eligibleKey := types.NamespacedName{Namespace: nsName, Name: "pp-e2e-eligible"} + ownedKey := types.NamespacedName{Namespace: nsName, Name: "pp-e2e-owned"} + // The owner carries no label, so the policy does not select it. + ownerKey := types.NamespacedName{Namespace: nsName, Name: "pp-e2e-owner"} + cmData := map[string]string{"data": "labeled"} + selectedLabels := map[string]string{placementPolicySelectedLabelKey: "true"} + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a labeled ConfigMap, and another one that a ConfigMap controls") + eligible := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: eligibleKey.Name, Namespace: eligibleKey.Namespace, Labels: selectedLabels}, + Data: cmData, + } + Expect(hubClient.Create(ctx, eligible)).To(Succeed(), "Failed to create ConfigMap %s", eligibleKey) + owner := &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ownerKey.Name, Namespace: ownerKey.Namespace}} + Expect(hubClient.Create(ctx, owner)).To(Succeed(), "Failed to create ConfigMap %s", ownerKey) + owned := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: ownedKey.Name, + Namespace: ownedKey.Namespace, + Labels: selectedLabels, + // The owner must exist (with this UID); otherwise the garbage collector removes the ConfigMap. + OwnerReferences: []metav1.OwnerReference{ + {APIVersion: "v1", Kind: "ConfigMap", Name: owner.Name, UID: owner.UID, Controller: ptr.To(true)}, + }, + }, + Data: cmData, + } + Expect(hubClient.Create(ctx, owned)).To(Succeed(), "Failed to create ConfigMap %s", ownedKey) + + By("creating a placement policy that selects ConfigMaps by label") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", LabelSelector: &metav1.LabelSelector{MatchLabels: selectedLabels}}, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should select the labeled ConfigMap only, skipping the controller-owned one", func() { + // A label selector may match resources that the user did not pick explicitly, so an ineligible one is + // skipped rather than failing the selection. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster with the eligible ConfigMap alone") + Eventually(configMapManagedOnClustersActual(eligibleKey, cmData, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the eligible ConfigMap on every member cluster") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should not place the controller-owned ConfigMap, or its owner", func() { + Consistently(configMapAbsentOnClustersActual(ownedKey, allMemberClusters...), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The controller-owned ConfigMap was placed") + Expect(configMapAbsentOnClustersActual(ownerKey, allMemberClusters...)()).To(Succeed(), "The unlabeled owner ConfigMap was placed") + }) + }) +}) diff --git a/test/e2e/placement_policy_resources_snapshots_test.go b/test/e2e/placement_policy_resources_snapshots_test.go new file mode 100644 index 000000000..4fbbb445e --- /dev/null +++ b/test/e2e/placement_policy_resources_snapshots_test.go @@ -0,0 +1,321 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "strconv" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils" +) + +const ( + clusterPlacementPolicyManyResourcesNameTemplate = "cpp-many-resources-%d" + clusterPlacementPolicyHistoryLimitNameTemplate = "cpp-history-limit-%d" + clusterPlacementPolicyHistoryLimitStrategyNameTemplate = "cpp-history-limit-strategy-%d" + clusterPlacementPolicyHistoryLimitRunNameTemplate = "cpp-history-limit-%d-run-%d" + + // placementPolicyBulkLabelKey is the label that the ConfigMaps selected in bulk carry. + placementPolicyBulkLabelKey = "placement-policy-e2e/bulk" + placementPolicyBulkConfigMapNameFormat = "pp-e2e-bulk-%02d" +) + +const ( + // placementPolicyBulkConfigMapCount is the number of ConfigMaps selected in bulk: more than a resource snapshot + // holds. + placementPolicyBulkConfigMapCount = 60 + // placementPolicyMaxResourcesPerSnapshot is the number of resources that the hub agent puts in a resource snapshot + // at most (placementresourcesnapshot.DefaultMaxPerSnapshotResourceCnt). + placementPolicyMaxResourcesPerSnapshot = 50 + // placementPolicyBulkConfigMapsInSecondarySnapshot is the number of bulk ConfigMaps that overflow into the + // secondary resource snapshot. + placementPolicyBulkConfigMapsInSecondarySnapshot = placementPolicyBulkConfigMapCount - placementPolicyMaxResourcesPerSnapshot + + // placementPolicyHistoryLimitRunCount is the number of runs that roll out to a policy that keeps one revision. + placementPolicyHistoryLimitRunCount = 3 +) + +// placementResourceSnapshotView is the part of a resource snapshot that the specs compare in one shot. +type placementResourceSnapshotView struct { + Index string + SubIndexedCount string + ControlledByPolicy bool + SelectedResourceCnt int +} + +// linkedWorksView is the part of a binding's Works, all in the namespace of one member cluster, that the specs +// compare in one shot: the primary Work, which carries the linked Work count, and the secondary Works it controls. +type linkedWorksView struct { + PrimaryLinkedWorkCount string + PrimaryManifestCnt int + SecondaryManifestCnts []int + UnlinkedSecondaryWorkCnt int +} + +var _ = Describe("placement policy resource snapshots", Label("placementpolicy"), func() { + Context("cluster placement policy selecting more resources than fit in one snapshot", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyManyResourcesNameTemplate, GinkgoParallelProcess()), + } + bulkLabels := map[string]string{placementPolicyBulkLabelKey: "true"} + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By(fmt.Sprintf("creating %d labeled ConfigMaps", placementPolicyBulkConfigMapCount)) + for i := range placementPolicyBulkConfigMapCount { + cm := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: fmt.Sprintf(placementPolicyBulkConfigMapNameFormat, i), + Namespace: nsName, + Labels: bulkLabels, + }, + Data: map[string]string{"index": strconv.Itoa(i)}, + } + Expect(hubClient.Create(ctx, cm)).To(Succeed(), "Failed to create ConfigMap %s", client.ObjectKeyFromObject(cm)) + } + + By("creating a cluster placement policy that selects the labeled ConfigMaps") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Namespace: nsName, LabelSelector: &metav1.LabelSelector{MatchLabels: bulkLabels}}, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources, the labeled ConfigMaps among them, must go even if removing the policy fails; + // otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should split the resources across a primary and a secondary resource snapshot", func() { + // The hub agent puts at most 50 resources in a resource snapshot; the ConfigMaps are small enough that the + // count, not the size, splits them. + want := map[string]placementResourceSnapshotView{ + "0": {Index: "0", SubIndexedCount: "2", ControlledByPolicy: true, SelectedResourceCnt: placementPolicyMaxResourcesPerSnapshot}, + "1": {Index: "0", ControlledByPolicy: true, SelectedResourceCnt: placementPolicyBulkConfigMapsInSecondarySnapshot}, + } + Eventually(func() error { + policy, err := retrievePlacementPolicy(policyKey) + if err != nil { + return err + } + snapshots, err := placementPolicySnapshots(policyKey) + if err != nil { + return err + } + got := make(map[string]placementResourceSnapshotView, len(snapshots)) + for _, snapshot := range snapshots { + labels := snapshot.GetLabels() + got[labels[kfplacementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey]] = placementResourceSnapshotView{ + Index: labels[kfplacementv1alpha1.PlacementResourceSnapshotIndexLabelKey], + SubIndexedCount: labels[kfplacementv1alpha1.SubIndexedPlacementResourceSnapshotCountLabelKey], + ControlledByPolicy: metav1.IsControlledBy(snapshot, policy), + SelectedResourceCnt: len(snapshot.GetSpec().Resources), + } + } + if diff := cmp.Diff(got, want); diff != "" { + return fmt.Errorf("resource snapshots of cluster placement policy %s diff, keyed by sub-index (-got, +want):\n%s", policyKey, diff) + } + return nil + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to split the resources across two resource snapshots") + }) + + It("should generate a primary Work and a linked secondary Work for every member cluster", func() { + want := make(map[string]linkedWorksView, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + want[fmt.Sprintf(utils.NamespaceNameFormat, clusterName)] = linkedWorksView{ + PrimaryLinkedWorkCount: "2", + PrimaryManifestCnt: placementPolicyMaxResourcesPerSnapshot, + SecondaryManifestCnts: []int{placementPolicyBulkConfigMapsInSecondarySnapshot}, + } + } + Eventually(func() error { + works, err := placementPolicyWorks(policyKey) + if err != nil { + return err + } + // The primary Work is the one that carries the linked Work count; it controls the others. + primaryNames := make(map[string]string, len(want)) + got := make(map[string]linkedWorksView, len(want)) + for i := range works { + work := &works[i] + if count, found := work.GetAnnotations()[kfplacementv1alpha1.LinkedWorkCountAnnotationKey]; found { + primaryNames[work.Namespace] = work.Name + view := got[work.Namespace] + view.PrimaryLinkedWorkCount, view.PrimaryManifestCnt = count, len(work.Spec.Manifests) + got[work.Namespace] = view + } + } + for i := range works { + work := &works[i] + if primaryNames[work.Namespace] == work.Name { + continue + } + view := got[work.Namespace] + if controller := metav1.GetControllerOf(work); controller != nil && controller.Kind == "Work" && controller.Name == primaryNames[work.Namespace] { + view.SecondaryManifestCnts = append(view.SecondaryManifestCnts, len(work.Spec.Manifests)) + } else { + view.UnlinkedSecondaryWorkCnt++ + } + got[work.Namespace] = view + } + if diff := cmp.Diff(got, want); diff != "" { + return fmt.Errorf("works of cluster placement policy %s diff, keyed by namespace (-got, +want):\n%s", policyKey, diff) + } + return nil + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate the linked Works") + }) + + It("should place every ConfigMap on every member cluster", func() { + Eventually(func() error { + for _, cluster := range allMemberClusters { + cms := &corev1.ConfigMapList{} + if err := cluster.KubeClient.List(ctx, cms, client.InNamespace(nsName), client.MatchingLabels(bulkLabels)); err != nil { + return fmt.Errorf("cluster %s: failed to list ConfigMaps: %w", cluster.ClusterName, err) + } + if len(cms.Items) != placementPolicyBulkConfigMapCount { + return fmt.Errorf("cluster %s has %d labeled ConfigMaps, want %d", cluster.ClusterName, len(cms.Items), placementPolicyBulkConfigMapCount) + } + } + return nil + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to place every ConfigMap on every member cluster") + }) + + It("should report every resource as synchronized and available on every binding", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), placementPolicyBulkConfigMapCount), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the resources as synchronized and available") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) + + Context("cluster placement policy keeping one resource revision, rolled out by a strategy that leaves out a cluster", Ordered, func() { + policyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyHistoryLimitNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyHistoryLimitStrategyNameTemplate, GinkgoParallelProcess())} + runKeys := make([]types.NamespacedName, placementPolicyHistoryLimitRunCount) + for i := range runKeys { + runKeys[i] = types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyHistoryLimitRunNameTemplate, GinkgoParallelProcess(), i+1)} + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + + var initialSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that keeps a single resource revision") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + ResourceRevisionHistoryLimit: ptr.To(int32(1)), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + + By("creating a strategy whose stages cover the east region only") + // As in the strategy that leaves out a cluster: the east stage gets cluster-1 alone, and cluster-3, in the + // west, is in no stage at all. + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyCanaryStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envCanary}}, + }, + { + Name: placementPolicyEastStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{regionLabelName: regionEast}}, + }, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create cluster staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + for _, runKey := range runKeys { + ensureStagedUpdateRunRemoved(runKey) + } + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + for i, runKey := range runKeys { + data := map[string]string{"data": fmt.Sprintf("test-v%d", i+1)} + var newSnapshotName string + + It(fmt.Sprintf("can roll out change %d with a cluster staged update run", i+1), func() { + updateConfigMapDataOnHub(cmKey, data) + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create cluster staged update run %s", runKey) + }) + + It(fmt.Sprintf("should complete run %d, leaving the cluster outside every stage on the initial revision", i+1), func() { + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete cluster staged update run %s", runKey) + newSnapshotName = resourceSnapshotNameToRolloutOf(runKey) + Eventually(placementPolicyBindingSnapshotsActual(policyKey, map[string]string{ + memberCluster2EastCanaryName: newSnapshotName, + memberCluster1EastProdName: newSnapshotName, + memberCluster3WestProdName: initialSnapshotName, + }), workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to leave cluster-3 on the initial resource snapshot") + Eventually(configMapDataOnClustersActual(cmKey, data, memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place change %d on the east clusters", i+1) + }) + + It(fmt.Sprintf("should keep only the latest resource snapshot and the one still in use, after run %d", i+1), func() { + // With a history limit of 1, only the latest snapshot is kept, except that a snapshot a binding still + // points to is never collected. The snapshots of earlier runs are no longer in use. + Eventually(placementPolicyPrimarySnapshotNamesActual(policyKey, initialSnapshotName, newSnapshotName), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to garbage-collect the unused resource snapshots") + }) + } + + It("should leave cluster-3 on the initial version, reported as on an older resource revision", func() { + Expect(configMapDataOnClustersActual(cmKey, v0Data, memberCluster3WestProd)()).To(Succeed(), "A change reached cluster-3") + wantStatus := someClustersOnOlderResourceRevisionPlacementPolicyStatus(int32(len(allMemberClusters)), 2) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) +}) diff --git a/test/e2e/placement_policy_resources_test.go b/test/e2e/placement_policy_resources_test.go new file mode 100644 index 000000000..3a0e6e6d7 --- /dev/null +++ b/test/e2e/placement_policy_resources_test.go @@ -0,0 +1,643 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + placementPolicyLabelSelectedNameTemplate = "pp-label-selected-%d" + placementPolicyPendingScaleUpNameTemplate = "pp-pending-scale-up-%d" + clusterPlacementPolicyNamespaceNameTemplate = "cpp-namespace-%d" + placementPolicyOtherNamespaceNameTemplate = "pp-other-namespace-%d" + placementPolicyMissingResourceNameTemplate = "pp-missing-resource-%d" + + // placementPolicyClusterRoleNameTemplate names the ClusterRole that the specs place; it differs from the names + // that the CRP specs use, so the two never collide. + placementPolicyClusterRoleNameTemplate = "pp-e2e-reader-%d" + // placementPolicyMissingConfigMapName names a ConfigMap that a policy selects before it exists. + placementPolicyMissingConfigMapName = "pp-e2e-missing-config" + + // placementPolicySelectedLabelKey is the label that the ConfigMaps selected by label carry. + placementPolicySelectedLabelKey = "placement-policy-e2e/selected" + + placementPolicyResourceDeletedNameTemplate = "pp-resource-deleted-%d" +) + +// newPlacementPolicySelectedConfigMap returns a ConfigMap that carries the label the specs select ConfigMaps by. +func newPlacementPolicySelectedConfigMap(key types.NamespacedName, data map[string]string) *corev1.ConfigMap { + return &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{ + Name: key.Name, + Namespace: key.Namespace, + Labels: map[string]string{placementPolicySelectedLabelKey: "true"}, + }, + Data: maps.Clone(data), + } +} + +// placementPolicyLabelSelectedResourceSelectors returns the resource selectors that select ConfigMaps by the label +// that newPlacementPolicySelectedConfigMap sets. +func placementPolicyLabelSelectedResourceSelectors() []kfplacementv1alpha1.ResourceSelector { + return []kfplacementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + LabelSelector: &metav1.LabelSelector{ + MatchLabels: map[string]string{placementPolicySelectedLabelKey: "true"}, + }, + }, + } +} + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy resource selection", Label("placementpolicy"), func() { + Context("placement policy selecting ConfigMaps by label, with resource changes pending a rollout", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyLabelSelectedNameTemplate, GinkgoParallelProcess()), + } + cmAKey := types.NamespacedName{Namespace: nsName, Name: "selected-config-a"} + cmBKey := types.NamespacedName{Namespace: nsName, Name: "selected-config-b"} + cmCKey := types.NamespacedName{Namespace: nsName, Name: "selected-config-c"} + // The app ConfigMap that createWorkResources creates carries no label, so the policy must leave it out. + unselectedCMKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + v0Data := map[string]string{"data": "v0"} + v1Data := map[string]string{"data": "v1"} + + var initialSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating two ConfigMaps that carry the selected label") + for _, key := range []types.NamespacedName{cmAKey, cmBKey} { + Expect(hubClient.Create(ctx, newPlacementPolicySelectedConfigMap(key, v0Data))).To(Succeed(), "Failed to create ConfigMap %s", key) + } + + By("creating a placement policy that selects ConfigMaps by label") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyLabelSelectedResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the labeled ConfigMaps, and only those, on every member cluster", func() { + wantSelectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantSelectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster, 2), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + for _, key := range []types.NamespacedName{cmAKey, cmBKey} { + Eventually(configMapDataOnClustersActual(key, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place ConfigMap %s on every member cluster", key) + } + Expect(configMapAbsentOnClustersActual(unselectedCMKey, allMemberClusters...)()).To(Succeed(), "An unlabeled ConfigMap was placed") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of placement policy %s", policyKey) + initialSnapshotName = snapshot.GetName() + }) + + It("should report the policy as synchronized on every member cluster", func() { + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("can change a selected ConfigMap and add another one on the hub", func() { + updateConfigMapDataOnHub(cmAKey, v1Data) + Expect(hubClient.Create(ctx, newPlacementPolicySelectedConfigMap(cmCKey, v1Data))).To(Succeed(), "Failed to create ConfigMap %s", cmCKey) + }) + + It("should report the resource changes as pending a rollout, without taking a new resource snapshot", func() { + // The clusters still hold, synchronized and available, the revision that remains the latest one; only the + // Synchronized condition tells that the hub has moved on since. + wantStatus := resourcesChangedPendingRolloutPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the resource changes as pending a rollout") + }) + + It("should keep the member clusters on the initial resource revision", func() { + wantBindings := make(map[string]wantPlacementBinding, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: initialSnapshotName, + status: synchronizedPlacementBindingStatus(2), + } + } + bindingsActual := placementPolicyBindingsActual(policyKey, wantBindings) + changedCMActual := configMapDataOnClustersActual(cmAKey, v0Data, allMemberClusters...) + addedCMActual := configMapAbsentOnClustersActual(cmCKey, allMemberClusters...) + Consistently(func() error { + policy, err := retrievePlacementPolicy(policyKey) + if err != nil { + return err + } + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + if err != nil { + return err + } + if snapshot.GetName() != initialSnapshotName { + return fmt.Errorf("a new resource snapshot %s was taken without a rollout", snapshot.GetName()) + } + if err := bindingsActual(); err != nil { + return err + } + if err := changedCMActual(); err != nil { + return err + } + return addedCMActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The resource changes reached the member clusters without a rollout") + }) + }) + + Context("placement policy scaled up while resource changes are pending a rollout", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyPendingScaleUpNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + eastLabels := map[string]string{regionLabelName: regionEast} + eastSelectors := []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{clusterSelectorTermsMatchingLabels(eastLabels)} + + // Which east cluster ranks first depends on live capacity, so the specs learn it rather than assume it. + var firstClusterName, secondClusterName, initialSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects one east cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(eastLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on one east cluster", func() { + firstClusterName = waitForSinglePlacementPolicyBinding(policyKey).GetSpec().ClusterName + secondClusterName = otherEastClusterName(firstClusterName) + + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + firstClusterName: eastSelectors, + }, 1), workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to synchronize the binding") + Eventually(configMapDataOnClustersActual(cmKey, v0Data, memberClusterNamed(firstClusterName)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on the bound cluster") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of placement policy %s", policyKey) + initialSnapshotName = snapshot.GetName() + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("should report the change as pending a rollout", func() { + wantStatus := resourcesChangedPendingRolloutPlacementPolicyStatus(1) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the change as pending a rollout") + }) + + It("can scale the selector up to 2 clusters", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ClusterSelectors[0].Count = ptr.To(intstr.FromInt32(2)) + }) + }) + + It("should bind the other east cluster to the initial resource snapshot, not to the changed ConfigMap", func() { + // A new binding gets the latest existing resource snapshot; only a rollout takes a new one. + wantBindings := make(map[string]wantPlacementBinding, 2) + for _, clusterName := range []string{firstClusterName, secondClusterName} { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: eastSelectors, + snapshotName: initialSnapshotName, + status: synchronizedPlacementBindingStatus(1), + } + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind the other east cluster to the initial resource snapshot") + Eventually(configMapDataOnClustersActual(cmKey, v0Data, memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on both east clusters") + + wantStatus := resourcesChangedPendingRolloutPlacementPolicyStatus(2) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the change as still pending a rollout") + }) + }) + + Context("cluster placement policy selecting a namespace, re-created to select a ConfigMap in it too", Ordered, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyNamespaceNameTemplate, GinkgoParallelProcess()), + } + nsName := appNamespace().Name + clusterRoleName := fmt.Sprintf(placementPolicyClusterRoleNameTemplate, GinkgoParallelProcess()) + + namespaceAndClusterRoleSelectors := func() []kfplacementv1alpha1.ResourceSelector { + return []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "Namespace", Name: nsName}, + {APIGroup: rbacv1.GroupName, APIVersion: "v1", Kind: "ClusterRole", Name: clusterRoleName}, + } + } + allClustersSelectors := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + wantSelectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantSelectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + return wantSelectorsByCluster + } + + BeforeAll(func() { + By("creating the work resources and a ClusterRole") + createWorkResources() + clusterRole := &rbacv1.ClusterRole{ + ObjectMeta: metav1.ObjectMeta{Name: clusterRoleName}, + Rules: []rbacv1.PolicyRule{ + {APIGroups: []string{""}, Resources: []string{"configmaps"}, Verbs: []string{"get", "list"}}, + }, + } + Expect(hubClient.Create(ctx, clusterRole)).To(Succeed(), "Failed to create ClusterRole %s", clusterRoleName) + + By("creating a cluster placement policy that selects the namespace and the ClusterRole by name") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: namespaceAndClusterRoleSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources and the ClusterRole must go even if removing the policy fails; otherwise they leak + // into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + defer func() { + clusterRole := &rbacv1.ClusterRole{ObjectMeta: metav1.ObjectMeta{Name: clusterRoleName}} + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, clusterRole))).To(Succeed(), "Failed to delete ClusterRole %s", clusterRoleName) + }() + ensurePlacementPolicyRemoved(policyKey) + Eventually(clusterRoleAbsentOnClustersActual(clusterRoleName, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ClusterRole from the member clusters") + }) + + It("should place the namespace and the ClusterRole on every member cluster", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, allClustersSelectors(), 2), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(namespacePlacedOnClustersActual(nsName, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the namespace on every member cluster") + Eventually(clusterRolePlacedOnClustersActual(clusterRoleName, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ClusterRole on every member cluster") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + + It("should not place the ConfigMap in the namespace", func() { + // Unlike a CRP, a cluster placement policy that selects a namespace places the namespace object alone. + Consistently(appConfigMapRemovedFromClustersActual(allMemberClusters...), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The ConfigMap in the selected namespace was placed") + }) + + It("can delete the cluster placement policy", func() { + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should remove the namespace and the ClusterRole from every member cluster", func() { + Eventually(namespaceAbsentOnClustersActual(nsName, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the namespace from the member clusters") + Eventually(clusterRoleAbsentOnClustersActual(clusterRoleName, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ClusterRole from the member clusters") + }) + + It("should leave no binding or resource snapshot of the deleted policy behind", func() { + Eventually(placementPolicyLeftoversRemovedActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the bindings and resource snapshots of the deleted policy") + }) + + It("can re-create the cluster placement policy, selecting the ConfigMap in the namespace too", func() { + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: append(namespaceAndClusterRoleSelectors(), kfplacementv1alpha1.ResourceSelector{ + APIVersion: "v1", + Kind: "ConfigMap", + Name: appConfigMap().Name, + Namespace: nsName, + }), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to re-create cluster placement policy %s", policyKey) + }) + + It("should place the namespace, the ClusterRole, and the ConfigMap on every member cluster", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, allClustersSelectors(), 3), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(namespacePlacedOnClustersActual(nsName, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the namespace on every member cluster") + Eventually(clusterRolePlacedOnClustersActual(clusterRoleName, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ClusterRole on every member cluster") + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) + + // The two contexts below hit the two kinds of selection errors that only the controller can catch: a selection the + // user got wrong, and a selected resource that does not exist. + Context("placement policy selecting a ConfigMap from another namespace", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyOtherNamespaceNameTemplate, GinkgoParallelProcess()), + } + otherNamespace := nsName + "-other" + + notCollectedActual := func() func() error { + statusActual := placementPolicyStatusWithoutResourceRevisionActual(policyKey, resourcesNotCollectedPlacementPolicyStatus(int32(len(allMemberClusters)))) + bindingsActual := noPlacementPolicyBindingsActual(policyKey) + return func() error { + if err := statusActual(); err != nil { + return err + } + return bindingsActual() + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy whose resource selector names another namespace") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Name: appConfigMap().Name, Namespace: otherNamespace}, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should report the resources as not collected, binding no cluster", func() { + Eventually(notCollectedActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the resources as not collected") + }) + + It("should not bind any cluster or place the ConfigMap", func() { + placedActual := appConfigMapRemovedFromClustersActual(allMemberClusters...) + Consistently(func() error { + if err := notCollectedActual()(); err != nil { + return err + } + return placedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The policy bound a cluster, or placed the ConfigMap") + }) + + It("can fix the resource selector to name the policy's own namespace", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ResourceSelectors[0].Namespace = nsName + }) + }) + + It("should place the ConfigMap on every member cluster", func() { + wantSelectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantSelectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster, 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("placement policy selecting a ConfigMap that does not exist yet", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyMissingResourceNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + missingCMKey := types.NamespacedName{Namespace: nsName, Name: placementPolicyMissingConfigMapName} + missingCMData := map[string]string{"data": "created-later"} + + notCollectedActual := func() func() error { + statusActual := placementPolicyStatusWithoutResourceRevisionActual(policyKey, resourcesNotCollectedPlacementPolicyStatus(int32(len(allMemberClusters)))) + bindingsActual := noPlacementPolicyBindingsActual(policyKey) + return func() error { + if err := statusActual(); err != nil { + return err + } + return bindingsActual() + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects an existing ConfigMap and a missing one by name") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Name: cmKey.Name}, + {APIVersion: "v1", Kind: "ConfigMap", Name: missingCMKey.Name}, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources, the once-missing ConfigMap among them, must go even if removing the policy fails; + // otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should report the resources as not collected, binding no cluster", func() { + Eventually(notCollectedActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the resources as not collected") + }) + + It("should not bind any cluster or place the existing ConfigMap", func() { + placedActual := appConfigMapRemovedFromClustersActual(allMemberClusters...) + Consistently(func() error { + if err := notCollectedActual()(); err != nil { + return err + } + return placedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The policy bound a cluster, or placed the existing ConfigMap") + }) + + It("can create the missing ConfigMap on the hub", func() { + cm := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: missingCMKey.Name, Namespace: missingCMKey.Namespace}, + Data: maps.Clone(missingCMData), + } + Expect(hubClient.Create(ctx, cm)).To(Succeed(), "Failed to create ConfigMap %s", missingCMKey) + }) + + It("should place both ConfigMaps on every member cluster", func() { + // The resource watcher re-queues the policy as soon as the selected ConfigMap appears. + wantSelectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantSelectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster, 2), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the existing ConfigMap on every member cluster") + Eventually(configMapDataOnClustersActual(missingCMKey, missingCMData, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the once-missing ConfigMap on every member cluster") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("placement policy auto-rolling out the deletion of a ConfigMap it selects by label", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyResourceDeletedNameTemplate, GinkgoParallelProcess()), + } + keptCMKey := types.NamespacedName{Namespace: nsName, Name: "selected-config-a"} + deletedCMKey := types.NamespacedName{Namespace: nsName, Name: "selected-config-b"} + v0Data := map[string]string{"data": "v0"} + + var initialSnapshotName, newSnapshotName string + var keptCMUIDsByCluster map[string]types.UID + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating two ConfigMaps that carry the selected label") + for _, key := range []types.NamespacedName{keptCMKey, deletedCMKey} { + Expect(hubClient.Create(ctx, newPlacementPolicySelectedConfigMap(key, v0Data))).To(Succeed(), "Failed to create ConfigMap %s", key) + } + + By("creating a placement policy that selects ConfigMaps by label, with auto-rollout enabled") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyLabelSelectedResourceSelectors(), + }) + policy.SetAnnotations(autoRolloutAnnotations()) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place both ConfigMaps on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 2) + for _, key := range []types.NamespacedName{keptCMKey, deletedCMKey} { + Eventually(configMapDataOnClustersActual(key, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place ConfigMap %s on every member cluster", key) + } + + var err error + keptCMUIDsByCluster, err = configMapUIDsByCluster(keptCMKey, allMemberClusters...) + Expect(err).NotTo(HaveOccurred(), "Failed to read ConfigMap %s on the member clusters", keptCMKey) + }) + + It("can delete one of the ConfigMaps on the hub", func() { + cm := &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: deletedCMKey.Name, Namespace: deletedCMKey.Namespace}} + Expect(hubClient.Delete(ctx, cm)).To(Succeed(), "Failed to delete ConfigMap %s on the hub cluster", deletedCMKey) + }) + + It("should roll the deletion out, removing the ConfigMap from every member cluster", func() { + // The deletion is a resource change like any other: auto-rollout takes a resource snapshot without the + // ConfigMap, and the work applier removes what the Works no longer carry. + Eventually(placementPolicyLatestSnapshotChangedActual(policyKey, initialSnapshotName, &newSnapshotName), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to take a resource snapshot without the deleted ConfigMap") + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new resource snapshot out to every binding") + Eventually(configMapAbsentOnClustersActual(deletedCMKey, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the deleted ConfigMap from the member clusters") + + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should leave the other ConfigMap on every member cluster as it was", func() { + dataActual := configMapDataOnClustersActual(keptCMKey, v0Data, allMemberClusters...) + Consistently(func() error { + if err := dataActual(); err != nil { + return err + } + uids, err := configMapUIDsByCluster(keptCMKey, allMemberClusters...) + if err != nil { + return err + } + if diff := cmp.Diff(uids, keptCMUIDsByCluster); diff != "" { + return fmt.Errorf("ConfigMap %s UIDs diff, keyed by cluster (-got, +want):\n%s", keptCMKey, diff) + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The other ConfigMap was changed or re-created") + }) + }) +}) diff --git a/test/e2e/placement_policy_scheduling_cluster_changes_test.go b/test/e2e/placement_policy_scheduling_cluster_changes_test.go new file mode 100644 index 000000000..7d06ed55b --- /dev/null +++ b/test/e2e/placement_policy_scheduling_cluster_changes_test.go @@ -0,0 +1,240 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" +) + +const ( + placementPolicyTolerationsNameTemplate = "pp-tolerations-%d" + placementPolicyTaintAfterBindNameTemplate = "pp-taint-after-bind-%d" + clusterPlacementPolicyRelabeledMatchNameTemplate = "cpp-relabeled-match-%d" +) + +// Every context here taints or relabels a member cluster, so it must not run alongside other specs, and must undo +// the change afterwards, as every spec relies on the member clusters that the suite sets up. +var _ = Describe("placement policy scheduling as member clusters change", Label("placementpolicy"), func() { + Context("placement policy tolerating a tainted cluster", Ordered, Serial, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyTolerationsNameTemplate, GinkgoParallelProcess()), + } + eastTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: regionEast}}, + } + taintedClusterNames := []string{memberCluster2EastCanaryName} + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("tainting the canary cluster") + addTaintsToMemberClusters(taintedClusterNames, buildTaints(taintedClusterNames)) + + By("creating a placement policy that selects all east clusters") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(eastTerms, intstr.FromString(clusterSelectorCountAll)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go, and the taint too, even if removing the policy fails; otherwise they leak + // into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + defer removeTaintsFromMemberClusters(taintedClusterNames) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind the untainted east cluster only", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, eastTerms, memberCluster1EastProdName) + }) + + It("should not bind or place the ConfigMap on the tainted cluster", func() { + bindingsActual := allSynchronizedPlacementPolicyBindingsActual(policyKey, sameSelectorsForClusters(eastTerms, memberCluster1EastProdName), 1) + taintedActual := appConfigMapRemovedFromClustersActual(memberCluster2EastCanary) + Consistently(func() error { + if err := bindingsActual(); err != nil { + return err + } + return taintedActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The tainted cluster was bound, or got the ConfigMap") + }) + + It("can add a toleration for the taint", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().Tolerations = placementPolicyTolerationsFor(taintedClusterNames...) + }) + }) + + It("should bind the tainted east cluster as well", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, eastTerms, memberCluster1EastProdName, memberCluster2EastCanaryName) + }) + }) + + Context("placement policy keeping a cluster that is tainted after it is bound", Ordered, Serial, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyTaintAfterBindNameTemplate, GinkgoParallelProcess()), + } + prodTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{envLabelName: envProd}}, + } + + // Which prod cluster ranks first depends on live capacity, so the specs learn it rather than assume it. + var taintedClusterName, otherClusterName string + var taintedBindingUID types.UID + var taintedBindingCreatedAt metav1.Time + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects one prod cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(prodTerms, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go, and the taint too, even if removing the policy fails; otherwise they leak + // into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + defer func() { + if taintedClusterName != "" { + removeTaintsFromMemberClusters([]string{taintedClusterName}) + } + }() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind one prod cluster", func() { + binding := waitForSinglePlacementPolicyBinding(policyKey) + taintedClusterName, taintedBindingUID, taintedBindingCreatedAt = binding.GetSpec().ClusterName, binding.GetUID(), binding.GetCreationTimestamp() + otherClusterName = otherProdClusterName(taintedClusterName) + + waitForPlacementPolicyPlacedOnlyOn(policyKey, prodTerms, taintedClusterName) + }) + + It("can taint the bound cluster", func() { + // Once the taint is gone, the trim needs a well-defined newest binding: the one the taint causes. + waitPastCreationSecond(taintedBindingCreatedAt) + addTaintsToMemberClusters([]string{taintedClusterName}, buildTaints([]string{taintedClusterName})) + }) + + It("should keep the tainted cluster bound, and bind the other prod cluster in its place", func() { + // The tainted cluster keeps its binding and its resources, so it is still a target of the resource + // status; it no longer counts toward the selector, though, so the other prod cluster fills the gap. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, sameSelectorsForClusters(prodTerms, taintedClusterName, otherClusterName), 1), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind the other prod cluster") + Expect(placementPolicyBindingsKeptActual(policyKey, map[string]types.UID{taintedClusterName: taintedBindingUID})()). + To(Succeed(), "Failed to keep the binding of the tainted cluster") + Eventually(appConfigMapPlacedOnClustersActual(memberClusterNamed(taintedClusterName), memberClusterNamed(otherClusterName)), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to place the ConfigMap on both prod clusters") + + wantStatus := allSynchronizedPlacementPolicyStatus(2) + wantStatus.DesiredClusters = ptr.To(int32(1)) + wantStatus.ScheduledClusters = ptr.To(int32(1)) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("can remove the taint", func() { + removeTaintsFromMemberClusters([]string{taintedClusterName}) + }) + + It("should unbind the other prod cluster, and keep the originally bound one", func() { + // Back in the running, the originally bound cluster has the older binding, so the count trims the newer + // one. + waitForPlacementPolicyPlacedOnlyOn(policyKey, prodTerms, taintedClusterName) + Expect(placementPolicyBindingsKeptActual(policyKey, map[string]types.UID{taintedClusterName: taintedBindingUID})()). + To(Succeed(), "Failed to keep the binding of the originally bound cluster") + }) + }) + + Context("cluster placement policy whose claim an existing cluster fulfills by starting to match", Ordered, Serial, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyRelabeledMatchNameTemplate, GinkgoParallelProcess()), + } + // No member cluster is in the south region until cluster-3 is relabeled. + southTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: regionSouth}}, + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects one south cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(southTerms, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go, and cluster-3 must get its label back, even if removing the policy fails; + // otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + defer setMemberClusterLabel(memberCluster3WestProdName, regionLabelName, regionWest) + // Deleting the policy also withdraws its cluster claim, should the spec fail before the claim is. + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should add a cluster claim, binding no cluster", func() { + Eventually(placementPolicyClaimActual(policyKey, southTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + Expect(noPlacementPolicyBindingsActual(policyKey)()).To(Succeed(), "A cluster was bound") + }) + + It("can relabel cluster-3 into the south region", func() { + setMemberClusterLabel(memberCluster3WestProdName, regionLabelName, regionSouth) + }) + + It("should bind the relabeled cluster, and report the selector as fulfilled", func() { + // The wanted status has no active cluster claim. + waitForPlacementPolicyPlacedOnlyOn(policyKey, southTerms, memberCluster3WestProdName) + }) + + It("should withdraw the cluster claim", func() { + Eventually(noPlacementPolicyClaimsActual(policyKey), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to withdraw the cluster claim") + }) + }) +}) diff --git a/test/e2e/placement_policy_scheduling_selectors_test.go b/test/e2e/placement_policy_scheduling_selectors_test.go new file mode 100644 index 000000000..bbb68adad --- /dev/null +++ b/test/e2e/placement_policy_scheduling_selectors_test.go @@ -0,0 +1,510 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + "go.goms.io/fleet/pkg/propertyprovider" +) + +const ( + clusterPlacementPolicyMinCountNameTemplate = "cpp-min-count-%d" + placementPolicyKeepSearchingNameTemplate = "pp-keep-searching-%d" + placementPolicyLabelExpressionsNameTemplate = "pp-label-expressions-%d" + clusterPlacementPolicyPropertyExpressionsNameTemplate = "cpp-property-expressions-%d" + placementPolicyInvalidSelectorNameTemplate = "pp-invalid-selector-%d" + placementPolicyClaimTermsChangedNameTemplate = "pp-claim-terms-changed-%d" +) + +// setPlacementPolicySelectorTerms replaces the terms of the policy's first cluster selector. A change to the terms +// re-evaluates the clusters that the selector has picked from scratch. +func setPlacementPolicySelectorTerms(key types.NamespacedName, terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm) { + GinkgoHelper() + + updatePlacementPolicy(key, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ClusterSelectors[0].Terms = terms + }) +} + +var _ = Describe("placement policy scheduling with cluster selector options", Label("placementpolicy"), func() { + Context("cluster placement policy that reaches its minimum count but not its count", Ordered, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyMinCountNameTemplate, GinkgoParallelProcess()), + } + // Only two member clusters are in the east region. + eastTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: regionEast}}, + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that wants 3 east clusters, and at least 2") + selector := clusterSelectorWithTerms(eastTerms, intstr.FromInt32(3)) + selector.MinCount = ptr.To(int32(2)) + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{selector}, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // Deleting the policy also withdraws its cluster claim. + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind both east clusters and place the ConfigMap on them", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, sameSelectorsForClusters(eastTerms, memberCluster1EastProdName, memberCluster2EastCanaryName), 1), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind both east clusters") + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on both east clusters") + }) + + It("should add a cluster claim for the missing cluster", func() { + Eventually(placementPolicyClaimActual(policyKey, eastTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + }) + + It("should report the selector as short of its count, with its minimum count reached", func() { + // Reaching minCount but not count still reports the policy as not scheduled; only the message tells the two + // cases apart. + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(3, 2, 1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + Eventually(placementPolicyConditionMessageActual(policyKey, kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, "have reached their minimum cluster counts"), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "The Scheduled condition message does not report the minimum count as reached") + }) + }) + + Context("placement policy that keeps searching for an unfulfillable selector", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyKeepSearchingNameTemplate, GinkgoParallelProcess()), + } + // No member cluster is in this region. + nowhereTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: "nowhere"}}, + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that keeps searching for a cluster in a region that has none") + selector := clusterSelectorWithTerms(nowhereTerms, intstr.FromInt32(1)) + selector.WhenUnfulfilled = kfplacementv1alpha1.WhenUnfulfilledOptionKeepSearching + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{selector}, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should report the selector as unfulfilled, with no cluster claim", func() { + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 0)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should neither add a cluster claim nor bind a cluster", func() { + claimsActual := noPlacementPolicyClaimsActual(policyKey) + bindingsActual := noPlacementPolicyBindingsActual(policyKey) + Consistently(func() error { + if err := claimsActual(); err != nil { + return err + } + return bindingsActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "A cluster claim was added, or a cluster bound") + }) + + It("can switch the selector to add a cluster claim when unfulfilled", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ClusterSelectors[0].WhenUnfulfilled = kfplacementv1alpha1.WhenUnfulfilledOptionAddClusterClaim + }) + }) + + It("should add a cluster claim carrying the selector terms", func() { + Eventually(placementPolicyClaimActual(policyKey, nowhereTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("placement policy selecting clusters by label expressions", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyLabelExpressionsNameTemplate, GinkgoParallelProcess()), + } + // Every member cluster carries the env label, and none carries this one, so Exists and DoesNotExist are + // each checked against a label that is present and one that is absent. + const neverSetLabelName = "fleet-e2e/never-set" + + labelExpressionTerm := func(exprs ...kfplacementv1alpha1.LabelClusterPropertyExpression) kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm { + return kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{MatchLabelExpressions: exprs} + } + inTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + labelExpressionTerm(kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: envLabelName, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorIn, Values: []string{envCanary}, + }), + } + notInTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + labelExpressionTerm(kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: envLabelName, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorNotIn, Values: []string{envCanary}, + }), + } + // Every member cluster satisfies both expressions. + matchingPresenceTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + labelExpressionTerm( + kfplacementv1alpha1.LabelClusterPropertyExpression{Key: envLabelName, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists}, + kfplacementv1alpha1.LabelClusterPropertyExpression{Key: neverSetLabelName, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist}, + ), + } + // No member cluster satisfies either term. + mismatchingPresenceTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + labelExpressionTerm(kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: envLabelName, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorDoesNotExist, + }), + labelExpressionTerm(kfplacementv1alpha1.LabelClusterPropertyExpression{ + Key: neverSetLabelName, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorExists, + }), + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects all clusters whose env label is in the given values") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(inTerms, intstr.FromString(clusterSelectorCountAll)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // Deleting the policy also withdraws its cluster claim. + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind the canary cluster for the In expression", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, inTerms, memberCluster2EastCanaryName) + }) + + It("can switch to a NotIn expression", func() { + setPlacementPolicySelectorTerms(policyKey, notInTerms) + }) + + It("should bind the prod clusters for the NotIn expression", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, notInTerms, memberCluster1EastProdName, memberCluster3WestProdName) + }) + + It("can switch to Exists and DoesNotExist expressions that every cluster satisfies", func() { + setPlacementPolicySelectorTerms(policyKey, matchingPresenceTerms) + }) + + It("should bind every member cluster", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, matchingPresenceTerms, allMemberClusterNames...) + }) + + It("can switch to Exists and DoesNotExist expressions that no cluster satisfies", func() { + setPlacementPolicySelectorTerms(policyKey, mismatchingPresenceTerms) + }) + + It("should unbind every member cluster and add a cluster claim", func() { + Eventually(noPlacementPolicyBindingsActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to unbind every member cluster") + Eventually(appConfigMapRemovedFromClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from every member cluster") + Eventually(placementPolicyClaimActual(policyKey, mismatchingPresenceTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + // A count: All selector wants at least one cluster. + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("cluster placement policy selecting clusters by cluster property expressions", Ordered, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyPropertyExpressionsNameTemplate, GinkgoParallelProcess()), + } + nodeCountTerms := func(operator kfplacementv1alpha1.LabelClusterPropertyExpressionOperator, value string) []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm { + return []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + { + MatchClusterPropertyExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: propertyprovider.NodeCountProperty, Operator: operator, Values: []string{value}}, + }, + }, + } + } + // The member clusters have 2, 3, and 4 nodes respectively, control plane included (see kindconfigs/). + atLeastThreeNodesTerms := nodeCountTerms(kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGe, "3") + exactlyTwoNodesTerms := nodeCountTerms(kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorEq, "2") + + BeforeAll(func() { + // The node count property comes from the property provider; the legacy property selector specs are + // gated the same way. + if !isAzurePropertyProviderEnabled { + Skip("Skipping this test spec as Azure property provider is not enabled in the test environment") + } + + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects all clusters with at least 3 nodes") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(atLeastThreeNodesTerms, intstr.FromString(clusterSelectorCountAll)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind the clusters with at least 3 nodes", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, atLeastThreeNodesTerms, memberCluster2EastCanaryName, memberCluster3WestProdName) + }) + + It("can switch to clusters with exactly 2 nodes", func() { + setPlacementPolicySelectorTerms(policyKey, exactlyTwoNodesTerms) + }) + + It("should bind the cluster with exactly 2 nodes only", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, exactlyTwoNodesTerms, memberCluster1EastProdName) + }) + }) + + Context("placement policy whose cluster selector cannot be evaluated", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyInvalidSelectorNameTemplate, GinkgoParallelProcess()), + } + prodTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{envLabelName: envProd}}, + } + // Admission accepts Gt on any key, but only a cluster property can be compared numerically; a label + // expression with Gt fails when the selector is evaluated. + invalidTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + { + MatchLabelExpressions: []kfplacementv1alpha1.LabelClusterPropertyExpression{ + {Key: envLabelName, Operator: kfplacementv1alpha1.LabelClusterPropertyExpressionOperatorGt, Values: []string{"1"}}, + }, + }, + } + + var bindingUIDs map[string]types.UID + // The policy generation that the last round with valid selectors observed. + var validGeneration int64 + + invalidSelectorStatus := func() kfplacementv1alpha1.PlacementPolicyStatus { + // Only the Scheduled condition is re-evaluated, and the cluster counts are cleared; the resource revision, + // the synchronization and availability counts, and their conditions are left as the last valid round + // reported them. + lastValid := func(cond metav1.Condition) metav1.Condition { + cond.ObservedGeneration = validGeneration + return cond + } + return kfplacementv1alpha1.PlacementPolicyStatus{ + Conditions: []metav1.Condition{ + lastValid(placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected)), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, metav1.ConditionFalse, + placementPolicyScheduledCondReasonInvalidClusterSelectors), + lastValid(placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized)), + lastValid(placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyAvailableCondReasonAllClustersAvailable)), + }, + SynchronizedClusters: ptr.To(int32(2)), + ResourcesAvailableClusters: ptr.To(int32(2)), + ActiveClusterClaims: ptr.To(int32(0)), + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects all prod clusters") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(prodTerms, intstr.FromString(clusterSelectorCountAll)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind the prod clusters", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, prodTerms, memberCluster1EastProdName, memberCluster3WestProdName) + bindingUIDs = placementPolicyBindingUIDsByCluster(policyKey) + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + validGeneration = policy.GetGeneration() + }) + + It("can change the selector to one that cannot be evaluated", func() { + setPlacementPolicySelectorTerms(policyKey, invalidTerms) + }) + + It("should report the cluster selectors as invalid, leaving the resource status as it was", func() { + Eventually(placementPolicyStatusActual(policyKey, invalidSelectorStatus()), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should keep the existing bindings and the ConfigMap on the prod clusters", func() { + // The bindings still record the last valid terms. + bindingsActual := allSynchronizedPlacementPolicyBindingsActual(policyKey, sameSelectorsForClusters(prodTerms, memberCluster1EastProdName, memberCluster3WestProdName), 1) + keptActual := placementPolicyBindingsKeptActual(policyKey, bindingUIDs) + placedActual := appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster3WestProd) + statusActual := placementPolicyStatusActual(policyKey, invalidSelectorStatus()) + Consistently(func() error { + for _, actual := range []func() error{bindingsActual, keptActual, placedActual, statusActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The bindings, the ConfigMap, or the status changed while the selector was invalid") + }) + + It("can fix the selector", func() { + setPlacementPolicySelectorTerms(policyKey, prodTerms) + }) + + It("should report the policy as scheduled again, keeping the bindings", func() { + waitForPlacementPolicyPlacedOnlyOn(policyKey, prodTerms, memberCluster1EastProdName, memberCluster3WestProdName) + Expect(placementPolicyBindingsKeptActual(policyKey, bindingUIDs)()).To(Succeed(), "Failed to keep the bindings") + }) + }) + + Context("placement policy whose unfulfilled selector changes its terms", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyClaimTermsChangedNameTemplate, GinkgoParallelProcess()), + } + // No member cluster is in either region. + nowhereTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: "nowhere"}}, + } + elsewhereTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: "elsewhere"}}, + } + + var claimName string + var claimUID types.UID + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects one cluster in a region that has none") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorWithTerms(nowhereTerms, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // Deleting the policy also withdraws its cluster claim. + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should add a cluster claim carrying the selector terms", func() { + Eventually(placementPolicyClaimActual(policyKey, nowhereTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + + claims, err := claimsOfPolicy(policyKey.Name, policyKey.Namespace) + Expect(err).NotTo(HaveOccurred(), "Failed to list the cluster claims of placement policy %s", policyKey) + Expect(claims.Items).To(HaveLen(1), "Placement policy %s does not have exactly one cluster claim", policyKey) + claimName, claimUID = claims.Items[0].Name, claims.Items[0].UID + }) + + It("can change the terms of the selector", func() { + setPlacementPolicySelectorTerms(policyKey, elsewhereTerms) + }) + + It("should replace the cluster claim under the same name, carrying the new terms", func() { + // A claim serves only the terms it was issued for: the policy withdraws it and issues a new one, under the + // same name, which derives from the policy and the selector index. + claimActual := placementPolicyClaimActual(policyKey, elsewhereTerms) + Eventually(func() error { + if err := claimActual(); err != nil { + return err + } + claims, err := claimsOfPolicy(policyKey.Name, policyKey.Namespace) + if err != nil { + return err + } + if len(claims.Items) != 1 { + return fmt.Errorf("placement policy %s has %d cluster claims, want 1", policyKey, len(claims.Items)) + } + claim := &claims.Items[0] + switch { + case claim.Name != claimName: + return fmt.Errorf("the cluster claim of placement policy %s is named %s, want %s", policyKey, claim.Name, claimName) + case claim.UID == claimUID: + return fmt.Errorf("the cluster claim %s of placement policy %s has not been replaced (UID %s)", claim.Name, policyKey, claim.UID) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to replace the cluster claim") + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) +}) diff --git a/test/e2e/placement_policy_scheduling_test.go b/test/e2e/placement_policy_scheduling_test.go new file mode 100644 index 000000000..054489a6c --- /dev/null +++ b/test/e2e/placement_policy_scheduling_test.go @@ -0,0 +1,531 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "strings" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils" +) + +const ( + placementPolicyNoSelectorsNameTemplate = "pp-no-selectors-%d" + clusterPlacementPolicyAllProdNameTemplate = "cpp-all-prod-%d" + placementPolicyOverlappingNameTemplate = "pp-overlapping-%d" + placementPolicyScaledNameTemplate = "pp-scaled-%d" + placementPolicyStickyNameTemplate = "pp-sticky-%d" + + clusterPlacementPolicyBindingDeletedNameTemplate = "cpp-binding-deleted-%d" +) + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy scheduling", Label("placementpolicy"), func() { + Context("placement policy with no cluster selectors", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyNoSelectorsNameTemplate, GinkgoParallelProcess()), + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy with no cluster selectors") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind every member cluster to the latest resource snapshot", func() { + // With no cluster selectors, the policy selects all clusters through a single selector without terms. + wantSelectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantSelectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster, 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + }) + + It("should place the ConfigMap on every member cluster", func() { + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + }) + + It("should report the policy as scheduled, synchronized, and available on every member cluster", func() { + // The fleet also has virtual member clusters that are never eligible for scheduling (see setup_test.go); + // the counts cover the eligible clusters only. + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("cluster placement policy selecting all prod clusters", Ordered, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyAllProdNameTemplate, GinkgoParallelProcess()), + } + prodClusters := []string{memberCluster1EastProdName, memberCluster3WestProdName} + + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + wantSelectors := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(prodClusters)) + for _, clusterName := range prodClusters { + wantSelectors[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + { + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{envLabelName: envProd}}, + }, + }, + } + } + return wantSelectors + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects all prod clusters") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(map[string]string{envLabelName: envProd}, intstr.FromString(clusterSelectorCountAll)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind the prod clusters only, to the latest resource snapshot", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind the prod clusters") + }) + + It("should place the ConfigMap on the prod clusters", func() { + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster3WestProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on the prod clusters") + }) + + It("should report the policy as scheduled, synchronized, and available on the prod clusters", func() { + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(prodClusters))) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + + It("should not bind or place the ConfigMap on the canary cluster", func() { + bindingsActual := allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1) + canaryActual := appConfigMapRemovedFromClustersActual(memberCluster2EastCanary) + Consistently(func() error { + if err := bindingsActual(); err != nil { + return err + } + return canaryActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The canary cluster was bound, or got the ConfigMap") + }) + }) + + Context("placement policy with overlapping cluster selectors", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyOverlappingNameTemplate, GinkgoParallelProcess()), + } + // Selector A picks cluster-1 by its alias; selector B picks both prod clusters, cluster-1 among them. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + prodLabels := map[string]string{envLabelName: envProd} + + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + // A cluster that satisfies several selectors records each of them, in selector order. + return map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster1EastProdName: {clusterSelectorTermsMatchingLabels(aliasLabels), clusterSelectorTermsMatchingLabels(prodLabels)}, + memberCluster3WestProdName: {clusterSelectorTermsMatchingLabels(prodLabels)}, + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy whose two cluster selectors both select cluster-1") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + clusterSelectorMatchingLabels(prodLabels, intstr.FromInt32(2)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind cluster-1 once for both selectors, and cluster-3 for the prod selector", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind the selected clusters") + }) + + It("should place the ConfigMap on the prod clusters", func() { + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster3WestProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on the prod clusters") + }) + + It("should count clusters per selector, and synchronized clusters per cluster", func() { + // The desired and scheduled counts are sums over the selectors (1 + 2), so cluster-1 counts twice; the + // synchronized and available counts are over the bound clusters. + wantStatus := allSynchronizedPlacementPolicyStatus(2) + wantStatus.DesiredClusters = ptr.To(int32(3)) + wantStatus.ScheduledClusters = ptr.To(int32(3)) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should not bind or place the ConfigMap on the canary cluster", func() { + bindingsActual := allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1) + canaryActual := appConfigMapRemovedFromClustersActual(memberCluster2EastCanary) + Consistently(func() error { + if err := bindingsActual(); err != nil { + return err + } + return canaryActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The canary cluster was bound, or got the ConfigMap") + }) + }) + + Context("placement policy scaled up and back down", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyScaledNameTemplate, GinkgoParallelProcess()), + } + eastLabels := map[string]string{regionLabelName: regionEast} + eastSelectors := []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{clusterSelectorTermsMatchingLabels(eastLabels)} + + // Which east cluster ranks first depends on live capacity, so the specs learn it rather than assume it. + var firstClusterName, secondClusterName string + var firstBindingUID types.UID + var firstBindingCreatedAt metav1.Time + + firstBindingKeptActual := func() func() error { + return func() error { + bindings, err := placementPolicyBindingsByCluster(policyKey) + if err != nil { + return err + } + binding, found := bindings[firstClusterName] + if !found { + return fmt.Errorf("placement policy %s has no binding for the first picked cluster %s", policyKey, firstClusterName) + } + if binding.GetUID() != firstBindingUID { + return fmt.Errorf("the binding for the first picked cluster %s was replaced (UID %s, want %s)", firstClusterName, binding.GetUID(), firstBindingUID) + } + return nil + } + } + + setEastSelectorCount := func(count int32) { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ClusterSelectors[0].Count = ptr.To(intstr.FromInt32(count)) + }) + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects one east cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(eastLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind one east cluster", func() { + binding := waitForSinglePlacementPolicyBinding(policyKey) + firstClusterName, firstBindingUID, firstBindingCreatedAt = binding.GetSpec().ClusterName, binding.GetUID(), binding.GetCreationTimestamp() + secondClusterName = otherEastClusterName(firstClusterName) + + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + firstClusterName: eastSelectors, + }, 1), workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to synchronize the binding") + Eventually(appConfigMapPlacedOnClustersActual(memberClusterNamed(firstClusterName)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on the bound cluster") + }) + + It("can scale the selector up to 2 clusters", func() { + // The scale-down below needs a well-defined newest binding. + waitPastCreationSecond(firstBindingCreatedAt) + setEastSelectorCount(2) + }) + + It("should bind the other east cluster as well, keeping the first binding", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + firstClusterName: eastSelectors, + secondClusterName: eastSelectors, + }, 1), workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind both east clusters") + Expect(firstBindingKeptActual()()).To(Succeed(), "Failed to keep the first binding") + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on both east clusters") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(2)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("can scale the selector back down to 1 cluster", func() { + setEastSelectorCount(1) + }) + + It("should unbind the newer cluster and keep the first one", func() { + // A scale-down undoes the latest picks, so the binding made for the scale-up goes, whichever cluster ranks + // better now. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + firstClusterName: eastSelectors, + }, 1), workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to unbind the newer cluster") + Expect(firstBindingKeptActual()()).To(Succeed(), "Failed to keep the first binding") + // Deleting a binding deletes its Works, which removes the resources from the cluster by default. + Eventually(appConfigMapRemovedFromClustersActual(memberClusterNamed(secondClusterName)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from the unbound cluster") + Expect(appConfigMapPlacedOnClustersActual(memberClusterNamed(firstClusterName))()).To(Succeed(), "The ConfigMap is gone from the cluster that kept its binding") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + // The context relabels a member cluster, so it must not run alongside other specs. + Context("placement policy sticking to a cluster that stops matching its selector", Ordered, Serial, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyStickyNameTemplate, GinkgoParallelProcess()), + } + canaryLabels := map[string]string{envLabelName: envCanary} + // The changed terms still ask for the canary, which no cluster is once cluster-2 is relabeled. + changedTerms := []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{envLabelName: envCanary, regionLabelName: regionEast}}, + } + + stickyActual := func() func() error { + bindingsActual := allSynchronizedPlacementPolicyBindingsActual(policyKey, map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster2EastCanaryName: {clusterSelectorTermsMatchingLabels(canaryLabels)}, + }, 1) + placedActual := appConfigMapPlacedOnClustersActual(memberCluster2EastCanary) + statusActual := placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(1)) + return func() error { + if err := bindingsActual(); err != nil { + return err + } + if err := placedActual(); err != nil { + return err + } + return statusActual() + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that selects the canary cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(canaryLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go, and cluster-2 must get its label back, even if removing the policy fails; + // otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + defer setMemberClusterLabel(memberCluster2EastCanaryName, envLabelName, labelsByClusterName[memberCluster2EastCanaryName][envLabelName]) + // Deleting the policy also withdraws its cluster claim. + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should bind the canary cluster and place the ConfigMap on it", func() { + Eventually(stickyActual(), workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind the canary cluster") + }) + + It("can relabel the canary cluster as a prod cluster", func() { + setMemberClusterLabel(memberCluster2EastCanaryName, envLabelName, envProd) + }) + + It("should keep the binding and the ConfigMap on the relabeled cluster, still counting it", func() { + // Only a change to the selector's terms re-evaluates the clusters it has picked. + Consistently(stickyActual(), consistentlyDuration, consistentlyInterval).Should(Succeed(), "Failed to keep the relabeled cluster bound") + }) + + It("can change the selector terms", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ClusterSelectors[0].Terms = changedTerms + }) + }) + + It("should unbind the relabeled cluster and remove the ConfigMap from it", func() { + Eventually(noPlacementPolicyBindingsActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to unbind the relabeled cluster") + Eventually(appConfigMapRemovedFromClustersActual(memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from the relabeled cluster") + }) + + It("should add a cluster claim carrying the changed terms", func() { + Eventually(placementPolicyClaimActual(policyKey, changedTerms), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to add the cluster claim") + }) + + It("should report the selector as unfulfilled, with the claim outstanding", func() { + // With no target cluster left, nothing is out of sync or unavailable. + Eventually(placementPolicyStatusActual(policyKey, unfulfilledPlacementPolicyStatus(1, 0, 1)), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("cluster placement policy whose binding is deleted out of band", Ordered, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyBindingDeletedNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + deletedClusterName := memberCluster2EastCanaryName + + var bindingUIDsByCluster map[string]types.UID + var workUIDsByWork map[string]types.UID + var cmUIDsByCluster map[string]types.UID + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the ConfigMap on every member cluster", func() { + waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster") + + bindingUIDsByCluster = placementPolicyBindingUIDsByCluster(policyKey) + var err error + workUIDsByWork, err = placementPolicyWorkUIDs(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the Works of cluster placement policy %s", policyKey) + cmUIDsByCluster, err = configMapUIDsByCluster(cmKey, allMemberClusters...) + Expect(err).NotTo(HaveOccurred(), "Failed to read the ConfigMap on the member clusters") + }) + + It("can delete the binding for member cluster 2", func() { + bindings, err := placementPolicyBindingsByCluster(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to list the bindings of cluster placement policy %s", policyKey) + binding, found := bindings[deletedClusterName] + Expect(found).To(BeTrue(), "Cluster placement policy %s has no binding for member cluster 2", policyKey) + Expect(hubClient.Delete(ctx, binding)).To(Succeed(), "Failed to delete binding %s", binding.GetName()) + }) + + It("should re-create the binding, and with it the Works and the ConfigMap, on member cluster 2 only", func() { + // Deleting the binding deletes its Works, and the work applier removes the ConfigMap they placed (the default + // whenPlacementDeleted=CleanUpResources). The policy then binds the cluster again, under the same binding + // name, and the ConfigMap comes back as a new object. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster again") + Eventually(appConfigMapPlacedOnClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on every member cluster again") + + deletedClusterNamespace := fmt.Sprintf(utils.NamespaceNameFormat, deletedClusterName) + Eventually(func() error { + bindings, err := placementPolicyBindingsByCluster(policyKey) + if err != nil { + return err + } + workUIDs, err := placementPolicyWorkUIDs(policyKey) + if err != nil { + return err + } + cmUIDs, err := configMapUIDsByCluster(cmKey, allMemberClusters...) + if err != nil { + return err + } + for clusterName, uid := range bindingUIDsByCluster { + binding, found := bindings[clusterName] + if !found { + return fmt.Errorf("cluster placement policy %s has no binding for cluster %s", policyKey, clusterName) + } + if replaced := binding.GetUID() != uid; replaced != (clusterName == deletedClusterName) { + return fmt.Errorf("the binding for cluster %s replaced = %t (UID %s, before %s)", clusterName, replaced, binding.GetUID(), uid) + } + } + for clusterName, uid := range cmUIDsByCluster { + if replaced := cmUIDs[clusterName] != uid; replaced != (clusterName == deletedClusterName) { + return fmt.Errorf("the ConfigMap on cluster %s replaced = %t (UID %s, before %s)", clusterName, replaced, cmUIDs[clusterName], uid) + } + } + // The Works of the other clusters are kept as they were; those of the re-created binding are new. + for workKey, uid := range workUIDsByWork { + onDeletedCluster := strings.HasPrefix(workKey, deletedClusterNamespace+"/") + if gotUID, found := workUIDs[workKey]; found && (gotUID == uid) == onDeletedCluster { + return fmt.Errorf("work %s replaced = %t (UID %s, before %s)", workKey, gotUID != uid, gotUID, uid) + } else if !found && !onDeletedCluster { + return fmt.Errorf("work %s is gone", workKey) + } + } + return nil + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to replace the binding, its Works, and the ConfigMap of member cluster 2 only") + + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) +}) diff --git a/test/e2e/placement_policy_staged_update_run_failures_test.go b/test/e2e/placement_policy_staged_update_run_failures_test.go new file mode 100644 index 000000000..67a62d736 --- /dev/null +++ b/test/e2e/placement_policy_staged_update_run_failures_test.go @@ -0,0 +1,425 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" +) + +const ( + placementPolicyFailurePolicyNameTemplate = "pp-failure-%s-%d" + clusterPlacementPolicyFailurePolicyNameTemplate = "cpp-failure-%s-%d" + placementPolicyFailurePolicyRunNameTemplate = "pp-failure-%s-run-%d" + clusterPlacementPolicyFailurePolicyRunNameTemplate = "cpp-failure-%s-run-%d" + placementPolicyFailurePolicyStrategyNameTemplate = "pp-failure-strategy-%d" + clusterPlacementPolicyFailurePolicyStrategyNameTemplate = "cpp-failure-strategy-%d" + placementPolicyFailurePolicyDeploymentNameTemplate = "pp-failure-%s" + clusterPlacementPolicyFailurePolicyDeploymentNameTemplate = "cpp-failure-%s" + + placementPolicyInitFailureNameTemplate = "pp-init-failure-%d" + placementPolicyInitFailureOtherNameTemplate = "pp-init-failure-other-%d" + placementPolicyInitFailureStrategyNameTemplate = "pp-init-failure-strategy-%d" + placementPolicyInitFailureSortingStrategyNameTemplate = "pp-init-failure-sorting-strategy-%d" + placementPolicyNoStrategyRunNameTemplate = "pp-init-failure-no-strategy-%d" + placementPolicyNoPolicyRunNameTemplate = "pp-init-failure-no-policy-%d" + placementPolicyForeignSnapshotRunNameTemplate = "pp-init-failure-foreign-snapshot-%d" + placementPolicyNoSortingLabelRunNameTemplate = "pp-init-failure-no-sorting-label-%d" + + // placementPolicyUnpullableImage is an image that no node can pull: the registry host does not resolve. + placementPolicyUnpullableImage = "fleet-e2e.invalid/no-such-image:v1" + // placementPolicyNoPodsQuotaName names the resource quota that blocks new pods in a namespace on a member cluster. + placementPolicyNoPodsQuotaName = "pp-e2e-no-pods" + // placementPolicyOtherConfigMapName names the ConfigMap that a second policy places. + placementPolicyOtherConfigMapName = "pp-e2e-other-config" + + // placementPolicyFailedRunEventuallyDuration bounds the wait on a run whose cluster times out: the run gives a + // cluster 1 minute, and checks on it in reconciles that each last up to the run controller's 30s refresh period. + placementPolicyFailedRunEventuallyDuration = 3 * time.Minute +) + +// failurePolicyCase is one of the runs that the failure policy spec rolls out, one at a time. +type failurePolicyCase struct { + description string + policyKey types.NamespacedName + strategyKey types.NamespacedName + runKey types.NamespacedName + deployKey types.NamespacedName + + initialSnapshotName string +} + +// newFailurePolicyDeployment returns a Deployment for the failure policy spec; each has its own pod labels, so that +// the Deployments in the namespace never select each other's pods. +func newFailurePolicyDeployment(key types.NamespacedName) *appsv1.Deployment { + deploy := appDeployment() + deploy.Name = key.Name + deploy.Namespace = key.Namespace + podLabels := map[string]string{"app": key.Name} + deploy.Spec.Selector = &metav1.LabelSelector{MatchLabels: podLabels} + deploy.Spec.Template.Labels = maps.Clone(podLabels) + return &deploy +} + +// updateDeploymentOnHub applies the mutation to a Deployment on the hub cluster, retrying on conflicts. +func updateDeploymentOnHub(key types.NamespacedName, mutate func(deploy *appsv1.Deployment)) { + GinkgoHelper() + + Eventually(func() error { + deploy := &appsv1.Deployment{} + if err := hubClient.Get(ctx, key, deploy); err != nil { + return err + } + mutate(deploy) + return hubClient.Update(ctx, deploy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update Deployment %s on the hub cluster", key) +} + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy staged update run failures", Label("placementpolicy"), func() { + // The four runs below each hold a staged update run worker while a cluster times out, and the workers are shared + // by every run in the fleet; running them one at a time keeps the rest of the lane going. + Context("placement policies rolled out by staged update runs with failure policies, one run at a time", Ordered, func() { + nsName := appNamespace().Name + ppStrategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyFailurePolicyStrategyNameTemplate, GinkgoParallelProcess())} + cppStrategyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyFailurePolicyStrategyNameTemplate, GinkgoParallelProcess())} + + newCase := func(clusterScoped bool, variant string) *failurePolicyCase { + if clusterScoped { + return &failurePolicyCase{ + description: fmt.Sprintf("cluster placement policy (%s)", variant), + policyKey: types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyFailurePolicyNameTemplate, variant, GinkgoParallelProcess())}, + strategyKey: cppStrategyKey, + runKey: types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyFailurePolicyRunNameTemplate, variant, GinkgoParallelProcess())}, + deployKey: types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(clusterPlacementPolicyFailurePolicyDeploymentNameTemplate, variant)}, + } + } + return &failurePolicyCase{ + description: fmt.Sprintf("placement policy (%s)", variant), + policyKey: types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyFailurePolicyNameTemplate, variant, GinkgoParallelProcess())}, + strategyKey: ppStrategyKey, + runKey: types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyFailurePolicyRunNameTemplate, variant, GinkgoParallelProcess())}, + deployKey: types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyFailurePolicyDeploymentNameTemplate, variant)}, + } + } + // Variant a allows no failure; variant b allows one. + failingCases := []*failurePolicyCase{newCase(false, "a"), newCase(true, "a")} + partialCases := []*failurePolicyCase{newCase(false, "b"), newCase(true, "b")} + allCases := append(append([]*failurePolicyCase{}, failingCases...), partialCases...) + + createRun := func(c *failurePolicyCase, maxAllowedClusterFailures int32) { + run := newStagedUpdateRunObject(c.runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: c.policyKey.Name, + StagedUpdateStrategyName: c.strategyKey.Name, + FailurePolicy: &kfrolloutv1alpha1.StagedUpdateRunFailurePolicy{ + MaxAllowedClusterFailures: ptr.To(maxAllowedClusterFailures), + MaxWaitTimePerClusterMinutes: 1, + }, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", c.runKey) + } + + BeforeAll(func() { + By("creating the work resources and a Deployment per run") + createWorkResources() + for _, c := range allCases { + Expect(hubClient.Create(ctx, newFailurePolicyDeployment(c.deployKey))).To(Succeed(), "Failed to create Deployment %s", c.deployKey) + } + + By("creating a policy per run, each placing its own Deployment on every member cluster") + for _, c := range allCases { + selector := kfplacementv1alpha1.ResourceSelector{APIGroup: appsv1.GroupName, APIVersion: "v1", Kind: "Deployment", Name: c.deployKey.Name} + if c.policyKey.Namespace == "" { + selector.Namespace = c.deployKey.Namespace + } + policy := newPlacementPolicyObject(c.policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{selector}, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create %s %s", c.description, c.policyKey) + } + + By("creating a strategy per scope with a canary stage before a prod stage") + for _, strategyKey := range []types.NamespacedName{ppStrategyKey, cppStrategyKey} { + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + {Name: placementPolicyCanaryStageName, LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envCanary}}}, + {Name: placementPolicyProdStageName, LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envProd}}}, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + } + }) + + AfterAll(func() { + // The work resources, and the resource quota with the member namespace, must go even if the cleanup + // below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + for _, c := range allCases { + ensureStagedUpdateRunRemoved(c.runKey) + } + ensureStagedUpdateStrategyRemoved(ppStrategyKey) + ensureStagedUpdateStrategyRemoved(cppStrategyKey) + for _, c := range allCases { + ensurePlacementPolicyRemoved(c.policyKey) + } + }) + + It("should place every Deployment on every member cluster", func() { + for _, c := range allCases { + // The first placement pulls the image on every node, so it gets more time. + Eventually(allSynchronizedPlacementPolicyBindingsActual(c.policyKey, noTermsSelectorsForAllMemberClusters(), 1), longEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the Deployment of %s %s", c.description, c.policyKey) + c.initialSnapshotName = latestPrimarySnapshotNameOfPlacementPolicy(c.policyKey) + Expect(deploymentAvailableOnClustersActual(c.deployKey, allMemberClusters...)()).To(Succeed(), "Deployment %s is not available", c.deployKey) + } + }) + + for _, c := range failingCases { + It(fmt.Sprintf("can start a staged update run for the %s that rolls out an image that cannot be pulled, allowing no failure", c.description), func() { + updateDeploymentOnHub(c.deployKey, func(deploy *appsv1.Deployment) { + deploy.Spec.Template.Spec.Containers[0].Image = placementPolicyUnpullableImage + }) + createRun(c, 0) + }) + + It(fmt.Sprintf("should fail the run of the %s at the canary cluster, leaving the prod stage untouched", c.description), func() { + Eventually(stagedUpdateRunConditionActual(c.runKey, kfrolloutv1alpha1.StagedUpdateRunCondTypeCompleted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunCompletedCondReasonFailed), placementPolicyFailedRunEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to fail staged update run %s", c.runKey) + Expect(stagedUpdateRunClusterResultsActual(c.runKey, map[string]string{ + memberCluster2EastCanaryName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed, + memberCluster1EastProdName: "", + memberCluster3WestProdName: "", + })()).To(Succeed(), "Staged update run %s did not stop at the canary cluster", c.runKey) + Expect(placementPolicyBindingSnapshotsActual(c.policyKey, map[string]string{ + memberCluster2EastCanaryName: resourceSnapshotNameToRolloutOf(c.runKey), + memberCluster1EastProdName: c.initialSnapshotName, + memberCluster3WestProdName: c.initialSnapshotName, + })()).To(Succeed(), "The run of %s %s touched the prod clusters", c.description, c.policyKey) + Eventually(placementPolicyBindingManagerActual(c.policyKey, nil), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to release the binding manager role of %s %s", c.description, c.policyKey) + }) + } + + It("can block new pods in the namespace on the canary cluster with a resource quota", func() { + quota := &corev1.ResourceQuota{ + ObjectMeta: metav1.ObjectMeta{Name: placementPolicyNoPodsQuotaName, Namespace: nsName}, + Spec: corev1.ResourceQuotaSpec{ + Hard: corev1.ResourceList{corev1.ResourcePods: resource.MustParse("0")}, + }, + } + // The quota blocks only pods created from now on; the Deployments keep the pods they have. + Expect(client.IgnoreAlreadyExists(memberCluster2EastCanary.KubeClient.Create(ctx, quota))).To(Succeed(), + "Failed to create resource quota %s on cluster %s", client.ObjectKeyFromObject(quota), memberCluster2EastCanary.ClusterName) + }) + + for _, c := range partialCases { + It(fmt.Sprintf("can start a staged update run for the %s that changes the pod template, allowing one failure", c.description), func() { + updateDeploymentOnHub(c.deployKey, func(deploy *appsv1.Deployment) { + deploy.Spec.Template.Annotations = map[string]string{"placement-policy-e2e/revision": "v1"} + }) + createRun(c, 1) + }) + + It(fmt.Sprintf("should complete the run of the %s with a partial success, only the canary cluster failing", c.description), func() { + Eventually(stagedUpdateRunConditionActual(c.runKey, kfrolloutv1alpha1.StagedUpdateRunCondTypeCompleted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunCompletedCondReasonPartiallySucceeded), placementPolicyFailedRunEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s with a partial success", c.runKey) + Expect(stagedUpdateRunClusterResultsActual(c.runKey, map[string]string{ + memberCluster2EastCanaryName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonFailed, + memberCluster1EastProdName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + memberCluster3WestProdName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + })()).To(Succeed(), "Staged update run %s did not fail the canary cluster alone", c.runKey) + newSnapshotName := resourceSnapshotNameToRolloutOf(c.runKey) + Expect(placementPolicyBindingSnapshotsActual(c.policyKey, map[string]string{ + memberCluster2EastCanaryName: newSnapshotName, + memberCluster1EastProdName: newSnapshotName, + memberCluster3WestProdName: newSnapshotName, + })()).To(Succeed(), "The run of %s %s did not move every binding to the new resource snapshot", c.description, c.policyKey) + Eventually(placementPolicyBindingManagerActual(c.policyKey, nil), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to release the binding manager role of %s %s", c.description, c.policyKey) + }) + } + }) + + Context("placement policy with staged update runs that fail to initialize", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyInitFailureNameTemplate, GinkgoParallelProcess())} + otherPolicyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyInitFailureOtherNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyInitFailureStrategyNameTemplate, GinkgoParallelProcess())} + sortingStrategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyInitFailureSortingStrategyNameTemplate, GinkgoParallelProcess())} + noStrategyRunKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyNoStrategyRunNameTemplate, GinkgoParallelProcess())} + noPolicyRunKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyNoPolicyRunNameTemplate, GinkgoParallelProcess())} + foreignSnapshotRunKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyForeignSnapshotRunNameTemplate, GinkgoParallelProcess())} + noSortingLabelRunKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyNoSortingLabelRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName, otherSnapshotName string + + createRunAndWaitForInitFailure := func(runKey types.NamespacedName, spec kfrolloutv1alpha1.StagedUpdateRunSpec) { + GinkgoHelper() + + Expect(hubClient.Create(ctx, newStagedUpdateRunObject(runKey, spec))).To(Succeed(), "Failed to create staged update run %s", runKey) + // A run that fails to initialize reports that alone: it picks no resource snapshot and sets up no stage. + wantStatus := kfrolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: []metav1.Condition{ + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunCondTypeInitialized, metav1.ConditionFalse, + kfrolloutv1alpha1.StagedUpdateRunInitializedCondReasonFailed), + }, + } + Eventually(stagedUpdateRunStatusActual(runKey, wantStatus), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report that staged update run %s failed to initialize", runKey) + } + + BeforeAll(func() { + By("creating the work resources and a second ConfigMap") + createWorkResources() + otherCM := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: placementPolicyOtherConfigMapName, Namespace: nsName}, + Data: map[string]string{"data": "other"}, + } + Expect(hubClient.Create(ctx, otherCM)).To(Succeed(), "Failed to create ConfigMap %s", client.ObjectKeyFromObject(otherCM)) + + By("creating a policy for the app ConfigMap, and another one for the second ConfigMap") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + otherPolicy := newPlacementPolicyObject(otherPolicyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: []kfplacementv1alpha1.ResourceSelector{ + {APIVersion: "v1", Kind: "ConfigMap", Name: placementPolicyOtherConfigMapName}, + }, + }) + Expect(hubClient.Create(ctx, otherPolicy)).To(Succeed(), "Failed to create placement policy %s", otherPolicyKey) + + By("creating a valid strategy, and one that sorts clusters by a label that no cluster has") + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{{Name: placementPolicyAllClustersStageName, LabelSelector: &metav1.LabelSelector{}}}, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + sortingStrategy := newStagedUpdateStrategyObject(sortingStrategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyAllClustersStageName, + LabelSelector: &metav1.LabelSelector{}, + SortingLabelKey: ptr.To("placement-policy-e2e/no-such-label"), + }, + }, + }) + Expect(hubClient.Create(ctx, sortingStrategy)).To(Succeed(), "Failed to create staged update strategy %s", sortingStrategyKey) + }) + + AfterAll(func() { + // The work resources, the second ConfigMap among them, must go even if the cleanup below fails; otherwise + // they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + for _, runKey := range []types.NamespacedName{noStrategyRunKey, noPolicyRunKey, foreignSnapshotRunKey, noSortingLabelRunKey} { + ensureStagedUpdateRunRemoved(runKey) + } + ensureStagedUpdateStrategyRemoved(strategyKey) + ensureStagedUpdateStrategyRemoved(sortingStrategyKey) + // ensurePlacementPolicyRemoved waits for the app ConfigMap to go, which only the main policy places. + ensurePlacementPolicyRemoved(policyKey) + deletePlacementPolicyAndWait(otherPolicyKey) + Eventually(configMapAbsentOnClustersActual(types.NamespacedName{Namespace: nsName, Name: placementPolicyOtherConfigMapName}, allMemberClusters...), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove the second ConfigMap from the member clusters") + }) + + It("should place both ConfigMaps on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + otherSnapshotName = waitForPlacementPolicySynchronized(otherPolicyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("should fail to initialize a run whose strategy does not exist", func() { + createRunAndWaitForInitFailure(noStrategyRunKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: "pp-e2e-no-such-strategy", + }) + }) + + It("should fail to initialize a run whose policy does not exist", func() { + createRunAndWaitForInitFailure(noPolicyRunKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: "pp-e2e-no-such-policy", + StagedUpdateStrategyName: strategyKey.Name, + }) + }) + + It("should fail to initialize a run that names a resource snapshot of another policy", func() { + createRunAndWaitForInitFailure(foreignSnapshotRunKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + ResourceSnapshotName: otherSnapshotName, + }) + }) + + It("should fail to initialize a run whose strategy sorts clusters by a label they lack", func() { + createRunAndWaitForInitFailure(noSortingLabelRunKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: sortingStrategyKey.Name, + }) + }) + + It("should roll nothing out, with the binding manager role released", func() { + // None of the runs got as far as taking a new resource snapshot, so the change is still pending. + wantStatus := resourcesChangedPendingRolloutPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + snapshotsActual := placementPolicyBindingSnapshotsActual(policyKey, map[string]string{ + memberCluster1EastProdName: initialSnapshotName, + memberCluster2EastCanaryName: initialSnapshotName, + memberCluster3WestProdName: initialSnapshotName, + }) + dataActual := configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...) + unchangedActual := func() error { + if err := statusActual(); err != nil { + return err + } + if err := snapshotsActual(); err != nil { + return err + } + return dataActual() + } + Eventually(unchangedActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the change as still pending a rollout") + Consistently(unchangedActual, consistentlyDuration, consistentlyInterval). + Should(Succeed(), "A run that failed to initialize rolled something out") + }) + }) +}) diff --git a/test/e2e/placement_policy_staged_update_run_interruptions_test.go b/test/e2e/placement_policy_staged_update_run_interruptions_test.go new file mode 100644 index 000000000..c48a9778b --- /dev/null +++ b/test/e2e/placement_policy_staged_update_run_interruptions_test.go @@ -0,0 +1,635 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" +) + +const ( + placementPolicySuspendedNameTemplate = "pp-suspended-%d" + clusterPlacementPolicySuspendedNameTemplate = "cpp-suspended-%d" + placementPolicySuspendedStrategyNameTemplate = "pp-suspended-strategy-%d" + clusterPlacementPolicySuspendedStrategyNameTemplate = "cpp-suspended-strategy-%d" + placementPolicySuspendedRunNameTemplate = "pp-suspended-run-%d" + clusterPlacementPolicySuspendedRunNameTemplate = "cpp-suspended-run-%d" + + placementPolicyRunDeletedNameTemplate = "pp-run-deleted-%d" + placementPolicyRunDeletedStrategyNameTemplate = "pp-run-deleted-strategy-%d" + placementPolicyDeletedRunNameTemplate = "pp-deleted-run-%d" + placementPolicyFollowUpRunNameTemplate = "pp-follow-up-run-%d" + + placementPolicyHandMadeRequestNameTemplate = "pp-hand-made-request-%d" + placementPolicyHandMadeRequestStrategyNameTemplate = "pp-hand-made-request-strategy-%d" + placementPolicyHandMadeRequestRunNameTemplate = "pp-hand-made-request-run-%d" + + clusterPlacementPolicySuspendedMidRunNameTemplate = "cpp-suspended-mid-run-%d" + clusterPlacementPolicySuspendedMidRunStrategyNameTemplate = "cpp-suspended-mid-run-strategy-%d" + clusterPlacementPolicySuspendedMidRunRunNameTemplate = "cpp-suspended-mid-run-run-%d" +) + +// canaryApprovalStrategySpec returns a strategy with a canary stage, gated by an approval, before a prod stage. +func canaryApprovalStrategySpec() kfrolloutv1alpha1.StagedUpdateStrategySpec { + return kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyCanaryStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envCanary}}, + AfterStageTasks: []kfrolloutv1alpha1.StageTask{{Type: kfrolloutv1alpha1.StageTaskTypeApproval}}, + }, + { + Name: placementPolicyProdStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envProd}}, + }, + }, + } +} + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy staged update run interruptions", Label("placementpolicy"), func() { + // A suspended run gives up the binding manager role, so the two scopes run the same journey through the + // namespaced and the cluster-scoped APIs. + for _, clusterScoped := range []bool{false, true} { + scopeDescription := "placement policy" + policyNameTemplate, strategyNameTemplate, runNameTemplate := placementPolicySuspendedNameTemplate, placementPolicySuspendedStrategyNameTemplate, placementPolicySuspendedRunNameTemplate + if clusterScoped { + scopeDescription = "cluster placement policy" + policyNameTemplate, strategyNameTemplate, runNameTemplate = clusterPlacementPolicySuspendedNameTemplate, clusterPlacementPolicySuspendedStrategyNameTemplate, clusterPlacementPolicySuspendedRunNameTemplate + } + + Context(fmt.Sprintf("%s rolled out by a staged update run that is suspended and resumed", scopeDescription), Ordered, func() { + objNamespace := "" + if !clusterScoped { + objNamespace = appNamespace().Name + } + policyKey := types.NamespacedName{Namespace: objNamespace, Name: fmt.Sprintf(policyNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Namespace: objNamespace, Name: fmt.Sprintf(strategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Namespace: objNamespace, Name: fmt.Sprintf(runNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName, newSnapshotName string + var approvalRequestKey types.NamespacedName + + setSuspended := func(suspended bool) { + updateStagedUpdateRun(runKey, func(run kfrolloutv1alpha1.StagedUpdateRunAccessor) { + run.GetSpec().Suspended = suspended + }) + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create %s %s", scopeDescription, policyKey) + + By("creating a strategy with a canary stage, gated by an approval, before a prod stage") + strategy := newStagedUpdateStrategyObject(strategyKey, canaryApprovalStrategySpec()) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("can create a staged update run that is suspended", func() { + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + Suspended: true, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", runKey) + }) + + It("should initialize the run with a new resource snapshot, then give up the binding manager role", func() { + Eventually(stagedUpdateRunConditionActual(runKey, kfrolloutv1alpha1.StagedUpdateRunCondTypeInitialized, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunInitializedCondReasonPreppedResourceSnapshotAndAllStages), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to initialize staged update run %s", runKey) + Eventually(stagedUpdateRunConditionActual(runKey, kfrolloutv1alpha1.StagedUpdateRunCondTypeSuspended, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunSuspendedCondReasonSuspended), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to mark staged update run %s as suspended", runKey) + newSnapshotName = resourceSnapshotNameToRolloutOf(runKey) + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Staged update run %s rolls out the initial resource snapshot", runKey) + Eventually(placementPolicyBindingManagerActual(policyKey, nil), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to give up the binding manager role") + }) + + It("should not roll anything out while the run is suspended", func() { + snapshotsActual := placementPolicyBindingSnapshotsActual(policyKey, map[string]string{ + memberCluster1EastProdName: initialSnapshotName, + memberCluster2EastCanaryName: initialSnapshotName, + memberCluster3WestProdName: initialSnapshotName, + }) + dataActual := configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...) + Consistently(func() error { + if err := snapshotsActual(); err != nil { + return err + } + if err := dataActual(); err != nil { + return err + } + run, err := retrieveStagedUpdateRun(runKey) + if err != nil { + return err + } + if cond := meta.FindStatusCondition(run.GetStatus().Conditions, kfrolloutv1alpha1.StagedUpdateRunCondTypeStarted); cond != nil { + return fmt.Errorf("suspended staged update run %s has started (condition: %+v)", runKey, cond) + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The suspended staged update run rolled something out") + }) + + It("can resume the run", func() { + setSuspended(false) + }) + + It("should mark the run as resumed, roll the new version out to the canary cluster, and wait for an approval", func() { + Eventually(stagedUpdateRunConditionActual(runKey, kfrolloutv1alpha1.StagedUpdateRunCondTypeSuspended, metav1.ConditionFalse, + kfrolloutv1alpha1.StagedUpdateRunSuspendedCondReasonResumed), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to mark staged update run %s as resumed", runKey) + approvalRequestKey = waitForStageApprovalRequest(runKey, 0, false) + Eventually(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new version out to the canary cluster only") + Eventually(placementPolicyBindingManagerActual(policyKey, &kfplacementv1alpha1.BindingManager{ + ControllerName: stagedUpdateRunBindingManagerName, + ObjectRefs: []kfplacementv1alpha1.ObjectReference{stagedUpdateRunObjectReference(runKey)}, + }), eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to hold the binding manager role again") + }) + + It("can suspend the run while it waits for the approval", func() { + setSuspended(true) + Eventually(stagedUpdateRunConditionActual(runKey, kfrolloutv1alpha1.StagedUpdateRunCondTypeSuspended, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunSuspendedCondReasonSuspended), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to mark staged update run %s as suspended again", runKey) + Eventually(placementPolicyBindingManagerActual(policyKey, nil), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to give up the binding manager role again") + }) + + It("can approve the canary stage", func() { + approvePlacementPolicyApprovalRequest(approvalRequestKey) + }) + + It("should not accept the approval or move on to the prod stage while the run is suspended", func() { + stateActual := onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data) + Consistently(func() error { + if err := stateActual(); err != nil { + return err + } + run, err := retrieveStagedUpdateRun(runKey) + if err != nil { + return err + } + task := run.GetStatus().Stages[0].AfterStageTasks[0] + if cond := meta.FindStatusCondition(task.Conditions, kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestApproved); cond != nil { + return fmt.Errorf("suspended staged update run %s has taken the approval (condition: %+v)", runKey, cond) + } + request, err := retrieveApprovalRequest(approvalRequestKey) + if err != nil { + return err + } + if cond := meta.FindStatusCondition(request.GetStatus().Conditions, kfrolloutv1alpha1.ApprovalRequestCondTypeApprovalAccepted); cond != nil { + return fmt.Errorf("approval request %s was accepted while the run is suspended (condition: %+v)", approvalRequestKey, cond) + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The suspended staged update run acted on the approval") + }) + + It("can resume the run again", func() { + setSuspended(false) + }) + + It("should accept the approval and complete the run", func() { + Eventually(approvalRequestActual(approvalRequestKey, runKey, placementPolicyCanaryStageName, true), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to accept the approval") + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s", runKey) + Expect(stagedUpdateRunConditionActual(runKey, kfrolloutv1alpha1.StagedUpdateRunCondTypeSuspended, metav1.ConditionFalse, + kfrolloutv1alpha1.StagedUpdateRunSuspendedCondReasonResumed)()).To(Succeed(), "Staged update run %s is not marked as resumed", runKey) + }) + + It("should place the new version on every member cluster, with the binding manager role released", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new resource snapshot out to every binding") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected %s status", scopeDescription) + }) + }) + } + + Context("placement policy whose staged update run is deleted while it waits for an approval", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyRunDeletedNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyRunDeletedStrategyNameTemplate, GinkgoParallelProcess())} + deletedRunKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyDeletedRunNameTemplate, GinkgoParallelProcess())} + followUpRunKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyFollowUpRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName, newSnapshotName string + var deletedRunApprovalRequestKey types.NamespacedName + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + + By("creating a strategy with a canary stage, gated by an approval, before a prod stage") + strategy := newStagedUpdateStrategyObject(strategyKey, canaryApprovalStrategySpec()) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(followUpRunKey) + ensureStagedUpdateRunRemoved(deletedRunKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("can create a staged update run", func() { + run := newStagedUpdateRunObject(deletedRunKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", deletedRunKey) + }) + + It("should roll the new version out to the canary cluster, then wait for an approval", func() { + deletedRunApprovalRequestKey = waitForStageApprovalRequest(deletedRunKey, 0, false) + newSnapshotName = resourceSnapshotNameToRolloutOf(deletedRunKey) + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Staged update run %s rolls out the initial resource snapshot", deletedRunKey) + Eventually(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new version out to the canary cluster only") + }) + + It("can delete the run", func() { + ensureStagedUpdateRunRemoved(deletedRunKey) + }) + + It("should remove the approval request and release the binding manager role", func() { + Eventually(approvalRequestAbsentActual(deletedRunApprovalRequestKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the approval request of the deleted run") + Eventually(placementPolicyBindingManagerActual(policyKey, nil), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to release the binding manager role") + }) + + It("should keep the clusters where the deleted run left them", func() { + wantStatus := someClustersOnOlderResourceRevisionPlacementPolicyStatus(int32(len(allMemberClusters)), 1) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the canary cluster alone as on the latest resource revision") + Consistently(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The clusters moved after the run was deleted") + }) + + It("can finish the rollout with a new staged update run", func() { + run := newStagedUpdateRunObject(followUpRunKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", followUpRunKey) + + // The hub has not changed since, so the new run rolls out the same resource snapshot, and the canary + // stage, already up to date, goes straight to its approval. + approvePlacementPolicyApprovalRequest(waitForStageApprovalRequest(followUpRunKey, 0, false)) + Eventually(stagedUpdateRunSucceededActual(followUpRunKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s", followUpRunKey) + Expect(resourceSnapshotNameToRolloutOf(followUpRunKey)).To(Equal(newSnapshotName), "Staged update run %s rolls out another resource snapshot", followUpRunKey) + }) + + It("should place the new version on every member cluster, with the binding manager role released", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new resource snapshot out to every binding") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("placement policy whose staged update run finds a hand-made approval request under the name it uses", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyHandMadeRequestNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyHandMadeRequestStrategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyHandMadeRequestRunNameTemplate, GinkgoParallelProcess())} + // The name that the run gives the approval request of the canary stage's after-stage task; see + // formatAfterStageApprovalRequestName in the staged update run controller (short names are not hashed). + requestKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf("%s-after-stage-%s", runKey.Name, placementPolicyCanaryStageName)} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName, newSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + + By("creating a strategy with a canary stage, gated by an approval, before a prod stage") + strategy := newStagedUpdateStrategyObject(strategyKey, canaryApprovalStrategySpec()) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(runKey) + // The hand-made request outlives the run, should the spec fail before deleting it. + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, newApprovalRequestObject(requestKey)))).To(Succeed(), "Failed to delete approval request %s", requestKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can create, and approve, an approval request under the name that the run will use for the canary stage", func() { + request := newApprovalRequestObject(requestKey) + request.SetSpec(kfrolloutv1alpha1.ApprovalRequestSpec{ + StagedUpdateRunName: runKey.Name, + StageName: placementPolicyCanaryStageName, + }) + Expect(hubClient.Create(ctx, request)).To(Succeed(), "Failed to create approval request %s", requestKey) + approvePlacementPolicyApprovalRequest(requestKey) + }) + + It("can change the ConfigMap on the hub, and roll the change out with a staged update run", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", runKey) + }) + + It("should roll the new version out to the canary cluster, then report that it does not own the approval request", func() { + Eventually(stagedUpdateRunApprovalTaskActual(runKey, 0, false, requestKey.Name, + kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated, metav1.ConditionFalse, + kfrolloutv1alpha1.StagedUpdateRunTaskApprovalRequestCreatedCondReasonOwnershipMismatch), + placementPolicyRolloutEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the approval request as not owned by the run") + newSnapshotName = resourceSnapshotNameToRolloutOf(runKey) + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Staged update run %s rolls out the initial resource snapshot", runKey) + Eventually(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new version out to the canary cluster only") + }) + + It("should not roll out to the prod clusters, although the hand-made request is approved", func() { + Consistently(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The new version reached the prod clusters through an approval request that the run does not own") + }) + + It("can delete the hand-made approval request", func() { + Expect(hubClient.Delete(ctx, newApprovalRequestObject(requestKey))).To(Succeed(), "Failed to delete approval request %s", requestKey) + }) + + It("should create its own approval request, which needs an approval as usual", func() { + Eventually(approvalRequestActual(requestKey, runKey, placementPolicyCanaryStageName, false), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to create an approval request controlled by the run") + Eventually(stagedUpdateRunApprovalTaskActual(runKey, 0, false, requestKey.Name, + kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunTaskApprovalRequestCreatedCondReasonCreated), + placementPolicyRolloutEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the approval request as created") + Consistently(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The new version reached the prod clusters before the approval") + }) + + It("can approve the canary stage", func() { + approvePlacementPolicyApprovalRequest(requestKey) + }) + + It("should roll the new version out to every member cluster", func() { + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s", runKey) + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("cluster placement policy whose binding is suspended by hand while a staged update run is in progress", Ordered, func() { + policyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicySuspendedMidRunNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicySuspendedMidRunStrategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicySuspendedMidRunRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + // A prod cluster, which the run reaches only after the approval. + suspendedClusterName := memberCluster1EastProdName + updatedClusterNames := []string{memberCluster2EastCanaryName, memberCluster3WestProdName} + // The member clusters are known only once the suite is set up, after the spec tree is built. + clusterCount := func() int32 { return int32(len(allMemberClusterNames)) } + + var initialSnapshotName, newSnapshotName string + var bindingUIDsByCluster map[string]types.UID + var approvalRequestKey types.NamespacedName + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + + By("creating a strategy with a canary stage, gated by an approval, before a prod stage") + strategy := newStagedUpdateStrategyObject(strategyKey, canaryApprovalStrategySpec()) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create cluster staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + bindingUIDsByCluster = placementPolicyBindingUIDsByCluster(policyKey) + }) + + It("can change the ConfigMap on the hub, and roll the change out with a cluster staged update run", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create cluster staged update run %s", runKey) + }) + + It("should roll the new version out to the canary cluster, then wait for an approval", func() { + approvalRequestKey = waitForStageApprovalRequest(runKey, 0, false) + newSnapshotName = resourceSnapshotNameToRolloutOf(runKey) + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Cluster staged update run %s rolls out the initial resource snapshot", runKey) + Eventually(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new version out to the canary cluster only") + }) + + It("can suspend the binding for member cluster 1", func() { + setPlacementPolicyBindingSuspended(policyKey, suspendedClusterName, true) + }) + + It("should remove the ConfigMap from member cluster 1", func() { + Eventually(suspendedPlacementPolicyBindingActual(policyKey, suspendedClusterName, bindingUIDsByCluster[suspendedClusterName], initialSnapshotName), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to process the suspension of the binding") + Eventually(appConfigMapRemovedFromClustersActual(memberCluster1EastProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from member cluster 1") + }) + + It("can approve the canary stage", func() { + approvePlacementPolicyApprovalRequest(approvalRequestKey) + }) + + It("should complete the run, counting the suspended cluster as updated", func() { + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete cluster staged update run %s", runKey) + Expect(stagedUpdateRunClusterResultsActual(runKey, map[string]string{ + memberCluster2EastCanaryName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + memberCluster1EastProdName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + memberCluster3WestProdName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + })()).To(Succeed(), "Cluster staged update run %s did not update every cluster", runKey) + + // The run skips the wait on a suspended binding; only the message of the cluster's Completed condition tells + // that path apart. + run, err := retrieveStagedUpdateRun(runKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get cluster staged update run %s", runKey) + var completedCond *metav1.Condition + for _, stage := range run.GetStatus().Stages { + for _, cluster := range stage.Clusters { + if cluster.ClusterName == suspendedClusterName { + completedCond = meta.FindStatusCondition(cluster.Conditions, kfrolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted) + } + } + } + Expect(completedCond).NotTo(BeNil(), "Cluster staged update run %s reports no Completed condition for member cluster 1", runKey) + Expect(completedCond.Message).To(ContainSubstring("binding is suspended"), "The Completed condition of member cluster 1 does not report the binding as suspended") + }) + + It("should have moved the suspended binding to the new resource snapshot, without placing it on member cluster 1", func() { + suspendedActual := suspendedPlacementPolicyBindingActual(policyKey, suspendedClusterName, bindingUIDsByCluster[suspendedClusterName], newSnapshotName) + Eventually(suspendedActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to move the suspended binding to the new resource snapshot") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, memberCluster2EastCanary, memberCluster3WestProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on the other member clusters") + + // The suspended binding names the latest revision but has processed none, so it counts as neither + // synchronized nor on an older revision. + wantStatus := someClustersNotSynchronizedPlacementPolicyStatus(clusterCount(), clusterCount()-1) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + statusActual := placementPolicyStatusActual(policyKey, wantStatus) + Eventually(statusActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + + worksActual := placementPolicyWorkClustersActual(policyKey, updatedClusterNames...) + removedActual := appConfigMapRemovedFromClustersActual(memberCluster1EastProd) + Consistently(func() error { + for _, actual := range []func() error{suspendedActual, worksActual, removedActual, statusActual} { + if err := actual(); err != nil { + return err + } + } + return nil + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The new version reached member cluster 1, or the suspended binding changed") + }) + + It("can resume the binding for member cluster 1", func() { + setPlacementPolicyBindingSuspended(policyKey, suspendedClusterName, false) + }) + + It("should place the new version on member cluster 1", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to synchronize every binding on the new resource snapshot") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + wantStatus := allSynchronizedPlacementPolicyStatus(clusterCount()) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) +}) diff --git a/test/e2e/placement_policy_staged_update_run_stages_test.go b/test/e2e/placement_policy_staged_update_run_stages_test.go new file mode 100644 index 000000000..0d9d2529a --- /dev/null +++ b/test/e2e/placement_policy_staged_update_run_stages_test.go @@ -0,0 +1,521 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "slices" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" +) + +const ( + placementPolicyTimedWaitNameTemplate = "pp-timed-wait-%d" + placementPolicyTimedWaitStrategyNameTemplate = "pp-timed-wait-strategy-%d" + placementPolicyTimedWaitRunNameTemplate = "pp-timed-wait-run-%d" + + clusterPlacementPolicyBeforeStageApprovalNameTemplate = "cpp-before-stage-approval-%d" + clusterPlacementPolicyBeforeStageApprovalStrategyNameTemplate = "cpp-before-stage-approval-strategy-%d" + clusterPlacementPolicyBeforeStageApprovalRunNameTemplate = "cpp-before-stage-approval-run-%d" + + clusterPlacementPolicyPartialStrategyNameTemplate = "cpp-partial-strategy-%d" + clusterPlacementPolicyPartialStrategyStrategyNameTemplate = "cpp-partial-strategy-strategy-%d" + clusterPlacementPolicyPartialStrategyRunNameTemplate = "cpp-partial-strategy-run-%d" + + clusterPlacementPolicyConcurrencyNameTemplate = "cpp-concurrency-%d" + clusterPlacementPolicyConcurrencyStrategyNameTemplate = "cpp-concurrency-%d-strategy-%d" + clusterPlacementPolicyConcurrencyRunNameTemplate = "cpp-concurrency-%d-run-%d" + + placementPolicyEastStageName = "east" + + // placementPolicyTimedWaitDuration is the wait time of the timed wait task in the specs; it is long enough to + // observe the hold on the prod stage, and short enough to keep the lane fast. + placementPolicyTimedWaitDuration = 30 * time.Second +) + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy staged update run stages", Label("placementpolicy"), func() { + Context("placement policy rolled out with a timed wait after the canary stage", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyTimedWaitNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyTimedWaitStrategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyTimedWaitRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + prodClusters := []string{memberCluster1EastProdName, memberCluster3WestProdName} + + var initialSnapshotName, newSnapshotName string + var bindingNamesByCluster map[string]string + // waitStartedAt is when the timed wait started, as the run records it (to the second). + var waitStartedAt time.Time + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + + By("creating a strategy with a timed wait between the canary and prod stages") + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyCanaryStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envCanary}}, + AfterStageTasks: []kfrolloutv1alpha1.StageTask{ + {Type: kfrolloutv1alpha1.StageTaskTypeTimedWait, WaitTime: &metav1.Duration{Duration: placementPolicyTimedWaitDuration}}, + }, + }, + { + Name: placementPolicyProdStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envProd}}, + }, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + bindingNamesByCluster = placementPolicyBindingNamesByCluster(policyKey) + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("can create a staged update run", func() { + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", runKey) + }) + + It("should roll the new version out to the canary cluster, then start the timed wait", func() { + Eventually(func() error { + run, err := retrieveStagedUpdateRun(runKey) + if err != nil { + return err + } + stages := run.GetStatus().Stages + if len(stages) == 0 || len(stages[0].AfterStageTasks) == 0 { + return fmt.Errorf("staged update run %s has no after-stage task status for the canary stage yet", runKey) + } + cond := meta.FindStatusCondition(stages[0].AfterStageTasks[0].Conditions, kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeTimedWaitStarted) + if cond == nil || cond.Status != metav1.ConditionTrue { + return fmt.Errorf("staged update run %s has not started the timed wait yet (task status: %+v)", runKey, stages[0].AfterStageTasks[0]) + } + waitStartedAt = cond.LastTransitionTime.Time + return nil + }, placementPolicyRolloutEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to start the timed wait") + newSnapshotName = resourceSnapshotNameToRolloutOf(runKey) + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Staged update run %s rolls out the initial resource snapshot", runKey) + + Eventually(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new version out to the canary cluster only") + }) + + It("should not update the prod clusters before the wait time has elapsed", func() { + // Hold until shortly before the wait ends, leaving a margin for the run to act on time. + if holdFor := time.Until(waitStartedAt.Add(placementPolicyTimedWaitDuration - 5*time.Second)); holdFor > 0 { + Consistently(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), holdFor, consistentlyInterval). + Should(Succeed(), "The new version reached the prod clusters before the wait time elapsed") + } + }) + + It("should complete the run once the wait time has elapsed", func() { + wantStatus := kfrolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: succeededStagedUpdateRunConditions(), + ResourceSnapshotNameToRollout: newSnapshotName, + Stages: []kfrolloutv1alpha1.PerStageStatus{ + { + StageName: placementPolicyCanaryStageName, + Clusters: []kfrolloutv1alpha1.PerClusterStatus{ + succeededStagedUpdateRunClusterStatus(memberCluster2EastCanaryName, bindingNamesByCluster[memberCluster2EastCanaryName]), + }, + AfterStageTasks: []kfrolloutv1alpha1.PerStageTaskStatus{ + { + Type: kfrolloutv1alpha1.StageTaskTypeTimedWait, + WaitTime: &metav1.Duration{Duration: placementPolicyTimedWaitDuration}, + Conditions: []metav1.Condition{ + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeTimedWaitStarted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunTaskTimedWaitStartedCondReasonTimerStarted), + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeWaitTimeElapsed, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunTaskWaitTimeElapsedCondReasonTimerElapsed), + }, + }, + }, + Conditions: succeededStagedUpdateRunStageConditions(), + StartedTimestamp: &metav1.Time{}, + CompletedTimestamp: &metav1.Time{}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: placementPolicyProdStageName, + Clusters: []kfrolloutv1alpha1.PerClusterStatus{ + succeededStagedUpdateRunClusterStatus(prodClusters[0], bindingNamesByCluster[prodClusters[0]]), + succeededStagedUpdateRunClusterStatus(prodClusters[1], bindingNamesByCluster[prodClusters[1]]), + }, + Conditions: succeededStagedUpdateRunStageConditions(), + StartedTimestamp: &metav1.Time{}, + CompletedTimestamp: &metav1.Time{}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + Eventually(stagedUpdateRunStatusActual(runKey, wantStatus), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s", runKey) + + // The run measures the wait from the start time it recorded, which is kept to the second, so it may move + // on up to a second before the full wait time has passed. + run, err := retrieveStagedUpdateRun(runKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get staged update run %s", runKey) + prodStartedAt := run.GetStatus().Stages[1].StartedTimestamp.Time + Expect(prodStartedAt.Sub(waitStartedAt)).To(BeNumerically(">=", placementPolicyTimedWaitDuration-time.Second), + "The prod stage started at %v, before the timed wait that started at %v elapsed", prodStartedAt, waitStartedAt) + }) + + It("should place the new version on every member cluster", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new resource snapshot out to every binding") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + }) + }) + + Context("cluster placement policy rolled out with an approval before the prod stage", Ordered, func() { + policyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyBeforeStageApprovalNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyBeforeStageApprovalStrategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyBeforeStageApprovalRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName, newSnapshotName string + var approvalRequestKey types.NamespacedName + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + + By("creating a strategy whose prod stage waits for an approval before it starts") + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyCanaryStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envCanary}}, + }, + { + Name: placementPolicyProdStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envProd}}, + BeforeStageTasks: []kfrolloutv1alpha1.StageTask{{Type: kfrolloutv1alpha1.StageTaskTypeApproval}}, + }, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create cluster staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("can create a cluster staged update run", func() { + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create cluster staged update run %s", runKey) + }) + + It("should roll the new version out to the canary cluster, then ask for an approval before the prod stage", func() { + approvalRequestKey = waitForStageApprovalRequest(runKey, 1, true) + Eventually(approvalRequestActual(approvalRequestKey, runKey, placementPolicyProdStageName, false), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to find the approval request for the prod stage") + newSnapshotName = resourceSnapshotNameToRolloutOf(runKey) + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Cluster staged update run %s rolls out the initial resource snapshot", runKey) + + Eventually(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new version out to the canary cluster only") + }) + + It("should not update the prod clusters before the approval", func() { + Consistently(onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The new version reached the prod clusters before the approval") + }) + + It("can approve the prod stage", func() { + approvePlacementPolicyApprovalRequest(approvalRequestKey) + }) + + It("should complete the run and place the new version on every member cluster", func() { + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete cluster staged update run %s", runKey) + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new resource snapshot out to every binding") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + }) + }) + + Context("cluster placement policy rolled out by a strategy whose stages leave out a cluster", Ordered, func() { + policyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyPartialStrategyNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyPartialStrategyStrategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyPartialStrategyRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName, newSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + + By("creating a strategy whose stages cover the east region only") + // A cluster belongs to the first stage that selects it, so the east stage gets cluster-1 alone; cluster-3, + // in the west, is in no stage at all. + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyCanaryStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envCanary}}, + }, + { + Name: placementPolicyEastStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{regionLabelName: regionEast}}, + }, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create cluster staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("can roll the change out with a cluster staged update run", func() { + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create cluster staged update run %s", runKey) + }) + + It("should complete the run, updating only the clusters in its stages", func() { + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete cluster staged update run %s", runKey) + Expect(stagedUpdateRunClusterResultsActual(runKey, map[string]string{ + memberCluster2EastCanaryName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + memberCluster1EastProdName: kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded, + })()).To(Succeed(), "Cluster staged update run %s did not update exactly the clusters in its stages", runKey) + newSnapshotName = resourceSnapshotNameToRolloutOf(runKey) + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Cluster staged update run %s rolled out the initial resource snapshot", runKey) + }) + + It("should leave the cluster outside every stage on the initial version", func() { + Eventually(placementPolicyBindingSnapshotsActual(policyKey, map[string]string{ + memberCluster2EastCanaryName: newSnapshotName, + memberCluster1EastProdName: newSnapshotName, + memberCluster3WestProdName: initialSnapshotName, + }), workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to leave cluster-3 on the initial resource snapshot") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on the east clusters") + Expect(configMapDataOnClustersActual(cmKey, v0Data, memberCluster3WestProd)()).To(Succeed(), "The new version reached cluster-3") + }) + + It("should report that cluster as on an older resource revision, with the binding manager role released", func() { + wantStatus := someClustersOnOlderResourceRevisionPlacementPolicyStatus(int32(len(allMemberClusters)), 2) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) + + Context("cluster placement policy rolled out with limits on how many clusters a stage updates at once", Ordered, func() { + policyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyConcurrencyNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + // Each case rolls out a change with its own strategy, whose single stage covers all three member clusters. + cases := []struct { + description string + maxConcurrency intstr.IntOrString + wantResolved int32 + data map[string]string + }{ + {description: "3", maxConcurrency: intstr.FromInt32(3), wantResolved: 3, data: map[string]string{"data": "test-v1"}}, + // A percentage of the stage's clusters is rounded down: 70% of 3 is 2. + {description: "70%", maxConcurrency: intstr.FromString("70%"), wantResolved: 2, data: map[string]string{"data": "test-v2"}}, + } + strategyKey := func(i int) types.NamespacedName { + return types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyConcurrencyStrategyNameTemplate, GinkgoParallelProcess(), i)} + } + runKey := func(i int) types.NamespacedName { + return types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyConcurrencyRunNameTemplate, GinkgoParallelProcess(), i)} + } + + var bindingNamesByCluster map[string]string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + + By("creating a strategy for each concurrency limit, with a single stage that covers every cluster") + for i, c := range cases { + strategy := newStagedUpdateStrategyObject(strategyKey(i), kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyAllClustersStageName, + LabelSelector: &metav1.LabelSelector{}, + MaxConcurrency: ptr.To(c.maxConcurrency), + }, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create cluster staged update strategy %s", strategyKey(i)) + } + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + for i := range cases { + ensureStagedUpdateRunRemoved(runKey(i)) + ensureStagedUpdateStrategyRemoved(strategyKey(i)) + } + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + waitForPlacementPolicySynchronized(policyKey, noTermsSelectorsForAllMemberClusters(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + bindingNamesByCluster = placementPolicyBindingNamesByCluster(policyKey) + }) + + for i, c := range cases { + It(fmt.Sprintf("can change the ConfigMap and roll the change out with a stage that updates at most %s of its clusters at once", c.description), func() { + updateConfigMapDataOnHub(cmKey, c.data) + run := newStagedUpdateRunObject(runKey(i), kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey(i).Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create cluster staged update run %s", runKey(i)) + }) + + It(fmt.Sprintf("should resolve the limit of %s to %d clusters, and update every cluster", c.description, c.wantResolved), func() { + Eventually(stagedUpdateRunSucceededActual(runKey(i)), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete cluster staged update run %s", runKey(i)) + + // Clusters in a stage with no sorting label key are ordered by name. + clusters := make([]kfrolloutv1alpha1.PerClusterStatus, 0, len(allMemberClusterNames)) + for _, clusterName := range slices.Sorted(slices.Values(allMemberClusterNames)) { + clusters = append(clusters, succeededStagedUpdateRunClusterStatus(clusterName, bindingNamesByCluster[clusterName])) + } + // Timestamps are ignored in the comparison; non-nil wanted values check that they are set. + wantStatus := kfrolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: succeededStagedUpdateRunConditions(), + ResourceSnapshotNameToRollout: resourceSnapshotNameToRolloutOf(runKey(i)), + Stages: []kfrolloutv1alpha1.PerStageStatus{ + { + StageName: placementPolicyAllClustersStageName, + Clusters: clusters, + Conditions: succeededStagedUpdateRunStageConditions(), + StartedTimestamp: &metav1.Time{}, + CompletedTimestamp: &metav1.Time{}, + MaxConcurrency: ptr.To(c.wantResolved), + }, + }, + } + Eventually(stagedUpdateRunStatusActual(runKey(i), wantStatus), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Cluster staged update run %s does not report the expected status", runKey(i)) + Eventually(configMapDataOnClustersActual(cmKey, c.data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + }) + } + }) +}) diff --git a/test/e2e/placement_policy_staged_update_run_test.go b/test/e2e/placement_policy_staged_update_run_test.go new file mode 100644 index 000000000..1a4957632 --- /dev/null +++ b/test/e2e/placement_policy_staged_update_run_test.go @@ -0,0 +1,793 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + "slices" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" +) + +const ( + placementPolicyStagedNameTemplate = "pp-staged-%d" + clusterPlacementPolicyStagedNameTemplate = "cpp-staged-%d" + placementPolicyStrategyNameTemplate = "pp-strategy-%d" + clusterPlacementPolicyStrategyNameTemplate = "cpp-strategy-%d" + placementPolicyRunNameTemplate = "pp-run-%d" + clusterPlacementPolicyRunNameTemplate = "cpp-run-%d" + + placementPolicyRollbackNameTemplate = "pp-rollback-%d" + placementPolicyRollbackStrategyNameTemplate = "pp-rollback-strategy-%d" + placementPolicyRollForwardRunNameTemplate = "pp-roll-forward-run-%d" + placementPolicyRollBackRunNameTemplate = "pp-roll-back-run-%d" + + clusterPlacementPolicyMutualExclusionNameTemplate = "cpp-mutual-exclusion-%d" + clusterPlacementPolicyMutualExclusionStrategyNameTemplate = "cpp-mutual-exclusion-strategy-%d" + clusterPlacementPolicyMutualExclusionRunNameTemplate = "cpp-mutual-exclusion-run-%d" + + placementPolicyGeneratedStrategyNameTemplate = "pp-generated-strategy-%d" + placementPolicyGeneratedRunNameTemplate = "pp-generated-run-%d" + + placementPolicyCanaryStageName = "canary" + placementPolicyProdStageName = "prod" + placementPolicyAllClustersStageName = "all" +) + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy staged update runs", Label("placementpolicy"), func() { + // The two scopes run the same journey through different APIs: a PlacementPolicy with a StagedUpdateRun, + // StagedUpdateStrategy, and ApprovalRequest in its namespace, and a ClusterPlacementPolicy with their + // cluster-scoped counterparts. + for _, clusterScoped := range []bool{false, true} { + scopeDescription := "placement policy" + policyNameTemplate, strategyNameTemplate, runNameTemplate := placementPolicyStagedNameTemplate, placementPolicyStrategyNameTemplate, placementPolicyRunNameTemplate + if clusterScoped { + scopeDescription = "cluster placement policy" + policyNameTemplate, strategyNameTemplate, runNameTemplate = clusterPlacementPolicyStagedNameTemplate, clusterPlacementPolicyStrategyNameTemplate, clusterPlacementPolicyRunNameTemplate + } + + Context(fmt.Sprintf("%s rolled out by a staged update run, with an approval between the canary and prod stages", scopeDescription), Ordered, func() { + // Objects of the namespaced APIs live in the app namespace; the cluster-scoped ones have no namespace. + objNamespace := "" + if !clusterScoped { + objNamespace = appNamespace().Name + } + policyKey := types.NamespacedName{Namespace: objNamespace, Name: fmt.Sprintf(policyNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Namespace: objNamespace, Name: fmt.Sprintf(strategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Namespace: objNamespace, Name: fmt.Sprintf(runNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + prodClusters := []string{memberCluster1EastProdName, memberCluster3WestProdName} + + var initialSnapshotName, newSnapshotName string + var approvalRequestKey types.NamespacedName + // The binding of each cluster, as recorded in the run status. + bindingNamesByCluster := map[string]string{} + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create %s %s", scopeDescription, policyKey) + + By("creating a strategy with a canary stage, gated by an approval, before a prod stage") + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyCanaryStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envCanary}}, + AfterStageTasks: []kfrolloutv1alpha1.StageTask{{Type: kfrolloutv1alpha1.StageTaskTypeApproval}}, + }, + { + Name: placementPolicyProdStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: map[string]string{envLabelName: envProd}}, + }, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // The run goes first, so that it gives up the binding manager role before the policy goes. + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + wantSelectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantSelectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster, 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get %s %s", scopeDescription, policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of %s %s", scopeDescription, policyKey) + initialSnapshotName = snapshot.GetName() + bindings, err := placementPolicyBindings(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to list the bindings of %s %s", scopeDescription, policyKey) + for _, binding := range bindings { + bindingNamesByCluster[binding.GetSpec().ClusterName] = binding.GetName() + } + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("can create a staged update run that names no resource snapshot", func() { + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", runKey) + }) + + It("should initialize the run with a new resource snapshot of the changed ConfigMap", func() { + Eventually(func() error { + run, err := retrieveStagedUpdateRun(runKey) + if err != nil { + return err + } + if !meta.IsStatusConditionTrue(run.GetStatus().Conditions, kfrolloutv1alpha1.StagedUpdateRunCondTypeInitialized) { + return fmt.Errorf("staged update run %s is not initialized yet (conditions: %+v)", runKey, run.GetStatus().Conditions) + } + policy, err := retrievePlacementPolicy(policyKey) + if err != nil { + return err + } + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + if err != nil { + return err + } + switch { + case snapshot.GetName() == initialSnapshotName: + return fmt.Errorf("no new resource snapshot has been taken yet") + case run.GetStatus().ResourceSnapshotNameToRollout != snapshot.GetName(): + return fmt.Errorf("staged update run %s rolls out resource snapshot %q, want the latest one %q", + runKey, run.GetStatus().ResourceSnapshotNameToRollout, snapshot.GetName()) + case ptr.Deref(policy.GetStatus().LatestResourceRevisionName, "") != snapshot.GetName(): + return fmt.Errorf("%s %s reports latest resource revision %v, want %q", + scopeDescription, policyKey, policy.GetStatus().LatestResourceRevisionName, snapshot.GetName()) + } + newSnapshotName = snapshot.GetName() + return nil + }, placementPolicyRolloutEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to initialize the staged update run") + }) + + // prodOnInitialRevisionActual checks that the canary cluster has the new revision while the prod clusters + // still have the initial one. + prodOnInitialRevisionActual := func() func() error { + wantBindings := map[string]wantPlacementBinding{ + memberCluster2EastCanaryName: { + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: newSnapshotName, + status: synchronizedPlacementBindingStatus(1), + }, + } + for _, clusterName := range prodClusters { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: initialSnapshotName, + status: synchronizedPlacementBindingStatus(1), + } + } + bindingsActual := placementPolicyBindingsActual(policyKey, wantBindings) + canaryActual := configMapDataOnClustersActual(cmKey, v1Data, memberCluster2EastCanary) + prodActual := configMapDataOnClustersActual(cmKey, v0Data, memberCluster1EastProd, memberCluster3WestProd) + return func() error { + if err := bindingsActual(); err != nil { + return err + } + if err := canaryActual(); err != nil { + return err + } + return prodActual() + } + } + + It("should roll out the new version to the canary cluster only", func() { + Eventually(prodOnInitialRevisionActual(), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll out the new version to the canary cluster only") + }) + + It("should report the prod clusters as on an older resource revision, with the run holding the binding manager role", func() { + clusterCount := int32(len(allMemberClusters)) + wantStatus := kfplacementv1alpha1.PlacementPolicyStatus{ + Conditions: []metav1.Condition{ + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision), + // Availability is judged only on clusters that have the latest revision synchronized. + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable), + }, + LatestResourceRevisionName: ptr.To(newSnapshotName), + DesiredClusters: ptr.To(clusterCount), + ScheduledClusters: ptr.To(clusterCount), + SynchronizedClusters: ptr.To(int32(1)), + ResourcesAvailableClusters: ptr.To(int32(1)), + ActiveClusterClaims: ptr.To(int32(0)), + BindingManager: &kfplacementv1alpha1.BindingManager{ + ControllerName: stagedUpdateRunBindingManagerName, + ObjectRefs: []kfplacementv1alpha1.ObjectReference{stagedUpdateRunObjectReference(runKey)}, + }, + } + Eventually(placementPolicyStatusActual(policyKey, wantStatus), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected %s status", scopeDescription) + }) + + It("should create an approval request for the canary stage, controlled by the run", func() { + approvalRequestKey = waitForStageApprovalRequest(runKey, 0, false) + Eventually(approvalRequestActual(approvalRequestKey, runKey, placementPolicyCanaryStageName, false), eventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to find the approval request") + }) + + It("should not roll out to the prod clusters before the approval", func() { + Consistently(prodOnInitialRevisionActual(), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The new version reached the prod clusters before the approval") + }) + + It("can approve the canary stage", func() { + approvePlacementPolicyApprovalRequest(approvalRequestKey) + }) + + It("should accept the approval", func() { + Eventually(approvalRequestActual(approvalRequestKey, runKey, placementPolicyCanaryStageName, true), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to accept the approval") + }) + + It("should roll out the new version to every member cluster", func() { + wantSelectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantSelectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster, 1), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll out the new version to every member cluster") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + }) + + It("should complete the run successfully", func() { + succeededCluster := func(clusterName string) kfrolloutv1alpha1.PerClusterStatus { + return succeededStagedUpdateRunClusterStatus(clusterName, bindingNamesByCluster[clusterName]) + } + // Timestamps are ignored in the comparison; non-nil wanted values check that they are set. + wantStatus := kfrolloutv1alpha1.StagedUpdateRunStatus{ + Conditions: succeededStagedUpdateRunConditions(), + ResourceSnapshotNameToRollout: newSnapshotName, + Stages: []kfrolloutv1alpha1.PerStageStatus{ + { + StageName: placementPolicyCanaryStageName, + Clusters: []kfrolloutv1alpha1.PerClusterStatus{succeededCluster(memberCluster2EastCanaryName)}, + AfterStageTasks: []kfrolloutv1alpha1.PerStageTaskStatus{ + { + Type: kfrolloutv1alpha1.StageTaskTypeApproval, + ApprovalRequestName: approvalRequestKey.Name, + Conditions: []metav1.Condition{ + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunTaskApprovalRequestCreatedCondReasonCreated), + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestApproved, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunTaskApprovalRequestApprovedCondReasonApproved), + }, + }, + }, + Conditions: succeededStagedUpdateRunStageConditions(), + StartedTimestamp: &metav1.Time{}, + CompletedTimestamp: &metav1.Time{}, + MaxConcurrency: ptr.To(int32(1)), + }, + { + StageName: placementPolicyProdStageName, + // Clusters in a stage with no sorting label key are ordered by name. + Clusters: []kfrolloutv1alpha1.PerClusterStatus{ + succeededCluster(memberCluster1EastProdName), + succeededCluster(memberCluster3WestProdName), + }, + Conditions: succeededStagedUpdateRunStageConditions(), + StartedTimestamp: &metav1.Time{}, + CompletedTimestamp: &metav1.Time{}, + MaxConcurrency: ptr.To(int32(1)), + }, + }, + } + Eventually(stagedUpdateRunStatusActual(runKey, wantStatus), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete the staged update run") + }) + + It("should report the policy as synchronized on every member cluster, with the binding manager role released", func() { + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected %s status", scopeDescription) + }) + }) + } + + Context("placement policy rolled back to its initial resource snapshot by a staged update run", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyRollbackNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyRollbackStrategyNameTemplate, GinkgoParallelProcess())} + forwardRunKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyRollForwardRunNameTemplate, GinkgoParallelProcess())} + backRunKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyRollBackRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + + var initialSnapshotName, newSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a policy that places the ConfigMap on every member cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + + By("creating a strategy with a single stage that covers every cluster") + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + // An empty (not nil) label selector selects every cluster. + {Name: placementPolicyAllClustersStageName, LabelSelector: &metav1.LabelSelector{}}, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensureStagedUpdateRunRemoved(backRunKey) + ensureStagedUpdateRunRemoved(forwardRunKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on every member cluster", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of placement policy %s", policyKey) + initialSnapshotName = snapshot.GetName() + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("can roll the change out with a staged update run", func() { + run := newStagedUpdateRunObject(forwardRunKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", forwardRunKey) + Eventually(stagedUpdateRunSucceededActual(forwardRunKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s", forwardRunKey) + + forwardRun, err := retrieveStagedUpdateRun(forwardRunKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get staged update run %s", forwardRunKey) + newSnapshotName = forwardRun.GetStatus().ResourceSnapshotNameToRollout + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Staged update run %s rolled out the initial resource snapshot", forwardRunKey) + }) + + It("should place the new version on every member cluster", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new resource snapshot out to every binding") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + }) + + It("can roll back with a staged update run that names the initial resource snapshot", func() { + run := newStagedUpdateRunObject(backRunKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + ResourceSnapshotName: initialSnapshotName, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", backRunKey) + Eventually(stagedUpdateRunSucceededActual(backRunKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s", backRunKey) + }) + + It("should place the initial version on every member cluster again", func() { + wantBindings := make(map[string]wantPlacementBinding, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: initialSnapshotName, + status: synchronizedPlacementBindingStatus(1), + } + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the initial resource snapshot back out to every binding") + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster again") + }) + + It("should report every cluster as on an older resource revision, the change being still the latest one", func() { + // A rollback takes no new snapshot: the hub still matches the changed revision, which stays the latest, so + // every cluster counts as outdated rather than as synchronized. Availability, too, is judged only on + // clusters that have the latest revision synchronized. + clusterCount := int32(len(allMemberClusters)) + wantStatus := kfplacementv1alpha1.PlacementPolicyStatus{ + Conditions: []metav1.Condition{ + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable), + }, + LatestResourceRevisionName: ptr.To(newSnapshotName), + DesiredClusters: ptr.To(clusterCount), + ScheduledClusters: ptr.To(clusterCount), + SynchronizedClusters: ptr.To(int32(0)), + ResourcesAvailableClusters: ptr.To(int32(0)), + ActiveClusterClaims: ptr.To(int32(0)), + } + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("cluster placement policy scaled up while a cluster staged update run holds the binding manager role", Ordered, func() { + policyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyMutualExclusionNameTemplate, GinkgoParallelProcess())} + strategyKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyMutualExclusionStrategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Name: fmt.Sprintf(clusterPlacementPolicyMutualExclusionRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + canaryLabels := map[string]string{envLabelName: envCanary} + prodLabels := map[string]string{envLabelName: envProd} + canarySelectors := []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{clusterSelectorTermsMatchingLabels(canaryLabels)} + prodSelectors := []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{clusterSelectorTermsMatchingLabels(prodLabels)} + + // Which prod cluster ranks first depends on live capacity, so the specs learn it rather than assume it. + var firstProdClusterName, secondProdClusterName, initialSnapshotName, newSnapshotName string + var approvalRequestKey types.NamespacedName + + // heldBackActual checks the state while the run holds the binding manager role: the canary cluster has the new + // revision, the first prod cluster still has the initial one, and the second prod cluster, which the raised + // count now wants, is neither bound nor given the ConfigMap. The policy status already counts it as wanted. + heldBackActual := func() func() error { + bindingsActual := placementPolicyBindingsActual(policyKey, map[string]wantPlacementBinding{ + memberCluster2EastCanaryName: {selectors: canarySelectors, snapshotName: newSnapshotName, status: synchronizedPlacementBindingStatus(1)}, + firstProdClusterName: {selectors: prodSelectors, snapshotName: initialSnapshotName, status: synchronizedPlacementBindingStatus(1)}, + }) + secondProdActual := configMapAbsentOnClustersActual(cmKey, memberClusterNamed(secondProdClusterName)) + statusActual := placementPolicyStatusActual(policyKey, kfplacementv1alpha1.PlacementPolicyStatus{ + Conditions: []metav1.Condition{ + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable), + }, + LatestResourceRevisionName: ptr.To(newSnapshotName), + // The canary selector wants 1 cluster and the raised prod selector 2. + DesiredClusters: ptr.To(int32(3)), + ScheduledClusters: ptr.To(int32(3)), + // Only the canary cluster has the latest revision; the first prod cluster is on an older one, and the + // second has no binding yet. + SynchronizedClusters: ptr.To(int32(1)), + ResourcesAvailableClusters: ptr.To(int32(1)), + ActiveClusterClaims: ptr.To(int32(0)), + BindingManager: &kfplacementv1alpha1.BindingManager{ + ControllerName: stagedUpdateRunBindingManagerName, + ObjectRefs: []kfplacementv1alpha1.ObjectReference{stagedUpdateRunObjectReference(runKey)}, + }, + }) + return func() error { + if err := bindingsActual(); err != nil { + return err + } + if err := secondProdActual(); err != nil { + return err + } + return statusActual() + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a cluster placement policy that selects the canary cluster and one prod cluster") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(canaryLabels, intstr.FromInt32(1)), + clusterSelectorMatchingLabels(prodLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + + By("creating a strategy with a canary stage, gated by an approval, before a prod stage") + strategy := newStagedUpdateStrategyObject(strategyKey, kfrolloutv1alpha1.StagedUpdateStrategySpec{ + Stages: []kfrolloutv1alpha1.Stage{ + { + Name: placementPolicyCanaryStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: canaryLabels}, + AfterStageTasks: []kfrolloutv1alpha1.StageTask{{Type: kfrolloutv1alpha1.StageTaskTypeApproval}}, + }, + { + Name: placementPolicyProdStageName, + LabelSelector: &metav1.LabelSelector{MatchLabels: prodLabels}, + }, + }, + }) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create cluster staged update strategy %s", strategyKey) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // The run goes first, so that it gives up the binding manager role before the policy goes. + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the initial version on the canary cluster and one prod cluster", func() { + Eventually(func() error { + bindings, err := placementPolicyBindingsByCluster(policyKey) + if err != nil { + return err + } + if _, found := bindings[memberCluster2EastCanaryName]; !found || len(bindings) != 2 { + return fmt.Errorf("cluster placement policy %s binds clusters %v, want the canary cluster and one prod cluster", + policyKey, slices.Sorted(maps.Keys(bindings))) + } + for clusterName := range bindings { + if clusterName != memberCluster2EastCanaryName { + firstProdClusterName = clusterName + } + } + return nil + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind the canary cluster and one prod cluster") + secondProdClusterName = otherProdClusterName(firstProdClusterName) + + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster2EastCanaryName: canarySelectors, + firstProdClusterName: prodSelectors, + }, 1), workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to synchronize the bindings") + Eventually(configMapDataOnClustersActual(cmKey, v0Data, memberCluster2EastCanary, memberClusterNamed(firstProdClusterName)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on the bound clusters") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get cluster placement policy %s", policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of cluster placement policy %s", policyKey) + initialSnapshotName = snapshot.GetName() + }) + + It("can change the ConfigMap on the hub", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + }) + + It("can create a cluster staged update run", func() { + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create cluster staged update run %s", runKey) + }) + + It("should roll the new version out to the canary cluster, then wait for an approval", func() { + approvalRequestKey = waitForStageApprovalRequest(runKey, 0, false) + run, err := retrieveStagedUpdateRun(runKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get cluster staged update run %s", runKey) + newSnapshotName = run.GetStatus().ResourceSnapshotNameToRollout + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Cluster staged update run %s rolls out the initial resource snapshot", runKey) + + Eventually(placementPolicyBindingsActual(policyKey, map[string]wantPlacementBinding{ + memberCluster2EastCanaryName: {selectors: canarySelectors, snapshotName: newSnapshotName, status: synchronizedPlacementBindingStatus(1)}, + firstProdClusterName: {selectors: prodSelectors, snapshotName: initialSnapshotName, status: synchronizedPlacementBindingStatus(1)}, + }), placementPolicyRolloutEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to roll the new version out to the canary cluster only") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on the canary cluster") + }) + + It("can raise the prod selector's count to 2 while the run holds the binding manager role", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().ClusterSelectors[1].Count = ptr.To(intstr.FromInt32(2)) + }) + }) + + It("should want the second prod cluster but hold off binding it while the run holds the role", func() { + // Scheduling decisions and rollouts are mutually exclusive: the policy defers the new binding (retrying on its + // backoff) until the run gives up the binding manager role. + Eventually(heldBackActual(), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the second prod cluster as wanted") + Consistently(heldBackActual(), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The second prod cluster was bound while the run held the binding manager role") + }) + + It("can approve the canary stage", func() { + approvePlacementPolicyApprovalRequest(approvalRequestKey) + }) + + It("should complete the run", func() { + // The prod stage covers only the prod cluster that was bound when the run started. + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete cluster staged update run %s", runKey) + }) + + It("should bind the second prod cluster to the new resource snapshot once the run gives up the role", func() { + // A new binding gets the latest resource snapshot, which the run has just rolled out everywhere else. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster2EastCanaryName: canarySelectors, + firstProdClusterName: prodSelectors, + secondProdClusterName: prodSelectors, + }, 1), placementPolicyRolloutEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind the second prod cluster") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) + + Context("annotation-generated placement policy rolled out by a staged update run", Ordered, func() { + nsName := appNamespace().Name + strategyKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyGeneratedStrategyNameTemplate, GinkgoParallelProcess())} + runKey := types.NamespacedName{Namespace: nsName, Name: fmt.Sprintf(placementPolicyGeneratedRunNameTemplate, GinkgoParallelProcess())} + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + v0Data := appConfigMap().Data + v1Data := map[string]string{"data": "test-v1"} + canaryTerms := clusterSelectorTermsMatchingLabels(map[string]string{envLabelName: envCanary}) + prodTerms := clusterSelectorTermsMatchingLabels(map[string]string{envLabelName: envProd}) + // The generated policy has two selectors, the canary cluster and every prod cluster; each binding records the + // one that picked its cluster. + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + return map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster2EastCanaryName: {canaryTerms}, + memberCluster1EastProdName: {prodTerms}, + memberCluster3WestProdName: {prodTerms}, + } + } + + var configMap corev1.ConfigMap + var policyKey types.NamespacedName + var initialSnapshotName, newSnapshotName string + var approvalRequestKey types.NamespacedName + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a strategy with a canary stage, gated by an approval, before a prod stage") + strategy := newStagedUpdateStrategyObject(strategyKey, canaryApprovalStrategySpec()) + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create staged update strategy %s", strategyKey) + + By("annotating the ConfigMap so that the hub agent generates a placement policy for the canary and prod clusters") + configMap = appConfigMap() + setClusterSelectorsAnnotation(&configMap, fmt.Sprintf("%s=%s;%s=%s,count=%s", envLabelName, envCanary, envLabelName, envProd, clusterSelectorCountAll)) + }) + + AfterAll(func() { + // The work resources must go even if the cleanup below fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + // The run goes first, so that it gives up the binding manager role before the policy goes. + ensureStagedUpdateRunRemoved(runKey) + ensureStagedUpdateStrategyRemoved(strategyKey) + // Removing the annotation makes the hub agent delete the generated policy. + setClusterSelectorsAnnotation(&configMap, "") + if policyKey.Name != "" { + ensurePlacementPolicyRemoved(policyKey) + } + }) + + It("should generate a placement policy that places the ConfigMap on every member cluster", func() { + // The provenance labels are how a user finds the generated policy, whose name is the hub agent's to choose. + Eventually(func() error { + policies := &kfplacementv1alpha1.PlacementPolicyList{} + if err := hubClient.List(ctx, policies, client.InNamespace(cmKey.Namespace), generatedPolicyLabels("", "ConfigMap", cmKey.Name)); err != nil { + return err + } + if len(policies.Items) != 1 { + return fmt.Errorf("%d placement policies are generated for ConfigMap %s, want 1", len(policies.Items), cmKey) + } + policyKey = types.NamespacedName{Namespace: policies.Items[0].Namespace, Name: policies.Items[0].Name} + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to generate a placement policy for ConfigMap %s", cmKey) + initialSnapshotName = waitForPlacementPolicySynchronized(policyKey, wantSelectorsByCluster(), 1) + Eventually(configMapDataOnClustersActual(cmKey, v0Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the initial version on every member cluster") + }) + + It("can change the ConfigMap on the hub, and roll the change out with a staged update run on the generated policy", func() { + updateConfigMapDataOnHub(cmKey, v1Data) + run := newStagedUpdateRunObject(runKey, kfrolloutv1alpha1.StagedUpdateRunSpec{ + PlacementPolicyName: policyKey.Name, + StagedUpdateStrategyName: strategyKey.Name, + }) + Expect(hubClient.Create(ctx, run)).To(Succeed(), "Failed to create staged update run %s", runKey) + }) + + It("should roll the new version out to the canary cluster, then wait for an approval", func() { + approvalRequestKey = waitForStageApprovalRequest(runKey, 0, false) + newSnapshotName = resourceSnapshotNameToRolloutOf(runKey) + Expect(newSnapshotName).NotTo(Equal(initialSnapshotName), "Staged update run %s rolls out the initial resource snapshot", runKey) + onlyCanaryActual := onlyCanaryUpdatedActual(policyKey, cmKey, initialSnapshotName, newSnapshotName, v0Data, v1Data) + Eventually(onlyCanaryActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new version out to the canary cluster only") + Consistently(onlyCanaryActual, consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The new version reached the prod clusters before the approval") + }) + + It("can approve the canary stage", func() { + approvePlacementPolicyApprovalRequest(approvalRequestKey) + }) + + It("should roll the new version out to every member cluster, keeping the generated policy as it was", func() { + Eventually(stagedUpdateRunSucceededActual(runKey), placementPolicyRolloutEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to complete staged update run %s", runKey) + // The bindings still record the selectors that the annotation generated. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the new resource snapshot out to every binding") + Eventually(configMapDataOnClustersActual(cmKey, v1Data, allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the new version on every member cluster") + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(newSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) +}) diff --git a/test/e2e/placement_policy_sync_strategy_failures_test.go b/test/e2e/placement_policy_sync_strategy_failures_test.go new file mode 100644 index 000000000..b96afa443 --- /dev/null +++ b/test/e2e/placement_policy_sync_strategy_failures_test.go @@ -0,0 +1,358 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "encoding/json" + "fmt" + "maps" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/test/e2e/framework" +) + +const ( + placementPolicyUnavailableNameTemplate = "pp-unavailable-%d" + placementPolicyNoNamespaceNameTemplate = "pp-no-namespace-%d" + placementPolicyServerSideApplyNameTemplate = "pp-server-side-apply-%d" +) + +const ( + // placementPolicyOtherFieldManagerName is the field manager that the specs write member cluster objects as, in + // place of another controller. + placementPolicyOtherFieldManagerName = "pp-e2e-other-manager" + // placementPolicyWorkApplierFieldManagerName is the field manager that the placement policy work applier applies + // manifests as. It is spelled out rather than imported so that a rename shows up as a failed spec, the same way it + // would show up for a user inspecting the managed fields. + placementPolicyWorkApplierFieldManagerName = "kubefleet-member-agent" +) + +// configMapDataKeyOwnersActual checks that the ConfigMap on the cluster has exactly the given data, and that its managed +// fields record exactly the given field managers as owners of its data.data field, the one key the app ConfigMap has. +func configMapDataKeyOwnersActual(cluster *framework.Cluster, key types.NamespacedName, wantData map[string]string, wantOwners ...string) func() error { + dataActual := configMapDataOnClustersActual(key, wantData, cluster) + return func() error { + if err := dataActual(); err != nil { + return err + } + cm := &corev1.ConfigMap{} + if err := cluster.KubeClient.Get(ctx, key, cm); err != nil { + return fmt.Errorf("cluster %s: failed to get ConfigMap %s: %w", cluster.ClusterName, key, err) + } + var gotOwners []string + for _, entry := range cm.ManagedFields { + if entry.FieldsV1 == nil { + continue + } + fields := map[string]map[string]any{} + if err := json.Unmarshal(entry.FieldsV1.Raw, &fields); err != nil { + return fmt.Errorf("cluster %s: failed to parse the fields that %s manages on ConfigMap %s: %w", cluster.ClusterName, entry.Manager, key, err) + } + if _, found := fields["f:data"]["f:data"]; found { + gotOwners = append(gotOwners, entry.Manager) + } + } + if diff := cmp.Diff(gotOwners, wantOwners, cmpopts.SortSlices(func(a, b string) bool { return a < b })); diff != "" { + return fmt.Errorf("cluster %s: ConfigMap %s data.data field managers diff (-got, +want):\n%s", cluster.ClusterName, key, diff) + } + return nil + } +} + +var _ = Describe("placement policy sync strategy failures", Label("placementpolicy"), func() { + Context("placement policy placing a Deployment whose image cannot be pulled", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyUnavailableNameTemplate, GinkgoParallelProcess()), + } + deployKey := types.NamespacedName{Namespace: nsName, Name: appDeployment().Name} + // The policy places the resources on cluster-1 alone. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + + BeforeAll(func() { + By("creating the work resources and a Deployment whose image cannot be pulled") + createWorkResources() + deploy := appDeployment() + deploy.Spec.Template.Spec.Containers[0].Image = placementPolicyUnpullableImage + Expect(hubClient.Create(ctx, &deploy)).To(Succeed(), "Failed to create Deployment %s", deployKey) + + By("creating a placement policy that places the ConfigMap and the Deployment on cluster-1") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: append(placementPolicyResourceSelectors(), kfplacementv1alpha1.ResourceSelector{ + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + Name: deployKey.Name, + }), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources, the Deployment among them, must go even if removing the policy fails; otherwise they + // leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should apply both resources on member cluster 1", func() { + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on member cluster 1") + Eventually(func() error { + return memberCluster1EastProd.KubeClient.Get(ctx, deployKey, &appsv1.Deployment{}) + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to place the Deployment on member cluster 1") + }) + + It("should report the Deployment as applied but unavailable on the binding", func() { + Eventually(func() error { + // The condition copied from the work applier carries the generation of the Deployment on the member + // cluster. + deploy := &appsv1.Deployment{} + if err := memberCluster1EastProd.KubeClient.Get(ctx, deployKey, deploy); err != nil { + return err + } + unavailableCond := placementPolicyCondition(kfplacementv1alpha1.ManifestCondTypeAvailable, metav1.ConditionFalse, "Unavailable") + unavailableCond.ObservedGeneration = deploy.Generation + wantBindings := map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{clusterSelectorTermsMatchingLabels(aliasLabels)}, + status: kfplacementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + placementPolicyCondition(kfplacementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized), + placementPolicyCondition(kfplacementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable), + }, + SelectedResources: ptr.To(int32(2)), + SynchronizedResources: ptr.To(int32(2)), + AvailableResources: ptr.To(int32(1)), + FailedResources: []kfplacementv1alpha1.FailedResource{ + { + ObjectRef: kfplacementv1alpha1.ObjectReference{ + Namespace: deployKey.Namespace, + Name: deployKey.Name, + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + }, + Conditions: []metav1.Condition{unavailableCond}, + }, + }, + }, + }, + } + return placementPolicyBindingsActual(policyKey, wantBindings)() + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the Deployment as unavailable on the binding") + }) + + It("should report the policy as synchronized, but unavailable on member cluster 1", func() { + wantStatus := allSynchronizedPlacementPolicyStatus(1) + meta.SetStatusCondition(&wantStatus.Conditions, placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, + metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable)) + wantStatus.ResourcesAvailableClusters = ptr.To(int32(0)) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("placement policy reporting a missing namespace instead of creating it", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyNoNamespaceNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + // The policy places the ConfigMap on cluster-1 alone. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + return map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster1EastProdName: {clusterSelectorTermsMatchingLabels(aliasLabels)}, + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("checking that the namespace is absent on cluster-1") + // Each process reuses its namespace name, and an earlier spec of this process removes the namespace from + // the member clusters as it cleans up. + Eventually(namespaceAbsentOnClustersActual(nsName, memberCluster1EastProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "The namespace is still on member cluster 1") + + By("creating a placement policy that does not create missing namespaces") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + SyncStrategy: &kfplacementv1alpha1.SyncStrategy{ + WhenNamespaceDoesNotExist: kfplacementv1alpha1.WhenNamespaceDoesNotExistOptionReportError, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should report the ConfigMap as not applied", func() { + // The ConfigMap is not on the member cluster, so the condition copied from the work applier observes no + // generation. + wantBindings := map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: wantSelectorsByCluster()[memberCluster1EastProdName], + status: configMapNotAppliedPlacementBindingStatus(cmKey, "FailedToApply", nil), + }, + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as not applied") + Eventually(placementPolicyStatusActual(policyKey, someClustersNotSynchronizedPlacementPolicyStatus(1, 0)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should create neither the namespace nor the ConfigMap on member cluster 1", func() { + namespaceActual := namespaceAbsentOnClustersActual(nsName, memberCluster1EastProd) + configMapActual := configMapAbsentOnClustersActual(cmKey, memberCluster1EastProd) + Consistently(func() error { + if err := namespaceActual(); err != nil { + return err + } + return configMapActual() + }, consistentlyDuration, consistentlyInterval).Should(Succeed(), "The namespace or the ConfigMap was created") + }) + + It("can create the namespace on member cluster 1", func() { + ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: nsName}} + Expect(memberCluster1EastProd.KubeClient.Create(ctx, ns)).To(Succeed(), "Failed to create namespace %s on member cluster 1", nsName) + }) + + It("should apply the ConfigMap, leaving the namespace unmanaged", func() { + // The work applier retries the failed apply on its backoff, which the E2E setup caps at 5s. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as applied") + Eventually(configMapTakenOverOnClusterActual(memberCluster1EastProd, cmKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to apply the ConfigMap on member cluster 1") + Expect(namespaceLeftUnmanagedOnClustersActual(nsName, memberCluster1EastProd)()).To(Succeed(), "The namespace is managed by KubeFleet") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("placement policy server-side applying a field that another field manager owns", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyServerSideApplyNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + otherManagerData := map[string]string{"data": "set-by-another-manager"} + // The policy places the ConfigMap on cluster-1 alone. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + return map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster1EastProdName: {clusterSelectorTermsMatchingLabels(aliasLabels)}, + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating the same ConfigMap, with different data, directly on member cluster 1, as another field manager") + preExisting := appConfigMap() + preExisting.Data = maps.Clone(otherManagerData) + createConfigMapOnMemberCluster(memberCluster1EastProd, &preExisting, client.FieldOwner(placementPolicyOtherFieldManagerName)) + + By("creating a placement policy that server-side applies its resources, without forcing conflicts") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + SyncStrategy: &kfplacementv1alpha1.SyncStrategy{ + ApplyMethod: kfplacementv1alpha1.ApplyMethodServerSideApply, + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The ConfigMap on cluster-1 goes with the member namespace, which the cleanup removes even if the policy + // never applied it. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should fail to apply the ConfigMap, as the field it sets belongs to another field manager", func() { + wantBindings := map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: wantSelectorsByCluster()[memberCluster1EastProdName], + status: configMapNotAppliedPlacementBindingStatus(cmKey, "FailedToApply", nil), + }, + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as not applied") + Eventually(placementPolicyStatusActual(policyKey, someClustersNotSynchronizedPlacementPolicyStatus(1, 0)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should leave the field on member cluster 1 as the other field manager set it", func() { + Consistently(configMapDataKeyOwnersActual(memberCluster1EastProd, cmKey, otherManagerData, placementPolicyOtherFieldManagerName), + consistentlyDuration, consistentlyInterval).Should(Succeed(), "The field was changed, or its owner") + }) + + It("can switch the policy to force conflicts", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().SyncStrategy.ServerSideApplyOptions = &kfplacementv1alpha1.ServerSideApplyOptions{ForceConflicts: true} + }) + }) + + It("should apply the ConfigMap, taking the field over from the other field manager", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as synchronized") + Eventually(configMapDataKeyOwnersActual(memberCluster1EastProd, cmKey, appConfigMap().Data, placementPolicyWorkApplierFieldManagerName), + workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to take the field over") + Eventually(configMapTakenOverOnClusterActual(memberCluster1EastProd, cmKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to take the ConfigMap over on member cluster 1") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) +}) diff --git a/test/e2e/placement_policy_sync_strategy_takeover_test.go b/test/e2e/placement_policy_sync_strategy_takeover_test.go new file mode 100644 index 000000000..7e236a2be --- /dev/null +++ b/test/e2e/placement_policy_sync_strategy_takeover_test.go @@ -0,0 +1,281 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + + "github.com/google/go-cmp/cmp" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" +) + +const ( + placementPolicyTakeOverIfNoDiffNameTemplate = "pp-take-over-if-no-diff-%d" + placementPolicyOwnedByOthersNameTemplate = "pp-owned-by-others-%d" +) + +var _ = Describe("placement policy taking over existing resources", Label("placementpolicy"), func() { + Context("placement policy taking over a ConfigMap only if it has no differences", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyTakeOverIfNoDiffNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + hubValue := appConfigMap().Data["data"] + // The member copy differs from the hub copy in the key the hub sets, and has a key of its own. + preExistingData := map[string]string{"data": "pre-existing", "member-only": "kept"} + matchingData := map[string]string{"data": hubValue, "member-only": "kept"} + // The policy places the ConfigMap on cluster-1 alone. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + return map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster1EastProdName: {clusterSelectorTermsMatchingLabels(aliasLabels)}, + } + } + + dataDiff := kfplacementv1alpha1.PatchDetail{Path: "/data/data", ValueInMember: preExistingData["data"], ValueInHub: hubValue} + memberOnlyDiff := kfplacementv1alpha1.PatchDetail{Path: "/data/member-only", ValueInMember: preExistingData["member-only"]} + notTakenOverActual := func(diffs ...kfplacementv1alpha1.PatchDetail) func() error { + // A ConfigMap carries no generation, so the generation observed on the member cluster is 0. + diffDetails := &kfplacementv1alpha1.DiffDetails{ObservedInMemberClusterGeneration: ptr.To(int64(0)), ObservedDiffs: diffs} + return placementPolicyBindingsActual(policyKey, map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: wantSelectorsByCluster()[memberCluster1EastProdName], + status: configMapNotAppliedPlacementBindingStatus(cmKey, "FailedToTakeOver", diffDetails), + }, + }) + } + setComparisonOption := func(option kfplacementv1alpha1.ComparisonOption) { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().SyncStrategy = &kfplacementv1alpha1.SyncStrategy{ + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff, + ComparisonOption: option, + } + }) + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating the same ConfigMap, with different data, directly on member cluster 1") + preExisting := appConfigMap() + preExisting.Data = maps.Clone(preExistingData) + createConfigMapOnMemberCluster(memberCluster1EastProd, &preExisting) + + By("creating a placement policy that takes over existing resources only if they have no differences, in full") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + SyncStrategy: &kfplacementv1alpha1.SyncStrategy{ + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionTakeOverIfNoDiff, + ComparisonOption: kfplacementv1alpha1.ComparisonOptionFullComparison, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The pre-existing ConfigMap goes with the member namespace, which the cleanup removes even if deleting the + // policy fails; it stays unmanaged unless the specs got as far as the takeover. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should report every difference under full comparison, the member-only key included", func() { + Eventually(notTakenOverActual(dataDiff, memberOnlyDiff), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the differences on the binding") + Eventually(placementPolicyStatusActual(policyKey, someClustersNotSynchronizedPlacementPolicyStatus(1, 0)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should leave the pre-existing ConfigMap on member cluster 1 as it is", func() { + Consistently(configMapOrphanedOnClustersActual(cmKey, preExistingData, memberCluster1EastProd), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The pre-existing ConfigMap was changed or taken over") + }) + + It("can switch the policy to partial comparison", func() { + setComparisonOption(kfplacementv1alpha1.ComparisonOptionPartialComparison) + }) + + It("should report only the difference in the key that the hub sets", func() { + // Partial comparison judges only the fields that the hub copy specifies. + Eventually(notTakenOverActual(dataDiff), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the difference on the binding") + }) + + It("can make the key that the hub sets match on member cluster 1", func() { + updateConfigMapDataOnMemberCluster(memberCluster1EastProd, cmKey, matchingData) + }) + + It("should take the ConfigMap over, keeping the member-only key", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as synchronized") + Eventually(configMapManagedOnClustersActual(cmKey, matchingData, memberCluster1EastProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to take the ConfigMap over on member cluster 1") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) + + Context("placement policy taking over a ConfigMap that another object owns", Ordered, func() { + nsName := appNamespace().Name + policyKey := types.NamespacedName{ + Namespace: nsName, + Name: fmt.Sprintf(placementPolicyOwnedByOthersNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: nsName, Name: appConfigMap().Name} + ownerKey := types.NamespacedName{Namespace: nsName, Name: "pp-e2e-member-owner"} + hubData := appConfigMap().Data + preExistingData := map[string]string{"data": "pre-existing"} + // The policy places the ConfigMap on cluster-1 alone. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + return map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster1EastProdName: {clusterSelectorTermsMatchingLabels(aliasLabels)}, + } + } + + var ownerRef metav1.OwnerReference + + // ownershipActual checks the data of the ConfigMap on cluster-1 and that the other object still owns it, along + // with an AppliedWork of the placement policy APIs or not. + ownershipActual := func(wantData map[string]string, wantCoOwnedByKubeFleet bool) func() error { + dataActual := configMapDataOnClustersActual(cmKey, wantData, memberCluster1EastProd) + return func() error { + if err := dataActual(); err != nil { + return err + } + cm := &corev1.ConfigMap{} + if err := memberCluster1EastProd.KubeClient.Get(ctx, cmKey, cm); err != nil { + return err + } + var otherOwnerRefs []metav1.OwnerReference + for _, ref := range cm.OwnerReferences { + if ref.Kind != kfplacementv1alpha1.AppliedWorkKind { + otherOwnerRefs = append(otherOwnerRefs, ref) + } + } + if diff := cmp.Diff(otherOwnerRefs, []metav1.OwnerReference{ownerRef}); diff != "" { + return fmt.Errorf("ConfigMap %s non-KubeFleet owner references diff (-got, +want):\n%s", cmKey, diff) + } + if got := ownedByPlacementPolicyAppliedWork(cm); got != wantCoOwnedByKubeFleet { + return fmt.Errorf("ConfigMap %s owned by an AppliedWork = %t, want %t (owner references: %+v)", cmKey, got, wantCoOwnedByKubeFleet, cm.OwnerReferences) + } + return nil + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating the same ConfigMap directly on member cluster 1, owned by another ConfigMap") + owner := &corev1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ownerKey.Name, Namespace: ownerKey.Namespace}} + createConfigMapOnMemberCluster(memberCluster1EastProd, owner) + Expect(memberCluster1EastProd.KubeClient.Get(ctx, ownerKey, owner)).To(Succeed(), "Failed to get ConfigMap %s on member cluster 1", ownerKey) + // The owner must exist (with this UID); otherwise the garbage collector removes the ConfigMap. + ownerRef = metav1.OwnerReference{APIVersion: "v1", Kind: "ConfigMap", Name: owner.Name, UID: owner.UID} + preExisting := appConfigMap() + preExisting.Data = maps.Clone(preExistingData) + preExisting.OwnerReferences = []metav1.OwnerReference{ownerRef} + createConfigMapOnMemberCluster(memberCluster1EastProd, &preExisting) + + By("creating a placement policy that always takes over existing resources, but not co-owned ones") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + SyncStrategy: &kfplacementv1alpha1.SyncStrategy{ + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The ConfigMaps on cluster-1 go with the member namespace, which the cleanup removes even if deleting the + // policy fails; the other owner keeps the ConfigMap there after the policy is gone. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should fail to take the ConfigMap over, as another object owns it", func() { + // The default whenOwnedByOthers=ReportError forbids co-ownership, even with whenAlreadyExists=AlwaysTakeOver. + wantBindings := map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: wantSelectorsByCluster()[memberCluster1EastProdName], + status: configMapNotAppliedPlacementBindingStatus(cmKey, "FailedToTakeOver", nil), + }, + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as not taken over") + Eventually(placementPolicyStatusActual(policyKey, someClustersNotSynchronizedPlacementPolicyStatus(1, 0)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("should leave the ConfigMap on member cluster 1 as it is, owned by the other object alone", func() { + Consistently(ownershipActual(preExistingData, false), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The ConfigMap was changed or taken over") + }) + + It("can switch the policy to share ownership", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().SyncStrategy = &kfplacementv1alpha1.SyncStrategy{ + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + WhenOwnedByOthers: kfplacementv1alpha1.WhenOwnedByOthersOptionShareOwnership, + } + }) + }) + + It("should take the ConfigMap over as a co-owner", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as synchronized") + Eventually(ownershipActual(hubData, true), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to take the ConfigMap over as a co-owner") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + + It("can delete the placement policy", func() { + deletePlacementPolicyAndWait(policyKey) + }) + + It("should leave the ConfigMap on member cluster 1, owned by the other object alone", func() { + // The garbage collector deletes an object only once all of its owners are gone; with another owner left, it + // drops the reference to the deleted AppliedWork instead. + Eventually(ownershipActual(hubData, false), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to release the ConfigMap") + Consistently(ownershipActual(hubData, false), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The ConfigMap was removed, or KubeFleet took it back") + }) + }) +}) diff --git a/test/e2e/placement_policy_sync_strategy_test.go b/test/e2e/placement_policy_sync_strategy_test.go new file mode 100644 index 000000000..8d425f7da --- /dev/null +++ b/test/e2e/placement_policy_sync_strategy_test.go @@ -0,0 +1,371 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + appsv1 "k8s.io/api/apps/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" +) + +const ( + clusterPlacementPolicyTakeOverNameTemplate = "cpp-take-over-%d" + placementPolicyDriftNameTemplate = "pp-drift-%d" + placementPolicyOrphanNameTemplate = "pp-orphan-%d" + placementPolicyDeploymentNameTemplate = "pp-deployment-%d" +) + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis hub and member +// agent flags, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy sync strategy", Label("placementpolicy"), func() { + Context("cluster placement policy placing a ConfigMap that already exists, unmanaged, on a member cluster", Ordered, func() { + policyKey := types.NamespacedName{ + Name: fmt.Sprintf(clusterPlacementPolicyTakeOverNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + preExistingData := map[string]string{"data": "pre-existing"} + otherClusters := []string{memberCluster2EastCanaryName, memberCluster3WestProdName} + + var initialSnapshotName string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating the same ConfigMap, with different data, directly on member cluster 1") + preExisting := appConfigMap() + preExisting.Data = maps.Clone(preExistingData) + createConfigMapOnMemberCluster(memberCluster1EastProd, &preExisting) + + By("creating a cluster placement policy with the default sync strategy") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create cluster placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the ConfigMap on the other member clusters", func() { + Eventually(appConfigMapPlacedOnClustersActual(memberCluster2EastCanary, memberCluster3WestProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on the other member clusters") + }) + + It("should report the ConfigMap as not taken over on member cluster 1", func() { + // The default whenAlreadyExists=ReportError leaves an unmanaged object alone. + wantBindings := map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + status: notTakenOverPlacementBindingStatus(cmKey), + }, + } + for _, clusterName := range otherClusters { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + status: synchronizedPlacementBindingStatus(1), + } + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the ConfigMap as not taken over") + + policy, err := retrievePlacementPolicy(policyKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get cluster placement policy %s", policyKey) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of cluster placement policy %s", policyKey) + initialSnapshotName = snapshot.GetName() + }) + + It("should report the policy as not synchronized on member cluster 1", func() { + clusterCount := int32(len(allMemberClusters)) + wantStatus := someClustersNotSynchronizedPlacementPolicyStatus(clusterCount, clusterCount-1) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + + It("should leave the pre-existing ConfigMap on member cluster 1 as it is", func() { + Consistently(configMapDataOnClustersActual(cmKey, preExistingData, memberCluster1EastProd), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The pre-existing ConfigMap was overwritten") + }) + + It("can switch the policy to take over existing resources", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().SyncStrategy = &kfplacementv1alpha1.SyncStrategy{ + WhenAlreadyExists: kfplacementv1alpha1.WhenAlreadyExistsOptionAlwaysTakeOver, + } + }) + }) + + It("should carry the new sync strategy to every binding without a rollout", func() { + // The bindings must stay on the same resource snapshot; the actual also checks that each one carries the + // policy's (API server defaulted) sync strategy. + wantBindings := make(map[string]wantPlacementBinding, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + wantBindings[clusterName] = wantPlacementBinding{ + selectors: []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}}, + snapshotName: initialSnapshotName, + status: synchronizedPlacementBindingStatus(1), + } + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to carry the new sync strategy to every binding") + }) + + It("should take over the ConfigMap on member cluster 1", func() { + Eventually(configMapTakenOverOnClusterActual(memberCluster1EastProd, cmKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to take over the ConfigMap on member cluster 1") + }) + + It("should report the policy as synchronized on every member cluster, on the same resource revision", func() { + wantStatus := allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters))) + wantStatus.LatestResourceRevisionName = ptr.To(initialSnapshotName) + Eventually(placementPolicyStatusActual(policyKey, wantStatus), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected cluster placement policy status") + }) + }) + + Context("placement policy overwriting drifts by default, then reporting them", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyDriftNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + hubData := appConfigMap().Data + driftedData := map[string]string{"data": "drifted"} + // The policy places the ConfigMap on cluster-1 alone, where the specs edit it. + aliasLabels := map[string]string{placementv1beta1.ClusterAliasLabel: memberCluster1EastProdName} + wantSelectorsByCluster := func() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + return map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + memberCluster1EastProdName: {clusterSelectorTermsMatchingLabels(aliasLabels)}, + } + } + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy with the default sync strategy that selects cluster-1") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: []kfplacementv1alpha1.ClusterSelector{ + clusterSelectorMatchingLabels(aliasLabels, intstr.FromInt32(1)), + }, + ResourceSelectors: placementPolicyResourceSelectors(), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources must go even if removing the policy fails; otherwise they leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should place the ConfigMap on member cluster 1", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind member cluster 1") + Eventually(configMapDataOnClustersActual(cmKey, hubData, memberCluster1EastProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on member cluster 1") + }) + + It("can change the ConfigMap on member cluster 1", func() { + updateConfigMapDataOnMemberCluster(memberCluster1EastProd, cmKey, driftedData) + }) + + It("should overwrite the drift, as the default whenDrifted=ApplyAnyway asks", func() { + // The work applier notices member-side changes on its periodic requeue, which the E2E setup caps at 5s. + Eventually(configMapDataOnClustersActual(cmKey, hubData, memberCluster1EastProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to overwrite the drift") + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to keep the binding synchronized") + }) + + It("can switch the policy to report drifts", func() { + updatePlacementPolicy(policyKey, func(policy kfplacementv1alpha1.PlacementPolicyAccessor) { + policy.GetSpec().SyncStrategy = &kfplacementv1alpha1.SyncStrategy{ + WhenDrifted: kfplacementv1alpha1.WhenDriftedOptionReportError, + } + }) + }) + + It("should carry the new sync strategy to the binding without a rollout", func() { + // The actual also checks that the binding carries the policy's (API server defaulted) sync strategy. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to carry the new sync strategy to the binding") + }) + + It("can change the ConfigMap on member cluster 1 again", func() { + updateConfigMapDataOnMemberCluster(memberCluster1EastProd, cmKey, driftedData) + }) + + It("should report the drift on the binding and the policy", func() { + // A ConfigMap carries no generation, so the generation observed on the member cluster is 0. + wantDiffDetails := &kfplacementv1alpha1.DiffDetails{ + ObservedInMemberClusterGeneration: ptr.To(int64(0)), + ObservedDiffs: []kfplacementv1alpha1.PatchDetail{ + {Path: "/data/data", ValueInMember: driftedData["data"], ValueInHub: hubData["data"]}, + }, + } + wantBindings := map[string]wantPlacementBinding{ + memberCluster1EastProdName: { + selectors: wantSelectorsByCluster()[memberCluster1EastProdName], + status: configMapNotAppliedPlacementBindingStatus(cmKey, "FoundDrifts", wantDiffDetails), + }, + } + Eventually(placementPolicyBindingsActual(policyKey, wantBindings), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the drift on the binding") + Eventually(placementPolicyStatusActual(policyKey, someClustersNotSynchronizedPlacementPolicyStatus(1, 0)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the drift on the policy") + }) + + It("should leave the drift on member cluster 1 as it is", func() { + Consistently(configMapDataOnClustersActual(cmKey, driftedData, memberCluster1EastProd), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The drift was overwritten") + }) + + It("can revert the drift on member cluster 1", func() { + updateConfigMapDataOnMemberCluster(memberCluster1EastProd, cmKey, hubData) + }) + + It("should report the policy as synchronized again", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, wantSelectorsByCluster(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the binding as synchronized again") + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(1)), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the policy as synchronized again") + }) + }) + + Context("placement policy orphaning its resources when deleted", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyOrphanNameTemplate, GinkgoParallelProcess()), + } + cmKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appConfigMap().Name} + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + + By("creating a placement policy that orphans its resources when deleted") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: placementPolicyResourceSelectors(), + SyncStrategy: &kfplacementv1alpha1.SyncStrategy{ + WhenPlacementDeleted: kfplacementv1alpha1.WhenPlacementDeletedOptionOrphanResources, + }, + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The orphaned ConfigMaps go with the member namespaces, which the cleanup removes even if deleting the + // policy fails. + defer cleanupWorkResourcesOnHubAndMembers() + deletePlacementPolicyAndWait(policyKey) + }) + + It("should place the ConfigMap on every member cluster, managed by KubeFleet", func() { + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind every member cluster") + for _, cluster := range allMemberClusters { + Eventually(configMapTakenOverOnClusterActual(cluster, cmKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on cluster %s", cluster.ClusterName) + } + }) + + It("can delete the placement policy", func() { + deletePlacementPolicyAndWait(policyKey) + }) + + It("should remove the bindings, Works, and resource snapshots of the policy from the hub", func() { + Eventually(placementPolicyLeftoversRemovedActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the bindings and resource snapshots of the policy") + Eventually(noPlacementPolicyWorksActual(policyKey), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the Works of the policy") + }) + + It("should leave the ConfigMap on every member cluster, no longer owned by KubeFleet", func() { + orphanedActual := configMapOrphanedOnClustersActual(cmKey, appConfigMap().Data, allMemberClusters...) + Eventually(orphanedActual, workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to orphan the ConfigMap on the member clusters") + Consistently(orphanedActual, consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The orphaned ConfigMap was removed, or KubeFleet took it back") + }) + }) + + Context("placement policy placing a Deployment", Ordered, func() { + policyKey := types.NamespacedName{ + Namespace: appNamespace().Name, + Name: fmt.Sprintf(placementPolicyDeploymentNameTemplate, GinkgoParallelProcess()), + } + deployKey := types.NamespacedName{Namespace: appNamespace().Name, Name: appDeployment().Name} + + BeforeAll(func() { + By("creating the work resources and a Deployment") + createWorkResources() + deploy := appDeployment() + Expect(hubClient.Create(ctx, &deploy)).To(Succeed(), "Failed to create Deployment %s", deployKey) + + By("creating a placement policy that selects the ConfigMap and the Deployment") + policy := newPlacementPolicyObject(policyKey, kfplacementv1alpha1.PlacementPolicySpec{ + ResourceSelectors: append(placementPolicyResourceSelectors(), kfplacementv1alpha1.ResourceSelector{ + APIGroup: appsv1.GroupName, + APIVersion: "v1", + Kind: "Deployment", + Name: deployKey.Name, + }), + }) + Expect(hubClient.Create(ctx, policy)).To(Succeed(), "Failed to create placement policy %s", policyKey) + }) + + AfterAll(func() { + // The work resources, the Deployment among them, must go even if removing the policy fails; otherwise they + // leak into later specs. + defer cleanupWorkResourcesOnHubAndMembers() + ensurePlacementPolicyRemoved(policyKey) + }) + + It("should roll the Deployment out on every member cluster", func() { + // The Deployment becomes available only once its image is pulled and its pod is ready. + Eventually(deploymentAvailableOnClustersActual(deployKey, allMemberClusters...), longEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll the Deployment out on every member cluster") + }) + + It("should report both resources as synchronized and available on every binding", func() { + // The work applier tracks the availability of a Deployment by its rollout; a ConfigMap is available once + // applied. + Eventually(allSynchronizedPlacementPolicyBindingsActual(policyKey, noTermsSelectorsForAllMemberClusters(), 2), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the resources as available on every binding") + }) + + It("should report the policy as available on every member cluster", func() { + Eventually(placementPolicyStatusActual(policyKey, allSynchronizedPlacementPolicyStatus(int32(len(allMemberClusters)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected placement policy status") + }) + }) +}) diff --git a/test/e2e/placement_policy_test.go b/test/e2e/placement_policy_test.go new file mode 100644 index 000000000..5252ad36c --- /dev/null +++ b/test/e2e/placement_policy_test.go @@ -0,0 +1,401 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/test/e2e/framework" +) + +// The referenced ConfigMap is the one the shared work resources already provide. +// +// The namespace is set for the cluster-scoped API, which selects namespaced resources by +// qualifying them; the namespaced API ignores the field beyond requiring it to match its own +// namespace. +func placementPolicyResourceSelectors() []kfplacementv1alpha1.ResourceSelector { + return []kfplacementv1alpha1.ResourceSelector{ + { + APIVersion: "v1", + Kind: "ConfigMap", + Name: fmt.Sprintf(appConfigMapNameTemplate, GinkgoParallelProcess()), + Namespace: appNamespace().Name, + }, + } +} + +func placementPolicyRegionSelector(region string, count int32) kfplacementv1alpha1.ClusterSelector { + return kfplacementv1alpha1.ClusterSelector{ + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: map[string]string{regionLabelName: region}}, + }, + Count: ptr.To(intstr.FromInt32(count)), + } +} + +// appConfigMapPlacedOnClustersActual checks that the app ConfigMap on each given cluster matches +// the hub copy. +func appConfigMapPlacedOnClustersActual(clusters ...*framework.Cluster) func() error { + cm := appConfigMap() + return func() error { + for _, c := range clusters { + if err := validateConfigMapOnCluster(c, types.NamespacedName{Name: cm.Name, Namespace: cm.Namespace}); err != nil { + return fmt.Errorf("cluster %s: %w", c.ClusterName, err) + } + } + return nil + } +} + +// appConfigMapRemovedFromClustersActual checks that none of the given clusters has the app ConfigMap. +func appConfigMapRemovedFromClustersActual(clusters ...*framework.Cluster) func() error { + return func() error { + for _, c := range clusters { + if err := namespacedResourcesRemovedFromClusterActual(c)(); err != nil { + return err + } + } + return nil + } +} + +// cleanupWorkResourcesOnHubAndMembers removes the work resources from the hub and every member +// cluster. The placement policy work applier creates the namespaces of the resources it applies +// and leaves them behind on purpose, so the member clusters need cleaning too. +// +// The hub goes first: once its namespace is gone, so are the Works, and nothing re-creates the +// member namespaces. The member cleanup is deferred so that a failed hub cleanup does not leak +// them into later specs. +func cleanupWorkResourcesOnHubAndMembers() { + defer func() { + for _, c := range allMemberClusters { + cleanWorkResourcesOnCluster(c) + } + }() + cleanupWorkResources() +} + +// claimsOfPolicy lists the cluster claims added for a policy through the ownership labels; the +// namespace label is empty for claims of a cluster-scoped policy. +func claimsOfPolicy(name, namespace string) (*kfplacementv1alpha1.ClusterClaimList, error) { + claims := &kfplacementv1alpha1.ClusterClaimList{} + err := hubClient.List(ctx, claims, client.MatchingLabels{ + kfplacementv1alpha1.ClusterClaimPlacementPolicyNameLabel: name, + kfplacementv1alpha1.ClusterClaimPlacementPolicyNamespaceLabel: namespace, + }) + return claims, err +} + +// The FEP-0001 placement experience is alpha and runs behind the enable-placement-policy-apis +// hub agent flag, which test/e2e/setup.sh turns on. +var _ = Describe("placement policy scheduling and cluster claims", Label("placementpolicy"), Ordered, Serial, func() { + policyName := fmt.Sprintf("pp-e2e-%d", GinkgoParallelProcess()) + var policyNS string + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + policyNS = appNamespace().Name + }) + + AfterAll(func() { + cleanupWorkResourcesOnHubAndMembers() + }) + + scheduledConditionActual := func(wantStatus metav1.ConditionStatus, wantReason string, wantScheduled int32) func() error { + return func() error { + policy := &kfplacementv1alpha1.PlacementPolicy{} + if err := hubClient.Get(ctx, types.NamespacedName{Name: policyName, Namespace: policyNS}, policy); err != nil { + return err + } + cond := meta.FindStatusCondition(policy.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeScheduled) + if cond == nil { + return fmt.Errorf("the Scheduled condition is not set yet") + } + if cond.Status != wantStatus || cond.Reason != wantReason { + return fmt.Errorf("Scheduled condition = %s/%s, want %s/%s", cond.Status, cond.Reason, wantStatus, wantReason) + } + if policy.Status.ScheduledClusters == nil || *policy.Status.ScheduledClusters != wantScheduled { + return fmt.Errorf("scheduledClusters = %v, want %d", policy.Status.ScheduledClusters, wantScheduled) + } + return nil + } + } + + newPolicy := func(selectors ...kfplacementv1alpha1.ClusterSelector) *kfplacementv1alpha1.PlacementPolicy { + return &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: policyName, Namespace: policyNS}, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: selectors, + ResourceSelectors: placementPolicyResourceSelectors(), + }, + } + } + + AfterEach(func() { + policy := &kfplacementv1alpha1.PlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: policyName, Namespace: policyNS}} + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, policy))).Should(Succeed(), "Failed to delete the placement policy") + // Policy deletion is finalizer-gated on cluster claim withdrawal, so waiting for the + // policy to disappear also confirms its claims are gone. + Eventually(func() error { + err := hubClient.Get(ctx, types.NamespacedName{Name: policyName, Namespace: policyNS}, &kfplacementv1alpha1.PlacementPolicy{}) + if err == nil { + return fmt.Errorf("the placement policy still exists") + } + return client.IgnoreNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to clean up the placement policy") + Eventually(appConfigMapRemovedFromClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the placed ConfigMap from the member clusters") + }) + + It("should report fulfilled scheduling for a selector the fleet satisfies", func() { + // Two of the three E2E member clusters carry region=east. + Expect(hubClient.Create(ctx, newPolicy(placementPolicyRegionSelector(regionEast, 2)))).Should(Succeed(), "Failed to create the placement policy") + + Eventually(scheduledConditionActual(metav1.ConditionTrue, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, 2), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report fulfilled scheduling") + + By("checking that each selected cluster gets a binding naming the resource snapshot") + Eventually(func() error { + bindings := &kfplacementv1alpha1.PlacementBindingList{} + if err := hubClient.List(ctx, bindings, client.InNamespace(policyNS)); err != nil { + return err + } + got := 0 + for _, b := range bindings.Items { + if b.Spec.PlacementPolicyName != policyName { + continue + } + if b.Spec.ResourceSnapshotName == "" { + return fmt.Errorf("binding %s names no resource snapshot", b.Name) + } + got++ + } + if got != 2 { + return fmt.Errorf("got %d bindings, want 2", got) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind the selected clusters") + + By("checking that the ConfigMap is applied on the east clusters only") + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to apply the ConfigMap on the east clusters") + Consistently(appConfigMapRemovedFromClustersActual(memberCluster3WestProd), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The ConfigMap was applied on a cluster that was not selected") + }) + + It("should roll out resource changes automatically when auto-rollout is enabled", func() { + policy := newPolicy(placementPolicyRegionSelector(regionEast, 2)) + policy.Annotations = map[string]string{kfrolloutv1alpha1.AutoRolloutEnabledAnnotationKey: kfrolloutv1alpha1.AutoRolloutEnabledAnnotationValue} + Expect(hubClient.Create(ctx, policy)).Should(Succeed(), "Failed to create the placement policy") + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster2EastCanary), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to apply the ConfigMap on the east clusters") + + By("updating the ConfigMap on the hub") + cm := appConfigMap() + Eventually(func() error { + hubCM := &corev1.ConfigMap{} + if err := hubClient.Get(ctx, types.NamespacedName{Name: cm.Name, Namespace: cm.Namespace}, hubCM); err != nil { + return err + } + hubCM.Data["data"] = "auto-rolled-out" + return hubClient.Update(ctx, hubCM) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the ConfigMap on the hub") + + // No staged update run is created; only the auto-rollout controller can deliver the change. + By("checking that the change reaches the east clusters") + Eventually(appConfigMapPlacedOnClustersActual(memberCluster1EastProd, memberCluster2EastCanary), longEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to roll out the ConfigMap change") + // The timestamp is written after the bindings are refreshed, so it can trail the change. + Eventually(func() error { + if err := hubClient.Get(ctx, types.NamespacedName{Name: policyName, Namespace: policyNS}, policy); err != nil { + return err + } + if _, ok := policy.Annotations[kfrolloutv1alpha1.AutoRolloutLastRolloutAttemptTimestampAnnotationKey]; !ok { + return fmt.Errorf("the policy has no last rollout attempt timestamp") + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "The auto-rollout controller did not record the rollout") + }) + + It("should add a cluster claim for an unfulfillable selector and withdraw it with the policy", func() { + Expect(hubClient.Create(ctx, newPolicy(placementPolicyRegionSelector("nowhere", 1)))).Should(Succeed(), "Failed to create the placement policy") + + By("checking that the policy reports unfulfilled scheduling") + Eventually(scheduledConditionActual(metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters, 0), + eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report unfulfilled scheduling") + + By("checking that a cluster claim carrying the selector terms is added") + var claimName string + Eventually(func() error { + claims, err := claimsOfPolicy(policyName, policyNS) + if err != nil { + return err + } + if len(claims.Items) != 1 { + return fmt.Errorf("got %d cluster claims, want 1", len(claims.Items)) + } + claim := claims.Items[0] + if claim.Spec.PlacementPolicyRef == nil || + claim.Spec.PlacementPolicyRef.Name != policyName || + claim.Spec.PlacementPolicyRef.Kind != kfplacementv1alpha1.PlacementPolicyKind { + return fmt.Errorf("the cluster claim does not reference the policy") + } + if len(claim.Spec.ClusterSelectorTerms) != 1 || claim.Spec.ClusterSelectorTerms[0].MatchLabels[regionLabelName] != "nowhere" { + return fmt.Errorf("the cluster claim does not carry the selector terms") + } + claimName = claim.Name + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cluster claim") + + By("checking that deleting the policy withdraws the claim") + Expect(hubClient.Delete(ctx, &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: policyName, Namespace: policyNS}, + })).Should(Succeed(), "Failed to delete the placement policy") + Eventually(func() error { + err := hubClient.Get(ctx, types.NamespacedName{Name: claimName}, &kfplacementv1alpha1.ClusterClaim{}) + if err == nil { + return fmt.Errorf("the cluster claim still exists") + } + return client.IgnoreNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to withdraw the cluster claim") + }) +}) + +// The cluster-scoped policy has its own watch wiring and claim reference path, so it gets its +// own end-to-end coverage. +var _ = Describe("cluster placement policy scheduling and cluster claims", Label("placementpolicy"), Ordered, Serial, func() { + policyName := fmt.Sprintf("cpp-e2e-%d", GinkgoParallelProcess()) + + BeforeAll(func() { + By("creating the work resources") + createWorkResources() + }) + + AfterAll(func() { + cleanupWorkResourcesOnHubAndMembers() + }) + + newClusterPolicy := func(selectors ...kfplacementv1alpha1.ClusterSelector) *kfplacementv1alpha1.ClusterPlacementPolicy { + return &kfplacementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: policyName}, + Spec: kfplacementv1alpha1.PlacementPolicySpec{ + ClusterSelectors: selectors, + ResourceSelectors: placementPolicyResourceSelectors(), + }, + } + } + + AfterEach(func() { + policy := &kfplacementv1alpha1.ClusterPlacementPolicy{ObjectMeta: metav1.ObjectMeta{Name: policyName}} + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, policy))).Should(Succeed(), "Failed to delete the cluster placement policy") + Eventually(func() error { + err := hubClient.Get(ctx, types.NamespacedName{Name: policyName}, &kfplacementv1alpha1.ClusterPlacementPolicy{}) + if err == nil { + return fmt.Errorf("the cluster placement policy still exists") + } + return client.IgnoreNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to clean up the cluster placement policy") + Eventually(appConfigMapRemovedFromClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the placed ConfigMap from the member clusters") + }) + + It("should schedule against the fleet and claim for an unfulfillable selector", func() { + Expect(hubClient.Create(ctx, newClusterPolicy( + placementPolicyRegionSelector(regionWest, 1), + placementPolicyRegionSelector("nowhere", 1), + ))).Should(Succeed(), "Failed to create the cluster placement policy") + + By("checking that the fulfillable selector is scheduled while the policy stays unfulfilled overall") + Eventually(func() error { + policy := &kfplacementv1alpha1.ClusterPlacementPolicy{} + if err := hubClient.Get(ctx, types.NamespacedName{Name: policyName}, policy); err != nil { + return err + } + cond := meta.FindStatusCondition(policy.Status.Conditions, kfplacementv1alpha1.PlacementPolicyCondTypeScheduled) + if cond == nil { + return fmt.Errorf("the Scheduled condition is not set yet") + } + if cond.Status != metav1.ConditionFalse || cond.Reason != kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters { + return fmt.Errorf("Scheduled condition = %s/%s, want False/%s", cond.Status, cond.Reason, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters) + } + // One cluster carries region=west; the second selector matches nothing. + if policy.Status.ScheduledClusters == nil || *policy.Status.ScheduledClusters != 1 { + return fmt.Errorf("scheduledClusters = %v, want 1", policy.Status.ScheduledClusters) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to report the expected scheduling status") + + By("checking that a cluster claim referencing the cluster-scoped policy is added") + Eventually(func() error { + claims, err := claimsOfPolicy(policyName, "") + if err != nil { + return err + } + if len(claims.Items) != 1 { + return fmt.Errorf("got %d cluster claims, want 1", len(claims.Items)) + } + ref := claims.Items[0].Spec.PlacementPolicyRef + if ref == nil || ref.Kind != kfplacementv1alpha1.ClusterPlacementPolicyKind || ref.Namespace != "" { + return fmt.Errorf("the cluster claim does not reference the cluster placement policy") + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to add the cluster claim") + + By("checking that the west cluster gets a cluster binding naming the resource snapshot") + Eventually(func() error { + bindings := &kfplacementv1alpha1.ClusterPlacementBindingList{} + if err := hubClient.List(ctx, bindings); err != nil { + return err + } + var owned []kfplacementv1alpha1.ClusterPlacementBinding + for _, b := range bindings.Items { + if b.Spec.PlacementPolicyName == policyName { + owned = append(owned, b) + } + } + if len(owned) != 1 { + return fmt.Errorf("got %d cluster bindings, want 1", len(owned)) + } + if owned[0].Spec.ClusterName != memberCluster3WestProdName { + return fmt.Errorf("cluster binding %s targets %s, want %s", owned[0].Name, owned[0].Spec.ClusterName, memberCluster3WestProdName) + } + if owned[0].Spec.ResourceSnapshotName == "" { + return fmt.Errorf("cluster binding %s names no resource snapshot", owned[0].Name) + } + return nil + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind the west cluster") + + By("checking that the ConfigMap is applied on the west cluster only") + Eventually(appConfigMapPlacedOnClustersActual(memberCluster3WestProd), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to apply the ConfigMap on the west cluster") + Consistently(appConfigMapRemovedFromClustersActual(memberCluster1EastProd, memberCluster2EastCanary), consistentlyDuration, consistentlyInterval). + Should(Succeed(), "The ConfigMap was applied on a cluster that was not selected") + }) +}) diff --git a/test/e2e/placement_policy_utils_test.go b/test/e2e/placement_policy_utils_test.go new file mode 100644 index 000000000..36e614a7e --- /dev/null +++ b/test/e2e/placement_policy_utils_test.go @@ -0,0 +1,1025 @@ +/* +Copyright 2026 The KubeFleet Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package e2e + +import ( + "fmt" + "maps" + "slices" + "strconv" + "time" + + "github.com/google/go-cmp/cmp" + "github.com/google/go-cmp/cmp/cmpopts" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/intstr" + "k8s.io/utils/ptr" + "sigs.k8s.io/controller-runtime/pkg/client" + + clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" + "go.goms.io/fleet/pkg/utils" + "go.goms.io/fleet/test/e2e/framework" +) + +// The helpers in this file work with both scopes of the placement policy APIs: a key with an empty +// namespace refers to a ClusterPlacementPolicy, and one with a namespace to a PlacementPolicy. + +// clusterSelectorCountAll is the cluster selector count that selects every matching cluster. +const clusterSelectorCountAll = "All" + +// placementPolicyCmpOptions compares placement policy and binding statuses, ignoring the condition fields +// that carry no contract (transition times and messages). +var placementPolicyCmpOptions = cmp.Options{ + cmpopts.SortSlices(lessFuncCondition), + utils.IgnoreConditionLTTAndMessageFields, + cmpopts.EquateEmpty(), +} + +// newPlacementPolicyObject returns a PlacementPolicy or a ClusterPlacementPolicy, depending on the key. +func newPlacementPolicyObject(key types.NamespacedName, spec kfplacementv1alpha1.PlacementPolicySpec) kfplacementv1alpha1.PlacementPolicyAccessor { + if key.Namespace == "" { + return &kfplacementv1alpha1.ClusterPlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: key.Name}, + Spec: spec, + } + } + return &kfplacementv1alpha1.PlacementPolicy{ + ObjectMeta: metav1.ObjectMeta{Name: key.Name, Namespace: key.Namespace}, + Spec: spec, + } +} + +// retrievePlacementPolicy reads a PlacementPolicy or a ClusterPlacementPolicy, depending on the key. +func retrievePlacementPolicy(key types.NamespacedName) (kfplacementv1alpha1.PlacementPolicyAccessor, error) { + policy := newPlacementPolicyObject(key, kfplacementv1alpha1.PlacementPolicySpec{}) + if err := hubClient.Get(ctx, key, policy); err != nil { + return nil, err + } + return policy, nil +} + +// clusterSelectorMatchingLabels builds a cluster selector with a single term that matches the given labels. +func clusterSelectorMatchingLabels(matchLabels map[string]string, count intstr.IntOrString) kfplacementv1alpha1.ClusterSelector { + return kfplacementv1alpha1.ClusterSelector{ + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: matchLabels}, + }, + Count: ptr.To(count), + } +} + +// noTermsSelectorsForAllMemberClusters returns, keyed by cluster, how every member cluster's binding records the +// cluster selectors of a policy that has none: as a single selector without terms. +func noTermsSelectorsForAllMemberClusters() map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + selectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + selectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{}} + } + return selectorsByCluster +} + +// clusterSelectorTermsMatchingLabels returns how a binding records a cluster selector built by +// clusterSelectorMatchingLabels: its terms, without the counts. +func clusterSelectorTermsMatchingLabels(matchLabels map[string]string) kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + return kfplacementv1alpha1.ClusterSelectorWithTermsOnly{ + Terms: []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm{ + {MatchLabels: matchLabels}, + }, + } +} + +// clusterSelectorWithTerms builds a cluster selector with the given terms and count. +func clusterSelectorWithTerms(terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm, count intstr.IntOrString) kfplacementv1alpha1.ClusterSelector { + return kfplacementv1alpha1.ClusterSelector{ + Terms: terms, + Count: ptr.To(count), + } +} + +// sameSelectorsForClusters returns, keyed by cluster, how the binding of each given cluster records the cluster +// selectors of a policy whose only selector, with the given terms, selects them all. +func sameSelectorsForClusters( + terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm, + clusterNames ...string, +) map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly { + selectorsByCluster := make(map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, len(clusterNames)) + for _, clusterName := range clusterNames { + selectorsByCluster[clusterName] = []kfplacementv1alpha1.ClusterSelectorWithTermsOnly{{Terms: terms}} + } + return selectorsByCluster +} + +// waitForPlacementPolicyPlacedOnlyOn waits until the policy, whose only cluster selector has the given terms, binds +// exactly the given clusters, has the app ConfigMap placed on them and removed from every other member cluster, and +// reports all of them as synchronized and available. +func waitForPlacementPolicyPlacedOnlyOn( + key types.NamespacedName, + terms []kfplacementv1alpha1.ClusterLabelAndPropertySelectorTerm, + clusterNames ...string, +) { + GinkgoHelper() + + var bound, unbound []*framework.Cluster + for _, cluster := range allMemberClusters { + if slices.Contains(clusterNames, cluster.ClusterName) { + bound = append(bound, cluster) + } else { + unbound = append(unbound, cluster) + } + } + + Eventually(allSynchronizedPlacementPolicyBindingsActual(key, sameSelectorsForClusters(terms, clusterNames...), 1), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to bind exactly clusters %v for placement policy %s", clusterNames, key) + Eventually(appConfigMapPlacedOnClustersActual(bound...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to place the ConfigMap on clusters %v", clusterNames) + Eventually(appConfigMapRemovedFromClustersActual(unbound...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the ConfigMap from the clusters other than %v", clusterNames) + Eventually(placementPolicyStatusActual(key, allSynchronizedPlacementPolicyStatus(int32(len(clusterNames)))), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to report the expected status for placement policy %s", key) +} + +// placementPolicyBindingUIDsByCluster returns, keyed by cluster, the UIDs of the bindings that name the policy, so +// that specs can check later that the bindings were kept rather than replaced. +func placementPolicyBindingUIDsByCluster(key types.NamespacedName) map[string]types.UID { + GinkgoHelper() + + bindings, err := placementPolicyBindingsByCluster(key) + Expect(err).NotTo(HaveOccurred(), "Failed to list the bindings of placement policy %s", key) + uids := make(map[string]types.UID, len(bindings)) + for clusterName, binding := range bindings { + uids[clusterName] = binding.GetUID() + } + return uids +} + +// waitPastCreationSecond waits until the second after the one in which a binding was created has begun, with a +// margin (the Kind clusters share the host clock). +// +// When a selector picks too many clusters, the placement policy controller trims the newest bindings first, by +// creation timestamp, which has a resolution of one second, and breaks ties by cluster name. Specs that rely on which +// of two bindings is the newer one call this before causing the second binding, so that it is stamped a later second +// than the first. +func waitPastCreationSecond(createdAt metav1.Time) { + if wait := time.Until(createdAt.Add(time.Second + 250*time.Millisecond)); wait > 0 { + time.Sleep(wait) + } +} + +// placementPolicyTolerationsFor returns the tolerations of the taints that buildTaints puts on the given member +// clusters. +func placementPolicyTolerationsFor(clusterNames ...string) []kfplacementv1alpha1.Toleration { + tolerations := make([]kfplacementv1alpha1.Toleration, 0, len(clusterNames)) + for _, clusterName := range clusterNames { + tolerations = append(tolerations, kfplacementv1alpha1.Toleration{ + Key: regionLabelName, + Operator: corev1.TolerationOpEqual, + Value: taintTolerationMap[clusterName][regionLabelName], + Effect: corev1.TaintEffectNoSchedule, + }) + } + return tolerations +} + +// placementPolicyBindingsByCluster returns, keyed by cluster, the bindings that name the policy. Unlike +// placementPolicyBindings, it needs no policy object, so it also works before the policy has a snapshot or after +// it is gone. +func placementPolicyBindingsByCluster(key types.NamespacedName) (map[string]kfplacementv1alpha1.PlacementBindingAccessor, error) { + bindings, err := placementPolicyBindings(newPlacementPolicyObject(key, kfplacementv1alpha1.PlacementPolicySpec{})) + if err != nil { + return nil, err + } + byCluster := make(map[string]kfplacementv1alpha1.PlacementBindingAccessor, len(bindings)) + for _, binding := range bindings { + clusterName := binding.GetSpec().ClusterName + if _, found := byCluster[clusterName]; found { + return nil, fmt.Errorf("placement policy %s has more than one binding for cluster %s", key, clusterName) + } + byCluster[clusterName] = binding + } + return byCluster, nil +} + +// waitForSinglePlacementPolicyBinding waits until the policy has exactly one binding, and returns it. Specs use it to +// learn which cluster a ranked pick chose, since the ranking depends on live capacity. +func waitForSinglePlacementPolicyBinding(key types.NamespacedName) kfplacementv1alpha1.PlacementBindingAccessor { + GinkgoHelper() + + var single kfplacementv1alpha1.PlacementBindingAccessor + Eventually(func() error { + bindings, err := placementPolicyBindingsByCluster(key) + if err != nil { + return err + } + if len(bindings) != 1 { + return fmt.Errorf("placement policy %s has %d bindings, want 1", key, len(bindings)) + } + for _, binding := range bindings { + single = binding + } + return nil + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to bind a single cluster for placement policy %s", key) + return single +} + +// otherEastClusterName returns the name of the east member cluster other than the given one. +func otherEastClusterName(clusterName string) string { + GinkgoHelper() + + switch clusterName { + case memberCluster1EastProdName: + return memberCluster2EastCanaryName + case memberCluster2EastCanaryName: + return memberCluster1EastProdName + default: + Fail(fmt.Sprintf("member cluster %s is not in the east region", clusterName)) + return "" + } +} + +// resourcesChangedPendingRolloutPlacementPolicyStatus returns the status of a policy whose clusters all hold, and have +// synchronized and available, its latest resource revision, while the selected resources on the hub have changed +// since and wait for a rollout. +// +// The latest resource revision name and the observed generations are left for the actual to fill in. +func resourcesChangedPendingRolloutPlacementPolicyStatus(clusterCount int32) kfplacementv1alpha1.PlacementPolicyStatus { + status := allSynchronizedPlacementPolicyStatus(clusterCount) + meta.SetStatusCondition(&status.Conditions, placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, + metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonResourcesChangedPendingRollout)) + return status +} + +// resourcesNotCollectedPlacementPolicyStatus returns the status of a policy that has found all of its clusterCount +// desired clusters but has never collected its selected resources, so it has no resource snapshot, no latest +// resource revision, and no binding. +func resourcesNotCollectedPlacementPolicyStatus(clusterCount int32) kfplacementv1alpha1.PlacementPolicyStatus { + return kfplacementv1alpha1.PlacementPolicyStatus{ + Conditions: []metav1.Condition{ + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonFailedToCollectSomeResources), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters), + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters), + // Availability is still judged, on the bindings alone, and there are none. + placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable), + }, + DesiredClusters: ptr.To(clusterCount), + ScheduledClusters: ptr.To(clusterCount), + SynchronizedClusters: ptr.To(int32(0)), + ResourcesAvailableClusters: ptr.To(int32(0)), + ActiveClusterClaims: ptr.To(int32(0)), + } +} + +// someClustersNotSynchronizedPlacementPolicyStatus returns the status of a policy that has found all of its +// clusterCount desired clusters, of which only synchronizedCount have the latest resource revision synchronized +// (and available), while the others failed to apply it. +// +// The latest resource revision name and the observed generations are left for the actual to fill in. +func someClustersNotSynchronizedPlacementPolicyStatus(clusterCount, synchronizedCount int32) kfplacementv1alpha1.PlacementPolicyStatus { + status := allSynchronizedPlacementPolicyStatus(clusterCount) + meta.SetStatusCondition(&status.Conditions, placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, + metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonFailedToSynchronizeSomeClusters)) + // Availability is judged only on clusters that have the latest revision synchronized. + meta.SetStatusCondition(&status.Conditions, placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, + metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyAvailableCondReasonSomeClustersUnavailable)) + status.SynchronizedClusters = ptr.To(synchronizedCount) + status.ResourcesAvailableClusters = ptr.To(synchronizedCount) + return status +} + +// waitForPlacementPolicySynchronized waits until the policy has exactly the wanted bindings, each recording the given +// cluster selectors and reporting all of its resourceCount resources as synchronized and available on the policy's +// latest resource snapshot; it returns the name of that snapshot. +func waitForPlacementPolicySynchronized( + key types.NamespacedName, + wantSelectorsByCluster map[string][]kfplacementv1alpha1.ClusterSelectorWithTermsOnly, + resourceCount int32, +) string { + GinkgoHelper() + + Eventually(allSynchronizedPlacementPolicyBindingsActual(key, wantSelectorsByCluster, resourceCount), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to synchronize the bindings of placement policy %s", key) + return latestPrimarySnapshotNameOfPlacementPolicy(key) +} + +// latestPrimarySnapshotNameOfPlacementPolicy returns the name of the policy's latest primary resource snapshot. +func latestPrimarySnapshotNameOfPlacementPolicy(key types.NamespacedName) string { + GinkgoHelper() + + policy, err := retrievePlacementPolicy(key) + Expect(err).NotTo(HaveOccurred(), "Failed to get placement policy %s", key) + snapshot, err := latestPrimarySnapshotOfPlacementPolicy(policy) + Expect(err).NotTo(HaveOccurred(), "Failed to find the resource snapshot of placement policy %s", key) + return snapshot.GetName() +} + +// resourceSnapshotNameToRolloutOf returns the resource snapshot that an initialized staged update run (or cluster +// staged update run) rolls out. +func resourceSnapshotNameToRolloutOf(runKey types.NamespacedName) string { + GinkgoHelper() + + run, err := retrieveStagedUpdateRun(runKey) + Expect(err).NotTo(HaveOccurred(), "Failed to get staged update run %s", runKey) + name := run.GetStatus().ResourceSnapshotNameToRollout + Expect(name).NotTo(BeEmpty(), "Staged update run %s names no resource snapshot to roll out", runKey) + return name +} + +// someClustersOnOlderResourceRevisionPlacementPolicyStatus returns the status of a policy that has found all of its +// clusterCount desired clusters, of which only synchronizedCount have the latest resource revision synchronized (and +// available), while the others are still on an older revision. +// +// The latest resource revision name and the observed generations are left for the actual to fill in. +func someClustersOnOlderResourceRevisionPlacementPolicyStatus(clusterCount, synchronizedCount int32) kfplacementv1alpha1.PlacementPolicyStatus { + status := someClustersNotSynchronizedPlacementPolicyStatus(clusterCount, synchronizedCount) + meta.SetStatusCondition(&status.Conditions, placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, + metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonSomeClustersOnOlderResourceRevision)) + return status +} + +// unfulfilledPlacementPolicyStatus returns the status of a policy that wants desiredCount clusters but has found only +// scheduledCount, with activeClaims cluster claims outstanding for the missing ones. Each found cluster has the +// latest resource revision synchronized and available; with no target cluster at all, nothing is out of sync or +// unavailable either. +// +// The latest resource revision name and the observed generations are left for the actual to fill in. +func unfulfilledPlacementPolicyStatus(desiredCount, scheduledCount, activeClaims int32) kfplacementv1alpha1.PlacementPolicyStatus { + status := allSynchronizedPlacementPolicyStatus(scheduledCount) + meta.SetStatusCondition(&status.Conditions, placementPolicyCondition(kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, + metav1.ConditionFalse, kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFailedToFindSomeClusters)) + status.DesiredClusters = ptr.To(desiredCount) + status.ActiveClusterClaims = ptr.To(activeClaims) + return status +} + +// placementPolicyScheduledCondReasonInvalidClusterSelectors is the Scheduled condition reason of a policy whose +// cluster selectors cannot be evaluated. The API defines no constant for it, so it is spelled out the way the +// placement policy controller reports it. +const placementPolicyScheduledCondReasonInvalidClusterSelectors = "InvalidClusterSelectors" + +// otherProdClusterName returns the name of the prod member cluster other than the given one. +func otherProdClusterName(clusterName string) string { + GinkgoHelper() + + switch clusterName { + case memberCluster1EastProdName: + return memberCluster3WestProdName + case memberCluster3WestProdName: + return memberCluster1EastProdName + default: + Fail(fmt.Sprintf("member cluster %s is not a prod cluster", clusterName)) + return "" + } +} + +// waitForStageApprovalRequest waits until a staged update run (or a cluster staged update run) has created the +// approval request of the approval task of the stage at the given index, before or after the stage, and returns its +// key. +func waitForStageApprovalRequest(runKey types.NamespacedName, stageIdx int, beforeStage bool) types.NamespacedName { + GinkgoHelper() + + var requestKey types.NamespacedName + Eventually(func() error { + run, err := retrieveStagedUpdateRun(runKey) + if err != nil { + return err + } + stages := run.GetStatus().Stages + if len(stages) <= stageIdx { + return fmt.Errorf("staged update run %s has no status for stage %d yet", runKey, stageIdx) + } + tasks := stages[stageIdx].AfterStageTasks + if beforeStage { + tasks = stages[stageIdx].BeforeStageTasks + } + taskIdx := slices.IndexFunc(tasks, func(task kfrolloutv1alpha1.PerStageTaskStatus) bool { + return task.Type == kfrolloutv1alpha1.StageTaskTypeApproval + }) + if taskIdx < 0 { + return fmt.Errorf("staged update run %s has no approval task status for stage %d yet (before stage: %t)", runKey, stageIdx, beforeStage) + } + task := tasks[taskIdx] + if task.ApprovalRequestName == "" || + !meta.IsStatusConditionTrue(task.Conditions, kfrolloutv1alpha1.StagedUpdateRunTaskCondTypeApprovalRequestCreated) { + return fmt.Errorf("staged update run %s has not created the approval request for stage %d yet (task status: %+v)", runKey, stageIdx, task) + } + // An approval request lives in the scope of its run. + requestKey = types.NamespacedName{Namespace: runKey.Namespace, Name: task.ApprovalRequestName} + return nil + }, placementPolicyRolloutEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to create the approval request of staged update run %s", runKey) + return requestKey +} + +// updateStagedUpdateRun applies the mutation to a staged update run (or a cluster staged update run), retrying on +// conflicts with the run controller's own writes. +func updateStagedUpdateRun(key types.NamespacedName, mutate func(run kfrolloutv1alpha1.StagedUpdateRunAccessor)) { + GinkgoHelper() + + Eventually(func() error { + run, err := retrieveStagedUpdateRun(key) + if err != nil { + return err + } + mutate(run) + return hubClient.Update(ctx, run) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update staged update run %s", key) +} + +// succeededStagedUpdateRunConditions returns the conditions of a staged update run that has completed successfully. +func succeededStagedUpdateRunConditions() []metav1.Condition { + return []metav1.Condition{ + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunCondTypeInitialized, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunInitializedCondReasonPreppedResourceSnapshotAndAllStages), + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunCondTypeStarted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunStartedCondReasonUpdateStarted), + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunCondTypeCompleted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunCompletedCondReasonSucceeded), + } +} + +// succeededStagedUpdateRunStageConditions returns the conditions of a stage that has completed successfully. +func succeededStagedUpdateRunStageConditions() []metav1.Condition { + return []metav1.Condition{ + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunPerStageCondTypeStarted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunPerStageStartedCondReasonUpdateStarted), + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunPerStageCondTypeCompleted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunPerStageCompletedCondReasonSucceeded), + } +} + +// succeededStagedUpdateRunClusterStatus returns the status of a cluster that a staged update run has updated +// successfully through the given binding. +func succeededStagedUpdateRunClusterStatus(clusterName, bindingName string) kfrolloutv1alpha1.PerClusterStatus { + return kfrolloutv1alpha1.PerClusterStatus{ + ClusterName: clusterName, + PlacementBindingName: bindingName, + Conditions: []metav1.Condition{ + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunPerClusterCondTypeStarted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunPerClusterStartedCondReasonUpdateStarted), + placementPolicyCondition(kfrolloutv1alpha1.StagedUpdateRunPerClusterCondTypeCompleted, metav1.ConditionTrue, + kfrolloutv1alpha1.StagedUpdateRunPerClusterCompletedCondReasonSucceeded), + }, + } +} + +// placementPolicyBindingNamesByCluster returns, keyed by cluster, the names of the bindings that name the policy. +func placementPolicyBindingNamesByCluster(key types.NamespacedName) map[string]string { + GinkgoHelper() + + bindings, err := placementPolicyBindingsByCluster(key) + Expect(err).NotTo(HaveOccurred(), "Failed to list the bindings of placement policy %s", key) + names := make(map[string]string, len(bindings)) + for clusterName, binding := range bindings { + names[clusterName] = binding.GetName() + } + return names +} + +// placementPolicyWorks lists the Works on the hub that were generated for the policy's bindings. Works sit in the +// reserved namespaces of the member clusters and record their policy in an owner annotation (the owner label may +// hold a shortened name) and the policy's namespace in an owner label. +func placementPolicyWorks(key types.NamespacedName) ([]kfplacementv1alpha1.Work, error) { + works := &kfplacementv1alpha1.WorkList{} + if err := hubClient.List(ctx, works, client.MatchingLabels{kfplacementv1alpha1.WorkOwnerNamespaceLabelKey: key.Namespace}); err != nil { + return nil, err + } + var owned []kfplacementv1alpha1.Work + for i := range works.Items { + if works.Items[i].GetAnnotations()[kfplacementv1alpha1.WorkOwnedByPlacementPolicyAnnotationKey] == key.Name { + owned = append(owned, works.Items[i]) + } + } + return owned, nil +} + +// placementPolicyWorkNamesByCluster returns, keyed by the member cluster they are for, the names of the Works on the +// hub that were generated for the policy's bindings. An AppliedWork carries the name of its Work. +func placementPolicyWorkNamesByCluster(key types.NamespacedName) (map[string][]string, error) { + works, err := placementPolicyWorks(key) + if err != nil { + return nil, err + } + clusterNamesByReservedNamespace := make(map[string]string, len(allMemberClusterNames)) + for _, clusterName := range allMemberClusterNames { + clusterNamesByReservedNamespace[fmt.Sprintf(utils.NamespaceNameFormat, clusterName)] = clusterName + } + namesByCluster := make(map[string][]string, len(allMemberClusterNames)) + for i := range works { + clusterName, found := clusterNamesByReservedNamespace[works[i].Namespace] + if !found { + return nil, fmt.Errorf("work %s of placement policy %s is not in the reserved namespace of a member cluster", client.ObjectKeyFromObject(&works[i]), key) + } + namesByCluster[clusterName] = append(namesByCluster[clusterName], works[i].Name) + } + return namesByCluster, nil +} + +// setPlacementPolicyBindingSuspended suspends or resumes the binding of the policy for the given cluster, retrying on +// conflicts. Users do not normally edit the field; upcoming APIs build on it, and specs set it as a stand-in for them. +func setPlacementPolicyBindingSuspended(key types.NamespacedName, clusterName string, suspended bool) { + GinkgoHelper() + + Eventually(func() error { + bindings, err := placementPolicyBindingsByCluster(key) + if err != nil { + return err + } + binding, found := bindings[clusterName] + if !found { + return fmt.Errorf("placement policy %s has no binding for cluster %s", key, clusterName) + } + binding.GetSpec().Suspended = suspended + return hubClient.Update(ctx, binding) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to set suspended=%t on the binding of placement policy %s for cluster %s", suspended, key, clusterName) +} + +// memberClusterNamed returns the member cluster of the given name. +func memberClusterNamed(name string) *framework.Cluster { + GinkgoHelper() + + for _, cluster := range allMemberClusters { + if cluster.ClusterName == name { + return cluster + } + } + Fail(fmt.Sprintf("no member cluster is named %s", name)) + return nil +} + +// setMemberClusterLabel sets a label on a MemberCluster object on the hub, retrying on conflicts. Specs that call +// it must be Serial and restore the label afterwards, as every spec relies on the labels that the suite sets up. +func setMemberClusterLabel(clusterName, key, value string) { + GinkgoHelper() + + Eventually(func() error { + mc := &clusterv1beta1.MemberCluster{} + if err := hubClient.Get(ctx, types.NamespacedName{Name: clusterName}, mc); err != nil { + return err + } + if mc.Labels == nil { + mc.Labels = make(map[string]string) + } + mc.Labels[key] = value + return hubClient.Update(ctx, mc) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to set label %s=%s on member cluster %s", key, value, clusterName) +} + +// ownedByPlacementPolicyAppliedWork reports whether an object on a member cluster is owned by an AppliedWork of the +// placement policy APIs, i.e., the placement policy work applier manages it. +func ownedByPlacementPolicyAppliedWork(obj metav1.Object) bool { + for _, ownerRef := range obj.GetOwnerReferences() { + if ownerRef.APIVersion == kfplacementv1alpha1.GroupVersion.String() && ownerRef.Kind == kfplacementv1alpha1.AppliedWorkKind { + return true + } + } + return false +} + +// placementPolicyBindings lists the bindings that name the policy, whether or not the policy controls them; +// callers check the controller reference, so that a binding left behind by an earlier policy of the same name +// is reported rather than silently skipped. +func placementPolicyBindings(policy kfplacementv1alpha1.PlacementPolicyAccessor) ([]kfplacementv1alpha1.PlacementBindingAccessor, error) { + var all []kfplacementv1alpha1.PlacementBindingAccessor + if policy.GetNamespace() == "" { + bindings := &kfplacementv1alpha1.ClusterPlacementBindingList{} + if err := hubClient.List(ctx, bindings); err != nil { + return nil, err + } + for i := range bindings.Items { + all = append(all, &bindings.Items[i]) + } + } else { + bindings := &kfplacementv1alpha1.PlacementBindingList{} + if err := hubClient.List(ctx, bindings, client.InNamespace(policy.GetNamespace())); err != nil { + return nil, err + } + for i := range bindings.Items { + all = append(all, &bindings.Items[i]) + } + } + + named := make([]kfplacementv1alpha1.PlacementBindingAccessor, 0, len(all)) + for _, binding := range all { + if binding.GetSpec().PlacementPolicyName == policy.GetName() { + named = append(named, binding) + } + } + return named, nil +} + +// placementPolicySnapshots lists the resource snapshots, primary and secondary, that name the policy through the owner +// annotation, which, unlike the owner label, always holds the full policy name. It needs no policy object, so it +// also finds the snapshots of a policy that is gone. +func placementPolicySnapshots(key types.NamespacedName) ([]kfplacementv1alpha1.PlacementResourceSnapshotAccessor, error) { + var all []kfplacementv1alpha1.PlacementResourceSnapshotAccessor + if key.Namespace == "" { + list := &kfplacementv1alpha1.ClusterPlacementResourceSnapshotList{} + if err := hubClient.List(ctx, list); err != nil { + return nil, err + } + for i := range list.Items { + all = append(all, &list.Items[i]) + } + } else { + list := &kfplacementv1alpha1.PlacementResourceSnapshotList{} + if err := hubClient.List(ctx, list, client.InNamespace(key.Namespace)); err != nil { + return nil, err + } + for i := range list.Items { + all = append(all, &list.Items[i]) + } + } + + named := make([]kfplacementv1alpha1.PlacementResourceSnapshotAccessor, 0, len(all)) + for _, snapshot := range all { + if snapshot.GetAnnotations()[kfplacementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] == key.Name { + named = append(named, snapshot) + } + } + return named, nil +} + +// latestPrimarySnapshotOfPlacementPolicy returns the primary resource snapshot of the highest index that the +// policy controls. +// +// Snapshots are matched on the owner annotation, which, unlike the owner label, always holds the full +// policy name. +func latestPrimarySnapshotOfPlacementPolicy(policy kfplacementv1alpha1.PlacementPolicyAccessor) (kfplacementv1alpha1.PlacementResourceSnapshotAccessor, error) { + primaryOnly := client.MatchingLabels{kfplacementv1alpha1.PlacementResourceSnapshotSubIndexLabelKey: "0"} + var all []kfplacementv1alpha1.PlacementResourceSnapshotAccessor + if policy.GetNamespace() == "" { + snapshots := &kfplacementv1alpha1.ClusterPlacementResourceSnapshotList{} + if err := hubClient.List(ctx, snapshots, primaryOnly); err != nil { + return nil, err + } + for i := range snapshots.Items { + all = append(all, &snapshots.Items[i]) + } + } else { + snapshots := &kfplacementv1alpha1.PlacementResourceSnapshotList{} + if err := hubClient.List(ctx, snapshots, client.InNamespace(policy.GetNamespace()), primaryOnly); err != nil { + return nil, err + } + for i := range snapshots.Items { + all = append(all, &snapshots.Items[i]) + } + } + + var latest kfplacementv1alpha1.PlacementResourceSnapshotAccessor + latestIdx := -1 + for _, snapshot := range all { + if snapshot.GetAnnotations()[kfplacementv1alpha1.PlacementResourceSnapshotOwnedByAnnotationKey] != policy.GetName() || + !metav1.IsControlledBy(snapshot, policy) { + continue + } + idxStr := snapshot.GetLabels()[kfplacementv1alpha1.PlacementResourceSnapshotIndexLabelKey] + idx, err := strconv.Atoi(idxStr) + if err != nil { + return nil, fmt.Errorf("snapshot %s has an invalid index label %q: %w", snapshot.GetName(), idxStr, err) + } + if idx > latestIdx { + latest, latestIdx = snapshot, idx + } + } + if latest == nil { + return nil, fmt.Errorf("no primary resource snapshot found for placement policy %s", client.ObjectKeyFromObject(policy)) + } + return latest, nil +} + +// allSynchronizedPlacementPolicyStatus returns the status of a policy that has found all of its desired clusters +// and has the latest resource revision synchronized and available on each of them. +// +// The latest resource revision name and the observed generations are left for the actual to fill in. +func allSynchronizedPlacementPolicyStatus(clusterCount int32) kfplacementv1alpha1.PlacementPolicyStatus { + return kfplacementv1alpha1.PlacementPolicyStatus{ + Conditions: []metav1.Condition{ + { + Type: kfplacementv1alpha1.PlacementPolicyCondTypeResourceCollected, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicyResourceCollectedCondReasonAllResourcesCollected, + }, + { + Type: kfplacementv1alpha1.PlacementPolicyCondTypeScheduled, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicyScheduledCondReasonFoundAllClusters, + }, + { + Type: kfplacementv1alpha1.PlacementPolicyCondTypeSynchronized, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicySynchronizedCondReasonAllClustersSynchronized, + }, + { + Type: kfplacementv1alpha1.PlacementPolicyCondTypeAvailable, + Status: metav1.ConditionTrue, + Reason: kfplacementv1alpha1.PlacementPolicyAvailableCondReasonAllClustersAvailable, + }, + }, + DesiredClusters: ptr.To(clusterCount), + ScheduledClusters: ptr.To(clusterCount), + SynchronizedClusters: ptr.To(clusterCount), + ResourcesAvailableClusters: ptr.To(clusterCount), + ActiveClusterClaims: ptr.To(int32(0)), + } +} + +// ensurePlacementPolicyRemoved deletes a placement policy and waits until it is gone and the app ConfigMap has +// been removed from every member cluster. +func ensurePlacementPolicyRemoved(key types.NamespacedName) { + GinkgoHelper() + + deletePlacementPolicyAndWait(key) + Eventually(appConfigMapRemovedFromClustersActual(allMemberClusters...), workloadEventuallyDuration, eventuallyInterval). + Should(Succeed(), "Failed to remove the placed ConfigMap from the member clusters") +} + +// deletePlacementPolicyAndWait deletes a placement policy, with the given delete options, and waits until it is gone, +// without checking what happens to the resources it placed; specs whose policy orphans its resources use it directly. +func deletePlacementPolicyAndWait(key types.NamespacedName, opts ...client.DeleteOption) { + GinkgoHelper() + + policy := newPlacementPolicyObject(key, kfplacementv1alpha1.PlacementPolicySpec{}) + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, policy, opts...))).To(Succeed(), "Failed to delete placement policy %s", key) + // Policy deletion is finalizer-gated on cluster claim withdrawal, so waiting for the policy to disappear also + // confirms that its claims are gone. + Eventually(func() error { + _, err := retrievePlacementPolicy(key) + if err == nil { + return fmt.Errorf("placement policy %s still exists", key) + } + return client.IgnoreNotFound(err) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove placement policy %s", key) +} + +// stagedUpdateRunBindingManagerName is the controller name that a staged update run records when it holds the +// binding manager role of its placement policy. It is spelled out rather than imported so that a rename shows up +// as a failed spec, the same way it would for a user inspecting the policy status. +const stagedUpdateRunBindingManagerName = "staged-update-run" + +// placementPolicyRolloutEventuallyDuration bounds the waits on rollouts (staged update runs and auto-rollouts). A +// staged update run reconcile can hold its worker for up to the run controller's 30s refresh period, and persists +// the run status only once it returns. +const placementPolicyRolloutEventuallyDuration = 90 * time.Second + +// placementPolicyCondition builds a wanted condition; observed generations and messages are left out, as the +// actuals fill in the former and ignore the latter. +func placementPolicyCondition(condType string, status metav1.ConditionStatus, reason string) metav1.Condition { + return metav1.Condition{Type: condType, Status: status, Reason: reason} +} + +// synchronizedPlacementBindingStatus returns the status of a binding whose resourceCount resources are all +// synchronized and available on its cluster. +func synchronizedPlacementBindingStatus(resourceCount int32) kfplacementv1alpha1.PlacementBindingStatus { + return kfplacementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + placementPolicyCondition(kfplacementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementBindingSynchronizedCondReasonAllResourcesSynchronized), + placementPolicyCondition(kfplacementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionTrue, + kfplacementv1alpha1.PlacementBindingAvailableCondReasonAllResourcesAvailable), + }, + SelectedResources: ptr.To(resourceCount), + SynchronizedResources: ptr.To(resourceCount), + AvailableResources: ptr.To(resourceCount), + } +} + +// notTakenOverPlacementBindingStatus returns the status of a binding whose only resource, a ConfigMap, already +// exists on its cluster unmanaged and has not been taken over, as the default whenAlreadyExists=ReportError asks. +func notTakenOverPlacementBindingStatus(configMapKey types.NamespacedName) kfplacementv1alpha1.PlacementBindingStatus { + return configMapNotAppliedPlacementBindingStatus(configMapKey, "NotTakenOver", nil) +} + +// configMapNotAppliedPlacementBindingStatus returns the status of a binding whose only resource, a ConfigMap, the +// work applier did not apply, for the reason given (the applier's apply result type), with the given diff details. +// +// A ConfigMap carries no generation, so the observed generation of the copied manifest condition is 0. +func configMapNotAppliedPlacementBindingStatus( + configMapKey types.NamespacedName, + reason string, + diffDetails *kfplacementv1alpha1.DiffDetails, +) kfplacementv1alpha1.PlacementBindingStatus { + return kfplacementv1alpha1.PlacementBindingStatus{ + Conditions: []metav1.Condition{ + placementPolicyCondition(kfplacementv1alpha1.PlacementBindingCondTypeSynchronized, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementBindingSynchronizedCondReasonFailedToSynchronizeSomeResources), + placementPolicyCondition(kfplacementv1alpha1.PlacementBindingCondTypeAvailable, metav1.ConditionFalse, + kfplacementv1alpha1.PlacementBindingAvailableCondReasonSomeResourcesUnavailable), + }, + SelectedResources: ptr.To(int32(1)), + SynchronizedResources: ptr.To(int32(0)), + AvailableResources: ptr.To(int32(0)), + FailedResources: []kfplacementv1alpha1.FailedResource{ + { + ObjectRef: kfplacementv1alpha1.ObjectReference{ + Namespace: configMapKey.Namespace, + Name: configMapKey.Name, + APIVersion: "v1", + Kind: "ConfigMap", + }, + Conditions: []metav1.Condition{ + // The manifest condition reason is the work applier's apply result type. + placementPolicyCondition(kfplacementv1alpha1.ManifestCondTypeApplied, metav1.ConditionFalse, reason), + }, + DiffDetails: diffDetails, + }, + }, + } +} + +// updatePlacementPolicy applies the mutation to the placement policy, retrying on conflicts. +func updatePlacementPolicy(key types.NamespacedName, mutate func(policy kfplacementv1alpha1.PlacementPolicyAccessor)) { + GinkgoHelper() + + Eventually(func() error { + policy, err := retrievePlacementPolicy(key) + if err != nil { + return err + } + mutate(policy) + return hubClient.Update(ctx, policy) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update placement policy %s", key) +} + +// updateConfigMapDataOnHub replaces the data of a ConfigMap on the hub cluster, retrying on conflicts. +func updateConfigMapDataOnHub(key types.NamespacedName, data map[string]string) { + GinkgoHelper() + + Eventually(func() error { + cm := &corev1.ConfigMap{} + if err := hubClient.Get(ctx, key, cm); err != nil { + return err + } + cm.Data = maps.Clone(data) + return hubClient.Update(ctx, cm) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update ConfigMap %s on the hub cluster", key) +} + +// updateConfigMapDataOnMemberCluster replaces the data of a ConfigMap directly on a member cluster, retrying on +// conflicts; the rest of the object, KubeFleet's owner references and annotations included, is kept as it is. +func updateConfigMapDataOnMemberCluster(cluster *framework.Cluster, key types.NamespacedName, data map[string]string) { + GinkgoHelper() + + Eventually(func() error { + cm := &corev1.ConfigMap{} + if err := cluster.KubeClient.Get(ctx, key, cm); err != nil { + return err + } + cm.Data = maps.Clone(data) + return cluster.KubeClient.Update(ctx, cm) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update ConfigMap %s on cluster %s", key, cluster.ClusterName) +} + +// createConfigMapOnMemberCluster creates a ConfigMap, and its namespace if needed, directly on a member cluster, so +// that the ConfigMap is not managed by KubeFleet. Creation is retried, as the namespace might still be terminating +// after an earlier spec's cleanup. +func createConfigMapOnMemberCluster(cluster *framework.Cluster, cm *corev1.ConfigMap, opts ...client.CreateOption) { + GinkgoHelper() + + Eventually(func() error { + ns := &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: cm.Namespace}} + if err := client.IgnoreAlreadyExists(cluster.KubeClient.Create(ctx, ns)); err != nil { + return err + } + return cluster.KubeClient.Create(ctx, cm.DeepCopy(), opts...) + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), + "Failed to create ConfigMap %s on cluster %s", client.ObjectKeyFromObject(cm), cluster.ClusterName) +} + +// newStagedUpdateStrategyObject returns a StagedUpdateStrategy or a ClusterStagedUpdateStrategy, depending on the key. +func newStagedUpdateStrategyObject(key types.NamespacedName, spec kfrolloutv1alpha1.StagedUpdateStrategySpec) kfrolloutv1alpha1.StagedUpdateStrategyAccessor { + if key.Namespace == "" { + return &kfrolloutv1alpha1.ClusterStagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{Name: key.Name}, + Spec: spec, + } + } + return &kfrolloutv1alpha1.StagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{Name: key.Name, Namespace: key.Namespace}, + Spec: spec, + } +} + +// newStagedUpdateRunObject returns a StagedUpdateRun or a ClusterStagedUpdateRun, depending on the key. +func newStagedUpdateRunObject(key types.NamespacedName, spec kfrolloutv1alpha1.StagedUpdateRunSpec) kfrolloutv1alpha1.StagedUpdateRunAccessor { + if key.Namespace == "" { + return &kfrolloutv1alpha1.ClusterStagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: key.Name}, + Spec: spec, + } + } + return &kfrolloutv1alpha1.StagedUpdateRun{ + ObjectMeta: metav1.ObjectMeta{Name: key.Name, Namespace: key.Namespace}, + Spec: spec, + } +} + +// retrieveStagedUpdateRun reads a StagedUpdateRun or a ClusterStagedUpdateRun, depending on the key. +func retrieveStagedUpdateRun(key types.NamespacedName) (kfrolloutv1alpha1.StagedUpdateRunAccessor, error) { + run := newStagedUpdateRunObject(key, kfrolloutv1alpha1.StagedUpdateRunSpec{}) + if err := hubClient.Get(ctx, key, run); err != nil { + return nil, err + } + return run, nil +} + +// newApprovalRequestObject returns an ApprovalRequest or a ClusterApprovalRequest, depending on the key. +func newApprovalRequestObject(key types.NamespacedName) kfrolloutv1alpha1.ApprovalRequestAccessor { + if key.Namespace == "" { + return &kfrolloutv1alpha1.ClusterApprovalRequest{ObjectMeta: metav1.ObjectMeta{Name: key.Name}} + } + return &kfrolloutv1alpha1.ApprovalRequest{ObjectMeta: metav1.ObjectMeta{Name: key.Name, Namespace: key.Namespace}} +} + +// retrieveApprovalRequest reads an ApprovalRequest or a ClusterApprovalRequest, depending on the key. +func retrieveApprovalRequest(key types.NamespacedName) (kfrolloutv1alpha1.ApprovalRequestAccessor, error) { + request := newApprovalRequestObject(key) + if err := hubClient.Get(ctx, key, request); err != nil { + return nil, err + } + return request, nil +} + +// approvePlacementPolicyApprovalRequest approves an approval request (or a cluster approval request) the way a user +// does without the kubectl-fleet plugin, which does not support the placement policy APIs: by adding an Approved +// condition to its status. Conflicts with the run controller's own writes are retried. +func approvePlacementPolicyApprovalRequest(key types.NamespacedName) { + GinkgoHelper() + + Eventually(func() error { + request, err := retrieveApprovalRequest(key) + if err != nil { + return err + } + meta.SetStatusCondition(&request.GetStatus().Conditions, metav1.Condition{ + Type: kfrolloutv1alpha1.ApprovalRequestCondTypeApproved, + Status: metav1.ConditionTrue, + Reason: "ApprovedByE2E", + Message: "Approved by the E2E test", + ObservedGeneration: request.GetGeneration(), + }) + return hubClient.Status().Update(ctx, request) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to approve approval request %s", key) +} + +// stagedUpdateRunObjectReference returns the reference that a staged update run (or a cluster staged update run) +// records on its placement policy when it holds the binding manager role. +func stagedUpdateRunObjectReference(key types.NamespacedName) kfplacementv1alpha1.ObjectReference { + kind := kfrolloutv1alpha1.StagedUpdateRunKind + if key.Namespace == "" { + kind = kfrolloutv1alpha1.ClusterStagedUpdateRunKind + } + return kfplacementv1alpha1.ObjectReference{ + APIGroup: kfrolloutv1alpha1.GroupVersion.Group, + APIVersion: kfrolloutv1alpha1.GroupVersion.Version, + Kind: kind, + Namespace: key.Namespace, + Name: key.Name, + } +} + +// ensureStagedUpdateRunRemoved deletes a staged update run (or a cluster staged update run) and waits until it is +// gone; the run controller releases its finalizer once it has given up the binding manager role. +func ensureStagedUpdateRunRemoved(key types.NamespacedName) { + GinkgoHelper() + + run := newStagedUpdateRunObject(key, kfrolloutv1alpha1.StagedUpdateRunSpec{}) + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, run))).To(Succeed(), "Failed to delete staged update run %s", key) + Eventually(func() error { + _, err := retrieveStagedUpdateRun(key) + if err == nil { + return fmt.Errorf("staged update run %s still exists", key) + } + return client.IgnoreNotFound(err) + }, workloadEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to remove staged update run %s", key) +} + +// ensureStagedUpdateStrategyRemoved deletes a staged update strategy (or a cluster staged update strategy). +func ensureStagedUpdateStrategyRemoved(key types.NamespacedName) { + GinkgoHelper() + + strategy := newStagedUpdateStrategyObject(key, kfrolloutv1alpha1.StagedUpdateStrategySpec{}) + Expect(client.IgnoreNotFound(hubClient.Delete(ctx, strategy))).To(Succeed(), "Failed to delete staged update strategy %s", key) +} diff --git a/test/e2e/setup.sh b/test/e2e/setup.sh index ad353b7a3..b54a7d1ce 100755 --- a/test/e2e/setup.sh +++ b/test/e2e/setup.sh @@ -160,6 +160,8 @@ helm install hub-agent ../../charts/hub-agent/ \ --set MaxConcurrentClusterPlacement=200 \ --set resourceSnapshotCreationMinimumInterval=$RESOURCE_SNAPSHOT_CREATION_MINIMUM_INTERVAL \ --set resourceChangesCollectionDuration=$RESOURCE_CHANGES_COLLECTION_DURATION \ + --set enablePlacementPolicyAPIs=true \ + --set enableAnnotationBasedPlacement=true \ --wait \ --timeout=2m @@ -234,6 +236,7 @@ do --set logVerbosity=5 \ --set namespace=fleet-system \ --set priorityQueue.enabled=true \ + --set enablePlacementPolicyAPIs=true \ --set workApplierRequeueRateLimiterMaxSlowBackoffDelaySeconds=5 \ --set workApplierRequeueRateLimiterMaxFastBackoffDelaySeconds=5 \ --set propertyProvider=$PROPERTY_PROVIDER \ @@ -260,12 +263,20 @@ do --set logVerbosity=5 \ --set namespace=fleet-system \ --set priorityQueue.enabled=true \ + --set enablePlacementPolicyAPIs=true \ --set workApplierRequeueRateLimiterMaxSlowBackoffDelaySeconds=5 \ --set workApplierRequeueRateLimiterMaxFastBackoffDelaySeconds=5 \ --set propertyProvider=$PROPERTY_PROVIDER \ --set enableNamespaceCollectionInPropertyProvider=true \ $( [ "$PROPERTY_PROVIDER" = "azure" ] && echo "-f azure_valid_config.yaml" ) fi + + # The placement policy work applier does not read the v1beta1 work applier flags above; cap its requeue delays + # through a test-only environment variable instead, for the same reason. The agent has not joined the fleet yet, + # so the restart this triggers is harmless. + kubectl set env deployment/member-agent -n fleet-system -c member-agent \ + TEST_ONLY_PLACEMENT_POLICY_WORK_APPLIER_MAX_REQUEUE_DELAY_SECONDS=5 + kubectl rollout status deployment/member-agent -n fleet-system --timeout=180s done # Create tools directory if it doesn't exist diff --git a/test/e2e/setup_test.go b/test/e2e/setup_test.go index 986819d9b..fdf862f78 100644 --- a/test/e2e/setup_test.go +++ b/test/e2e/setup_test.go @@ -47,6 +47,8 @@ import ( fleetnetworkingv1alpha1 "go.goms.io/fleet-networking/api/v1alpha1" clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" + kfplacementv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/placement/v1alpha1" + kfrolloutv1alpha1 "go.goms.io/fleet/apis/kubefleet.dev/rollout/v1alpha1" placementv1 "go.goms.io/fleet/apis/placement/v1" placementv1alpha1 "go.goms.io/fleet/apis/placement/v1alpha1" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" @@ -297,6 +299,12 @@ func TestMain(m *testing.M) { if err := placementv1beta1.AddToScheme(scheme); err != nil { log.Fatalf("failed to add custom APIs (placement) to the runtime scheme: %v", err) } + if err := kfplacementv1alpha1.AddToScheme(scheme); err != nil { + log.Fatalf("failed to add custom APIs (kubefleet.dev placement v1alpha1) to the runtime scheme: %v", err) + } + if err := kfrolloutv1alpha1.AddToScheme(scheme); err != nil { + log.Fatalf("failed to add custom APIs (kubefleet.dev rollout v1alpha1) to the runtime scheme: %v", err) + } if err := placementv1.AddToScheme(scheme); err != nil { log.Fatalf("failed to add custom APIs (placement v1) to the runtime scheme: %v", err) } diff --git a/test/e2e/staged_updaterun_test.go b/test/e2e/staged_updaterun_test.go index 58028c421..a6724483d 100644 --- a/test/e2e/staged_updaterun_test.go +++ b/test/e2e/staged_updaterun_test.go @@ -687,6 +687,126 @@ var _ = Describe("test RP rollout with staged update run", Label("resourceplacem }) }) + Context("Test cluster shrink with before-stage tasks for the delete stage", Ordered, func() { + var strategy *placementv1beta1.StagedUpdateStrategy + updateRunNames := []string{} + + It("should wait for namespace collection to sync on all member clusters", func() { + waitForNamespaceCollectionOnClusters(appNamespace().Name, allMemberClusterNames) + }) + + BeforeAll(func() { + // Create the RP with external rollout strategy and pick fixed policy. + rp := &placementv1beta1.ResourcePlacement{ + ObjectMeta: metav1.ObjectMeta{ + Name: rpName, + Namespace: testNamespace, + // Add a custom finalizer; this would allow us to better observe + // the behavior of the controllers. + Finalizers: []string{customDeletionBlockerFinalizer}, + }, + Spec: placementv1beta1.PlacementSpec{ + ResourceSelectors: configMapSelector(), + Policy: &placementv1beta1.PlacementPolicy{ + PlacementType: placementv1beta1.PickFixedPlacementType, + ClusterNames: []string{allMemberClusterNames[0], allMemberClusterNames[1]}, // member-cluster-1 and member-cluster-2 + }, + Strategy: placementv1beta1.RolloutStrategy{ + Type: placementv1beta1.ExternalRolloutStrategyType, + }, + }, + } + Expect(hubClient.Create(ctx, rp)).To(Succeed(), "Failed to create RP") + + // Create the stagedUpdateStrategy with a single stage and a gated delete stage. + strategy = &placementv1beta1.StagedUpdateStrategy{ + ObjectMeta: metav1.ObjectMeta{ + Name: strategyName, + Namespace: testNamespace, + }, + Spec: updateStrategySpecWithDeleteStageTasks(), + } + Expect(hubClient.Create(ctx, strategy)).To(Succeed(), "Failed to create StagedUpdateStrategy") + + for i := 0; i < 2; i++ { + updateRunNames = append(updateRunNames, fmt.Sprintf(stagedUpdateRunNameWithSubIndexTemplate, GinkgoParallelProcess(), i)) + } + }) + + AfterAll(func() { + // Remove the custom deletion blocker finalizer from the RP. + ensureRPAndRelatedResourcesDeleted(types.NamespacedName{Name: rpName, Namespace: testNamespace}, allMemberClusters) + + // Remove all the stagedUpdateRuns. + for _, name := range updateRunNames { + ensureStagedUpdateRunDeletion(name, testNamespace) + } + + // Delete the stagedUpdateStrategy. + ensureStagedUpdateRunStrategyDeletion(strategyName, testNamespace) + }) + + It("Should successfully schedule the rp", func() { + validateLatestSchedulingPolicySnapshot(rpName, testNamespace, policySnapshotIndex1st, 2) + }) + + It("Should create a staged update run successfully", func() { + createStagedUpdateRunWithAutoCreatedSnapshot(updateRunNames[0], testNamespace, rpName, strategyName, placementv1beta1.StateRun) + }) + + It("Should rollout resources to member-cluster-1 and member-cluster-2 and complete the staged update run without the delete stage tasks as there is no cluster to delete", func() { + surSucceededActual := testutilsupdaterun.StagedUpdateRunStatusSucceededActual(ctx, hubClient, updateRunNames[0], testNamespace, resourceSnapshotIndex1st, policySnapshotIndex1st, 2, defaultApplyStrategy, &strategy.Spec, [][]string{{allMemberClusterNames[0], allMemberClusterNames[1]}}, nil, nil, nil, true) + Eventually(surSucceededActual, updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to validate updateRun %s/%s succeeded", testNamespace, updateRunNames[0]) + checkIfPlacedWorkResourcesOnMemberClustersInUpdateRun([]*framework.Cluster{allMemberClusters[0], allMemberClusters[1]}) + }) + + It("Update the rp to only keep member-cluster-3", func() { + Eventually(func() error { + rp := &placementv1beta1.ResourcePlacement{} + if err := hubClient.Get(ctx, client.ObjectKey{Name: rpName, Namespace: testNamespace}, rp); err != nil { + return fmt.Errorf("failed to get the rp: %w", err) + } + rp.Spec.Policy.ClusterNames = []string{allMemberClusterNames[2]} + return hubClient.Update(ctx, rp) + }, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to update the rp to only keep member-cluster-3") + }) + + It("Should successfully schedule the rp", func() { + validateLatestSchedulingPolicySnapshot(rpName, testNamespace, policySnapshotIndex2nd, 1) + }) + + It("Should create a staged update run successfully", func() { + createStagedUpdateRunSucceed(updateRunNames[1], testNamespace, rpName, resourceSnapshotIndex1st, strategyName, placementv1beta1.StateRun) + }) + + It("Should rollout resources to member-cluster-3 and keep them on member-cluster-1 and member-cluster-2 until the delete stage tasks complete", func() { + checkIfPlacedWorkResourcesOnMemberClustersInUpdateRun([]*framework.Cluster{allMemberClusters[2]}) + + By("Validating the approval request of the delete stage is created") + Eventually(func() error { + appReq := &placementv1beta1.ApprovalRequest{} + return hubClient.Get(ctx, client.ObjectKey{Name: fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRunNames[1], placementv1beta1.UpdateRunDeleteStageTaskName), Namespace: testNamespace}, appReq) + }, updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to get the approval request of the delete stage") + + By("Validating the resources are kept while the delete stage waits for the approval") + checkIfPlacedWorkResourcesOnMemberClustersConsistently(allMemberClusters) + }) + + It("Should remove resources on member-cluster-1 and member-cluster-2 after approval and complete the staged update run successfully", func() { + validateAndApproveNamespacedApprovalRequests(updateRunNames[1], testNamespace, placementv1beta1.UpdateRunDeleteStageTaskName, placementv1beta1.BeforeStageApprovalTaskNameFmt, placementv1beta1.BeforeStageTaskLabelValue) + + surSucceededActual := testutilsupdaterun.StagedUpdateRunStatusSucceededActual(ctx, hubClient, updateRunNames[1], testNamespace, resourceSnapshotIndex1st, policySnapshotIndex2nd, 1, defaultApplyStrategy, &strategy.Spec, [][]string{{allMemberClusterNames[2]}}, []string{allMemberClusterNames[0], allMemberClusterNames[1]}, nil, nil, true) + Eventually(surSucceededActual, 2*updateRunEventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to validate updateRun %s/%s succeeded", testNamespace, updateRunNames[1]) + checkIfRemovedConfigMapFromMemberClusters([]*framework.Cluster{allMemberClusters[0], allMemberClusters[1]}) + checkIfPlacedWorkResourcesOnMemberClustersConsistently([]*framework.Cluster{allMemberClusters[2]}) + }) + + It("Should update rp status as completed with member-cluster-3 only", func() { + rpStatusUpdatedActual := rpStatusWithExternalStrategyActual(appConfigMapIdentifiers(), resourceSnapshotIndex1st, true, []string{allMemberClusterNames[2]}, []string{resourceSnapshotIndex1st}, []bool{true}, nil, nil) + Eventually(rpStatusUpdatedActual, eventuallyDuration, eventuallyInterval).Should(Succeed(), "Failed to keep RP %s/%s status as expected", testNamespace, rpName) + }) + }) + Context("Test cluster scale out and shrink using pickN policy with namespaced staged update run", Ordered, func() { var strategy *placementv1beta1.StagedUpdateStrategy updateRunNames := []string{} diff --git a/test/utils/informer/manager.go b/test/utils/informer/manager.go index 91eb17513..527ad169a 100644 --- a/test/utils/informer/manager.go +++ b/test/utils/informer/manager.go @@ -128,6 +128,8 @@ type FakeManager struct { // InformerSynced controls whether IsInformerSynced returns true or false. // If nil, defaults to true. If set, returns the value for all resources. InformerSynced *bool + // CreatedInformers records, in order, the resources passed to CreateInformerForResource. + CreatedInformers []informer.APIResourceMeta } func (m *FakeManager) AddDynamicResources(_ []informer.APIResourceMeta, _ cache.ResourceEventHandler, _ bool) { @@ -189,8 +191,8 @@ func (m *FakeManager) AddEventHandlerToInformer(_ schema.GroupVersionResource, _ // No-op for testing } -func (m *FakeManager) CreateInformerForResource(_ informer.APIResourceMeta) { - // No-op for testing +func (m *FakeManager) CreateInformerForResource(res informer.APIResourceMeta) { + m.CreatedInformers = append(m.CreatedInformers, res) } func (m *FakeManager) IsInformerSet(_ schema.GroupVersionKind) bool { diff --git a/test/utils/updaterun/updaterun_status.go b/test/utils/updaterun/updaterun_status.go index d3c01a0e5..4cfa3b6f7 100644 --- a/test/utils/updaterun/updaterun_status.go +++ b/test/utils/updaterun/updaterun_status.go @@ -75,11 +75,11 @@ func ClusterStagedUpdateRunStatusSucceededActual( if execute { wantStatus.StagesStatus = buildStageUpdatingStatuses(wantStrategySpec, wantSelectedClusters, wantCROs, wantROs, updateRun) - wantStatus.DeletionStageStatus = buildDeletionStageStatus(wantUnscheduledClusters, updateRun) + wantStatus.DeletionStageStatus = buildDeletionStageStatus(wantStrategySpec, wantUnscheduledClusters, updateRun) wantStatus.Conditions = updateRunSucceedConditions(updateRun.Generation) } else { wantStatus.StagesStatus = buildStageUpdatingStatusesForInitialized(wantStrategySpec, wantSelectedClusters, wantCROs, wantROs, updateRun) - wantStatus.DeletionStageStatus = buildDeletionStatusWithoutConditions(wantUnscheduledClusters, updateRun) + wantStatus.DeletionStageStatus = buildDeletionStatusWithoutConditions(wantStrategySpec, wantUnscheduledClusters, updateRun) wantStatus.Conditions = updateRunInitializedConditions(updateRun.Generation) } if diff := cmp.Diff(updateRun.Status, wantStatus, updateRunStatusCmpOption...); diff != "" { @@ -120,11 +120,11 @@ func StagedUpdateRunStatusSucceededActual( if execute { wantStatus.StagesStatus = buildStageUpdatingStatuses(wantStrategySpec, wantSelectedClusters, wantCROs, wantROs, updateRun) - wantStatus.DeletionStageStatus = buildDeletionStageStatus(wantUnscheduledClusters, updateRun) + wantStatus.DeletionStageStatus = buildDeletionStageStatus(wantStrategySpec, wantUnscheduledClusters, updateRun) wantStatus.Conditions = updateRunSucceedConditions(updateRun.Generation) } else { wantStatus.StagesStatus = buildStageUpdatingStatusesForInitialized(wantStrategySpec, wantSelectedClusters, wantCROs, wantROs, updateRun) - wantStatus.DeletionStageStatus = buildDeletionStatusWithoutConditions(wantUnscheduledClusters, updateRun) + wantStatus.DeletionStageStatus = buildDeletionStatusWithoutConditions(wantStrategySpec, wantUnscheduledClusters, updateRun) wantStatus.Conditions = updateRunInitializedConditions(updateRun.Generation) } if diff := cmp.Diff(updateRun.Status, wantStatus, updateRunStatusCmpOption...); diff != "" { @@ -150,20 +150,8 @@ func buildStageUpdatingStatusesForInitialized( stagesStatus[i].Clusters[j].ClusterResourceOverrideSnapshots = wantCROs[wantSelectedClusters[i][j]] stagesStatus[i].Clusters[j].ResourceOverrideSnapshots = wantROs[wantSelectedClusters[i][j]] } - stagesStatus[i].BeforeStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.BeforeStageTasks)) - for j, task := range stage.BeforeStageTasks { - stagesStatus[i].BeforeStageTaskStatus[j].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - stagesStatus[i].BeforeStageTaskStatus[j].ApprovalRequestName = fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - } - stagesStatus[i].AfterStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.AfterStageTasks)) - for j, task := range stage.AfterStageTasks { - stagesStatus[i].AfterStageTaskStatus[j].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - stagesStatus[i].AfterStageTaskStatus[j].ApprovalRequestName = fmt.Sprintf(placementv1beta1.AfterStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - } + stagesStatus[i].BeforeStageTaskStatus = buildStageTaskStatuses(stage.BeforeStageTasks, placementv1beta1.BeforeStageApprovalTaskNameFmt, stage.Name, updateRun, false) + stagesStatus[i].AfterStageTaskStatus = buildStageTaskStatuses(stage.AfterStageTasks, placementv1beta1.AfterStageApprovalTaskNameFmt, stage.Name, updateRun, false) } return stagesStatus } @@ -185,28 +173,10 @@ func buildStageUpdatingStatuses( stagesStatus[i].Clusters[j].ResourceOverrideSnapshots = wantROs[wantSelectedClusters[i][j]] stagesStatus[i].Clusters[j].Conditions = updateRunClusterRolloutSucceedConditions(updateRun.GetGeneration()) } - stagesStatus[i].BeforeStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.BeforeStageTasks)) - for j, task := range stage.BeforeStageTasks { - stagesStatus[i].BeforeStageTaskStatus[j].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - stagesStatus[i].BeforeStageTaskStatus[j].ApprovalRequestName = fmt.Sprintf(placementv1beta1.BeforeStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - // Skip populating task conditions if the stage has 0 clusters (stage is skipped). - if len(wantSelectedClusters[i]) > 0 { - stagesStatus[i].BeforeStageTaskStatus[j].Conditions = updateRunStageTaskSucceedConditions(updateRun.GetGeneration(), task.Type) - } - } - stagesStatus[i].AfterStageTaskStatus = make([]placementv1beta1.StageTaskStatus, len(stage.AfterStageTasks)) - for j, task := range stage.AfterStageTasks { - stagesStatus[i].AfterStageTaskStatus[j].Type = task.Type - if task.Type == placementv1beta1.StageTaskTypeApproval { - stagesStatus[i].AfterStageTaskStatus[j].ApprovalRequestName = fmt.Sprintf(placementv1beta1.AfterStageApprovalTaskNameFmt, updateRun.GetName(), stage.Name) - } - // Skip populating task conditions if the stage has 0 clusters (stage is skipped). - if len(wantSelectedClusters[i]) > 0 { - stagesStatus[i].AfterStageTaskStatus[j].Conditions = updateRunStageTaskSucceedConditions(updateRun.GetGeneration(), task.Type) - } - } + // Skip populating task conditions if the stage has 0 clusters (stage is skipped). + stageExecuted := len(wantSelectedClusters[i]) > 0 + stagesStatus[i].BeforeStageTaskStatus = buildStageTaskStatuses(stage.BeforeStageTasks, placementv1beta1.BeforeStageApprovalTaskNameFmt, stage.Name, updateRun, stageExecuted) + stagesStatus[i].AfterStageTaskStatus = buildStageTaskStatuses(stage.AfterStageTasks, placementv1beta1.AfterStageApprovalTaskNameFmt, stage.Name, updateRun, stageExecuted) // Use skipped conditions if the stage has 0 clusters. if len(wantSelectedClusters[i]) == 0 { stagesStatus[i].Conditions = updateRunStageSkippedNoClustersConditions(updateRun.GetGeneration()) @@ -217,21 +187,53 @@ func buildStageUpdatingStatuses( return stagesStatus } +// buildStageTaskStatuses builds the statuses of the before or after stage tasks of a stage, which have their +// conditions set if the tasks are completed. +func buildStageTaskStatuses( + tasks []placementv1beta1.StageTask, + approvalRequestNameFmt, stageName string, + updateRun placementv1beta1.UpdateRunObj, + completed bool, +) []placementv1beta1.StageTaskStatus { + taskStatuses := make([]placementv1beta1.StageTaskStatus, len(tasks)) + for i, task := range tasks { + taskStatuses[i].Type = task.Type + if task.Type == placementv1beta1.StageTaskTypeApproval { + taskStatuses[i].ApprovalRequestName = fmt.Sprintf(approvalRequestNameFmt, updateRun.GetName(), stageName) + } + if completed { + taskStatuses[i].Conditions = updateRunStageTaskSucceedConditions(updateRun.GetGeneration(), task.Type) + } + } + return taskStatuses +} + func buildDeletionStageStatus( + wantStrategySpec *placementv1beta1.UpdateStrategySpec, wantUnscheduledClusters []string, updateRun placementv1beta1.UpdateRunObj, ) *placementv1beta1.StageUpdatingStatus { - deleteStageStatus := buildDeletionStatusWithoutConditions(wantUnscheduledClusters, updateRun) + deleteStageStatus := buildDeletionStatusWithoutConditions(wantStrategySpec, wantUnscheduledClusters, updateRun) deleteStageStatus.Conditions = updateRunStageRolloutSucceedConditions(updateRun.GetGeneration()) + // The tasks are skipped if the delete stage has 0 clusters. + if len(wantUnscheduledClusters) > 0 { + for i := range deleteStageStatus.BeforeStageTaskStatus { + deleteStageStatus.BeforeStageTaskStatus[i].Conditions = updateRunStageTaskSucceedConditions(updateRun.GetGeneration(), deleteStageStatus.BeforeStageTaskStatus[i].Type) + } + } return deleteStageStatus } func buildDeletionStatusWithoutConditions( + wantStrategySpec *placementv1beta1.UpdateStrategySpec, wantUnscheduledClusters []string, updateRun placementv1beta1.UpdateRunObj, ) *placementv1beta1.StageUpdatingStatus { deleteStageStatus := &placementv1beta1.StageUpdatingStatus{ - StageName: "kubernetes-fleet.io/deleteStage", + StageName: placementv1beta1.UpdateRunDeleteStageName, + } + if wantStrategySpec.DeleteStage != nil { + deleteStageStatus.BeforeStageTaskStatus = buildStageTaskStatuses(wantStrategySpec.DeleteStage.BeforeStageTasks, placementv1beta1.BeforeStageApprovalTaskNameFmt, placementv1beta1.UpdateRunDeleteStageTaskName, updateRun, false) } deleteStageStatus.Clusters = make([]placementv1beta1.ClusterUpdatingStatus, len(wantUnscheduledClusters)) for i := range deleteStageStatus.Clusters { diff --git a/tools/fleet/cmd/draincluster/draincluster.go b/tools/fleet/cmd/draincluster/draincluster.go index 31b472165..f249531a7 100644 --- a/tools/fleet/cmd/draincluster/draincluster.go +++ b/tools/fleet/cmd/draincluster/draincluster.go @@ -186,38 +186,101 @@ func (o *drainOptions) drain(ctx context.Context) (bool, error) { return false, fmt.Errorf("failed to wait for eviction %s for CRP %s targeting member cluster %s to reach terminal state: %w", evictionName, crpName, o.clusterName, err) } - // TODO: add safeguards to check if eviction conditions are set to unknown. - validCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeValid)) - if validCondition != nil && validCondition.Status == metav1.ConditionFalse { - // check to see if CRP is missing or CRP is being deleted or CRB is missing. - if validCondition.Reason == condition.EvictionInvalidMissingCRPMessage || - validCondition.Reason == condition.EvictionInvalidDeletingCRPMessage || - validCondition.Reason == condition.EvictionInvalidMissingCRBMessage { - log.Printf("eviction %s is invalid with reason %s for CRP %s targeting member cluster %s, but drain will succeed", evictionName, validCondition.Reason, crpName, o.clusterName) - continue - } - } - executedCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeExecuted)) - if executedCondition == nil || executedCondition.Status == metav1.ConditionFalse { + // Classify the terminal eviction. A missing or Unknown Valid/Executed + // condition is never treated as success, so an uncertain eviction is not + // reported as a completed drain (see evaluateEviction). + switch evaluateEviction(&eviction) { + case evictionResultInvalidButDrained: + validCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeValid)) + log.Printf("eviction %s is invalid: %s for CRP %s targeting member cluster %s, but drain will succeed", evictionName, validCondition.Message, crpName, o.clusterName) + continue + case evictionResultIndeterminate: + isDrainSuccessful = false + log.Printf("eviction %s has a missing or unknown Valid or Executed condition for CRP %s targeting member cluster %s; cannot confirm drain", evictionName, crpName, o.clusterName) + continue + case evictionResultNotExecuted: isDrainSuccessful = false log.Printf("eviction %s was not executed successfully for CRP %s targeting member cluster %s", evictionName, crpName, o.clusterName) continue - } - log.Printf("eviction %s was executed successfully for CRP %s targeting member cluster %s", evictionName, crpName, o.clusterName) - // log each cluster scoped resource evicted for CRP. - clusterScopedResourceIdentifiers, err := o.collectClusterScopedResourcesSelectedByCRP(ctx, crpName) - if err != nil { - log.Printf("failed to collect cluster scoped resources selected by CRP %s: %v", crpName, err) + case evictionResultExecuted: + log.Printf("eviction %s was executed successfully for CRP %s targeting member cluster %s", evictionName, crpName, o.clusterName) + // log each cluster scoped resource evicted for CRP. + clusterScopedResourceIdentifiers, err := o.collectClusterScopedResourcesSelectedByCRP(ctx, crpName) + if err != nil { + log.Printf("failed to collect cluster scoped resources selected by CRP %s: %v", crpName, err) + continue + } + for _, resourceIdentifier := range clusterScopedResourceIdentifiers { + log.Printf("evicted resource %s propagated by CRP %s targeting member cluster %s", generateResourceIdentifierKey(resourceIdentifier), crpName, o.clusterName) + } + default: + // Any unexpected classification is treated as unconfirmed rather than + // success, so a future evictionResult value can never silently pass a drain. + isDrainSuccessful = false + log.Printf("eviction %s returned an unexpected evaluation result for CRP %s targeting member cluster %s; cannot confirm drain", evictionName, crpName, o.clusterName) continue } - for _, resourceIdentifier := range clusterScopedResourceIdentifiers { - log.Printf("evicted resource %s propagated by CRP %s targeting member cluster %s", generateResourceIdentifierKey(resourceIdentifier), crpName, o.clusterName) - } } return isDrainSuccessful, nil } +// evictionResult classifies the terminal state of a ClusterResourcePlacementEviction +// based on its Valid and Executed conditions. +type evictionResult int + +const ( + // evictionResultIndeterminate means the eviction's Valid or Executed condition + // is missing or Unknown, so the drain cannot be confirmed. + evictionResultIndeterminate evictionResult = iota + // evictionResultInvalidButDrained means the eviction is invalid for a reason + // that still leaves the cluster drained (missing/deleting CRP or missing CRB). + evictionResultInvalidButDrained + // evictionResultNotExecuted means the eviction did not execute. + evictionResultNotExecuted + // evictionResultExecuted means the eviction executed successfully. + evictionResultExecuted +) + +// evaluateEviction inspects a terminal eviction's Valid and Executed conditions and +// classifies the outcome. Only an explicit "True"/"False" status is treated as +// definitive; a missing, Unknown, or otherwise unexpected status is never treated as +// a successful drain. +func evaluateEviction(eviction *placementv1beta1.ClusterResourcePlacementEviction) evictionResult { + validCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeValid)) + if validCondition == nil { + return evictionResultIndeterminate + } + if validCondition.Status == metav1.ConditionFalse { + // An invalid eviction whose detail indicates a missing/deleting CRP or a + // missing CRB still means the cluster is drained. markEvictionInvalid stores + // that detail in the condition Message (Reason is a generic invalid reason). + if validCondition.Message == condition.EvictionInvalidMissingCRPMessage || + validCondition.Message == condition.EvictionInvalidDeletingCRPMessage || + validCondition.Message == condition.EvictionInvalidMissingCRBMessage { + return evictionResultInvalidButDrained + } + // Any other invalid eviction falls through to the Executed check. + } else if validCondition.Status != metav1.ConditionTrue { + // Unknown, empty, or any unexpected status: the drain cannot be confirmed. + return evictionResultIndeterminate + } + + executedCondition := eviction.GetCondition(string(placementv1beta1.PlacementEvictionConditionTypeExecuted)) + if executedCondition == nil { + return evictionResultIndeterminate + } + switch executedCondition.Status { + case metav1.ConditionTrue: + return evictionResultExecuted + case metav1.ConditionFalse: + return evictionResultNotExecuted + default: + // Unknown, empty, or any unexpected status: the drain cannot be confirmed. + return evictionResultIndeterminate + } +} + func (o *drainOptions) cordon(ctx context.Context) error { // add taint to member cluster to ensure resources aren't scheduled on it. return retry.RetryOnConflict(retry.DefaultRetry, func() error { diff --git a/tools/fleet/cmd/draincluster/draincluster_test.go b/tools/fleet/cmd/draincluster/draincluster_test.go index 4b8a017fb..c4461aa22 100644 --- a/tools/fleet/cmd/draincluster/draincluster_test.go +++ b/tools/fleet/cmd/draincluster/draincluster_test.go @@ -32,6 +32,7 @@ import ( clusterv1beta1 "go.goms.io/fleet/apis/cluster/v1beta1" placementv1beta1 "go.goms.io/fleet/apis/placement/v1beta1" + "go.goms.io/fleet/pkg/utils/condition" toolsutils "go.goms.io/fleet/tools/utils" ) @@ -510,3 +511,113 @@ func serviceScheme(t *testing.T) *runtime.Scheme { } return scheme } + +func TestEvaluateEviction(t *testing.T) { + validType := string(placementv1beta1.PlacementEvictionConditionTypeValid) + executedType := string(placementv1beta1.PlacementEvictionConditionTypeExecuted) + + tests := []struct { + name string + conditions []metav1.Condition + want evictionResult + }{ + { + name: "valid true, executed true", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + {Type: executedType, Status: metav1.ConditionTrue, Reason: "Executed"}, + }, + want: evictionResultExecuted, + }, + { + name: "valid true, executed false", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + {Type: executedType, Status: metav1.ConditionFalse, Reason: "NotExecuted"}, + }, + want: evictionResultNotExecuted, + }, + { + name: "valid true, executed unknown", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + {Type: executedType, Status: metav1.ConditionUnknown, Reason: "Unknown"}, + }, + want: evictionResultIndeterminate, + }, + { + name: "valid true, executed empty/invalid status", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + {Type: executedType, Status: "", Reason: "EmptyStatus"}, + }, + want: evictionResultIndeterminate, + }, + { + name: "valid true, executed missing", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionTrue, Reason: "Valid"}, + }, + want: evictionResultIndeterminate, + }, + { + name: "valid unknown", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionUnknown, Reason: "Unknown"}, + }, + want: evictionResultIndeterminate, + }, + { + name: "valid missing", + conditions: nil, + want: evictionResultIndeterminate, + }, + { + name: "valid false, message missing CRP", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: condition.EvictionInvalidMissingCRPMessage}, + }, + want: evictionResultInvalidButDrained, + }, + { + name: "valid false, message deleting CRP", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: condition.EvictionInvalidDeletingCRPMessage}, + }, + want: evictionResultInvalidButDrained, + }, + { + name: "valid false, message missing CRB", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: condition.EvictionInvalidMissingCRBMessage}, + }, + want: evictionResultInvalidButDrained, + }, + { + name: "valid false with unrecognized message, executed true", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: "some other invalid detail"}, + {Type: executedType, Status: metav1.ConditionTrue, Reason: "Executed"}, + }, + want: evictionResultExecuted, + }, + { + name: "valid false with unrecognized message, executed unknown", + conditions: []metav1.Condition{ + {Type: validType, Status: metav1.ConditionFalse, Reason: condition.ClusterResourcePlacementEvictionInvalidReason, Message: "some other invalid detail"}, + {Type: executedType, Status: metav1.ConditionUnknown, Reason: "Unknown"}, + }, + want: evictionResultIndeterminate, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + eviction := &placementv1beta1.ClusterResourcePlacementEviction{} + eviction.SetConditions(tc.conditions...) + if got := evaluateEviction(eviction); got != tc.want { + t.Errorf("evaluateEviction() = %v, want %v", got, tc.want) + } + }) + } +}