From 6af1214b3aa2fb15e7620d601da5d44888865486 Mon Sep 17 00:00:00 2001 From: yzc <127376228+yuzichen12123@users.noreply.github.com> Date: Tue, 20 Oct 2026 12:55:54 +0800 Subject: [PATCH 1/5] docs: add DevOps platform and delivery best-practice series (#857) --- .../Alauda_DevOps_Best_Practices_Overview.md | 124 +++++++ .../From_Commit_to_Production_on_ACP.md | 236 +++++++++++++ ...Projects_with_Argo_CD_and_Tekton_on_ACP.md | 325 ++++++++++++++++++ 3 files changed, 685 insertions(+) create mode 100644 docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md create mode 100644 docs/en/solutions/From_Commit_to_Production_on_ACP.md create mode 100644 docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md diff --git a/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md b/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md new file mode 100644 index 000000000..0100339da --- /dev/null +++ b/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md @@ -0,0 +1,124 @@ +--- +products: + - Alauda Container Platform + - Alauda DevOps +kind: + - Article +ProductsVersion: + - 4.x +--- + +# Alauda DevOps Best Practices: Overview and Scenario Guide + +This series helps application and platform teams combine Alauda DevOps capabilities into a delivery workflow. Start with a working pipeline, connect build outputs to controlled releases, and then standardize the platform for multiple isolated projects. + +The configuration baseline is Alauda DevOps Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and Alauda Container Platform GitOps 4.4 on a compatible ACP installation. Use the compatibility requirements of each component when selecting packages. Examples use GitLab, Harbor, Nexus, and SonarQube. They describe a reference implementation, with environment preparation and acceptance checks required before production use. + +The series has three entry points: + +| Your starting point | Read next | +| --- | --- | +| You need a build or test pipeline for one repository. | [Start a pipeline](#start-a-pipeline) below. | +| A pipeline already runs; you need a traceable path from a commit to production. | [From Commit to Production on ACP](./From_Commit_to_Production_on_ACP.md). | +| You operate a shared platform and need repeatable configuration and project onboarding. | [Managing Platform Configuration and Onboarding Isolated Projects](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md). | + +The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) contain the platform repository, project declaration, onboarding pipelines, GitOps repository template, application delivery examples, and acceptance scripts. Replace example addresses, image references, identities, and encrypted credential placeholders before using them. + +## The overall design + +The delivery layer moves one change from source to an immutable image and then to an environment. The platform layer supplies reusable definitions, tool access, namespaces, and operating settings. The control layer decides what may run, who approves a release, and what evidence must be retained. + +```mermaid +flowchart TB + subgraph Controls[Control layer] + Policy[Pipeline and image admission] + Approval[Release approval] + Audit[Run and log retention] + end + subgraph Delivery[Delivery layer] + Source[Source repository] --> Trigger[PaC or Triggers] + Trigger --> Build[Build and scan] + Build --> Image[Image identified by digest] + Image --> Test[Test deployment] + Test --> Promote[Promote the selected digest] + Promote --> Production[Production GitOps change] + end + subgraph Platform[Platform layer] + Catalog[Versioned catalog pipelines and tasks] + Connectors[Project and administrator Connectors] + Git[Platform configuration and project declarations] + end + Policy -.-> Trigger + Policy -.-> Production + Approval -.-> Promote + Audit -.-> Build + Catalog -.-> Build + Connectors -.-> Build + Git -.-> Connectors +``` + +Application developers own source and application delivery configuration. Platform administrators own shared components, admission rules, administrator credentials, and onboarding. Release approvers authorize a selected digest after examining its build and test evidence. Separate the release identity from the build identity. + +## Start a pipeline + +Choose the simplest definition model that meets the team's needs. The models can coexist. + +| Model | Definition and trigger | Use it when | +| --- | --- | --- | +| Pipeline in the cluster | Create a Pipeline in a project namespace. Save parameters, Workspaces, and Integrations in a Run Template. Start manually, through Triggers, or on a schedule. | A small number of pipelines need a straightforward execution entry. | +| Pipelines as Code (PaC) | Keep PipelineRun definitions in the application's `.tekton/` directory. Code events start runs and report results to the merge request. | Definitions should evolve with source and participate in merge-request checks. | +| Centrally managed template | Maintain a versioned Pipeline in the catalog, or a Pipeline in a controlled namespace. Projects supply parameters through a Run Template or a PaC PipelineRun using a resolver. | Multiple projects should share the same standard workflow. | + +For PaC retries, use `/test ` in the merge request and refer to [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html) for the complete syntax. Use one event route for each event: PaC or Triggers. Enabling both routes for the same event can start duplicate work. + +Start with [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html), [Configure GitLab Repository](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/guides/configure_gitlab_repo.html), or [Centralized Management with Hub Pipeline and Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/hub_pipeline_run_template_centralized_management.html). The [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html) provide a guided implementation. + +## Connect the delivery stages + +| Requirement | Recommended combination | Detailed guidance | +| --- | --- | --- | +| Build a traceable artifact | Reuse catalog Tasks, scan the output, and pass its digest to later stages. | [From Commit to Production](./From_Commit_to_Production_on_ACP.md). | +| Deploy to development or test | Update the workload by digest directly when Git history of environment state is unnecessary. | [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html). | +| Deploy to production | Review a change to the production GitOps repository; Argo CD reconciles the selected digest. | [Alauda Container Platform GitOps](https://docs.alauda.io/gitops/4.4/). | +| Separate production artifact access | Approve and copy the selected image by digest into a production registry project. | [Promote Artifacts with Skopeo Copy](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/promote_artifacts_with_skopeo_copy.html). | +| Record who approved a release | Add an ApprovalTask, use group resource names, and retain its approval response. | [Manual Approval Gate](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/manual_approval_gate.html). | +| Verify image signatures | Configure Chains keys and registry access centrally; verify the image by digest before promotion and at deployment admission. | [Signing Key Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | +| Hand off an artifact event | Map a Harbor event to the next pipeline, carrying the digest. Check scan and signing evidence separately. | [Use Harbor Event Triggers](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | + +Promotion copies a selected image; it does not establish that the image passed CI. The approver must match the digest with a successful trusted build and the test environment. Signatures establish the configured signer's statement; their meaning also depends on control over the signing key and the definitions allowed to run. + +The samples deliberately keep test CD, promotion, and production CD as manually started runs. Their inputs come from the preceding run's digest result. Harbor-driven automation is a separate integration step. The release tutorial's first deployment is performed by an administrator; subsequent releases follow the configured delivery path. + +## Prepare a shared platform + +Platform settings are administrator decisions. Roll them out in stages and check [Feature Maturity](https://docs.alauda.io/alauda-devops-pipelines/4.15/overview/feature_maturity.html) for the selected release. + +| Platform concern | Recommended approach | +| --- | --- | +| Tool access | Use project-scoped Connectors. Proxy access uses short-lived workload tokens. Manage image pull Secrets, PaC clone credentials, and Chains registry credentials as separate long-lived credential responsibilities. | +| Project onboarding | Orchestrate the four Connectors automatic creation Tasks, then seed a project GitOps repository and propose platform registration through a merge request. Complete the administrator prerequisites and acceptance checks. | +| Platform configuration | Keep component settings, policies, encrypted administrator credentials, and project declarations in a platform repository. Assign one owner to each resource or field. | +| Namespace isolation | Use separate project namespaces and tool-side tenants. Separate build, delivery, and workload namespaces and restrict access to administrator namespaces. | +| Execution policy | Use platform ClusterPolicies and project Policies. Restrict direct TaskRun and CustomRun creation. Optionally require approved resolver types and coordinates. Move from Audit to Enforce after checking intended workflows. | +| Cleanup | Select one cleanup mechanism. For event-based Pruner, review global defaults and namespace override settings before enabling deletion. | +| Audit history | Store run records and logs in external Results storage with an explicit retention period. Export approval evidence separately before deleting its objects. | +| Scheduled work | Use ScheduledTrigger or CronJob for reconciliation and rotation, with failure alerts and no overlapping runs for a tenant. | + +Results Watcher scope deserves a separate rollout decision. Its default watch scope can affect finished runs across the cluster and add `results.tekton.dev/*` finalizers. Scope it before enabling it. Disabling archiving does not remove existing finalizers; follow the product recovery procedure before cleanup. After Pruner removes cluster objects, the external Results store is the audit source. + +The [platform article](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) explains the 18 platform decisions, repository layout, ownership boundaries, onboarding workflow, rotation, and acceptance checks. + +## Operate without internet access + +Make the catalog, pinned tool and base images, package dependencies, certificates, and vulnerability databases available through the platform's approved internal services. Runtime pipelines must resolve Tasks and Pipelines from local catalog or cluster sources. The sample CI configuration points Trivy at `registry.example.com/mirror/trivy-db:2`; replace it with a reachable database mirror and grant the scanner access. + +The degree of network isolation, number of teams, and required approval evidence are deployment inputs that the platform team must decide. Capacity sizing, migration and decommissioning automation, platform disaster recovery, customer change-management integration, and offline package distribution procedures are outside this series. Define those procedures for the target environment. + +## Related information + +- [Pipelines 4.15](https://docs.alauda.io/alauda-devops-pipelines/4.15/) +- [Artifact Hub Shim 1.0](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/) +- [Connectors 1.14](https://docs.alauda.io/alauda-devops-connectors/1.14/) +- [GitOps 4.4](https://docs.alauda.io/gitops/4.4/) +- [Software Supply Chain Security with Tekton and Kyverno](./Software_Supply_Chain_Security_of_Alauda_Container_Platform_with_Tekton_and_Kyverno.md) +- [Pipeline Policy Constraints with Tekton and Kyverno](./Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md) diff --git a/docs/en/solutions/From_Commit_to_Production_on_ACP.md b/docs/en/solutions/From_Commit_to_Production_on_ACP.md new file mode 100644 index 000000000..07e14e44c --- /dev/null +++ b/docs/en/solutions/From_Commit_to_Production_on_ACP.md @@ -0,0 +1,236 @@ +--- +products: + - Alauda Container Platform + - Alauda DevOps +kind: + - Article +ProductsVersion: + - 4.x +--- + +# From Commit to Production on ACP: Choosing Triggers, Deployment, and Release Controls + +Use this article to choose where pipeline definitions live, how deployments consume build outputs, and where approvals, promotion, and signature checks belong. For a guided implementation, follow the [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html). + +This is the delivery part of [Alauda DevOps Best Practices](./Alauda_DevOps_Best_Practices_Overview.md). Its configuration baseline is Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and GitOps 4.4. Project namespaces, tool-side resources, and Connectors must already exist; the [platform article](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) explains their preparation. + +The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) use a fictional `shop` project. CI runs in `shop-build`; CD and promotion run in `shop-cd`; workloads run in `shop-test` and `shop-prod`. Replace every example address, tool image, group, credential placeholder, and policy identity for your environment. Make the vulnerability database, tool images, base images, and dependencies available locally for disconnected execution. + +## Follow one image identity through the workflow + +```mermaid +flowchart LR + Commit[Merge application change] --> CI[Build and scan] + CI --> Source[Source registry image by digest] + Source --> Test[Deploy and verify in test] + Test --> Approve[Approve selected digest] + Approve --> Copy[Copy image to production registry] + Copy --> Sign[Verify signature and provenance] + Sign --> MR[Review production GitOps change] + MR --> Sync[Argo CD sync and rollout] +``` + +Use `repository@sha256:` for deployment, promotion sources, signature verification, and audit correlation. Tags help people identify releases; a tag alone does not identify immutable content. A Harbor push event establishes that an image was pushed. It does not establish successful scanning, testing, signing, or approval. + +When a vulnerability is found in an already deployed image, fix it and use the same build, promotion, and deployment workflow to release a new digest. Keep the build evidence and the decision to replace the vulnerable image with the release record. + +## Choose where definitions live and how runs start + +| Model | Definition location | Trigger | Suitable use | +| --- | --- | --- | --- | +| Cluster Pipeline | Pipeline in the project namespace; execution settings in a Run Template | Manual, Triggers, or ScheduledTrigger | A small number of centrally operated pipelines | +| Pipelines as Code | PipelineRun files in the application's `.tekton/` directory | Push and merge-request events | Definitions versioned with source and merge-request checks | +| Central template | Versioned catalog Pipeline, or a Pipeline in a controlled cluster namespace | Run Template or a PaC PipelineRun with `pipelineRef` | Shared standards across projects | + +Start with a cluster Pipeline when that is sufficient. Add PaC when the change history and merge-request experience matter. Use central templates when projects should reuse a standard workflow. The same project can use all three for different jobs. Choose one route, PaC or Triggers, for a given event. + +For retries from a merge request, use `/test `. Keep the complete syntax in one place by linking to [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). + +### Combine PaC with a central template + +Keep event annotations and project inputs in `.tekton/`. Reference the platform's Pipeline with a resolver and a pinned version. The following fragment shows the reference used by the application sample; it is not a complete PipelineRun. Use the sample's full parameter and Workspace bindings. + +```yaml +spec: + pipelineRef: + resolver: hub + params: + - name: catalog + value: catalog-pipelines + - name: kind + value: pipeline + - name: name + value: python-image-build-scan-deploy + - name: version + value: "0.3" +``` + +Use `resolver: cluster` for a controlled cluster Pipeline. Avoid a bare `pipelineRef.name` in PaC definitions when admission must inspect the definition's source. See [Resolvers](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/concepts/resolver_concept.html) and [Define PipelineRuns in Git](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/maintain_pipeline_code.html). + +Upgrade a catalog Pipeline by reviewing a change to its pinned version. Projects do not automatically adopt a newly published version. For reproducible rollback of the definition, retain reviewed versions and define whether published coordinates are immutable. A Run Template's version change is a change to that existing template, rather than a reason to create another template. + +### Protect the definition and its runtime credentials + +For a self-managed GitLab instance, set `spec.git_provider.url` on every PaC `Repository`. Without that URL, the PaC behavior described in [Advanced Repository Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html) can send completion status requests and the Repository token to `gitlab.com`. Set the provider URL explicitly for application, platform, and project GitOps repositories. + +Set `spec.settings.pipelinerun_provenance: default_branch` where pre-merge runs must use reviewed definitions. This selects the definition, not trusted application code: a merge-request run still builds the proposed source with the credentials mounted by that definition. Limit those credentials and keep merge-request image outputs separate from release candidates. + +The sample separates the two outputs: + +| Definition | Event | Image repository | Delivery use | +| --- | --- | --- | --- | +| `sample-app/.tekton/ci.yaml` | Push to `main` | `registry.example.com/shop/hello` | Source for test CD and promotion | +| `sample-app/.tekton/ci-merge-request.yaml` | Merge request | `registry.example.com/shop/hello-mr` | Temporary merge-request artifacts | + +The sample's Harbor Connector has project-level access. Separating repository names does not itself enforce a credential boundary. For stronger separation, use a distinct temporary registry project and a credential that cannot write release-candidate images. Before delivery, check that the selected digest is the `IMAGE_DIGEST` of a successful trusted `main` CI run. + +Both sample CI definitions bind the Connector proxy CA through the `ca-bundle` Workspace. The proxy's CA is required even when the upstream registry uses a publicly trusted certificate. For offline scanning, set `trivyExtraArgs` to use the platform's mirrored vulnerability database, for example `--db-repository registry.example.com/mirror/trivy-db:2`. The scanner must be able to pull that image. + +The platform and GitOps repository validation samples compose `git-clone` and `run-script` inline. If the platform enables a policy that forbids inline `pipelineSpec`, publish that validation flow as a catalog Pipeline and reference it through a resolver. + +## Choose a deployment route per environment + +| Route | How the selected digest reaches the environment | Tradeoff | +| --- | --- | --- | +| Direct deployment | A task such as `workload-apply`, `helm-upgrade`, or `run-script` updates the workload. | Fewer moving parts; Git does not hold a second copy of environment state. | +| GitOps | A change updates the environment's image reference in Git; Argo CD reconciles it. | Reviewable environment state and history; requires a repository and a sync path. | + +A useful starting point is direct deployment for development or test and GitOps for production. The samples use GitOps for both test and production so that the same reconciliation path can be used in both environments. + +For direct deployment, use a digest in `workloadContainers`; use a lowercase `workloadKind`, such as `deployment`. The run identity needs access to read and update the target workload. Consult the catalog Task for its rollout, timeout, rollback, and cross-cluster options. + +For GitOps, use one directory and one Application per environment. Keep each image digest in one place, such as the Kustomize `images` entry: + +```text +gitops/ +├── base/ +└── envs/ + ├── test/kustomization.yaml + └── prod/kustomization.yaml +``` + +In the sample, test CD uses `submitMode: direct`; production CD uses `submitMode: pr`. The production overlay is seeded with the production registry repository, so a release changes only its digest. The first production deployment waits for a promoted image. + +Enforce production Git review in the code platform. Use protected branches and, where supported, code-owner approval. Other options include a separately protected production branch or repository, or a reviewed server-side policy. Branch protection alone does not create directory-level protection. Remove direct production write access that would bypass the chosen review path. + +### Pass results between build, promotion, and GitOps + +`gitops-repo-update` is a Pipeline and runs as a separate PipelineRun. Pass the selected digest as an argument to its update script. Use a fresh source Workspace volume per run and an image that contains every command used by the script. `submitMode: pr` also requires the configured GitLab CLI image and a token with API access. + +The sample updates the `digest:` line with `sed` and verifies the rendered image with Kustomize. This preserves a one-line production diff. Avoid using a rewrite operation that changes unrelated formatting or image fields. The GitOps writer token needs the scopes required for cloning, pushing, and creating the merge request; keep it separate from the application CI token. + +The sample starts test CD, promotion, and production CD manually from rendered PipelineRun YAML. Replace the template parameters, verify the image mapping, identity, and Workspaces, and submit with `kubectl create`. Use [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html) for other execution entries. Harbor event automation is an additional integration, not part of the sample's manual path. + +Production rollback follows the same review boundary: select a retained, previously promoted digest, verify its signature and application compatibility, and review the GitOps change. Database and data migrations need an application-specific rollback plan. Establish and rehearse the complete rollback runbook in the target environment. Configure [Harbor tag retention](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-retention-rules/) and [tag immutability](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-immutability-rules/) so that required rollback images remain available. + +## Select the release controls + +Promotion, approval, credential gating, and image admission solve separate decisions: + +| Decision | Recommended control | +| --- | --- | +| Should production use a separate artifact project? | Copy the selected digest into that project, then update the production Git reference. | +| Who authorizes the selected release? | An ApprovalTask, a production Git merge-request review, or both, according to policy. | +| Who may use the production Connector? | A dedicated release identity plus admission and RBAC restrictions. Add approval-gated proxy access when credential owners and pipeline owners are separate. | +| What may be deployed? | Admission policies for allowed production repositories, signatures, and any required approved-digest evidence. | + +Use the Core promotion admission rules in [Constrain Artifact Promotion with Kyverno](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/constrain_artifact_promotion_with_kyverno.html). Rules 1–3 constrain the promotion execution path. Add rule 4 when the promotion Pipeline is maintained in a namespace, to control changes to that definition. The companion samples do not install these promotion rules or a production-repository-only deployment policy; platform administrators must add them before using the reference design as a production control. + +Promotion validates the source and destination mapping and copies bytes. It does not query CI to establish that the image passed. The approver must correlate the digest with the trusted build, scan, and test evidence. The sample validates that the image mapping is first, the source uses a SHA-256 digest, and the repositories and release-label format match the configured workflow. Optional signature mappings must refer to the same image digest. + +### Add an approval node + +Use ACP group resource names in `approvers`, rather than a group's display name or an individual's name. The following is a task fragment; replace the example group with the actual resource name: + +```yaml +- name: wait-for-approval + taskRef: + apiVersion: openshift-pipelines.org/v1alpha1 + kind: ApprovalTask + timeout: "24h" + params: + - name: approvers + value: + - group:release-managers + - name: numberOfApprovalsRequired + value: "1" + - name: description + value: "Approve promotion into production" +``` + +PipelineRun and task timeout budgets must include work before approval, the entire approval window, and work after approval. Keep the approver group separate from identities allowed to initiate or modify release runs. Use [Manual Approval Gate](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/manual_approval_gate.html) for approval permissions, group responses, and timeout configuration. Retain the ApprovalTask responses separately before the objects are removed. + +Approvers can inspect pending work in the console or through `kubectl`. The optional sample `promotion/optional/console-approver.yaml` supplies a custom console role and group binding; review its scope before applying it. An approval notification can be sent with the catalog's `send-mail` Task. Email addresses, SMTP configuration, and credentials must be configured separately; the sample does not integrate that notification step. + +Customer change-management system integration is outside this series. The available building blocks are the approval node and [Incoming Webhooks](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/incoming_webhooks.html). Define the mapping and audit contract for that integration separately. + +### Gate the production Connector after approval + +Keep the production Connector and its Secret in a namespace where project members cannot create Pods or run arbitrary pipelines. A Pod can mount a Secret from its own namespace without a separate Secret-read authorization check. Consequently, permission to create Pods or PipelineRuns in a credential-bearing namespace is a credential boundary. + +The sample release ServiceAccount `shop-release` reads the source Connector and mounts the production Connector from `platform-connectors`. Its standing grants are in `sample-app/promotion/standing-access.yaml`. Use RBAC for the cross-namespace proxy grants described by the product guide. + +Approval gating uses two installation-wide feature flags. Roll them out in this order: + +1. Inventory every identity mounting a Connector that is not behind approval. Prepare its read authorization and `connectors/proxy` permission with verb `*`. A `get` grant on `connectors/proxy` is insufficient. Do not grant direct proxy access to the production Connector that approval must gate. +2. Enable `enable-connectors-approval`. Immediately create the required AccessPolicies, including the administrator-Connector grants and the production approval policy. Do not start affected runs in the interval between enabling the flag and creating the policies. +3. In the production AccessPolicy, configure the designated `approvers` and keep them aligned with the ApprovalTask and promotion admission policy. Remove any other matching default or direct proxy grants. +4. Enable `enable-connector-proxy-permissions`, which applies the proxy permission check to requests as well. + +Set feature flags through `ConnectorsConfig`. For the full configuration and verification path, see [Enable Proxy Permission Checks and Approval Gating](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/how_to/enable_proxy_permissions_and_approval.html), [AccessPolicy](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/access_policy.html), and [Skopeo Copy connector approval](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/promote_artifacts_with_skopeo_copy.html#connector-approval). + +`defaultPermission` policies have authorization prerequisites and are not a substitute for every cross-namespace RBAC grant. A Ready policy alone does not establish a working mount. Verify the actual identity, Pod events, mounted configuration, and AccessRequest result. + +## Sign and verify the image + +Configure Chains signing keys centrally and distribute only the public verification key to projects. Protect both the signing Secret and the ability to create Pods in its namespace. Application teams do not need access to the signing Secret. + +The sample stores signatures and attestations beside the image. Chains first uses matching registry credentials from the run's ServiceAccount. Those credentials must be able to push to the image's registry project. Where they are absent, configure a centrally managed registry credential on the Chains controller through `TektonConfig`. Chains does not use a Connector's CSI proxy Workspace as its registry credential. + +The central credential is created by an administrator, outside Git. Limit it to the registry projects that Chains must read and sign. When onboarding a project, the platform team must add its source and production registry projects to that credential's access. Rotate it and restrict its Secret and namespace. Apply controller configuration changes in a maintenance window, preserving other container options already configured. + +A successful PipelineRun is not proof of stored signatures. Inspect TaskRun signing annotations and events, and independently verify the image and its attestation. The acceptance object here is the image digest and the copy TaskRun's provenance; approval history is separate evidence. + +```bash +# IMAGE is the selected repository@sha256:digest; the platform supplies cosign.pub. +cosign verify --key cosign.pub "$IMAGE" +cosign verify-attestation --key cosign.pub --type slsaprovenance "$IMAGE" +``` + +For an offline installation configured without a transparency log, use the product's offline verification settings, including `--insecure-ignore-tlog=true` where required. This skips the external transparency-log check; retain the signature and attestation checks. Authenticate to private registries without committing the login credentials. + +If the platform chooses a separate signature repository, configure `storage.oci.repository` centrally and use the corresponding `COSIGN_REPOSITORY` in verification and the repository setting in Kyverno. Review storage and overwrite behavior before combining build and promotion attestations for the same digest. Copying build signature attachments during promotion is optional; use the documented `imageMappings` and `multiArch: all` configuration if selected. + +At deployment admission, start the image-verification policy in Audit, with the sample's `mutateDigest: false` setting, and review the matching images and reports before Enforce. The sample's production image list is explicit: add each newly onboarded production repository to the policy. See [Image Signature Verification](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/how_to/image_signature_verification.html). + +## Check the controls at their actual enforcement points + +| Point in the workflow | What is checked | Remaining responsibility | +| --- | --- | --- | +| Repository event and definition resolution | Event matching and reviewed definition source | Limit the credentials available to proposed source code. | +| PipelineRun admission | Allowed definition source and configured promotion path | Restrict who can edit referenced definitions and parameters. | +| TaskRun or CustomRun creation | Approved controller identities for direct-run restrictions | Protect automation identities and their RBAC. | +| Approval | Designated approver responses | Match the approved digest with CI and test evidence. | +| Production registry write | Promotion mapping and credential access | Control other writers, including the signing identity. | +| Workload admission | Allowed registry and required signature checks | Remove direct deployment paths that bypass the policy. | +| Retention and cleanup | Selected run/log retention | Export approval evidence and verify archived access. | + +Use platform ClusterPolicies for the baseline and project Policies for additional restrictions. Restrict PolicyException creation or disable exceptions according to platform policy. A resolver-type allow-list alone does not pin catalog, repository, namespace, version, or definition content. The inline validation exception must be handled explicitly when enabling a no-inline policy. + +## Accept the delivery workflow + +Run `acceptance/01-commit-to-production.sh` from the sample only in an isolated acceptance project after replacing its inputs. The checks are an execution procedure, not a claim that an arbitrary environment is ready. + +| Check | Required evidence | +| --- | --- | +| CI completed | The trusted `main` run succeeded; its image repository and digest match the selected release input. | +| Test ran the selected image | Compare the complete workload image reference and verify rollout completion. | +| Promotion copied that image | The promotion result reports the expected destination repository and digest. | +| Production uses the promotion result | The GitOps merge request is merged, the rendered image matches the destination digest, and the workload rollout completed. `Synced` alone can describe an older revision. | +| Signature and provenance exist | Both cosign commands succeed against the production image. A signing annotation alone is insufficient. | +| Admission enforces the selected policy | Use a restricted-compatible probe in the acceptance project and distinguish policy rejection from Pod Security or image-pull failure. Clean up the probe afterward. | +| Approval is attributable | Save the ApprovalTask response, approver group members, selected digest, and related run identifiers in the release record. | + +Before production, address the sample's explicit hardening work: restrict the GitOps writer's namespace, protect production Git changes, install the promotion Core rules, constrain production image sources, and limit direct cluster writes. Do not grant project members Pod creation in namespaces that contain administrator or release credentials. Establish local rollback and policy-service recovery runbooks. + +Continue with [Managing Platform Configuration and Onboarding Isolated Projects](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) for the platform configuration, project declarations, and operating boundaries behind this workflow. diff --git a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md new file mode 100644 index 000000000..a45b1c3cd --- /dev/null +++ b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md @@ -0,0 +1,325 @@ +--- +products: + - Alauda Container Platform + - Alauda DevOps +kind: + - Article +ProductsVersion: + - 4.x +--- + +# Managing Platform Configuration and Onboarding Isolated Projects with Argo CD and Tekton on ACP + +Use a platform Git repository to review and preserve shared configuration. Use an onboarding pipeline to create a project's namespaces and tool-side resources, then propose its deployment configuration through a merge request. Argo CD reconciles the objects owned by Git; the onboarding tasks reconcile the resources they create in external tools. + +This article is the platform part of [Alauda DevOps Best Practices](./Alauda_DevOps_Best_Practices_Overview.md). It assumes one ACP project and separate tool-side tenants per project. Its configuration baseline is Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and GitOps 4.4 on a compatible ACP installation. Teams sharing one namespace and one tool tenant need a different boundary model. + +The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) contain the complete reference files. Names such as `shop`, `payments`, `platform-connectors`, and `registry.example.com` are fictional. Replace them, the image placeholders, group resource names, policy identities, and encrypted Secret placeholders for the target environment. + +## Establish the ownership model + +Keep three kinds of repository separate: + +| Repository | Contents | Owner | +| --- | --- | --- | +| Platform repository | Component configuration, admission policies, encrypted administrator credentials, project declarations, and platform Application definitions | Platform team | +| Template repository and catalog | Versioned shared Pipelines and Tasks; initial project GitOps repository content | Platform team | +| Project repositories | Application source and `.tekton/`; environment manifests and digest references in the project GitOps repository | Project team | + +Run Templates are execution settings in project namespaces. They are distinct from versioned catalog Pipeline definitions. Keep application manifests out of the platform repository, and keep platform policy ownership out of application repositories. + +```mermaid +flowchart TB + MR[Review platform change] --> Git[Platform repository] + Git --> Argo[Argo CD] + Argo --> Components[Component settings] + Argo --> Policies[Platform policies] + Argo --> Credentials[Encrypted administrator credentials] + Argo --> Declarations[Published project declarations] + Declarations --> Onboard[Tekton onboarding] + Onboard --> Tools[GitLab Harbor Nexus SonarQube] + Onboard --> Namespaces[Project namespaces and identities] + Onboard --> Seed[Seed project GitOps repository] + Onboard --> Registration[Propose platform registration MR] + Registration --> Git + Seed --> ProjectArgo[Project Applications] +``` + +Assign one owner to each resource or field. Argo CD self-healing applies to objects it tracks; it does not reconcile every object in the cluster or reverse external tool operations. Removing a declaration is not an automated decommissioning procedure. + +## Prepare the platform before bootstrap + +Install components using their product installation instructions: + +- [Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/install.html), including the required [PaC component](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/install/manage_pac.html) and [Manual Approval Gate](https://docs.alauda.io/alauda-devops-pipelines/4.15/how_to/manual_approval_gate_operator.html). +- [Connectors](https://docs.alauda.io/alauda-devops-connectors/1.14/install/install.html). +- [GitOps](https://docs.alauda.io/gitops/4.4/install_gitops/index.html). +- [Kyverno](https://docs.alauda.io/container_platform/4.4/security/security_and_compliance/compliance/install.html) and the sealed-secrets controller. Their CRDs must exist before syncing policies and SealedSecrets. + +An administrator creates the management namespace, `platform-connectors` in the samples, and installs the four Connectors automatic creation Tasks there. These Tasks are installed from the Connectors documentation; they are not fetched from the general Task catalog. Prepare administrator access to the four tools and encrypt the credentials for the target sealed-secrets controller before committing them. + +Create the Chains signing key and central registry credential outside Git. The central registry credential must exist before syncing the component layer that mounts it. Give it access only to registry projects that Chains must read and sign, and update that access when a project is onboarded. + +Register the platform repository with Argo CD and provide read access to each project GitOps repository. A repository credential template can cover the approved GitLab URL prefix; narrower templates must be added per project. Use repository URLs ending in `.git` for Application and AppProject GitLab references. Use the cluster name registered by Alauda GitOps for both Application destinations and AppProject destination constraints. + +Create the ACP project through the platform before onboarding it. The pipeline exports that project's manifest for subsequent registration. Importing an existing project into Git needs the same review as creating one: the project Application can later prune that resource. + +Platform administrators are responsible for webhook setup and event-delivery verification for platform, project GitOps, and application repositories. Registration with a token authorized to manage repository webhooks supports controller-managed webhook creation and deletion. For centralized management, follow [Create a Git Secret for PAC](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/create_git_secret.html). The sample's Developer-level GitOps token cannot manage webhooks; complete the administrator webhook configuration and test a real event before accepting onboarding. + +Keep Harbor projects private. Configure Nexus anonymous access and SonarQube permissions according to the project-isolation requirements in their Task guides. Do not infer isolation merely from the presence of separate credentials. + +For a disconnected environment, prepare local catalog content, pinned tool and base images, dependency repositories, certificates, and vulnerability databases before running pipelines. + +## Decide the 18 platform baselines + +The table identifies the platform decision and its repository layer. Values and detailed steps belong in the linked product documentation. These are administrator-managed recommendations, with maturity and prerequisites determined by the selected release. + +| # | Decision | Why it is shared | Layer and guidance | +| --- | --- | --- | --- | +| 1 | Separate Repository access tokens from webhook administration; use reviewed default-branch definitions and an explicit GitLab provider URL. | Credential scope and pre-merge execution trust should be consistent. | Credentials and project: [PaC Repository configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html). | +| 2 | Standardize merge-request retries with `/test `. | Teams use one supported entry. | Project template: [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). | +| 3 | Separate platform-provided Repository parameters from application inputs. | Addresses and Connector names have a defined owner. | Project: [Advanced Repository Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html). | +| 4 | Decide Harbor event handoffs. | Event endpoints and validation need consistent ownership. | Component: [Harbor events](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | +| 5 | Schedule reconciliation and rotation with failure alerts. | The platform owns the interval and operational response. | Component: [ScheduledTrigger](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/scheduledtriggers.html). | +| 6 | Standardize EventListener exposure. | TLS, routes, and webhook validation belong to the platform. | Component: [Automatic exposure rules](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/trigger-wrapper-config.html). | +| 7 | Orchestrate tool-side tenant creation with the four product Tasks. | Administrators retain tool-wide credentials. | Credentials and project: tool Task guides below. | +| 8 | Define long-lived credential rotation for each tool. | Expiry and renewal behavior differ by tool. | Component and credentials: [Credential rotation](https://docs.alauda.io/alauda-devops-connectors/1.14/how_to/automatically_create_connectors_for_a_project.html#credential-rotation). | +| 9 | Configure Chains signing keys and image verification. | The platform owns the signer and distributes its public key. | Component, credentials, policy: [Signing keys](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | +| 10 | Use Connector proxy access while tracking long-lived credential exceptions. | Access boundaries and rotation need a central owner. | Credentials: [Connectors Proxy](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_proxy.html) and [CSI Driver](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_csi.html). | +| 11 | Apply namespace Pod Security and separate build namespaces. | Build tooling and workloads have different privilege needs. | Policy and project: [Restricted standard](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/comply_with_pod_security_restricted.html). | +| 12 | Install approval and define protected-environment groups and promotion rules. | Approval ownership is a platform control. | Component, project, policy: [Manual Approval Gate](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/manual_approval_gate.html). | +| 13 | Establish platform and project policy layers. | Projects can add restrictions without weakening the platform baseline. | Policy: [Constrain pipeline execution](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/constrain_pipeline_execution_with_kyverno.html). | +| 14 | Own the shared, locally available catalog. | Reuse and offline content need governance. | Component: [Author Catalog Resources](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/author_catalog_resources.html). | +| 15 | Publish versioned central Pipelines. | Projects adopt reviewed versions explicitly. | Component: [Centralized Pipeline management](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/hub_pipeline_run_template_centralized_management.html). | +| 16 | Select Pruner defaults and namespace override policy. | Cleanup affects shared cluster history and capacity. | Component: [Pruner configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pruner/configure/pruner_configuration.html). | +| 17 | Define Results storage, watch scope, authentication, and retention. | Audit records need storage independent of live objects. | Component: [Results retention](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/retention_policy_configuration.html). | +| 18 | Enable metrics and operational monitoring. | Operators need evidence of controller and execution health. | Component: [Metrics Collection](https://docs.alauda.io/alauda-devops-pipelines/4.15/configure/metrics_collection.html). | + +### Separate policy scope from definition trust + +Use platform ClusterPolicies for shared controls and namespaced Policies for project-specific restrictions. Restrict direct TaskRun and CustomRun creation by the actual submitting identity; client-supplied owner references alone are not an authorization boundary. Enumerate controller identities and review their permissions. + +Optionally restrict PipelineRun references to approved resolver types. This abbreviated policy illustrates only the source-type decision; adapt namespace exclusions for every platform system namespace: + +```yaml +apiVersion: kyverno.io/v1 +kind: ClusterPolicy +metadata: + name: pipelinerun-source-baseline +spec: + background: false + rules: + - name: approved-resolver-only + match: + any: + - resources: + kinds: [tekton.dev/v1/PipelineRun] + operations: [CREATE] + exclude: + any: + - resources: + namespaces: [platform-connectors] + validate: + failureAction: Audit + message: "pipelineRef must use an approved resolver" + deny: + conditions: + all: + - key: "{{ request.object.spec.pipelineRef.resolver || '' }}" + operator: AnyNotIn + value: [cluster, hub, git] +``` + +This does not constrain a resolver's URL, catalog, namespace, version, definition content, or runtime credential scope. It also rejects inline `pipelineSpec` when enforced. Convert the sample's inline validation flow to a catalog Pipeline before enabling that restriction for project repositories. Restrict PolicyException creation or disable exceptions according to platform policy. + +The sample's `argocd-project-namespace-boundary` policy constrains writes of Role, RoleBinding, ResourceQuota, LimitRange, and NetworkPolicy by the configured Argo CD identities in labelled project namespaces. It allows the platform-owned registration path and requires `allowExistingViolations: false`. Adapt both Argo CD identities for the target cluster and verify the intended allow and deny cases before relying on the boundary. Install and verify the policy on every destination cluster to which the boundary should apply. + +Kyverno availability and recovery are platform operating concerns. Prepare a local runbook; this series does not prescribe the cluster's write behavior during a policy-service outage. See [Pipeline Policy Constraints with Tekton and Kyverno](./Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md) for the policy design. + +## Organize and synchronize the platform repository + +The sample is split into four ownership layers and the onboarding installation: + +```text +platform-config/ +├── argocd/apps/ # Root and child Application definitions +├── components/ # TektonConfig and component options +├── policies/ # Platform admission policies +├── credentials/ # Administrator Connectors and sealed placeholders +├── projects/ # Declarations and reviewed registration output +├── templates/ # Registration rendering templates +├── onboarding/ # Pipelines, Run Templates, schedules, and RBAC +└── .tekton/ # Validation and changed-project dispatch +``` + +Copy `onboarding/` from the sample to the platform repository root as described by its README. Register the repository read-only with Argo CD and use one child Application per layer. The root Application manages those Application definitions. + +For partial singleton component configuration, use server-side apply so the platform repository owns only the specified fields. The samples also set `argocd.argoproj.io/compare-options: ServerSideDiff=true`. Review the generated diff and the operator's field ownership before synchronization. The component layer uses `prune: false` so that removing a fragment does not delete the singleton TektonConfig. Establish the component layer's automated synchronization behavior in the target environment before enabling ongoing self-healing. + +The credential layer does not enable self-healing: seal credentials for the target controller and plan how rotations update live Secrets and encrypted source. Other layers can use prune and self-heal according to their ownership. Namespaces are created by onboarding; do not add a second namespace creator with `CreateNamespace`. + +ACP maintains the AppProject associated with an ACP project. The registration patch manages only the intended repository and destination fields. Reconcile the effective destinations with the platform-managed entries, and use the admission boundary for object-kind restrictions. Do not rely on an Application's sync status alone to prove the deployment boundary. + +Review deletion carefully. Pruning a registered ACP project can delete its same-named namespace and its Secrets, Connectors, and Repository objects. It does not reverse tool-side creation or remove every build, delivery, or environment namespace. Decommissioning requires a separate approved procedure. + +### Validate changes before merging + +Both the platform and project GitOps templates include `.tekton/validate.yaml`, which composes catalog `git-clone` and `run-script` Tasks. Replace its tool image with a locally available image containing the required shell and Kustomize commands. Establish its webhook before the first merge-request run. With default-branch definition provenance, a change to the validation definition is first exercised after it has entered that branch. + +Validate every directory owned by an Application, and add new directories to the validation paths. Re-run a merge-request check with `/test validate`. YAML parsing and Kustomize rendering do not validate CRD admission behavior; use the target schemas and an isolated project for that acceptance step. + +## Declare a project and run onboarding + +The declaration is a sample schema, not an ACP API. The pipeline validates its supported naming conventions. A complete `shop` declaration includes the following fields; use the sample's full file: + +```yaml +name: shop +platformProject: shop +owners: + - shop-platform-owners +environments: [test, prod] +tools: [gitlab, harbor, nexus, sonarqube] +approvers: + prod: group:release-managers +imagePullFrom: [] +namespaces: + workloads: [shop, shop-test, shop-prod] + build: shop-build + delivery: shop-cd +gitlab: + group: shop + subgroups: [apps, gitops] +harbor: + projects: [shop, shop-prod] +nexus: + repositories: + - name: shop-maven + format: maven2 + type: hosted + - name: shop-npm + format: npm + type: hosted +sonarqube: + project: shop +``` + +`owners` contains ACP group resource names without a `group:` prefix; approval entries use `group:`. A console display name is not the resource name. Group membership changes belong to group administration, rather than edits naming individuals in every declaration. + +Submit the declaration, project-specific onboarding permissions, and schedule through a platform merge request. Confirm that the declaration and onboarding Applications have synchronized; synchronize them manually when using a manual sync policy. Then examine the dispatcher result and every child onboarding run. A successful merge or dispatcher does not by itself establish complete project readiness. + +The sample dispatches changed declarations after a merge. A path filter can also admit a run with no relevant declaration changes; the dispatcher is responsible for selecting actual declarations. A deleted declaration is skipped, rather than interpreted as an instruction to delete a project. + +### Understand the seven onboarding stages + +| Stage | Work | Owner or mechanism | +| --- | --- | --- | +| 1 | Export the existing ACP project without runtime metadata; create or reconcile labelled namespaces and delivery identities. Workloads use restricted Pod Security; the build namespace is separate. | Sample orchestration and Kubernetes APIs | +| 2 | Create the GitLab tenant group and a project access token; make a separate writer Connector for GitOps delivery. | GitLab automatic creation Task | +| 3 | Create private source and production Harbor projects, distribute image pull configuration, and create separate source-read and production-write Connectors for release. | Harbor automatic creation Task | +| 4 | Create Nexus repositories and a scoped user, then write the project Connector. | Nexus automatic creation Task | +| 5 | Create the SonarQube user, token, and project permission template, then write the Connector. | SonarQube automatic creation Task | +| 6 | Seed the empty project GitOps repository and create its PaC Repository and access Secret. Complete and verify webhook configuration separately. | Sample template seeding and PaC resources | +| 7 | Render project Applications, AppProject patch, owner and Connector bindings, and the exported ACP project. Open a platform registration merge request. | A separate `gitops-repo-update` PipelineRun | + +The registration step proposes Git changes instead of directly applying Argo CD resources. Review the entire diff, including RBAC and destination changes. Keep sufficient execution capacity for both the parent and child run: the waiting parent still reserves its Pod resources. Cancellation and cleanup must account for the independently created child run. + +The pipeline runs in the management namespace with administrator Connectors. Permission to create Pods or PipelineRuns there is effectively access to those credentials, even without API permission to read Secrets. Limit that namespace and the onboarding identity to platform automation and administrators. + +### Reuse the product Tasks + +The four Tasks have different parameters and rotation behavior. Read their installation, access, and prerequisite instructions: + +- [GitLab tenant group and Connector](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-gitlab/how_to/using_gitlab_connector_automatic_creation_task.html). +- [Harbor resources and Connector](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-harbor/how_to/using_harbor_connector_automatic_creation_task.html). +- [Nexus resources and Connector](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-nexus/how_to/using_nexus_connector_automatic_creation_task.html). +- [SonarQube tenant and Connector](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-sonarqube/how_to/using_sonarqube_connector_automatic_creation_task.html). + +Set the target `connector` as `namespace/name`. Set GitLab access levels explicitly: the project access token and the GitOps writer have different responsibilities. In the sample, project access uses `api` and `read_repository`; the writer additionally uses `write_repository`. Connectors CSI mounts use the `connectors` attribute, with the namespace included. + +The sample writes only to the current cluster using an explicitly authorized ServiceAccount. For cross-cluster application, follow each Task's Kubernetes Connector and `kubeconfig` Workspace instructions. Do not assume that the management namespace's default ServiceAccount has the required write permissions. + +An alternative product workflow creates project Connectors from namespace events: [Automatically Create Connectors for a Project](https://docs.alauda.io/alauda-devops-connectors/1.14/how_to/automatically_create_connectors_for_a_project.html). If choosing that path, assign one creator and rotation owner per tenant and resource. Avoid two independent reconcilers managing the same credentials. + +## Register repositories and verify events + +Set the self-managed provider URL and reviewed definition source on every PaC Repository. The following fragment omits credential data; create its referenced Secret through the approved credential path: + +```yaml +apiVersion: pipelinesascode.tekton.dev/v1alpha1 +kind: Repository +metadata: + name: shop-gitops + namespace: shop +spec: + url: https://git.example.com/shop/gitops/config + git_provider: + type: gitlab + url: https://git.example.com + secret: + name: shop-gitops-auth + webhook_secret: + name: shop-gitops-auth + settings: + pipelinerun_provenance: default_branch +``` + +Set the provider URL explicitly so completion requests and the Repository token use the intended GitLab instance. The PaC repository URL differs from the `.git` URL used by Argo CD. `gitops/config` is a repository beneath the `gitops` subgroup, rather than a repository path conflicting with that subgroup's name. + +The sample's project GitOps token is Developer-level and requires administrator-managed webhook setup. Verify a fresh event and its resulting run. Keep webhook management credentials available through repository decommissioning, and check for remaining code-platform webhooks afterward. The application's Repository belongs in the build namespace, where CI executes. + +## Reconcile and rotate deliberately + +Use repeated runs of the same supported parameters for reconciliation. Do not equate Connector `Ready=True` with an unexpired, usable tool token; perform an authorized tool request or workload operation before and after rotation. + +| Change | Required operating action | +| --- | --- | +| A supported declaration field changes | Review Task-specific recreation and rotation effects, then reconcile the project. | +| A new environment is added | Add its GitOps directory and validation path through a project merge request as well as changing the declaration. | +| A template changes | Propose updates to existing project repositories separately. Seeding copies only into an empty repository. | +| A credential is due for rotation | Run the relevant Task on a schedule shorter than expiry minus a safety margin; alert and retry on failure. | +| A credential has no expiry | Rotate at the interval required by policy; choose the Task's refresh behavior explicitly. | + +The sample's supported GitLab subgroups are `apps` and `gitops`. Changing that convention requires a reviewed pipeline change, rather than only a declaration edit. Tool-side identity changes can recreate accounts or credentials; inspect the product Task's contract before changing names or project sets. + +Do not overlap runs for the same tenant. Arrange rotation outside active pipeline windows and monitor failed runs before the current credential expires. CSI workload-token issuance is separate from rotating the tool's long-lived credential. Consult the Task guide for expiry recovery; define a local recovery procedure rather than depending on an unobserved natural-expiry path. + +If enabling Connector approval, prepare grants for every affected workload, enable `enable-connectors-approval`, immediately create the required AccessPolicies, and then enable `enable-connector-proxy-permissions`. The sample's optional administrator AccessPolicy is excluded from the default Kustomization and is applied only after the feature is enabled. Use RBAC for the documented cross-namespace grants and keep approval-gated production Connectors free of direct proxy grants. See the [delivery article](./From_Commit_to_Production_on_ACP.md#gate-the-production-connector-after-approval) for the complete rollout boundary. + +## Accept the project boundaries + +| Boundary | What must be demonstrated | +| --- | --- | +| Namespaces and Pod Security | Workload namespaces have the intended labels; builds and delivery use their separate namespaces and identities. | +| Deployment scope | Effective AppProject destinations and platform admission permit only the intended project deployment path. Test forbidden RBAC and quota writes with the actual Argo CD identities. | +| Tool access | GitLab, Harbor, Nexus, and SonarQube authorization matches the tenant. Use positive controls before negative cross-project tests. | +| Credential access | Project users cannot create Pods or runs in administrator or signing namespaces. Inventory image pull Secrets, PaC clone credentials, and central Chains credentials separately. | +| Event routing | A fresh repository event or merge-request retry produces the expected run in the intended namespace. | +| Rotation | A successful non-overlapping run updates the intended credential and an authorized operation works afterward. | + +Run `acceptance/02-platform-and-onboarding.sh` from the sample in an isolated acceptance project. Its companion delivery script verifies the image path after the first application build. Check every child onboarding run, the four Connectors, the seeded repository, registration merge request, and Applications. The initial placeholder digest is not deployable; workload health follows the first real deployment. + +For cross-project image denial, first verify that the exact image exists and can be pulled in its own project. Then test an unauthorized project, distinguish `unauthorized` from `not found` and Pod Security rejection, and repeat in the opposite direction using an image that exists there. Delete temporary probe resources after the checks. + +Test self-healing only on a disposable resource in the acceptance project. Record its UID and recovery manifest, perform the approved change, and confirm recreation with a different UID. Restore it if the check fails. This is separate from proving ongoing automatic reconciliation of the operator-managed component layer. + +## Preserve audit records and plan recovery + +Choose one cleanup mechanism and review its scope before enabling it. Use event-based Pruner according to the global and namespace configuration guides; disable the older periodic mechanism when migrating. Verify each selected retention mode in the target environment before it can remove shared run history. + +Use external PostgreSQL and the documented log storage for Results. Define watch scope, authentication, and one retention policy for the reference configuration. Verify read authorization through the Results API, rather than inferring it from Kubernetes resource visibility. Preserve approval responses separately before cleanup removes the ApprovalTask objects. + +Scope Results Watcher before enabling it: default watching can affect finished runs across namespaces and add `results.tekton.dev/*` finalizers. Disabling archiving does not remove finalizers already present. Follow the product recovery procedure before deletion and treat external Results storage as the audit source after cluster objects are pruned. See [S3 storage](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/s3_storage_configuration.html) and [Results retention](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/retention_policy_configuration.html). + +Git does not replace backups of credentials and external data. Before a platform rebuild, the platform team must prepare and rehearse recovery of these dependencies: + +1. Compatible platform components, Kyverno, and sealed-secrets CRDs and controllers. +2. The sealed-secrets decryption key, or re-sealed credentials for the new controller. +3. Chains signing keys and the central registry credential; coordinate any verification-key change with image admission. +4. The management namespace and installed automatic creation Tasks. +5. Argo CD repository access and project repository credential templates, plus PaC access and webhook configuration. + +After preparing those dependencies, synchronize the platform layers, run onboarding for the declared projects, review registration changes, and complete the acceptance checks. This is a dependency outline; a complete empty-cluster or second-cluster recovery runbook must be exercised in the target environment. + +Before production, complete the sample's hardening work: isolate pre-merge validation from administrator credentials, narrow the AppProject used for project registration, protect the GitOps writer's namespace, install promotion Core rules, and restrict production image sources. Restrict direct cluster writes alongside Git review. Template upgrades, decommissioning, capacity sizing, offline package distribution, and customer change-management integration require separate operating procedures. + +Continue with [From Commit to Production](./From_Commit_to_Production_on_ACP.md) to use the onboarded project's build, test, promotion, and production GitOps paths. From df9e34cd762f28f635e18c1b9320279311aec3e9 Mon Sep 17 00:00:00 2001 From: Zichen Yu <1062955096@qq.com> Date: Thu, 22 Oct 2026 12:31:28 +0800 Subject: [PATCH 2/5] docs: clarify DevOps best-practice boundaries --- .../Alauda_DevOps_Best_Practices_Overview.md | 18 ++-- .../From_Commit_to_Production_on_ACP.md | 44 ++++++---- ...Projects_with_Argo_CD_and_Tekton_on_ACP.md | 88 +++++++++++++++---- 3 files changed, 105 insertions(+), 45 deletions(-) diff --git a/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md b/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md index 0100339da..b698ebbe4 100644 --- a/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md +++ b/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md @@ -3,7 +3,7 @@ products: - Alauda Container Platform - Alauda DevOps kind: - - Article + - Best Practices ProductsVersion: - 4.x --- @@ -69,10 +69,12 @@ Choose the simplest definition model that meets the team's needs. The models can | Pipelines as Code (PaC) | Keep PipelineRun definitions in the application's `.tekton/` directory. Code events start runs and report results to the merge request. | Definitions should evolve with source and participate in merge-request checks. | | Centrally managed template | Maintain a versioned Pipeline in the catalog, or a Pipeline in a controlled namespace. Projects supply parameters through a Run Template or a PaC PipelineRun using a resolver. | Multiple projects should share the same standard workflow. | -For PaC retries, use `/test ` in the merge request and refer to [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html) for the complete syntax. Use one event route for each event: PaC or Triggers. Enabling both routes for the same event can start duplicate work. +For a PaC retry from a merge request, comment `/test branch:` on the merge request. For a push-triggered run, comment the same form on the commit at the head of that branch. The `branch` value makes the source explicit. Refer to [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html) for the complete syntax. Use one event route for each event: PaC or Triggers. Enabling both routes for the same event can start duplicate work. Start with [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html), [Configure GitLab Repository](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/guides/configure_gitlab_repo.html), or [Centralized Management with Hub Pipeline and Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/hub_pipeline_run_template_centralized_management.html). The [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html) provide a guided implementation. +For systems that do not use Git events, an Incoming Webhook can start a PipelineRun. Send an HTTP `POST` with a JSON body containing `repository`, `namespace`, `branch`, `pipelinerun`, and `secret` to the PaC `/incoming` endpoint. Treat the request as accepted only when it returns `202` and a new PipelineRun appears; a `200` response with an empty body can mean that no run was triggered. + ## Connect the delivery stages | Requirement | Recommended combination | Detailed guidance | @@ -85,9 +87,9 @@ Start with [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pi | Verify image signatures | Configure Chains keys and registry access centrally; verify the image by digest before promotion and at deployment admission. | [Signing Key Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | | Hand off an artifact event | Map a Harbor event to the next pipeline, carrying the digest. Check scan and signing evidence separately. | [Use Harbor Event Triggers](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | -Promotion copies a selected image; it does not establish that the image passed CI. The approver must match the digest with a successful trusted build and the test environment. Signatures establish the configured signer's statement; their meaning also depends on control over the signing key and the definitions allowed to run. +Promotion copies a selected image; it does not establish that the image passed CI. The approver must match the digest with a successful trusted build and the test environment. For the sample, a trusted CI result is the `ci` PipelineRun for a `push` event on `main`; a successful `/test ci branch:` run from another branch is not equivalent. Signatures establish the configured signer's statement, not which pipeline produced the image; their meaning also depends on control over the signing key and the definitions allowed to run. -The samples deliberately keep test CD, promotion, and production CD as manually started runs. Their inputs come from the preceding run's digest result. Harbor-driven automation is a separate integration step. The release tutorial's first deployment is performed by an administrator; subsequent releases follow the configured delivery path. +The samples deliberately keep test CD, promotion, and production CD as manually started runs. Their inputs come from the preceding run's digest result. Harbor-driven automation is a separate integration step. The first production deployment waits for a promoted image; after that image is promoted, follow the same reviewed production GitOps path for later releases. ## Prepare a shared platform @@ -95,17 +97,19 @@ Platform settings are administrator decisions. Roll them out in stages and check | Platform concern | Recommended approach | | --- | --- | -| Tool access | Use project-scoped Connectors. Proxy access uses short-lived workload tokens. Manage image pull Secrets, PaC clone credentials, and Chains registry credentials as separate long-lived credential responsibilities. | +| Tool access | Use project-scoped Connectors. Proxy access uses short-lived workload tokens. Manage image pull Secrets, PaC clone credentials, and Chains registry credentials as separate long-lived credential responsibilities. A project owner who can create Pods in a credential-bearing namespace can use the credentials in that namespace. | | Project onboarding | Orchestrate the four Connectors automatic creation Tasks, then seed a project GitOps repository and propose platform registration through a merge request. Complete the administrator prerequisites and acceptance checks. | | Platform configuration | Keep component settings, policies, encrypted administrator credentials, and project declarations in a platform repository. Assign one owner to each resource or field. | | Namespace isolation | Use separate project namespaces and tool-side tenants. Separate build, delivery, and workload namespaces and restrict access to administrator namespaces. | | Execution policy | Use platform ClusterPolicies and project Policies. Restrict direct TaskRun and CustomRun creation. Optionally require approved resolver types and coordinates. Move from Audit to Enforce after checking intended workflows. | -| Cleanup | Select one cleanup mechanism. For event-based Pruner, review global defaults and namespace override settings before enabling deletion. | +| Cleanup | Select one cleanup mechanism. For event-based Pruner, review global defaults and namespace override settings before enabling deletion. A global first sync can affect every namespace, with successful and failed history counted separately; inventory tenants and notify owners before changing the defaults. | | Audit history | Store run records and logs in external Results storage with an explicit retention period. Export approval evidence separately before deleting its objects. | -| Scheduled work | Use ScheduledTrigger or CronJob for reconciliation and rotation, with failure alerts and no overlapping runs for a tenant. | +| Scheduled work | Use ScheduledTrigger or CronJob for reconciliation and rotation, with failure alerts and no overlapping runs for a tenant. The sample has no distributed lock, so its operators enforce that rule. | Results Watcher scope deserves a separate rollout decision. Its default watch scope can affect finished runs across the cluster and add `results.tekton.dev/*` finalizers. Scope it before enabling it. Disabling archiving does not remove existing finalizers; follow the product recovery procedure before cleanup. After Pruner removes cluster objects, the external Results store is the audit source. +The platform repository is the durable, reviewable source for configuration and project boundaries. It makes changes auditable and repeatable, but it is not a backup for sealed-secrets keys, signing keys, external tool data, or the Results store. Keep those dependencies in a separate recovery plan. Reverting a registration commit can prune an ACP project and its same-named namespace, so treat that action as decommissioning rather than an ordinary rollback. + The [platform article](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) explains the 18 platform decisions, repository layout, ownership boundaries, onboarding workflow, rotation, and acceptance checks. ## Operate without internet access diff --git a/docs/en/solutions/From_Commit_to_Production_on_ACP.md b/docs/en/solutions/From_Commit_to_Production_on_ACP.md index 07e14e44c..2cc6d22df 100644 --- a/docs/en/solutions/From_Commit_to_Production_on_ACP.md +++ b/docs/en/solutions/From_Commit_to_Production_on_ACP.md @@ -3,7 +3,7 @@ products: - Alauda Container Platform - Alauda DevOps kind: - - Article + - Best Practices ProductsVersion: - 4.x --- @@ -14,7 +14,7 @@ Use this article to choose where pipeline definitions live, how deployments cons This is the delivery part of [Alauda DevOps Best Practices](./Alauda_DevOps_Best_Practices_Overview.md). Its configuration baseline is Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and GitOps 4.4. Project namespaces, tool-side resources, and Connectors must already exist; the [platform article](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) explains their preparation. -The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) use a fictional `shop` project. CI runs in `shop-build`; CD and promotion run in `shop-cd`; workloads run in `shop-test` and `shop-prod`. Replace every example address, tool image, group, credential placeholder, and policy identity for your environment. Make the vulnerability database, tool images, base images, and dependencies available locally for disconnected execution. +The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) use a fictional `shop` project. CI runs in `shop-build`; CD and promotion run in `shop-cd`; workloads run in `shop-test` and `shop-prod`. The sample binds the project owner group in `shop` only; applying the PaC Repository, CD and promotion objects, and starting their runs requires a platform administrator or a separately delegated grant in `shop-build` and `shop-cd`. See the sample application's ["Who applies what"](https://github.com/AlaudaDevops/devops-best-practices-samples/blob/main/sample-app/README.md#who-applies-what) table. Replace every example address, tool image, group, credential placeholder, and policy identity for your environment. Make the vulnerability database, tool images, base images, and dependencies available locally for disconnected execution. ## Follow one image identity through the workflow @@ -44,7 +44,9 @@ When a vulnerability is found in an already deployed image, fix it and use the s Start with a cluster Pipeline when that is sufficient. Add PaC when the change history and merge-request experience matter. Use central templates when projects should reuse a standard workflow. The same project can use all three for different jobs. Choose one route, PaC or Triggers, for a given event. -For retries from a merge request, use `/test `. Keep the complete syntax in one place by linking to [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). +For a retry from a merge request, comment `/test branch:` on the merge request. For a push-triggered run, comment the same form on the commit at the head of that branch. Keep the complete syntax in one place by linking to [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). + +A failed check blocks a merge only when the Git platform requires that check to pass. On GitLab, enable **Pipelines must succeed**. If Auto DevOps creates an additional pipeline and no runner picks it up, the GitLab pipeline can remain running and the merge request continues to show a running pipeline; disable Auto DevOps for these repositories or provide a runner. ### Combine PaC with a central template @@ -82,7 +84,7 @@ The sample separates the two outputs: | `sample-app/.tekton/ci.yaml` | Push to `main` | `registry.example.com/shop/hello` | Source for test CD and promotion | | `sample-app/.tekton/ci-merge-request.yaml` | Merge request | `registry.example.com/shop/hello-mr` | Temporary merge-request artifacts | -The sample's Harbor Connector has project-level access. Separating repository names does not itself enforce a credential boundary. For stronger separation, use a distinct temporary registry project and a credential that cannot write release-candidate images. Before delivery, check that the selected digest is the `IMAGE_DIGEST` of a successful trusted `main` CI run. +The sample's Harbor Connector has project-level access. Separating repository names does not itself enforce a credential boundary. For stronger separation, use a distinct temporary registry project and a credential that cannot write release-candidate images. Before delivery, check that the selected digest is the `IMAGE_DIGEST` of a successful trusted `main` CI run whose event type is `push`; a comment-triggered run on another branch is not a release candidate. Both sample CI definitions bind the Connector proxy CA through the `ca-bundle` Workspace. The proxy's CA is required even when the upstream registry uses a publicly trusted certificate. For offline scanning, set `trivyExtraArgs` to use the platform's mirrored vulnerability database, for example `--db-repository registry.example.com/mirror/trivy-db:2`. The scanner must be able to pull that image. @@ -109,7 +111,7 @@ gitops/ └── prod/kustomization.yaml ``` -In the sample, test CD uses `submitMode: direct`; production CD uses `submitMode: pr`. The production overlay is seeded with the production registry repository, so a release changes only its digest. The first production deployment waits for a promoted image. +In the sample, test CD uses `submitMode: direct` and `kustomize edit set image`, which may rewrite the test overlay. Production CD uses `submitMode: pr` and changes only the `digest:` line with `sed`; the production overlay is seeded with the production registry repository. The first production deployment waits for a promoted image. Enforce production Git review in the code platform. Use protected branches and, where supported, code-owner approval. Other options include a separately protected production branch or repository, or a reviewed server-side policy. Branch protection alone does not create directory-level protection. Remove direct production write access that would bypass the chosen review path. @@ -117,11 +119,11 @@ Enforce production Git review in the code platform. Use protected branches and, `gitops-repo-update` is a Pipeline and runs as a separate PipelineRun. Pass the selected digest as an argument to its update script. Use a fresh source Workspace volume per run and an image that contains every command used by the script. `submitMode: pr` also requires the configured GitLab CLI image and a token with API access. -The sample updates the `digest:` line with `sed` and verifies the rendered image with Kustomize. This preserves a one-line production diff. Avoid using a rewrite operation that changes unrelated formatting or image fields. The GitOps writer token needs the scopes required for cloning, pushing, and creating the merge request; keep it separate from the application CI token. +The sample updates the production `digest:` line with `sed` and verifies the rendered image with Kustomize. This preserves a one-line production diff. Avoid using a rewrite operation that changes unrelated formatting or image fields in production. The GitOps writer is a group access token with Maintainer access in the sample; that identity can change branch protection and code-owner settings, so use a project-scoped least-privilege token or a separately protected production repository in a hardened deployment. Keep it separate from the application CI token. -The sample starts test CD, promotion, and production CD manually from rendered PipelineRun YAML. Replace the template parameters, verify the image mapping, identity, and Workspaces, and submit with `kubectl create`. Use [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html) for other execution entries. Harbor event automation is an additional integration, not part of the sample's manual path. +The sample starts test CD, promotion, and production CD manually from rendered PipelineRun YAML. Replace the template parameters, verify the image mapping, identity, and Workspaces, and submit with `kubectl create`. A console Run Template form exposes Pipeline parameters: in the sample, remove the empty third `updateArgs` entry, use the full `repository@sha256:digest` for test CD, and use the bare `sha256:digest` for production CD because the production repository is fixed by the seeded overlay. Restrict who may create PipelineRuns in the delivery namespace; a console form can expose parameters that are fixed in the rendered YAML path. Use [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html) for other execution entries. Harbor event automation is an additional integration, not part of the sample's manual path. -Production rollback follows the same review boundary: select a retained, previously promoted digest, verify its signature and application compatibility, and review the GitOps change. Database and data migrations need an application-specific rollback plan. Establish and rehearse the complete rollback runbook in the target environment. Configure [Harbor tag retention](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-retention-rules/) and [tag immutability](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-immutability-rules/) so that required rollback images remain available. +Production rollback follows the same review boundary: select a retained, previously promoted digest, verify its signature and application compatibility, and review the GitOps change. A tested GitOps rollback consists of: find the previous digest in the production Git history; confirm it remains in the production registry and is signed; create a branch and change only the `digest:` line; open a merge request and wait for validation before review and merge; verify the image in Argo CD and the Deployment and wait for rollout; and return to the newer version through the same one-line change if needed. Database and data migrations need an application-specific rollback plan. Establish and rehearse the complete rollback runbook in the target environment. Configure [Harbor tag retention](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-retention-rules/) and [tag immutability](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-immutability-rules/) so that required rollback images remain available. ## Select the release controls @@ -129,12 +131,12 @@ Promotion, approval, credential gating, and image admission solve separate decis | Decision | Recommended control | | --- | --- | -| Should production use a separate artifact project? | Copy the selected digest into that project, then update the production Git reference. | +| Should production use a separate artifact project? | Copy the selected digest into that project, then update the production Git reference. If test and production consume the same image repository, promote only the Git reference and admit only digests approved for production, because a repository-based rule cannot distinguish the environments. | | Who authorizes the selected release? | An ApprovalTask, a production Git merge-request review, or both, according to policy. | | Who may use the production Connector? | A dedicated release identity plus admission and RBAC restrictions. Add approval-gated proxy access when credential owners and pipeline owners are separate. | | What may be deployed? | Admission policies for allowed production repositories, signatures, and any required approved-digest evidence. | -Use the Core promotion admission rules in [Constrain Artifact Promotion with Kyverno](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/constrain_artifact_promotion_with_kyverno.html). Rules 1–3 constrain the promotion execution path. Add rule 4 when the promotion Pipeline is maintained in a namespace, to control changes to that definition. The companion samples do not install these promotion rules or a production-repository-only deployment policy; platform administrators must add them before using the reference design as a production control. +Use the Core promotion admission rules in [Constrain Artifact Promotion with Kyverno](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/constrain_artifact_promotion_with_kyverno.html). Rules 1–3 constrain the promotion execution path. Add rule 4 when the promotion Pipeline is maintained in a namespace, to control changes to that definition. The companion samples do not install these promotion rules or a production-repository-only deployment policy; platform administrators must add them before using the reference design as a production control. A signed image proves possession of the signing key, not that this particular pipeline produced it. Promotion validates the source and destination mapping and copies bytes. It does not query CI to establish that the image passed. The approver must correlate the digest with the trusted build, scan, and test evidence. The sample validates that the image mapping is first, the source uses a SHA-256 digest, and the repositories and release-label format match the configured workflow. Optional signature mappings must refer to the same image digest. @@ -166,26 +168,26 @@ Customer change-management system integration is outside this series. The availa ### Gate the production Connector after approval -Keep the production Connector and its Secret in a namespace where project members cannot create Pods or run arbitrary pipelines. A Pod can mount a Secret from its own namespace without a separate Secret-read authorization check. Consequently, permission to create Pods or PipelineRuns in a credential-bearing namespace is a credential boundary. +Keep the production Connector and its Secret in a namespace where project members cannot create Pods or run arbitrary pipelines. A Pod can mount a Secret from its own namespace without a separate Secret-read authorization check. Consequently, permission to create Pods or PipelineRuns in a credential-bearing namespace is a credential boundary. With `enable-connectors-approval` disabled, a principal that can create a Pod or PipelineRun in `shop-cd` can use the production Connector granted to the release identity; an ApprovalTask does not protect that path. Treat the namespace and its ServiceAccounts as the boundary until proxy approval is enabled and verified. -The sample release ServiceAccount `shop-release` reads the source Connector and mounts the production Connector from `platform-connectors`. Its standing grants are in `sample-app/promotion/standing-access.yaml`. Use RBAC for the cross-namespace proxy grants described by the product guide. +The sample release ServiceAccount `shop-release` reads the source Connector and mounts the production Connector from `platform-connectors`. Its standing grants are in `sample-app/promotion/standing-access.yaml`. In the sample, the copy Task authenticates both ends with the production Connector's credentials; the source Connector mount supplies its CA certificate, so the production robot account must also be able to read the source repository. Use RBAC for the cross-namespace proxy grants described by the product guide. Approval gating uses two installation-wide feature flags. Roll them out in this order: 1. Inventory every identity mounting a Connector that is not behind approval. Prepare its read authorization and `connectors/proxy` permission with verb `*`. A `get` grant on `connectors/proxy` is insufficient. Do not grant direct proxy access to the production Connector that approval must gate. 2. Enable `enable-connectors-approval`. Immediately create the required AccessPolicies, including the administrator-Connector grants and the production approval policy. Do not start affected runs in the interval between enabling the flag and creating the policies. -3. In the production AccessPolicy, configure the designated `approvers` and keep them aligned with the ApprovalTask and promotion admission policy. Remove any other matching default or direct proxy grants. +3. In the production AccessPolicy, configure the designated `approvers` and keep them aligned with the ApprovalTask and promotion admission policy. The `approvers` field in a project declaration is an audited record only; it does not configure either object. Remove any other matching default or direct proxy grants. 4. Enable `enable-connector-proxy-permissions`, which applies the proxy permission check to requests as well. Set feature flags through `ConnectorsConfig`. For the full configuration and verification path, see [Enable Proxy Permission Checks and Approval Gating](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/how_to/enable_proxy_permissions_and_approval.html), [AccessPolicy](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/access_policy.html), and [Skopeo Copy connector approval](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/promote_artifacts_with_skopeo_copy.html#connector-approval). -`defaultPermission` policies have authorization prerequisites and are not a substitute for every cross-namespace RBAC grant. A Ready policy alone does not establish a working mount. Verify the actual identity, Pod events, mounted configuration, and AccessRequest result. +`defaultPermission` policies have authorization prerequisites and are not a substitute for every cross-namespace RBAC grant. A name-scoped `get` Role does not by itself satisfy a policy that requires namespace-wide reads. A `defaultPermission` policy applies only to identities with namespace-wide `get`; other identities can be silently skipped. The sample grants the onboarding identity namespace-wide `get` in the management namespace, while keeping proxy permissions limited to the named administrator Connectors; keep that broader read scoped to the management namespace. A Ready policy alone does not establish a working mount. Verify the actual identity, Pod events, mounted configuration, and AccessRequest result. ## Sign and verify the image Configure Chains signing keys centrally and distribute only the public verification key to projects. Protect both the signing Secret and the ability to create Pods in its namespace. Application teams do not need access to the signing Secret. -The sample stores signatures and attestations beside the image. Chains first uses matching registry credentials from the run's ServiceAccount. Those credentials must be able to push to the image's registry project. Where they are absent, configure a centrally managed registry credential on the Chains controller through `TektonConfig`. Chains does not use a Connector's CSI proxy Workspace as its registry credential. +The sample stores signatures and attestations beside the image. Chains first uses matching registry credentials from the run's ServiceAccount. In the sample, the image-pull Secret also has push permission and is used by Chains, so it is not a read-only credential; separate a read-only workload Secret from a signing credential in a hardened deployment. Those credentials must be able to push to the image's registry project. Where they are absent, configure a centrally managed registry credential on the Chains controller through `TektonConfig`. Chains does not use a Connector's CSI proxy Workspace as its registry credential. The central credential is created by an administrator, outside Git. Limit it to the registry projects that Chains must read and sign. When onboarding a project, the platform team must add its source and production registry projects to that credential's access. Rotate it and restrict its Secret and namespace. Apply controller configuration changes in a maintenance window, preserving other container options already configured. @@ -197,11 +199,13 @@ cosign verify --key cosign.pub "$IMAGE" cosign verify-attestation --key cosign.pub --type slsaprovenance "$IMAGE" ``` +The `--type` value must match the provenance predicate emitted by Chains. Keep the Chains format and the verification command aligned (the sample expects SLSA provenance v0.2); a different format requires the corresponding cosign type. + For an offline installation configured without a transparency log, use the product's offline verification settings, including `--insecure-ignore-tlog=true` where required. This skips the external transparency-log check; retain the signature and attestation checks. Authenticate to private registries without committing the login credentials. -If the platform chooses a separate signature repository, configure `storage.oci.repository` centrally and use the corresponding `COSIGN_REPOSITORY` in verification and the repository setting in Kyverno. Review storage and overwrite behavior before combining build and promotion attestations for the same digest. Copying build signature attachments during promotion is optional; use the documented `imageMappings` and `multiArch: all` configuration if selected. +If the platform chooses a separate signature repository, configure `storage.oci.repository` centrally and use the corresponding `COSIGN_REPOSITORY` in verification and the repository setting in Kyverno. Review storage and overwrite behavior before combining build and promotion attestations for the same digest. Copying build signature attachments during promotion is optional; the sample uses explicit `.sig` and `.att` mappings. For multi-architecture images, follow the Skopeo Copy Task contract and set `multiArch: all` when copying the index and its attachments; do not assume the single-image mappings cover an index automatically. -At deployment admission, start the image-verification policy in Audit, with the sample's `mutateDigest: false` setting, and review the matching images and reports before Enforce. The sample's production image list is explicit: add each newly onboarded production repository to the policy. See [Image Signature Verification](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/how_to/image_signature_verification.html). +At deployment admission, start the image-verification policy in Audit, with the sample's `mutateDigest: false` setting, and review the matching images and reports before Enforce. Kyverno also needs read credentials for private production registry projects through `imageRegistryCredentials.secrets`; otherwise a valid signature can be unreadable and Enforce will reject every production image. The sample's production image list is explicit: add each newly onboarded production repository to the policy. See [Image Signature Verification](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/how_to/image_signature_verification.html). ## Check the controls at their actual enforcement points @@ -217,9 +221,11 @@ At deployment admission, start the image-verification policy in Audit, with the Use platform ClusterPolicies for the baseline and project Policies for additional restrictions. Restrict PolicyException creation or disable exceptions according to platform policy. A resolver-type allow-list alone does not pin catalog, repository, namespace, version, or definition content. The inline validation exception must be handled explicitly when enabling a no-inline policy. +The sample's resolver and bare-run policies exclude `platform-connectors`, where onboarding uses administrator Connectors. Keep Pod and PipelineRun creation in that namespace limited to platform administrators and the platform ServiceAccount; the policy exclusions do not create that boundary. + ## Accept the delivery workflow -Run `acceptance/01-commit-to-production.sh` from the sample only in an isolated acceptance project after replacing its inputs. The checks are an execution procedure, not a claim that an arbitrary environment is ready. +Run `acceptance/01-commit-to-production.sh` from the sample only in an isolated acceptance project with a separate project, namespaces, and Harbor production repository after replacing its inputs. The script defaults to sample names; set every input explicitly and never point it at an ordinary production tenant. The checks are an execution procedure, not a claim that an arbitrary environment is ready. | Check | Required evidence | | --- | --- | @@ -228,7 +234,7 @@ Run `acceptance/01-commit-to-production.sh` from the sample only in an isolated | Promotion copied that image | The promotion result reports the expected destination repository and digest. | | Production uses the promotion result | The GitOps merge request is merged, the rendered image matches the destination digest, and the workload rollout completed. `Synced` alone can describe an older revision. | | Signature and provenance exist | Both cosign commands succeed against the production image. A signing annotation alone is insufficient. | -| Admission enforces the selected policy | Use a restricted-compatible probe in the acceptance project and distinguish policy rejection from Pod Security or image-pull failure. Clean up the probe afterward. | +| Admission reports or enforces the selected policy | In Audit, verify a known signed production digest as a positive control and review the report for an unsigned image; Audit reports are evidence, not enforcement. In Enforce, verify the signed positive control first, then distinguish rejection of an unsigned image from Pod Security or image-pull failure. Clean up both probes afterward. | | Approval is attributable | Save the ApprovalTask response, approver group members, selected digest, and related run identifiers in the release record. | Before production, address the sample's explicit hardening work: restrict the GitOps writer's namespace, protect production Git changes, install the promotion Core rules, constrain production image sources, and limit direct cluster writes. Do not grant project members Pod creation in namespaces that contain administrator or release credentials. Establish local rollback and policy-service recovery runbooks. diff --git a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md index a45b1c3cd..6b5637525 100644 --- a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md +++ b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md @@ -3,14 +3,14 @@ products: - Alauda Container Platform - Alauda DevOps kind: - - Article + - Best Practices ProductsVersion: - 4.x --- # Managing Platform Configuration and Onboarding Isolated Projects with Argo CD and Tekton on ACP -Use a platform Git repository to review and preserve shared configuration. Use an onboarding pipeline to create a project's namespaces and tool-side resources, then propose its deployment configuration through a merge request. Argo CD reconciles the objects owned by Git; the onboarding tasks reconcile the resources they create in external tools. +Use a platform Git repository to review and preserve shared configuration. This gives the platform team a durable, auditable, repeatable source, but it does not replace backups of sealed-secrets keys, signing keys, external tool data, or Results storage. Use an onboarding pipeline to create a project's namespaces and tool-side resources, then propose its deployment configuration through a merge request. Argo CD reconciles the objects owned by Git; the onboarding tasks reconcile the resources they create in external tools. This article is the platform part of [Alauda DevOps Best Practices](./Alauda_DevOps_Best_Practices_Overview.md). It assumes one ACP project and separate tool-side tenants per project. Its configuration baseline is Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and GitOps 4.4 on a compatible ACP installation. Teams sharing one namespace and one tool tenant need a different boundary model. @@ -64,7 +64,9 @@ Register the platform repository with Argo CD and provide read access to each pr Create the ACP project through the platform before onboarding it. The pipeline exports that project's manifest for subsequent registration. Importing an existing project into Git needs the same review as creating one: the project Application can later prune that resource. -Platform administrators are responsible for webhook setup and event-delivery verification for platform, project GitOps, and application repositories. Registration with a token authorized to manage repository webhooks supports controller-managed webhook creation and deletion. For centralized management, follow [Create a Git Secret for PAC](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/create_git_secret.html). The sample's Developer-level GitOps token cannot manage webhooks; complete the administrator webhook configuration and test a real event before accepting onboarding. +Onboarding creates the GitLab group and its service tokens; it does not decide which human users belong to the group or grant them GitLab project-creation rights. The platform or project administrators must add approved members and manage those memberships separately from the token used by Pipelines-as-Code. + +Platform administrators are responsible for webhook setup and event-delivery verification for platform, project GitOps, and application repositories. When automatic registration is enabled and the PaC controller URL is configured, creating a `Repository` can cause the registration controller to create and delete the project webhook. This requires a centralized webhook-registration GitLab credential with `api` scope and Maintainer access to the projects, stored in the controller's namespace; the PaC Repository's Developer-level group token cannot manage webhooks. For setup and verification, follow [Create a Git Secret for PAC](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/create_git_secret.html) and test a real event before accepting onboarding. Keep Harbor projects private. Configure Nexus anonymous access and SonarQube permissions according to the project-isolation requirements in their Task guides. Do not infer isolation merely from the presence of separate credentials. @@ -77,13 +79,13 @@ The table identifies the platform decision and its repository layer. Values and | # | Decision | Why it is shared | Layer and guidance | | --- | --- | --- | --- | | 1 | Separate Repository access tokens from webhook administration; use reviewed default-branch definitions and an explicit GitLab provider URL. | Credential scope and pre-merge execution trust should be consistent. | Credentials and project: [PaC Repository configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html). | -| 2 | Standardize merge-request retries with `/test `. | Teams use one supported entry. | Project template: [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). | +| 2 | Standardize merge-request retries with `/test branch:`. | Teams use one supported entry and make the source branch explicit. | Project template: [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). | | 3 | Separate platform-provided Repository parameters from application inputs. | Addresses and Connector names have a defined owner. | Project: [Advanced Repository Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html). | | 4 | Decide Harbor event handoffs. | Event endpoints and validation need consistent ownership. | Component: [Harbor events](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | | 5 | Schedule reconciliation and rotation with failure alerts. | The platform owns the interval and operational response. | Component: [ScheduledTrigger](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/scheduledtriggers.html). | | 6 | Standardize EventListener exposure. | TLS, routes, and webhook validation belong to the platform. | Component: [Automatic exposure rules](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/trigger-wrapper-config.html). | | 7 | Orchestrate tool-side tenant creation with the four product Tasks. | Administrators retain tool-wide credentials. | Credentials and project: tool Task guides below. | -| 8 | Define long-lived credential rotation for each tool. | Expiry and renewal behavior differ by tool. | Component and credentials: [Credential rotation](https://docs.alauda.io/alauda-devops-connectors/1.14/how_to/automatically_create_connectors_for_a_project.html#credential-rotation). | +| 8 | Define long-lived credential rotation for each tool. | Expiry and renewal behavior differ by tool. | Component and credentials: [GitLab automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-gitlab/how_to/using_gitlab_connector_automatic_creation_task.html#step-by-step--rotate-in-place), [Harbor automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-harbor/how_to/using_harbor_connector_automatic_creation_task.html#example-re-run-the-task-to-refresh-credentials-and-reconcile-resources), [Nexus automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-nexus/how_to/using_nexus_connector_automatic_creation_task.html#re-run-to-add-repositories-or-rotate-credentials), and [SonarQube automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-sonarqube/how_to/using_sonarqube_connector_automatic_creation_task.html#rotation). | | 9 | Configure Chains signing keys and image verification. | The platform owns the signer and distributes its public key. | Component, credentials, policy: [Signing keys](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | | 10 | Use Connector proxy access while tracking long-lived credential exceptions. | Access boundaries and rotation need a central owner. | Credentials: [Connectors Proxy](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_proxy.html) and [CSI Driver](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_csi.html). | | 11 | Apply namespace Pod Security and separate build namespaces. | Build tooling and workloads have different privilege needs. | Policy and project: [Restricted standard](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/comply_with_pod_security_restricted.html). | @@ -95,6 +97,42 @@ The table identifies the platform decision and its repository layer. Values and | 17 | Define Results storage, watch scope, authentication, and retention. | Audit records need storage independent of live objects. | Component: [Results retention](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/retention_policy_configuration.html). | | 18 | Enable metrics and operational monitoring. | Operators need evidence of controller and execution health. | Component: [Metrics Collection](https://docs.alauda.io/alauda-devops-pipelines/4.15/configure/metrics_collection.html). | +The baselines matter at different points in the operating model: + +| Baseline | When this matters | +| --- | --- | +| Repository and webhook ownership | Before onboarding the first repository or changing GitLab providers. | +| Retry and event conventions | When teams need reproducible merge-request or push-triggered reruns. | +| Parameter ownership | When a platform-managed address or Connector must change without editing every application. | +| Harbor event handoff | When an external registry event starts another pipeline. | +| Scheduled reconciliation | When credentials or declarations need periodic repair. | +| EventListener exposure | Before exposing webhook endpoints outside the cluster. | +| Automatic tenant creation | When a new project must receive the same tool-side resources. | +| Credential rotation | Before the first token reaches its expiry window. | +| Chains and image verification | Before a private production registry is enforced by Kyverno. | +| Connector proxy access | When workload access to long-lived credentials must be narrowed. | +| Pod Security and build separation | Before build tools and workloads share a cluster. | +| Approval and promotion rules | Before production credentials or artifacts can be released. | +| Policy layers | When projects need additional restrictions without weakening platform controls. | +| Offline catalog governance | When the cluster cannot fetch Tasks or Pipelines from the internet. | +| Versioned central Pipelines | When multiple projects consume the same workflow. | +| Pruner defaults | Before cleanup can remove shared run history. | +| Results storage and retention | Before live Tekton objects are pruned. | +| Metrics and monitoring | Before operating the platform without controller health evidence. | + +The following matrix describes the sample's intended six-role boundary. It is a reference target, not a claim that every production safeguard is already enabled: + +| Role | Platform repo | Project declaration | Project namespace | Delivery namespace | GitOps `envs/prod` | Production | Connectors | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform administrator | Write | Write | Full | Full | Configure branch protection and CODEOWNERS | Full | Administrator | +| Application developer | Read | Read | Project-scoped write | None by default | Open merge requests | None directly | Project-scoped through CI | +| Release approver | None | None | None | Approve ApprovalTask | Approve code-owner review | None | None | +| CI identity | None | None | Build only | None | None | None | Project tool Connectors | +| Delivery identity | None | None | None | Run CD | Open merge requests | Update references through GitOps | GitOps writer Connector | +| Promotion identity | None | None | None | Run promotion | None | Copy to production registry | Read source and write production Connector | + +In the sample, production branch protection and code-owner approval are still a platform setup task, the developer has no default delivery-namespace binding, and console approval requires the optional ACP role binding. Configure and verify those controls before treating the matrix as enforced. + ### Separate policy scope from definition trust Use platform ClusterPolicies for shared controls and namespaced Policies for project-specific restrictions. Restrict direct TaskRun and CustomRun creation by the actual submitting identity; client-supplied owner references alone are not an authorization boundary. Enumerate controller identities and review their permissions. @@ -132,16 +170,16 @@ spec: This does not constrain a resolver's URL, catalog, namespace, version, definition content, or runtime credential scope. It also rejects inline `pipelineSpec` when enforced. Convert the sample's inline validation flow to a catalog Pipeline before enabling that restriction for project repositories. Restrict PolicyException creation or disable exceptions according to platform policy. -The sample's `argocd-project-namespace-boundary` policy constrains writes of Role, RoleBinding, ResourceQuota, LimitRange, and NetworkPolicy by the configured Argo CD identities in labelled project namespaces. It allows the platform-owned registration path and requires `allowExistingViolations: false`. Adapt both Argo CD identities for the target cluster and verify the intended allow and deny cases before relying on the boundary. Install and verify the policy on every destination cluster to which the boundary should apply. +The sample's `argocd-project-namespace-boundary` policy constrains writes of Role, RoleBinding, ResourceQuota, LimitRange, and NetworkPolicy by the configured Argo CD identities in labelled project namespaces. It allows the platform-owned registration path and requires `allowExistingViolations: false`. Adapt both Argo CD identities for the target cluster and verify the intended allow and deny cases before relying on the boundary. The exception for the platform `projects` Application depends on Argo CD's `argocd.argoproj.io/tracking-id` annotation; label-only resource tracking is not covered. The policy does not cover Kyverno `PolicyException` or Connectors `AccessPolicy` resources; keep those kinds out of project Application allow-lists or add explicit deny rules after verifying the target platform's exception and namespace settings. Install and verify the policy on every destination cluster to which the boundary should apply. -Kyverno availability and recovery are platform operating concerns. Prepare a local runbook; this series does not prescribe the cluster's write behavior during a policy-service outage. See [Pipeline Policy Constraints with Tekton and Kyverno](./Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md) for the policy design. +Kyverno availability and recovery are platform operating concerns. For policies used as admission gates, keep the resource webhooks registered with `failurePolicy: Fail`; when Kyverno is unreachable, matching writes are then rejected. Operate Kyverno highly available and keep a controlled recovery procedure, including the conditions for temporarily removing and later restoring webhook configurations. A signing result proves the signing key, not the pipeline that produced an image. Restrict ClusterPolicy, resolver, webhook, and PolicyException changes to platform administrators. See [Pipeline Policy Constraints with Tekton and Kyverno](./Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md) for the policy design. ## Organize and synchronize the platform repository -The sample is split into four ownership layers and the onboarding installation: +Use `platform-repo/` as the tree root below; the repository itself can have any name. The sample is split into four ownership layers and the onboarding installation: ```text -platform-config/ +platform-repo/ ├── argocd/apps/ # Root and child Application definitions ├── components/ # TektonConfig and component options ├── policies/ # Platform admission policies @@ -152,9 +190,9 @@ platform-config/ └── .tekton/ # Validation and changed-project dispatch ``` -Copy `onboarding/` from the sample to the platform repository root as described by its README. Register the repository read-only with Argo CD and use one child Application per layer. The root Application manages those Application definitions. +Copy `onboarding/` from the sample to the platform repository root as described by its README. Register the repository read-only with Argo CD and use one child Application per layer. The root Application manages those Application definitions. On the first bootstrap, apply `argocd/apps/` once with `kubectl apply -k`; declarations already present in `projects/` are not picked up by the merge dispatcher until a later `main` change, so run the onboarding Run Template once for each existing declaration after the Applications are synchronized. -For partial singleton component configuration, use server-side apply so the platform repository owns only the specified fields. The samples also set `argocd.argoproj.io/compare-options: ServerSideDiff=true`. Review the generated diff and the operator's field ownership before synchronization. The component layer uses `prune: false` so that removing a fragment does not delete the singleton TektonConfig. Establish the component layer's automated synchronization behavior in the target environment before enabling ongoing self-healing. +For partial singleton component configuration, use server-side apply so the platform repository owns only the specified fields. The samples also set `argocd.argoproj.io/compare-options: ServerSideDiff=true`. Review the generated diff and the operator's field ownership before synchronization. The component layer uses `prune: false` so that removing a fragment does not delete the singleton TektonConfig. The sample currently enables `selfHeal` for this layer; if the operator's write-back behavior has not been verified, start with manual sync or `selfHeal: false` and enable it only after observing the effective ownership. Establish the component layer's automated synchronization behavior in the target environment before enabling ongoing self-healing. The credential layer does not enable self-healing: seal credentials for the target controller and plan how rotations update live Secrets and encrypted source. Other layers can use prune and self-heal according to their ownership. Namespaces are created by onboarding; do not add a second namespace creator with `CreateNamespace`. @@ -166,7 +204,11 @@ Review deletion carefully. Pruning a registered ACP project can delete its same- Both the platform and project GitOps templates include `.tekton/validate.yaml`, which composes catalog `git-clone` and `run-script` Tasks. Replace its tool image with a locally available image containing the required shell and Kustomize commands. Establish its webhook before the first merge-request run. With default-branch definition provenance, a change to the validation definition is first exercised after it has entered that branch. -Validate every directory owned by an Application, and add new directories to the validation paths. Re-run a merge-request check with `/test validate`. YAML parsing and Kustomize rendering do not validate CRD admission behavior; use the target schemas and an isolated project for that acceptance step. +Validate every directory owned by an Application, and add new directories to the validation paths. Re-run a merge-request check with `/test validate branch:`. YAML parsing and Kustomize rendering do not validate CRD admission behavior; use the target schemas and an isolated project for that acceptance step. + +A validation status blocks a merge only when the Git platform requires it. On GitLab, enable **Pipelines must succeed** for the platform and project GitOps repositories. If Auto DevOps starts an additional pipeline without an available runner, the merge request can remain in a running state; disable Auto DevOps for these repositories or provide a runner. With `pipelinerun_provenance: default_branch`, the definition is reviewed, but the merge-request content is still rendered by the validation run. Keep that dispatcher and its credentials isolated from administrator and production credentials. + +The sample onboarding dispatcher compares the pushed revision with its first parent. Use a merge or squash workflow that keeps all declaration changes in that comparison; a fast-forward push containing several independent commits can cause an earlier declaration change to be missed. The sample's `on-path-change` annotation is a Technology Preview feature in the 4.15 line, so verify its maturity and behavior before relying on it for the trigger boundary. ## Declare a project and run onboarding @@ -205,6 +247,10 @@ sonarqube: `owners` contains ACP group resource names without a `group:` prefix; approval entries use `group:`. A console display name is not the resource name. Group membership changes belong to group administration, rather than edits naming individuals in every declaration. +`imagePullFrom` is reserved: this sample only prints a warning when it is set and grants no cross-project pull access. + +The declaration's `approvers` field is a reviewed record only: onboarding validates its shape but does not apply or compare it. The approvers that take effect are the promotion Pipeline's `ApprovalTask` list and the production `AccessPolicy`; nothing keeps those three sources in sync, so update them together in one reviewed change. Keep the production `AccessPolicy` under platform-administrator control. + Submit the declaration, project-specific onboarding permissions, and schedule through a platform merge request. Confirm that the declaration and onboarding Applications have synchronized; synchronize them manually when using a manual sync policy. Then examine the dispatcher result and every child onboarding run. A successful merge or dispatcher does not by itself establish complete project readiness. The sample dispatches changed declarations after a merge. A path filter can also admit a run with no relevant declaration changes; the dispatcher is responsible for selecting actual declarations. A deleted declaration is skipped, rather than interpreted as an instruction to delete a project. @@ -214,14 +260,16 @@ The sample dispatches changed declarations after a merge. A path filter can also | Stage | Work | Owner or mechanism | | --- | --- | --- | | 1 | Export the existing ACP project without runtime metadata; create or reconcile labelled namespaces and delivery identities. Workloads use restricted Pod Security; the build namespace is separate. | Sample orchestration and Kubernetes APIs | -| 2 | Create the GitLab tenant group and a project access token; make a separate writer Connector for GitOps delivery. | GitLab automatic creation Task | +| 2 | Create the GitLab tenant group and two group access tokens: a Developer token for project operations and a separate Maintainer token for GitOps delivery. | GitLab automatic creation Task | | 3 | Create private source and production Harbor projects, distribute image pull configuration, and create separate source-read and production-write Connectors for release. | Harbor automatic creation Task | | 4 | Create Nexus repositories and a scoped user, then write the project Connector. | Nexus automatic creation Task | | 5 | Create the SonarQube user, token, and project permission template, then write the Connector. | SonarQube automatic creation Task | | 6 | Seed the empty project GitOps repository and create its PaC Repository and access Secret. Complete and verify webhook configuration separately. | Sample template seeding and PaC resources | | 7 | Render project Applications, AppProject patch, owner and Connector bindings, and the exported ACP project. Open a platform registration merge request. | A separate `gitops-repo-update` PipelineRun | -The registration step proposes Git changes instead of directly applying Argo CD resources. Review the entire diff, including RBAC and destination changes. Keep sufficient execution capacity for both the parent and child run: the waiting parent still reserves its Pod resources. Cancellation and cleanup must account for the independently created child run. +Onboarding creates the project's GitLab group but does not add people to it. Before handoff, a platform administrator must add the project's developers with a role that allows them to create repositories in that group, according to the platform's access policy. + +The registration step proposes Git changes instead of directly applying Argo CD resources. Review the entire diff, including RBAC and destination changes. The project declaration alone does not create the required `projects/kustomization.yaml` `configMapGenerator` entry, onboarding RBAC and schedule files, or their `onboarding/kustomization.yaml` entries; include those files in the same platform merge request. Keep sufficient execution capacity for both the parent and child run: the waiting parent still reserves its Pod resources. Cancellation and cleanup must account for the independently created child run. The pipeline runs in the management namespace with administrator Connectors. Permission to create Pods or PipelineRuns there is effectively access to those credentials, even without API permission to read Secrets. Limit that namespace and the onboarding identity to platform automation and administrators. @@ -234,7 +282,7 @@ The four Tasks have different parameters and rotation behavior. Read their insta - [Nexus resources and Connector](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-nexus/how_to/using_nexus_connector_automatic_creation_task.html). - [SonarQube tenant and Connector](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-sonarqube/how_to/using_sonarqube_connector_automatic_creation_task.html). -Set the target `connector` as `namespace/name`. Set GitLab access levels explicitly: the project access token and the GitOps writer have different responsibilities. In the sample, project access uses `api` and `read_repository`; the writer additionally uses `write_repository`. Connectors CSI mounts use the `connectors` attribute, with the namespace included. +Set the target `connector` as `namespace/name`. Set GitLab access levels explicitly: both tokens in the sample are group access tokens, with Developer for project operations and Maintainer for GitOps writing. The Developer token has `api`, so the absence of `write_repository` does not prevent API-based repository changes while `main` is unprotected; protect `main` before treating review as a control. Connectors CSI mounts use the `connectors` attribute, with the namespace included. The sample writes only to the current cluster using an explicitly authorized ServiceAccount. For cross-cluster application, follow each Task's Kubernetes Connector and `kubeconfig` Workspace instructions. Do not assume that the management namespace's default ServiceAccount has the required write permissions. @@ -265,7 +313,7 @@ spec: Set the provider URL explicitly so completion requests and the Repository token use the intended GitLab instance. The PaC repository URL differs from the `.git` URL used by Argo CD. `gitops/config` is a repository beneath the `gitops` subgroup, rather than a repository path conflicting with that subgroup's name. -The sample's project GitOps token is Developer-level and requires administrator-managed webhook setup. Verify a fresh event and its resulting run. Keep webhook management credentials available through repository decommissioning, and check for remaining code-platform webhooks afterward. The application's Repository belongs in the build namespace, where CI executes. +The project's PaC Repository uses the Developer-level group access token and requires administrator-managed webhook setup; the separate GitOps writer uses the Maintainer-level group access token. Verify a fresh event and its resulting run. Keep webhook management credentials available through repository decommissioning, and check for remaining code-platform webhooks afterward. The application's Repository belongs in the build namespace, where CI executes. Application CI objects belong in the build namespace; CD and promotion Run Templates and PipelineRuns belong in the delivery namespace and require an administrator or an explicit delegated grant. ## Reconcile and rotate deliberately @@ -281,7 +329,7 @@ Use repeated runs of the same supported parameters for reconciliation. Do not eq The sample's supported GitLab subgroups are `apps` and `gitops`. Changing that convention requires a reviewed pipeline change, rather than only a declaration edit. Tool-side identity changes can recreate accounts or credentials; inspect the product Task's contract before changing names or project sets. -Do not overlap runs for the same tenant. Arrange rotation outside active pipeline windows and monitor failed runs before the current credential expires. CSI workload-token issuance is separate from rotating the tool's long-lived credential. Consult the Task guide for expiry recovery; define a local recovery procedure rather than depending on an unobserved natural-expiry path. +Do not overlap runs for the same tenant. The sample has three possible entry points (scheduled reconciliation, merge dispatch, and a manual Run Template) and does not implement a distributed lock; enforce the no-overlap rule operationally or add a lock before using it for concurrent onboarding. Arrange rotation outside active pipeline windows and monitor failed runs before the current credential expires. CSI workload-token issuance is separate from rotating the tool's long-lived credential. Consult the Task guide for expiry recovery; define a local recovery procedure rather than depending on an unobserved natural-expiry path. If enabling Connector approval, prepare grants for every affected workload, enable `enable-connectors-approval`, immediately create the required AccessPolicies, and then enable `enable-connector-proxy-permissions`. The sample's optional administrator AccessPolicy is excluded from the default Kustomization and is applied only after the feature is enabled. Use RBAC for the documented cross-namespace grants and keep approval-gated production Connectors free of direct proxy grants. See the [delivery article](./From_Commit_to_Production_on_ACP.md#gate-the-production-connector-after-approval) for the complete rollout boundary. @@ -296,16 +344,18 @@ If enabling Connector approval, prepare grants for every affected workload, enab | Event routing | A fresh repository event or merge-request retry produces the expected run in the intended namespace. | | Rotation | A successful non-overlapping run updates the intended credential and an authorized operation works afterward. | -Run `acceptance/02-platform-and-onboarding.sh` from the sample in an isolated acceptance project. Its companion delivery script verifies the image path after the first application build. Check every child onboarding run, the four Connectors, the seeded repository, registration merge request, and Applications. The initial placeholder digest is not deployable; workload health follows the first real deployment. +Run `acceptance/02-platform-and-onboarding.sh` from the sample in an isolated acceptance project. Its companion delivery script verifies the image path after the first promoted production deployment. The acceptance script does not check onboarding runs: read the dispatcher's `onboarding-runs` result, confirm that it names a run for every project the change should onboard, and confirm that each child run has `Succeeded=True`. Check the four Connectors, the seeded repository, registration merge request, and Applications. The initial placeholder digest is not deployable; workload health follows the first real deployment. The platform `projects` Application can remain `OutOfSync` for platform-managed fields in this design; inspect the effective destinations and policy instead of treating that status alone as failure. For cross-project image denial, first verify that the exact image exists and can be pulled in its own project. Then test an unauthorized project, distinguish `unauthorized` from `not found` and Pod Security rejection, and repeat in the opposite direction using an image that exists there. Delete temporary probe resources after the checks. -Test self-healing only on a disposable resource in the acceptance project. Record its UID and recovery manifest, perform the approved change, and confirm recreation with a different UID. Restore it if the check fails. This is separate from proving ongoing automatic reconciliation of the operator-managed component layer. +Test self-healing only on a disposable resource in the acceptance project. The sample's default check deletes the project owners RoleBinding, so the project owner can temporarily lose access while the object is restored. Record its UID and recovery manifest, perform the approved change, and confirm recreation with a different UID. Restore it if the check fails. This is separate from proving ongoing automatic reconciliation of the operator-managed component layer. ## Preserve audit records and plan recovery Choose one cleanup mechanism and review its scope before enabling it. Use event-based Pruner according to the global and namespace configuration guides; disable the older periodic mechanism when migrating. Verify each selected retention mode in the target environment before it can remove shared run history. +Global values apply cluster-wide after the first synchronization, including other tenants' namespaces unless a namespace override applies, and successful and failed runs are counted separately per namespace. On a shared cluster, count each namespace's finished PipelineRuns and standalone TaskRuns before enabling cleanup, choose limits above the highest count or agree lower limits with the tenants, and notify owners first. + Use external PostgreSQL and the documented log storage for Results. Define watch scope, authentication, and one retention policy for the reference configuration. Verify read authorization through the Results API, rather than inferring it from Kubernetes resource visibility. Preserve approval responses separately before cleanup removes the ApprovalTask objects. Scope Results Watcher before enabling it: default watching can affect finished runs across namespaces and add `results.tekton.dev/*` finalizers. Disabling archiving does not remove finalizers already present. Follow the product recovery procedure before deletion and treat external Results storage as the audit source after cluster objects are pruned. See [S3 storage](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/s3_storage_configuration.html) and [Results retention](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/retention_policy_configuration.html). From ade5e657392dc73cb95fc36bd966271b99d69fb8 Mon Sep 17 00:00:00 2001 From: Zichen Yu <1062955096@qq.com> Date: Thu, 22 Oct 2026 13:19:17 +0800 Subject: [PATCH 3/5] docs: clarify webhook and proxy CA setup --- docs/en/solutions/From_Commit_to_Production_on_ACP.md | 2 +- ...boarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/solutions/From_Commit_to_Production_on_ACP.md b/docs/en/solutions/From_Commit_to_Production_on_ACP.md index 2cc6d22df..73412e92f 100644 --- a/docs/en/solutions/From_Commit_to_Production_on_ACP.md +++ b/docs/en/solutions/From_Commit_to_Production_on_ACP.md @@ -170,7 +170,7 @@ Customer change-management system integration is outside this series. The availa Keep the production Connector and its Secret in a namespace where project members cannot create Pods or run arbitrary pipelines. A Pod can mount a Secret from its own namespace without a separate Secret-read authorization check. Consequently, permission to create Pods or PipelineRuns in a credential-bearing namespace is a credential boundary. With `enable-connectors-approval` disabled, a principal that can create a Pod or PipelineRun in `shop-cd` can use the production Connector granted to the release identity; an ApprovalTask does not protect that path. Treat the namespace and its ServiceAccounts as the boundary until proxy approval is enabled and verified. -The sample release ServiceAccount `shop-release` reads the source Connector and mounts the production Connector from `platform-connectors`. Its standing grants are in `sample-app/promotion/standing-access.yaml`. In the sample, the copy Task authenticates both ends with the production Connector's credentials; the source Connector mount supplies its CA certificate, so the production robot account must also be able to read the source repository. Use RBAC for the cross-namespace proxy grants described by the product guide. +The sample release ServiceAccount `shop-release` reads the source Connector and mounts the production Connector from `platform-connectors`. Its standing grants are in `sample-app/promotion/standing-access.yaml`. In the sample, the copy Task authenticates both ends with their Connector credentials and routes both ends through the Connectors proxy. Bind the proxy CA bundles for both `src-ca-bundle` and `dst-ca-bundle`, even when the registries use publicly trusted certificates; without the source bundle, the source pull can fail TLS verification. The production robot account must also be able to read the source repository. Use RBAC for the cross-namespace proxy grants described by the product guide. Approval gating uses two installation-wide feature flags. Roll them out in this order: diff --git a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md index 6b5637525..401a97fe6 100644 --- a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md +++ b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md @@ -313,7 +313,7 @@ spec: Set the provider URL explicitly so completion requests and the Repository token use the intended GitLab instance. The PaC repository URL differs from the `.git` URL used by Argo CD. `gitops/config` is a repository beneath the `gitops` subgroup, rather than a repository path conflicting with that subgroup's name. -The project's PaC Repository uses the Developer-level group access token and requires administrator-managed webhook setup; the separate GitOps writer uses the Maintainer-level group access token. Verify a fresh event and its resulting run. Keep webhook management credentials available through repository decommissioning, and check for remaining code-platform webhooks afterward. The application's Repository belongs in the build namespace, where CI executes. Application CI objects belong in the build namespace; CD and promotion Run Templates and PipelineRuns belong in the delivery namespace and require an administrator or an explicit delegated grant. +The project's PaC Repository uses the Developer-level group access token. When automatic webhook registration is enabled, the controller uses the centralized webhook-registration credential configured by platform administrators to create and delete the webhook; verify a fresh event and its resulting run. The separate GitOps writer uses the Maintainer-level group access token. Keep the webhook-registration credential available through repository decommissioning, and check for remaining code-platform webhooks afterward. The application's Repository belongs in the build namespace, where CI executes. Application CI objects belong in the build namespace; CD and promotion Run Templates and PipelineRuns belong in the delivery namespace and require an administrator or an explicit delegated grant. ## Reconcile and rotate deliberately From cfb3cfe9ebb78fd9b5dee37fa3d59f640391c749 Mon Sep 17 00:00:00 2001 From: Zichen Yu <1062955096@qq.com> Date: Thu, 22 Oct 2026 15:28:12 +0800 Subject: [PATCH 4/5] docs: restore DevOps guidance and clarify operational controls --- .../Alauda_DevOps_Best_Practices_Overview.md | 14 ++- .../From_Commit_to_Production_on_ACP.md | 47 ++++--- ...Projects_with_Argo_CD_and_Tekton_on_ACP.md | 119 +++++++++--------- 3 files changed, 99 insertions(+), 81 deletions(-) diff --git a/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md b/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md index b698ebbe4..53134b870 100644 --- a/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md +++ b/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md @@ -73,7 +73,15 @@ For a PaC retry from a merge request, comment `/test branch:< Start with [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html), [Configure GitLab Repository](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/guides/configure_gitlab_repo.html), or [Centralized Management with Hub Pipeline and Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/hub_pipeline_run_template_centralized_management.html). The [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html) provide a guided implementation. -For systems that do not use Git events, an Incoming Webhook can start a PipelineRun. Send an HTTP `POST` with a JSON body containing `repository`, `namespace`, `branch`, `pipelinerun`, and `secret` to the PaC `/incoming` endpoint. Treat the request as accepted only when it returns `202` and a new PipelineRun appears; a `200` response with an empty body can mean that no run was triggered. +For systems that do not use Git events, an Incoming Webhook can start a PipelineRun. Send an HTTP `POST` with a JSON body containing `repository`, `namespace`, `branch`, `pipelinerun`, and `secret` to the PaC controller's `/incoming` endpoint. For example: + +```bash +curl -i -X POST "$PAC_CONTROLLER_URL/incoming" \ + -H 'Content-Type: application/json' \ + -d '{"repository":"","namespace":"","branch":"main","pipelinerun":"","secret":""}' +``` + +Treat the call as accepted only when it returns HTTP `202` and a new PipelineRun appears. An HTTP `200` response with an empty body can mean that no run was triggered, for example because the secret did not match. The controller URL, route exposure, and secret are platform prerequisites; see [Incoming Webhooks](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/incoming_webhooks.html). ## Connect the delivery stages @@ -87,7 +95,7 @@ For systems that do not use Git events, an Incoming Webhook can start a Pipeline | Verify image signatures | Configure Chains keys and registry access centrally; verify the image by digest before promotion and at deployment admission. | [Signing Key Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | | Hand off an artifact event | Map a Harbor event to the next pipeline, carrying the digest. Check scan and signing evidence separately. | [Use Harbor Event Triggers](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | -Promotion copies a selected image; it does not establish that the image passed CI. The approver must match the digest with a successful trusted build and the test environment. For the sample, a trusted CI result is the `ci` PipelineRun for a `push` event on `main`; a successful `/test ci branch:` run from another branch is not equivalent. Signatures establish the configured signer's statement, not which pipeline produced the image; their meaning also depends on control over the signing key and the definitions allowed to run. +Promotion copies a selected image; it does not establish that the image passed CI. The approver must match the digest with a successful trusted build and the test environment. For the sample, a trusted CI result is the `ci` PipelineRun for a `push` event on `main`; the sample rejects comment-triggered `ci` runs before building, and uses a separate definition for merge-request validation. The project credential still needs a separate scratch registry boundary to prevent proposed code from writing release candidates. Signatures establish the configured signer's statement, not which pipeline produced the image; their meaning also depends on control over the signing key and the definitions allowed to run. The samples deliberately keep test CD, promotion, and production CD as manually started runs. Their inputs come from the preceding run's digest result. Harbor-driven automation is a separate integration step. The first production deployment waits for a promoted image; after that image is promoted, follow the same reviewed production GitOps path for later releases. @@ -104,7 +112,7 @@ Platform settings are administrator decisions. Roll them out in stages and check | Execution policy | Use platform ClusterPolicies and project Policies. Restrict direct TaskRun and CustomRun creation. Optionally require approved resolver types and coordinates. Move from Audit to Enforce after checking intended workflows. | | Cleanup | Select one cleanup mechanism. For event-based Pruner, review global defaults and namespace override settings before enabling deletion. A global first sync can affect every namespace, with successful and failed history counted separately; inventory tenants and notify owners before changing the defaults. | | Audit history | Store run records and logs in external Results storage with an explicit retention period. Export approval evidence separately before deleting its objects. | -| Scheduled work | Use ScheduledTrigger or CronJob for reconciliation and rotation, with failure alerts and no overlapping runs for a tenant. The sample has no distributed lock, so its operators enforce that rule. | +| Scheduled work | Use ScheduledTrigger or CronJob for reconciliation and rotation, with failure alerts and no overlapping runs for a tenant. The sample uses an atomic per-project lock; operators must inspect failed or cancelled runs before clearing a retained lock. | Results Watcher scope deserves a separate rollout decision. Its default watch scope can affect finished runs across the cluster and add `results.tekton.dev/*` finalizers. Scope it before enabling it. Disabling archiving does not remove existing finalizers; follow the product recovery procedure before cleanup. After Pruner removes cluster objects, the external Results store is the audit source. diff --git a/docs/en/solutions/From_Commit_to_Production_on_ACP.md b/docs/en/solutions/From_Commit_to_Production_on_ACP.md index 73412e92f..1d75f89ae 100644 --- a/docs/en/solutions/From_Commit_to_Production_on_ACP.md +++ b/docs/en/solutions/From_Commit_to_Production_on_ACP.md @@ -12,6 +12,8 @@ ProductsVersion: Use this article to choose where pipeline definitions live, how deployments consume build outputs, and where approvals, promotion, and signature checks belong. For a guided implementation, follow the [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html). +A build that passes is only the start of delivery. The next questions are which image test actually ran, who authorized that image for production, and how the production deployment can be traced back to the same change. Work through those questions in order: choose the definition and trigger, choose the deployment route for each environment, then add the release controls required by your organization. A small project can start with a manual run; central templates and additional controls can be introduced as the workflow grows. + This is the delivery part of [Alauda DevOps Best Practices](./Alauda_DevOps_Best_Practices_Overview.md). Its configuration baseline is Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and GitOps 4.4. Project namespaces, tool-side resources, and Connectors must already exist; the [platform article](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) explains their preparation. The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) use a fictional `shop` project. CI runs in `shop-build`; CD and promotion run in `shop-cd`; workloads run in `shop-test` and `shop-prod`. The sample binds the project owner group in `shop` only; applying the PaC Repository, CD and promotion objects, and starting their runs requires a platform administrator or a separately delegated grant in `shop-build` and `shop-cd`. See the sample application's ["Who applies what"](https://github.com/AlaudaDevops/devops-best-practices-samples/blob/main/sample-app/README.md#who-applies-what) table. Replace every example address, tool image, group, credential placeholder, and policy identity for your environment. Make the vulnerability database, tool images, base images, and dependencies available locally for disconnected execution. @@ -84,7 +86,11 @@ The sample separates the two outputs: | `sample-app/.tekton/ci.yaml` | Push to `main` | `registry.example.com/shop/hello` | Source for test CD and promotion | | `sample-app/.tekton/ci-merge-request.yaml` | Merge request | `registry.example.com/shop/hello-mr` | Temporary merge-request artifacts | -The sample's Harbor Connector has project-level access. Separating repository names does not itself enforce a credential boundary. For stronger separation, use a distinct temporary registry project and a credential that cannot write release-candidate images. Before delivery, check that the selected digest is the `IMAGE_DIGEST` of a successful trusted `main` CI run whose event type is `push`; a comment-triggered run on another branch is not a release candidate. +The release-candidate definition also uses the catalog Pipeline's pre-build step to require a `push` event with source and target branch `main`. It reads PaC's actual Pod annotations, because comment arguments can override event and branch template variables. A named `/test ci` command can bypass PaC event annotations, so it fails this runtime check before `build-image`, even when commented on `main`. Retry release-candidate CI with a new reviewed push; use `ci-merge-request` for comment-triggered validation. + +The sample's Harbor Connector has project-level access. Separating repository names and adding the event check do not themselves enforce a credential boundary: merge-request code still runs with a credential that can write that project. For stronger separation, use a distinct temporary registry project and a credential that cannot write release-candidate images. Before delivery, match the selected digest to the `IMAGE_DIGEST` of a successful trusted `main` CI run whose event type is `push`, then examine its scan and test evidence. + +If delivery starts from a Harbor event, expose the EventListener only where Harbor can reach it and authenticate every request, for example with an interceptor that compares the webhook's authorization header with a Secret. An EventListener without source validation starts a run for any request that reaches it. See [Use Harbor Event Triggers](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). Both sample CI definitions bind the Connector proxy CA through the `ca-bundle` Workspace. The proxy's CA is required even when the upstream registry uses a publicly trusted certificate. For offline scanning, set `trivyExtraArgs` to use the platform's mirrored vulnerability database, for example `--db-repository registry.example.com/mirror/trivy-db:2`. The scanner must be able to pull that image. @@ -123,7 +129,16 @@ The sample updates the production `digest:` line with `sed` and verifies the ren The sample starts test CD, promotion, and production CD manually from rendered PipelineRun YAML. Replace the template parameters, verify the image mapping, identity, and Workspaces, and submit with `kubectl create`. A console Run Template form exposes Pipeline parameters: in the sample, remove the empty third `updateArgs` entry, use the full `repository@sha256:digest` for test CD, and use the bare `sha256:digest` for production CD because the production repository is fixed by the seeded overlay. Restrict who may create PipelineRuns in the delivery namespace; a console form can expose parameters that are fixed in the rendered YAML path. Use [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html) for other execution entries. Harbor event automation is an additional integration, not part of the sample's manual path. -Production rollback follows the same review boundary: select a retained, previously promoted digest, verify its signature and application compatibility, and review the GitOps change. A tested GitOps rollback consists of: find the previous digest in the production Git history; confirm it remains in the production registry and is signed; create a branch and change only the `digest:` line; open a merge request and wait for validation before review and merge; verify the image in Argo CD and the Deployment and wait for rollout; and return to the newer version through the same one-line change if needed. Database and data migrations need an application-specific rollback plan. Establish and rehearse the complete rollback runbook in the target environment. Configure [Harbor tag retention](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-retention-rules/) and [tag immutability](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-immutability-rules/) so that required rollback images remain available. +Production rollback follows the same review boundary. Rebuilds can produce different content, so select a retained, previously promoted digest and use this GitOps path: + +1. Find the previous digest in the production Git history. +2. Confirm that it remains in the production registry, verify its signature, and check compatibility with current configuration and data. +3. Create a branch and change only the production overlay's `digest:` line. +4. Open a merge request, wait for validation, then review and merge it. +5. Confirm the selected image in Argo CD and the Deployment, and wait for rollout completion. +6. If returning to the newer version is required, use the same reviewed one-line change. + +Database and data migrations need an application-specific rollback plan. Rehearse the complete runbook in the target environment. Configure [Harbor tag retention](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-retention-rules/) and [tag immutability](https://goharbor.io/docs/2.14.0/working-with-projects/working-with-images/create-tag-immutability-rules/) so that required rollback images remain available. ## Select the release controls @@ -164,19 +179,21 @@ PipelineRun and task timeout budgets must include work before approval, the enti Approvers can inspect pending work in the console or through `kubectl`. The optional sample `promotion/optional/console-approver.yaml` supplies a custom console role and group binding; review its scope before applying it. An approval notification can be sent with the catalog's `send-mail` Task. Email addresses, SMTP configuration, and credentials must be configured separately; the sample does not integrate that notification step. +To start a run through a Pipelines-as-Code Incoming Webhook, send a `POST` request whose JSON body contains `repository`, `namespace`, `branch`, `pipelinerun`, and `secret` to the controller's `/incoming` endpoint. Treat the call as successful only when it returns HTTP `202` and a new PipelineRun appears; an HTTP `200` with an empty body means that no run was started, for example because the secret did not match. See [Incoming Webhooks](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/incoming_webhooks.html). + Customer change-management system integration is outside this series. The available building blocks are the approval node and [Incoming Webhooks](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/incoming_webhooks.html). Define the mapping and audit contract for that integration separately. ### Gate the production Connector after approval Keep the production Connector and its Secret in a namespace where project members cannot create Pods or run arbitrary pipelines. A Pod can mount a Secret from its own namespace without a separate Secret-read authorization check. Consequently, permission to create Pods or PipelineRuns in a credential-bearing namespace is a credential boundary. With `enable-connectors-approval` disabled, a principal that can create a Pod or PipelineRun in `shop-cd` can use the production Connector granted to the release identity; an ApprovalTask does not protect that path. Treat the namespace and its ServiceAccounts as the boundary until proxy approval is enabled and verified. -The sample release ServiceAccount `shop-release` reads the source Connector and mounts the production Connector from `platform-connectors`. Its standing grants are in `sample-app/promotion/standing-access.yaml`. In the sample, the copy Task authenticates both ends with their Connector credentials and routes both ends through the Connectors proxy. Bind the proxy CA bundles for both `src-ca-bundle` and `dst-ca-bundle`, even when the registries use publicly trusted certificates; without the source bundle, the source pull can fail TLS verification. The production robot account must also be able to read the source repository. Use RBAC for the cross-namespace proxy grants described by the product guide. +The sample release ServiceAccount `shop-release` mounts the source and production Connectors from `platform-connectors`. Its standing grants are in `sample-app/promotion/standing-access.yaml`. The two registry configurations share the copy Task's proxy environment; the destination configuration overrides conflicting source settings. In this sample both ends therefore use the production Connector's proxy credential. The production robot must be able to pull the staging repository as well as write the production repository; onboarding grants both. The source mount supplies the source CA bundle and does not establish a separate read-only authentication path. Bind both `src-ca-bundle` and `dst-ca-bundle`, even when the registries use publicly trusted certificates, because the proxy re-signs TLS. Use RBAC for the cross-namespace proxy grants described by the product guide. Approval gating uses two installation-wide feature flags. Roll them out in this order: 1. Inventory every identity mounting a Connector that is not behind approval. Prepare its read authorization and `connectors/proxy` permission with verb `*`. A `get` grant on `connectors/proxy` is insufficient. Do not grant direct proxy access to the production Connector that approval must gate. 2. Enable `enable-connectors-approval`. Immediately create the required AccessPolicies, including the administrator-Connector grants and the production approval policy. Do not start affected runs in the interval between enabling the flag and creating the policies. -3. In the production AccessPolicy, configure the designated `approvers` and keep them aligned with the ApprovalTask and promotion admission policy. The `approvers` field in a project declaration is an audited record only; it does not configure either object. Remove any other matching default or direct proxy grants. +3. Keep the promotion Pipeline's ApprovalTask and production AccessPolicy `approvers` aligned. The `approvers` field in a project declaration is an audited record only; onboarding does not apply or compare it. Keep the AccessPolicy under platform-administrator control so a Pipeline editor cannot change both sides of the gate. Remove any other matching default or direct proxy grants. 4. Enable `enable-connector-proxy-permissions`, which applies the proxy permission check to requests as well. Set feature flags through `ConnectorsConfig`. For the full configuration and verification path, see [Enable Proxy Permission Checks and Approval Gating](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/how_to/enable_proxy_permissions_and_approval.html), [AccessPolicy](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/access_policy.html), and [Skopeo Copy connector approval](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/promote_artifacts_with_skopeo_copy.html#connector-approval). @@ -199,7 +216,7 @@ cosign verify --key cosign.pub "$IMAGE" cosign verify-attestation --key cosign.pub --type slsaprovenance "$IMAGE" ``` -The `--type` value must match the provenance predicate emitted by Chains. Keep the Chains format and the verification command aligned (the sample expects SLSA provenance v0.2); a different format requires the corresponding cosign type. +The `--type` value must match the provenance predicate emitted by Chains. This sample expects Chains `artifacts.taskrun.format: slsa/v1`, which emits SLSA provenance v0.2, and therefore uses cosign `--type slsaprovenance`. The `slsa/v1` configuration name does not mean the SLSA v1 predicate. A different Chains format requires the corresponding cosign type and attestation-policy paths. For an offline installation configured without a transparency log, use the product's offline verification settings, including `--insecure-ignore-tlog=true` where required. This skips the external transparency-log check; retain the signature and attestation checks. Authenticate to private registries without committing the login credentials. @@ -225,17 +242,17 @@ The sample's resolver and bare-run policies exclude `platform-connectors`, where ## Accept the delivery workflow -Run `acceptance/01-commit-to-production.sh` from the sample only in an isolated acceptance project with a separate project, namespaces, and Harbor production repository after replacing its inputs. The script defaults to sample names; set every input explicitly and never point it at an ordinary production tenant. The checks are an execution procedure, not a claim that an arbitrary environment is ready. +Run `acceptance/01-commit-to-production.sh` after the first promotion, production GitOps merge, and completed rollout. Use a disposable acceptance project with its own namespaces and Harbor production repository. `PROJECT` is required; inspect the remaining defaults and replace all environment-specific inputs. The script creates temporary admission probes in that project's production namespace. The table distinguishes its checks from the release evidence that you must inspect separately. -| Check | Required evidence | -| --- | --- | -| CI completed | The trusted `main` run succeeded; its image repository and digest match the selected release input. | -| Test ran the selected image | Compare the complete workload image reference and verify rollout completion. | -| Promotion copied that image | The promotion result reports the expected destination repository and digest. | -| Production uses the promotion result | The GitOps merge request is merged, the rendered image matches the destination digest, and the workload rollout completed. `Synced` alone can describe an older revision. | -| Signature and provenance exist | Both cosign commands succeed against the production image. A signing annotation alone is insufficient. | -| Admission reports or enforces the selected policy | In Audit, verify a known signed production digest as a positive control and review the report for an unsigned image; Audit reports are evidence, not enforcement. In Enforce, verify the signed positive control first, then distinguish rejection of an unsigned image from Pod Security or image-pull failure. Clean up both probes afterward. | -| Approval is attributable | Save the ApprovalTask response, approver group members, selected digest, and related run identifiers in the release record. | +| Check | Script coverage | Evidence to inspect separately | +| --- | --- | --- | +| CI completed | Uses the supplied `BUILD_DIGEST`; it does not select or validate a CI run. | The trusted `push` run on `main` succeeded; its repository, digest, scan, and test results match the release input. | +| Test ran the selected image | Compares the complete image reference and waits for rollout completion. | Application-level test results for that deployment. | +| Promotion copied that image | Compares the release tag's registry digest with `BUILD_DIGEST`. | The promotion run's `promoted-digest` and destination mapping match that digest and repository. | +| Production uses the promotion result | Checks Application sync/health, the exact release digest, and rollout completion. | Review and merge of the intended GitOps diff and its rendered image. `Synced` alone can describe an older revision. | +| Signature and provenance exist | Runs both cosign commands against the production image. | The attestation identifies the expected copy TaskRun; approval history is retained separately. | +| Admission reports or enforces the policy | First checks a signed production image, then an unsigned image; selects reports by probe UID and policy. | Audit reports demonstrate reporting. Claim enforcement only after the same controls pass in Enforce. Distinguish signature denial from Pod Security and registry-access failures. | +| Approval is attributable | Checks the approval object's ownership chain and responses, including group members. | Save those responses, the approved digest, and related run identifiers in the release record before cleanup. | Before production, address the sample's explicit hardening work: restrict the GitOps writer's namespace, protect production Git changes, install the promotion Core rules, constrain production image sources, and limit direct cluster writes. Do not grant project members Pod creation in namespaces that contain administrator or release credentials. Establish local rollback and policy-service recovery runbooks. diff --git a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md index 401a97fe6..a8d4d445d 100644 --- a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md +++ b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md @@ -12,6 +12,8 @@ ProductsVersion: Use a platform Git repository to review and preserve shared configuration. This gives the platform team a durable, auditable, repeatable source, but it does not replace backups of sealed-secrets keys, signing keys, external tool data, or Results storage. Use an onboarding pipeline to create a project's namespaces and tool-side resources, then propose its deployment configuration through a merge request. Argo CD reconciles the objects owned by Git; the onboarding tasks reconcile the resources they create in external tools. +This approach is useful when platform settings exist only in a running cluster, each new project requires repeated administration in four tools, or the team needs to explain who may change a shared boundary. Git provides a recorded starting point and a review trail. For tracked configuration, it also provides a way to inspect and revert a change. Keep the corresponding keys and external data recoverable separately: replaying YAML cannot recover them, and reverting a project registration can delete resources. + This article is the platform part of [Alauda DevOps Best Practices](./Alauda_DevOps_Best_Practices_Overview.md). It assumes one ACP project and separate tool-side tenants per project. Its configuration baseline is Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and GitOps 4.4 on a compatible ACP installation. Teams sharing one namespace and one tool tenant need a different boundary model. The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) contain the complete reference files. Names such as `shop`, `payments`, `platform-connectors`, and `registry.example.com` are fictional. Replace them, the image placeholders, group resource names, policy identities, and encrypted Secret placeholders for the target environment. @@ -66,9 +68,9 @@ Create the ACP project through the platform before onboarding it. The pipeline e Onboarding creates the GitLab group and its service tokens; it does not decide which human users belong to the group or grant them GitLab project-creation rights. The platform or project administrators must add approved members and manage those memberships separately from the token used by Pipelines-as-Code. -Platform administrators are responsible for webhook setup and event-delivery verification for platform, project GitOps, and application repositories. When automatic registration is enabled and the PaC controller URL is configured, creating a `Repository` can cause the registration controller to create and delete the project webhook. This requires a centralized webhook-registration GitLab credential with `api` scope and Maintainer access to the projects, stored in the controller's namespace; the PaC Repository's Developer-level group token cannot manage webhooks. For setup and verification, follow [Create a Git Secret for PAC](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/create_git_secret.html) and test a real event before accepting onboarding. +Platform administrators are responsible for webhook setup and event-delivery verification for platform, project GitOps, and application repositories. When automatic registration is enabled and the PaC controller URL is configured, creating a `Repository` can cause the registration controller to create and delete the project webhook. This requires a centralized webhook-registration GitLab credential with `api` scope and Maintainer access to the projects, stored in the controller's namespace; the PaC Repository's Developer-level group token cannot manage webhooks. Configure the controller URL so `controller-url` is populated in the `pipelines-as-code-info` ConfigMap, and verify the Repository's registration status and a real event before accepting onboarding. Follow [Advanced Repository Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html) and [Create a Git Secret for PAC](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/create_git_secret.html) for the product-specific credential and controller settings. -Keep Harbor projects private. Configure Nexus anonymous access and SonarQube permissions according to the project-isolation requirements in their Task guides. Do not infer isolation merely from the presence of separate credentials. +Keep Harbor projects private. Nexus anonymous access combined with an anonymous role that can read tenant repositories bypasses their user credentials; the onboarding Task only warns about anonymous access. Disable it (Settings → Security → Anonymous Access) or remove the anonymous role's access to tenant repositories, then confirm that an unauthenticated read is refused. Configure SonarQube permissions according to the project-isolation requirements in its Task guide. Do not infer isolation merely from the presence of separate credentials. For a disconnected environment, prepare local catalog content, pinned tool and base images, dependency repositories, certificates, and vulnerability databases before running pipelines. @@ -76,62 +78,41 @@ For a disconnected environment, prepare local catalog content, pinned tool and b The table identifies the platform decision and its repository layer. Values and detailed steps belong in the linked product documentation. These are administrator-managed recommendations, with maturity and prerequisites determined by the selected release. -| # | Decision | Why it is shared | Layer and guidance | -| --- | --- | --- | --- | -| 1 | Separate Repository access tokens from webhook administration; use reviewed default-branch definitions and an explicit GitLab provider URL. | Credential scope and pre-merge execution trust should be consistent. | Credentials and project: [PaC Repository configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html). | -| 2 | Standardize merge-request retries with `/test branch:`. | Teams use one supported entry and make the source branch explicit. | Project template: [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). | -| 3 | Separate platform-provided Repository parameters from application inputs. | Addresses and Connector names have a defined owner. | Project: [Advanced Repository Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html). | -| 4 | Decide Harbor event handoffs. | Event endpoints and validation need consistent ownership. | Component: [Harbor events](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | -| 5 | Schedule reconciliation and rotation with failure alerts. | The platform owns the interval and operational response. | Component: [ScheduledTrigger](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/scheduledtriggers.html). | -| 6 | Standardize EventListener exposure. | TLS, routes, and webhook validation belong to the platform. | Component: [Automatic exposure rules](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/trigger-wrapper-config.html). | -| 7 | Orchestrate tool-side tenant creation with the four product Tasks. | Administrators retain tool-wide credentials. | Credentials and project: tool Task guides below. | -| 8 | Define long-lived credential rotation for each tool. | Expiry and renewal behavior differ by tool. | Component and credentials: [GitLab automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-gitlab/how_to/using_gitlab_connector_automatic_creation_task.html#step-by-step--rotate-in-place), [Harbor automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-harbor/how_to/using_harbor_connector_automatic_creation_task.html#example-re-run-the-task-to-refresh-credentials-and-reconcile-resources), [Nexus automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-nexus/how_to/using_nexus_connector_automatic_creation_task.html#re-run-to-add-repositories-or-rotate-credentials), and [SonarQube automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-sonarqube/how_to/using_sonarqube_connector_automatic_creation_task.html#rotation). | -| 9 | Configure Chains signing keys and image verification. | The platform owns the signer and distributes its public key. | Component, credentials, policy: [Signing keys](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | -| 10 | Use Connector proxy access while tracking long-lived credential exceptions. | Access boundaries and rotation need a central owner. | Credentials: [Connectors Proxy](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_proxy.html) and [CSI Driver](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_csi.html). | -| 11 | Apply namespace Pod Security and separate build namespaces. | Build tooling and workloads have different privilege needs. | Policy and project: [Restricted standard](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/comply_with_pod_security_restricted.html). | -| 12 | Install approval and define protected-environment groups and promotion rules. | Approval ownership is a platform control. | Component, project, policy: [Manual Approval Gate](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/manual_approval_gate.html). | -| 13 | Establish platform and project policy layers. | Projects can add restrictions without weakening the platform baseline. | Policy: [Constrain pipeline execution](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/constrain_pipeline_execution_with_kyverno.html). | -| 14 | Own the shared, locally available catalog. | Reuse and offline content need governance. | Component: [Author Catalog Resources](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/author_catalog_resources.html). | -| 15 | Publish versioned central Pipelines. | Projects adopt reviewed versions explicitly. | Component: [Centralized Pipeline management](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/hub_pipeline_run_template_centralized_management.html). | -| 16 | Select Pruner defaults and namespace override policy. | Cleanup affects shared cluster history and capacity. | Component: [Pruner configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pruner/configure/pruner_configuration.html). | -| 17 | Define Results storage, watch scope, authentication, and retention. | Audit records need storage independent of live objects. | Component: [Results retention](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/retention_policy_configuration.html). | -| 18 | Enable metrics and operational monitoring. | Operators need evidence of controller and execution health. | Component: [Metrics Collection](https://docs.alauda.io/alauda-devops-pipelines/4.15/configure/metrics_collection.html). | - -The baselines matter at different points in the operating model: - -| Baseline | When this matters | -| --- | --- | -| Repository and webhook ownership | Before onboarding the first repository or changing GitLab providers. | -| Retry and event conventions | When teams need reproducible merge-request or push-triggered reruns. | -| Parameter ownership | When a platform-managed address or Connector must change without editing every application. | -| Harbor event handoff | When an external registry event starts another pipeline. | -| Scheduled reconciliation | When credentials or declarations need periodic repair. | -| EventListener exposure | Before exposing webhook endpoints outside the cluster. | -| Automatic tenant creation | When a new project must receive the same tool-side resources. | -| Credential rotation | Before the first token reaches its expiry window. | -| Chains and image verification | Before a private production registry is enforced by Kyverno. | -| Connector proxy access | When workload access to long-lived credentials must be narrowed. | -| Pod Security and build separation | Before build tools and workloads share a cluster. | -| Approval and promotion rules | Before production credentials or artifacts can be released. | -| Policy layers | When projects need additional restrictions without weakening platform controls. | -| Offline catalog governance | When the cluster cannot fetch Tasks or Pipelines from the internet. | -| Versioned central Pipelines | When multiple projects consume the same workflow. | -| Pruner defaults | Before cleanup can remove shared run history. | -| Results storage and retention | Before live Tekton objects are pruned. | -| Metrics and monitoring | Before operating the platform without controller health evidence. | - -The following matrix describes the sample's intended six-role boundary. It is a reference target, not a claim that every production safeguard is already enabled: +| # | When this matters | Decision | Why it is shared | Layer and guidance | +| --- | --- | --- | --- | --- | +| 1 | Before onboarding the first repository or changing GitLab providers. | Separate Repository access tokens from webhook administration; use reviewed default-branch definitions and an explicit GitLab provider URL. | Credential scope and pre-merge execution trust should be consistent. | Credentials and project: [PaC Repository configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html). | +| 2 | When teams need reproducible merge-request or push-triggered reruns. | Standardize merge-request retries with `/test branch:`. | Teams use one supported entry and make the source branch explicit. | Project template: [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). | +| 3 | When a platform-managed address or Connector must change without editing every application. | Separate platform-provided Repository parameters from application inputs. | Addresses and Connector names have a defined owner. | Project: [Advanced Repository Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/advanced_repository_config.html). | +| 4 | When an external registry event starts another pipeline. | Decide Harbor event handoffs. | Event endpoints and validation need consistent ownership. | Component: [Harbor events](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | +| 5 | When credentials or declarations need periodic repair. | Schedule reconciliation and rotation with failure alerts. | The platform owns the interval and operational response. | Component: [ScheduledTrigger](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/scheduledtriggers.html). | +| 6 | Before exposing webhook endpoints outside the cluster. | Standardize EventListener exposure. | TLS, routes, and webhook validation belong to the platform. | Component: [Automatic exposure rules](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/trigger-wrapper-config.html). | +| 7 | When a new project must receive the same tool-side resources. | Orchestrate tool-side tenant creation with the four product Tasks. | Administrators retain tool-wide credentials. | Credentials and project: tool Task guides below. | +| 8 | Before the first token reaches its expiry window. | Define long-lived credential rotation for each tool. | Expiry and renewal behavior differ by tool. | Component and credentials: [GitLab automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-gitlab/how_to/using_gitlab_connector_automatic_creation_task.html#step-by-step--rotate-in-place), [Harbor automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-harbor/how_to/using_harbor_connector_automatic_creation_task.html#example-re-run-the-task-to-refresh-credentials-and-reconcile-resources), [Nexus automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-nexus/how_to/using_nexus_connector_automatic_creation_task.html#re-run-to-add-repositories-or-rotate-credentials), and [SonarQube automatic-creation Task](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-sonarqube/how_to/using_sonarqube_connector_automatic_creation_task.html#rotation). | +| 9 | Before a private production registry is enforced by Kyverno. | Configure Chains signing keys and image verification. | The platform owns the signer and distributes its public key. | Component, credentials, policy: [Signing keys](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | +| 10 | When workload access to long-lived credentials must be narrowed. | Use Connector proxy access while tracking long-lived credential exceptions. | Access boundaries and rotation need a central owner. | Credentials: [Connectors Proxy](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_proxy.html) and [CSI Driver](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_csi.html). | +| 11 | Before build tools and workloads share a cluster. | Apply namespace Pod Security and separate build namespaces. | Build tooling and workloads have different privilege needs. | Policy and project: [Restricted standard](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/comply_with_pod_security_restricted.html). | +| 12 | Before production credentials or artifacts can be released. | Install approval and define protected-environment groups and promotion rules. | Approval ownership is a platform control. | Component, project, policy: [Manual Approval Gate](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/manual_approval_gate.html). | +| 13 | When projects need additional restrictions without weakening platform controls. | Establish platform and project policy layers. | Projects can add restrictions without weakening the platform baseline. | Policy: [Constrain pipeline execution](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/constrain_pipeline_execution_with_kyverno.html). | +| 14 | When the cluster cannot fetch Tasks or Pipelines from the internet. | Own the shared, locally available catalog. | Reuse and offline content need governance. | Component: [Author Catalog Resources](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/author_catalog_resources.html). | +| 15 | When multiple projects consume the same workflow. | Publish versioned central Pipelines. | Projects adopt reviewed versions explicitly. | Component: [Centralized Pipeline management](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/hub_pipeline_run_template_centralized_management.html). | +| 16 | Before cleanup can remove shared run history. | Select Pruner defaults and namespace override policy. | Cleanup affects shared cluster history and capacity. | Component: [Pruner configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/pruner/configure/pruner_configuration.html). | +| 17 | Before live Tekton objects are pruned. | Define Results storage, watch scope, authentication, and retention. | Audit records need storage independent of live objects. | Component: [Results retention](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/retention_policy_configuration.html). | +| 18 | Before operating the platform without controller health evidence. | Enable metrics and operational monitoring. | Operators need evidence of controller and execution health. | Component: [Metrics Collection](https://docs.alauda.io/alauda-devops-pipelines/4.15/configure/metrics_collection.html). | + +The following matrix summarizes the six roles and the grants supplied or required by this reference. Kubernetes grants are additive: an absent sample binding does not revoke permissions inherited from ACP or another role. Git permissions and production protection must be configured in GitLab separately. | Role | Platform repo | Project declaration | Project namespace | Delivery namespace | GitOps `envs/prod` | Production | Connectors | | --- | --- | --- | --- | --- | --- | --- | --- | -| Platform administrator | Write | Write | Full | Full | Configure branch protection and CODEOWNERS | Full | Administrator | -| Application developer | Read | Read | Project-scoped write | None by default | Open merge requests | None directly | Project-scoped through CI | -| Release approver | None | None | None | Approve ApprovalTask | Approve code-owner review | None | None | -| CI identity | None | None | Build only | None | None | None | Project tool Connectors | -| Delivery identity | None | None | None | Run CD | Open merge requests | Update references through GitOps | GitOps writer Connector | -| Promotion identity | None | None | None | Run promotion | None | Copy to production registry | Read source and write production Connector | +| Platform administrator | Write, prerequisite | Write, prerequisite | Administrator, prerequisite | Apply definitions and start runs | Configure protection separately | Administrator, prerequisite | Administrator Connectors | +| Application developer / project owner | No Git grant supplied | No Git grant supplied | Owners group gets `edit` in `shop` | No sample binding | No Git grant supplied | No workload binding supplied | Can use credentials in `shop` through Pods | +| Release approver | No grant supplied | No grant supplied | No binding supplied | ApprovalTask response; optional console read role | No Git review role supplied | No workload binding supplied | No mount grant supplied | +| CI ServiceAccount | No access supplied | No access supplied | Runs in `shop-build` | No binding supplied | No writer supplied | No workload binding supplied | Mount/proxy grants for project Harbor, SonarQube, Nexus | +| Delivery ServiceAccount | No access supplied | No access supplied | No binding supplied | Executes CD; no run-creation grant supplied | Maintainer group token through writer Connector | Changes Git references, reconciled by Argo CD | GitOps writer mount/proxy grant | +| Promotion ServiceAccount | No access supplied | No access supplied | No binding supplied | Executes promotion; no run-creation grant supplied | No writer supplied | Writes production registry through Connector | Mount both registry Connectors; direct proxy grant only for source | + +The release approver is not automatically a Git code owner. Configure the Git review role separately if both approvals are required. The supplied Maintainer writer can change GitLab protection settings; a production deployment needs a separately protected branch or repository and a narrower writer identity. Verify effective permissions with the actual users and ServiceAccounts before delegating delivery-namespace run creation. -In the sample, production branch protection and code-owner approval are still a platform setup task, the developer has no default delivery-namespace binding, and console approval requires the optional ACP role binding. Configure and verify those controls before treating the matrix as enforced. +Name-scoped Connector `get` grants record the mounts this workflow needs; they do not remove ACP's broader default Connector reads within a project. Identity-specific proxy restrictions require the proxy permission features and grants described in the [delivery article](./From_Commit_to_Production_on_ACP.md#gate-the-production-connector-after-approval). Keep the distinction between read authorization, proxy authorization, and access to raw Secrets when reviewing this matrix. ### Separate policy scope from definition trust @@ -170,9 +151,9 @@ spec: This does not constrain a resolver's URL, catalog, namespace, version, definition content, or runtime credential scope. It also rejects inline `pipelineSpec` when enforced. Convert the sample's inline validation flow to a catalog Pipeline before enabling that restriction for project repositories. Restrict PolicyException creation or disable exceptions according to platform policy. -The sample's `argocd-project-namespace-boundary` policy constrains writes of Role, RoleBinding, ResourceQuota, LimitRange, and NetworkPolicy by the configured Argo CD identities in labelled project namespaces. It allows the platform-owned registration path and requires `allowExistingViolations: false`. Adapt both Argo CD identities for the target cluster and verify the intended allow and deny cases before relying on the boundary. The exception for the platform `projects` Application depends on Argo CD's `argocd.argoproj.io/tracking-id` annotation; label-only resource tracking is not covered. The policy does not cover Kyverno `PolicyException` or Connectors `AccessPolicy` resources; keep those kinds out of project Application allow-lists or add explicit deny rules after verifying the target platform's exception and namespace settings. Install and verify the policy on every destination cluster to which the boundary should apply. +The sample's `argocd-project-namespace-boundary` policy constrains writes of Role, RoleBinding, ResourceQuota, LimitRange, NetworkPolicy, Kyverno PolicyException, and Connectors AccessPolicy by the configured Argo CD identities in labelled project namespaces. It allows the platform-owned registration path and requires `allowExistingViolations: false`. Adapt both Argo CD identities for the target cluster and verify the intended allow and deny cases for the installed APIs before relying on the boundary. The exception for the platform `projects` Application depends on Argo CD's `argocd.argoproj.io/tracking-id` annotation; label-only resource tracking is not covered. Restrict exception and AccessPolicy creation by other identities as well, and keep any platform exception namespace outside project destinations. Install and verify the policy on every destination cluster to which the boundary should apply. -Kyverno availability and recovery are platform operating concerns. For policies used as admission gates, keep the resource webhooks registered with `failurePolicy: Fail`; when Kyverno is unreachable, matching writes are then rejected. Operate Kyverno highly available and keep a controlled recovery procedure, including the conditions for temporarily removing and later restoring webhook configurations. A signing result proves the signing key, not the pipeline that produced an image. Restrict ClusterPolicy, resolver, webhook, and PolicyException changes to platform administrators. See [Pipeline Policy Constraints with Tekton and Kyverno](./Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md) for the policy design. +Kyverno availability and recovery are platform operating concerns. For policies used as admission gates, keep the resource webhooks registered with `failurePolicy: Fail`; when Kyverno is unreachable, matching writes are then rejected. An absent webhook provides no enforcement, and an orderly scale-to-zero can remove resource webhook registrations. Operate Kyverno highly available and keep a controlled recovery procedure, including the conditions for temporarily removing and later restoring webhook configurations. A signing result proves the signing key, not the pipeline that produced an image. Restrict ClusterPolicy, resolver, webhook, and PolicyException changes to platform administrators. See [Pipeline Policy Constraints with Tekton and Kyverno](./Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md) for the policy design. ## Organize and synchronize the platform repository @@ -190,15 +171,23 @@ platform-repo/ └── .tekton/ # Validation and changed-project dispatch ``` -Copy `onboarding/` from the sample to the platform repository root as described by its README. Register the repository read-only with Argo CD and use one child Application per layer. The root Application manages those Application definitions. On the first bootstrap, apply `argocd/apps/` once with `kubectl apply -k`; declarations already present in `projects/` are not picked up by the merge dispatcher until a later `main` change, so run the onboarding Run Template once for each existing declaration after the Applications are synchronized. +Copy `onboarding/` from the sample to the platform repository root as described by its README. Register the repository read-only with Argo CD and use one child Application per layer. The root Application manages those Application definitions. After replacing placeholders and preparing the credentials and CRDs above, bootstrap once from that repository's root: + +```bash +kubectl apply -k argocd/apps/ +``` -For partial singleton component configuration, use server-side apply so the platform repository owns only the specified fields. The samples also set `argocd.argoproj.io/compare-options: ServerSideDiff=true`. Review the generated diff and the operator's field ownership before synchronization. The component layer uses `prune: false` so that removing a fragment does not delete the singleton TektonConfig. The sample currently enables `selfHeal` for this layer; if the operator's write-back behavior has not been verified, start with manual sync or `selfHeal: false` and enable it only after observing the effective ownership. Establish the component layer's automated synchronization behavior in the target environment before enabling ongoing self-healing. +Declarations already present in `projects/` are not picked up by the merge dispatcher until a later `main` change. After the Applications synchronize, run the onboarding Run Template once for each existing declaration. Later changes follow the reviewed Git and dispatcher path. + +For partial singleton component configuration, use server-side apply so the platform repository owns only the specified fields. The samples also set `argocd.argoproj.io/compare-options: ServerSideDiff=true`. Review the generated diff and the operator's field ownership before synchronization. The component layer uses `prune: false` so that removing a fragment does not delete the singleton TektonConfig, and starts with `selfHeal: false`. Observe the operator's write-back behavior and effective ownership in the target environment before enabling ongoing self-healing. Automatic synchronization of a new Git revision and self-healing of a live edit are separate decisions. The credential layer does not enable self-healing: seal credentials for the target controller and plan how rotations update live Secrets and encrypted source. Other layers can use prune and self-heal according to their ownership. Namespaces are created by onboarding; do not add a second namespace creator with `CreateNamespace`. ACP maintains the AppProject associated with an ACP project. The registration patch manages only the intended repository and destination fields. Reconcile the effective destinations with the platform-managed entries, and use the admission boundary for object-kind restrictions. Do not rely on an Application's sync status alone to prove the deployment boundary. -Review deletion carefully. Pruning a registered ACP project can delete its same-named namespace and its Secrets, Connectors, and Repository objects. It does not reverse tool-side creation or remove every build, delivery, or environment namespace. Decommissioning requires a separate approved procedure. +Review deletion carefully. Reverting a registration commit, or removing `projects//` together with its `resources` entry, can make Argo CD prune the registered ACP project. That can delete its same-named namespace and the Secrets, Connectors, and Repository objects there. It does not reverse tool-side creation or automatically remove every build, delivery, or environment namespace. + +Deleting only `projects/.yaml` is different: it does not decommission the project, and leaving its `configMapGenerator` entry in place makes the declaration layer fail to render. Use a separate approved decommissioning procedure covering workloads, retained evidence, external tenants, credentials, webhooks, and each namespace. ### Validate changes before merging @@ -206,7 +195,9 @@ Both the platform and project GitOps templates include `.tekton/validate.yaml`, Validate every directory owned by an Application, and add new directories to the validation paths. Re-run a merge-request check with `/test validate branch:`. YAML parsing and Kustomize rendering do not validate CRD admission behavior; use the target schemas and an isolated project for that acceptance step. -A validation status blocks a merge only when the Git platform requires it. On GitLab, enable **Pipelines must succeed** for the platform and project GitOps repositories. If Auto DevOps starts an additional pipeline without an available runner, the merge request can remain in a running state; disable Auto DevOps for these repositories or provide a runner. With `pipelinerun_provenance: default_branch`, the definition is reviewed, but the merge-request content is still rendered by the validation run. Keep that dispatcher and its credentials isolated from administrator and production credentials. +A validation status blocks a merge only when the Git platform requires it. On GitLab, enable **Pipelines must succeed** for the platform and project GitOps repositories. If Auto DevOps starts an additional pipeline without an available runner, the merge request can remain in a running state; disable Auto DevOps for these repositories or provide a runner. + +With `pipelinerun_provenance: default_branch`, the definition is reviewed, but validation still renders proposed merge-request content. The sample's platform Repository and validation runs live in `platform-connectors`, which also holds administrator Secrets. That placement is suitable only for trusted platform maintainers: permission to create runs there can expose those Secrets. Before accepting less-trusted contributions, move the Repository, its limited token, and validation execution to a namespace without administrator credentials. Keep management dispatch separate and authorize only the reviewed dispatch path to create onboarding runs. The sample onboarding dispatcher compares the pushed revision with its first parent. Use a merge or squash workflow that keeps all declaration changes in that comparison; a fast-forward push containing several independent commits can cause an earlier declaration change to be missed. The sample's `on-path-change` annotation is a Technology Preview feature in the 4.15 line, so verify its maturity and behavior before relying on it for the trigger boundary. @@ -249,7 +240,7 @@ sonarqube: `imagePullFrom` is reserved: this sample only prints a warning when it is set and grants no cross-project pull access. -The declaration's `approvers` field is a reviewed record only: onboarding validates its shape but does not apply or compare it. The approvers that take effect are the promotion Pipeline's `ApprovalTask` list and the production `AccessPolicy`; nothing keeps those three sources in sync, so update them together in one reviewed change. Keep the production `AccessPolicy` under platform-administrator control. +The declaration's `approvers` field is a reviewed record only: onboarding validates its shape but does not apply or compare it. The approvers that take effect are the promotion Pipeline's `ApprovalTask` list and the production `AccessPolicy`; nothing keeps those three sources in sync, so update them together in one reviewed change. Keep the production `AccessPolicy` under platform-administrator control because a person who can edit the Pipeline can otherwise change its `ApprovalTask` list while the AccessPolicy continues to enforce its own approver set. Submit the declaration, project-specific onboarding permissions, and schedule through a platform merge request. Confirm that the declaration and onboarding Applications have synchronized; synchronize them manually when using a manual sync policy. Then examine the dispatcher result and every child onboarding run. A successful merge or dispatcher does not by itself establish complete project readiness. @@ -269,7 +260,7 @@ The sample dispatches changed declarations after a merge. A path filter can also Onboarding creates the project's GitLab group but does not add people to it. Before handoff, a platform administrator must add the project's developers with a role that allows them to create repositories in that group, according to the platform's access policy. -The registration step proposes Git changes instead of directly applying Argo CD resources. Review the entire diff, including RBAC and destination changes. The project declaration alone does not create the required `projects/kustomization.yaml` `configMapGenerator` entry, onboarding RBAC and schedule files, or their `onboarding/kustomization.yaml` entries; include those files in the same platform merge request. Keep sufficient execution capacity for both the parent and child run: the waiting parent still reserves its Pod resources. Cancellation and cleanup must account for the independently created child run. +The registration step proposes Git changes instead of directly applying Argo CD resources. Review the entire diff, including RBAC and destination changes. Submit one platform merge request that adds the declaration, its `configMapGenerator` entry in `projects/kustomization.yaml`, the project's onboarding RBAC and ScheduledTrigger files together with their entries in `onboarding/kustomization.yaml`, and the project's production repository in the image-verification policy. Without the `configMapGenerator` entry the dispatcher waits for the project's ConfigMap and then skips the project. Keep sufficient execution capacity for both the parent and child run: the waiting parent still reserves its Pod resources. Cancellation and cleanup must account for the independently created child run. The pipeline runs in the management namespace with administrator Connectors. Permission to create Pods or PipelineRuns there is effectively access to those credentials, even without API permission to read Secrets. Limit that namespace and the onboarding identity to platform automation and administrators. @@ -329,7 +320,9 @@ Use repeated runs of the same supported parameters for reconciliation. Do not eq The sample's supported GitLab subgroups are `apps` and `gitops`. Changing that convention requires a reviewed pipeline change, rather than only a declaration edit. Tool-side identity changes can recreate accounts or credentials; inspect the product Task's contract before changing names or project sets. -Do not overlap runs for the same tenant. The sample has three possible entry points (scheduled reconciliation, merge dispatch, and a manual Run Template) and does not implement a distributed lock; enforce the no-overlap rule operationally or add a lock before using it for concurrent onboarding. Arrange rotation outside active pipeline windows and monitor failed runs before the current credential expires. CSI workload-token issuance is separate from rotating the tool's long-lived credential. Consult the Task guide for expiry recovery; define a local recovery procedure rather than depending on an unobserved natural-expiry path. +Do not overlap runs for the same tenant. The three sample entry points—scheduled reconciliation, merge dispatch, and the manual Run Template—use the same onboarding Pipeline. Its first task atomically creates a per-project ConfigMap lock before any tenant changes; a competing run fails. A successful run releases its own lock in a final task. A failure, cancellation, or interrupted cleanup can leave the lock in place. Inspect its recorded PipelineRun and any independently started child runs, confirm that all work has stopped, then remove the lock before retrying. Never clear a lock just because the run is taking longer than expected. + +Arrange rotation outside active pipeline windows and monitor failed runs before the current credential expires. CSI workload-token issuance is separate from rotating the tool's long-lived credential. Consult the Task guide for expiry recovery; define a local recovery procedure rather than depending on an unobserved natural-expiry path. If enabling Connector approval, prepare grants for every affected workload, enable `enable-connectors-approval`, immediately create the required AccessPolicies, and then enable `enable-connector-proxy-permissions`. The sample's optional administrator AccessPolicy is excluded from the default Kustomization and is applied only after the feature is enabled. Use RBAC for the documented cross-namespace grants and keep approval-gated production Connectors free of direct proxy grants. See the [delivery article](./From_Commit_to_Production_on_ACP.md#gate-the-production-connector-after-approval) for the complete rollout boundary. From 19871f3b8467b51ecc174f7efced25c608dd86e9 Mon Sep 17 00:00:00 2001 From: Zichen Yu <1062955096@qq.com> Date: Thu, 22 Oct 2026 16:19:48 +0800 Subject: [PATCH 5/5] docs: restore DevOps decision guidance --- .../Alauda_DevOps_Best_Practices_Overview.md | 273 ++++++++-- .../From_Commit_to_Production_on_ACP.md | 432 +++++++++++++++- ...Projects_with_Argo_CD_and_Tekton_on_ACP.md | 475 +++++++++++++++++- 3 files changed, 1092 insertions(+), 88 deletions(-) diff --git a/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md b/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md index 53134b870..28ec3a8b1 100644 --- a/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md +++ b/docs/en/solutions/Alauda_DevOps_Best_Practices_Overview.md @@ -10,23 +10,48 @@ ProductsVersion: # Alauda DevOps Best Practices: Overview and Scenario Guide -This series helps application and platform teams combine Alauda DevOps capabilities into a delivery workflow. Start with a working pipeline, connect build outputs to controlled releases, and then standardize the platform for multiple isolated projects. +A team starting with one repository usually needs a build and a test result first. +Once that works, the questions change: how does the tested image reach production, +who approves it, and what must the platform prepare before another team can use the +same process? This series works through those questions in that order. It explains +how to choose and combine capabilities, then points to the product instructions +and reference files needed to implement the choice. -The configuration baseline is Alauda DevOps Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and Alauda Container Platform GitOps 4.4 on a compatible ACP installation. Use the compatibility requirements of each component when selecting packages. Examples use GitLab, Harbor, Nexus, and SonarQube. They describe a reference implementation, with environment preparation and acceptance checks required before production use. +The configuration baseline is Alauda DevOps Pipelines 4.15, Artifact Hub Shim 1.0, +Connectors 1.14, and Alauda Container Platform GitOps 4.4 on a compatible ACP +installation. Follow each component's compatibility requirements when selecting +packages. Examples use GitLab, Harbor, Nexus, and SonarQube. The reference design +requires environment preparation and acceptance checks before production use. -The series has three entry points: +## How to use this guide -| Your starting point | Read next | -| --- | --- | -| You need a build or test pipeline for one repository. | [Start a pipeline](#start-a-pipeline) below. | -| A pipeline already runs; you need a traceable path from a commit to production. | [From Commit to Production on ACP](./From_Commit_to_Production_on_ACP.md). | -| You operate a shared platform and need repeatable configuration and project onboarding. | [Managing Platform Configuration and Onboarding Isolated Projects](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md). | +Choose an entry according to the problem you have now: -The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) contain the platform repository, project declaration, onboarding pipelines, GitOps repository template, application delivery examples, and acceptance scripts. Replace example addresses, image references, identities, and encrypted credential placeholders before using them. +| Your starting point | What you need to decide | Read next | +| --- | --- | --- | +| You have one repository, or a simple job without a repository, and want a working pipeline. | Where to keep the definition and how to start it. | [Start a pipeline](#start-a-pipeline). | +| A pipeline already runs, but build, test, and release are separate manual activities. | How to carry the same image through environments and where to introduce release controls. | [Connect the delivery stages](#connect-the-delivery-stages), then [From Commit to Production on ACP](./From_Commit_to_Production_on_ACP.md). | +| Several teams will share the platform, and every project requires repeated administration. | Which settings and responsibilities to standardize before handing the platform to projects. | [Prepare a shared platform](#prepare-a-shared-platform), then [Managing Platform Configuration and Onboarding Isolated Projects](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md). | + +Each scenario below explains when a capability helps, how it fits with the others, +and where to continue. The delivery and platform articles develop the choices, +configuration points, and expected results. For a guided implementation of the +application workflow, use the [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html). + +The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) +contain a platform repository, project declarations, onboarding pipelines, a +project GitOps template, application CI/CD, and acceptance scripts. They are +reference files to adapt. Replace example addresses, images, identities, and +encrypted credential placeholders before use. The articles explain the decisions +and boundaries so that reading the files is not a prerequisite for understanding +the design. ## The overall design -The delivery layer moves one change from source to an immutable image and then to an environment. The platform layer supplies reusable definitions, tool access, namespaces, and operating settings. The control layer decides what may run, who approves a release, and what evidence must be retained. +The design has three layers. The **delivery layer** follows one change from source +to a running application. The **platform layer** provides the definitions, tool +access, and project environments that make that path repeatable. The **control +layer** places authorization and evidence checks at specific points on the path. ```mermaid flowchart TB @@ -57,74 +82,214 @@ flowchart TB Git -.-> Connectors ``` -Application developers own source and application delivery configuration. Platform administrators own shared components, admission rules, administrator credentials, and onboarding. Release approvers authorize a selected digest after examining its build and test evidence. Separate the release identity from the build identity. +| Layer | What belongs here | Who makes the decisions | +| --- | --- | --- | +| Delivery | Source, triggers, build and scan results, image digests, test deployment, and production changes. | Application developers and project maintainers. | +| Platform | Component settings, versioned catalog content, administrator Connectors, encrypted credentials, project declarations, and deployment boundaries. | Platform administrators. | +| Control | Required merge checks, release approval, promotion access, image admission, and retained audit evidence. | Platform administrators and designated release approvers. | + +The layers have different reconciliation owners. Argo CD reconciles the objects +tracked in the platform or project GitOps repository. Onboarding creates or +reconciles namespaces, project Connectors, and external tool tenants; Argo CD does +not automatically manage those external operations. ACP creates the namespace +with the same name as its project, which onboarding adopts. Assign one owner to +each resource or field so that two controllers do not repeatedly undo each other. + +Separating build and release identities also matters. A developer can produce a +candidate image; the release decision authorizes a particular digest after its +build and test evidence is examined. A signature, approval, and production Git +change each establish a different part of that record. ## Start a pipeline -Choose the simplest definition model that meets the team's needs. The models can coexist. - -| Model | Definition and trigger | Use it when | +For a single application, first decide who should maintain the definition and +whether definition changes need the same review history as source changes. A +simple manually operated job can start in the cluster. A team that wants +merge-request checks benefits from PaC. When several projects share the same +workflow, centralize its implementation and let projects supply inputs. + +| Model | Definition and execution | Choose it when | What you take on | +| --- | --- | --- | --- | +| Pipeline in the cluster | Maintain a Pipeline in a project namespace; save parameters, Workspaces, and Integrations in a Run Template. | A small number of jobs need a straightforward entry, including jobs without a code repository. | Definition history and permission control are maintained through the platform process. | +| Pipelines as Code (PaC) | Keep PipelineRun definitions in `.tekton/`; repository events start runs and report results. | Definitions should evolve with source, and reviewers need checks on a merge request. | Repository access, webhook delivery, definition provenance, and credentials available to proposed source must be governed. | +| Centrally managed template | Publish a versioned catalog Pipeline, or maintain a Pipeline in a controlled namespace; projects reference it. | Multiple projects need a shared process with deliberate upgrades. | The platform owns compatibility and published versions; projects still own their inputs and adoption. | + +These choices can coexist. PaC can reference the central Pipeline through a +resolver, leaving only trigger annotations and project inputs in `.tekton/`. +A Run Template can expose that same centrally maintained workflow to manual, +event-driven, and scheduled execution. Central reuse therefore does not require +giving up the repository's review experience. + +Start with [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html), +[Configure GitLab Repository](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/guides/configure_gitlab_repo.html), +[Configure GitHub Repository](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/guides/configure_github_repo.html), +or [Centralized Management with Hub Pipeline and Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/hub_pipeline_run_template_centralized_management.html). +The delivery article explains how the two event routes connect to these definitions. + +### Choose how work starts + +| When work should start | Entry | Typical reason | | --- | --- | --- | -| Pipeline in the cluster | Create a Pipeline in a project namespace. Save parameters, Workspaces, and Integrations in a Run Template. Start manually, through Triggers, or on a schedule. | A small number of pipelines need a straightforward execution entry. | -| Pipelines as Code (PaC) | Keep PipelineRun definitions in the application's `.tekton/` directory. Code events start runs and report results to the merge request. | Definitions should evolve with source and participate in merge-request checks. | -| Centrally managed template | Maintain a versioned Pipeline in the catalog, or a Pipeline in a controlled namespace. Projects supply parameters through a Run Template or a PaC PipelineRun using a resolver. | Multiple projects should share the same standard workflow. | - -For a PaC retry from a merge request, comment `/test branch:` on the merge request. For a push-triggered run, comment the same form on the commit at the head of that branch. The `branch` value makes the source explicit. Refer to [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html) for the complete syntax. Use one event route for each event: PaC or Triggers. Enabling both routes for the same event can start duplicate work. - -Start with [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html), [Configure GitLab Repository](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/guides/configure_gitlab_repo.html), or [Centralized Management with Hub Pipeline and Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/hub_pipeline_run_template_centralized_management.html). The [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html) provide a guided implementation. - -For systems that do not use Git events, an Incoming Webhook can start a PipelineRun. Send an HTTP `POST` with a JSON body containing `repository`, `namespace`, `branch`, `pipelinerun`, and `secret` to the PaC controller's `/incoming` endpoint. For example: +| A person requests it | Console or rendered PipelineRun from a Run Template. | First validation, a regression run, or a release decision. | +| Code changes | PaC, or Triggers with a Run Template. | Validate a merge request or build after a merge. | +| An artifact event arrives | Harbor event trigger. | Hand an image digest to a later stage. | +| An interval elapses | ScheduledTrigger. | Reconcile tool resources, rotate credentials, or run a periodic check. | +| Another system requests it | Incoming Webhook. | A change-management system or an internal portal starts work. | + +Use one route, PaC or Triggers, for the same code event. Both can create runs, so +connecting both to that event can duplicate the work. Prefer PaC when the +repository review experience is central; use Triggers when the platform owns the +event mapping and Run Template. A schedule is useful for reconciliation because +it reuses the same inputs and execution definition as a manual recovery run. + +For a PaC retry from a merge request, comment +`/test branch:` on the merge request. For a push-triggered +run, comment the same form on the commit at the head of that branch. The explicit +branch makes the requested source clear. The release-candidate `ci` sample has an +additional runtime restriction described below; it intentionally rejects comment +retries. See [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html) +for the complete syntax. + +For an Incoming Webhook, send an HTTP `POST` with a JSON body to the PaC +controller's `/incoming` endpoint. The five fields identify the Repository, +namespace, branch, PipelineRun definition, and shared secret: ```bash +# PAC_CONTROLLER_URL is the platform-provided controller endpoint. curl -i -X POST "$PAC_CONTROLLER_URL/incoming" \ -H 'Content-Type: application/json' \ -d '{"repository":"","namespace":"","branch":"main","pipelinerun":"","secret":""}' ``` -Treat the call as accepted only when it returns HTTP `202` and a new PipelineRun appears. An HTTP `200` response with an empty body can mean that no run was triggered, for example because the secret did not match. The controller URL, route exposure, and secret are platform prerequisites; see [Incoming Webhooks](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/incoming_webhooks.html). +Treat the request as accepted only when HTTP `202` is returned and a new +PipelineRun appears. An empty HTTP `200` can mean that no run matched, for example +because the secret did not match. Keep the secret in the caller's credential +management. Controller configuration, route exposure, and authentication are +platform prerequisites; see [Incoming Webhooks](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/incoming_webhooks.html). ## Connect the delivery stages -| Requirement | Recommended combination | Detailed guidance | -| --- | --- | --- | -| Build a traceable artifact | Reuse catalog Tasks, scan the output, and pass its digest to later stages. | [From Commit to Production](./From_Commit_to_Production_on_ACP.md). | -| Deploy to development or test | Update the workload by digest directly when Git history of environment state is unnecessary. | [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html). | -| Deploy to production | Review a change to the production GitOps repository; Argo CD reconciles the selected digest. | [Alauda Container Platform GitOps](https://docs.alauda.io/gitops/4.4/). | -| Separate production artifact access | Approve and copy the selected image by digest into a production registry project. | [Promote Artifacts with Skopeo Copy](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/promote_artifacts_with_skopeo_copy.html). | -| Record who approved a release | Add an ApprovalTask, use group resource names, and retain its approval response. | [Manual Approval Gate](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/manual_approval_gate.html). | -| Verify image signatures | Configure Chains keys and registry access centrally; verify the image by digest before promotion and at deployment admission. | [Signing Key Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | -| Hand off an artifact event | Map a Harbor event to the next pipeline, carrying the digest. Check scan and signing evidence separately. | [Use Harbor Event Triggers](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | +Once CI runs, the next problem is preserving the relationship between what was +built, what was tested, and what is released. Pass `repository@sha256:` +between stages. A tag helps someone recognize a release, but moving the tag does +not change the digest recorded in an environment's deployment configuration. -Promotion copies a selected image; it does not establish that the image passed CI. The approver must match the digest with a successful trusted build and the test environment. For the sample, a trusted CI result is the `ci` PipelineRun for a `push` event on `main`; the sample rejects comment-triggered `ci` runs before building, and uses a separate definition for merge-request validation. The project credential still needs a separate scratch registry boundary to prevent proposed code from writing release candidates. Signatures establish the configured signer's statement, not which pipeline produced the image; their meaning also depends on control over the signing key and the definitions allowed to run. +For development or test, a direct workload update can be sufficient. Production +often needs a reviewed record of desired state, which makes a GitOps change a +useful deployment entry. Separating the production artifact project adds another +decision: who may copy an approved candidate into that project? Add each control +for the decision it serves. -The samples deliberately keep test CD, promotion, and production CD as manually started runs. Their inputs come from the preceding run's digest result. Harbor-driven automation is a separate integration step. The first production deployment waits for a promoted image; after that image is promoted, follow the same reviewed production GitOps path for later releases. +| Situation | Combination and reason | Continue with | +| --- | --- | --- | +| One commit must produce a traceable, tested image. | Reuse catalog Tasks for build and scan, then retain the resulting digest and run evidence. | [From Commit to Production](./From_Commit_to_Production_on_ACP.md). | +| Test should update quickly and Git history of its state is unnecessary. | Deploy the digest directly with an authorized workload or Helm Task. | [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html). | +| Production state must be reviewed and recoverable from Git. | Update the production GitOps reference through a merge request, then let Argo CD reconcile it. | [GitOps](https://docs.alauda.io/gitops/4.4/). | +| Build and production registry access must be separate. | Copy the selected digest to a private production project with a release identity. | [Promote Artifacts with Skopeo Copy](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/promote_artifacts_with_skopeo_copy.html). | +| A person must authorize the release and leave a record. | Add an ApprovalTask and retain its responses; configure a separate Git review where required. | [Manual Approval Gate](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/manual_approval_gate.html). | +| Deployment must check a platform-controlled signer. | Configure Chains centrally and verify the digest before release and at admission. | [Signing Key Configuration](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/configure/signing_key_configuration.html). | +| The registry should hand an image to a later stage. | Carry the Harbor event's digest into the next run and check its CI evidence. | [Harbor event triggers](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/harbor_events.html). | + +An image push event establishes that an image arrived. Promotion establishes that +bytes were copied through a selected path. Neither establishes that CI passed. +The approver must correlate the digest with a successful trusted build, scanning, +and the test deployment. A signature establishes the configured signer's +statement; its meaning depends on who controls the key and which definitions and +source are allowed to use the signing path. + +### What the delivery sample chooses + +The sample uses GitOps for both test and production so that test exercises the +same synchronization path as production. Test changes commit directly; production +changes propose a merge request. It uses separate source and production registry +projects, with a separate release identity and an approval node before copying. +These are example choices to compare with your own review and artifact-access +requirements. + +Test CD, promotion, and production CD are manually started and take the preceding +stage's digest. This makes every handoff visible while establishing the workflow. +Harbor-driven automation can be added after event authentication, parameter +mapping, and CI-result correlation are configured. The first production deployment +waits for the first promoted image and its reviewed production GitOps change. + +For this sample, a trusted CI result is `ci` for a `push` on `main`. A pre-build +check reads the actual PaC event and branch annotations and rejects comment-triggered +`ci` runs before building, including comments on `main`. Merge requests use a +separate definition and scratch image repository. The shared project credential +still covers both repositories; a separate scratch registry project and restricted +credential are needed to prevent proposed source from writing release candidates. ## Prepare a shared platform -Platform settings are administrator decisions. Roll them out in stages and check [Feature Maturity](https://docs.alauda.io/alauda-devops-pipelines/4.15/overview/feature_maturity.html) for the selected release. - -| Platform concern | Recommended approach | -| --- | --- | -| Tool access | Use project-scoped Connectors. Proxy access uses short-lived workload tokens. Manage image pull Secrets, PaC clone credentials, and Chains registry credentials as separate long-lived credential responsibilities. A project owner who can create Pods in a credential-bearing namespace can use the credentials in that namespace. | -| Project onboarding | Orchestrate the four Connectors automatic creation Tasks, then seed a project GitOps repository and propose platform registration through a merge request. Complete the administrator prerequisites and acceptance checks. | -| Platform configuration | Keep component settings, policies, encrypted administrator credentials, and project declarations in a platform repository. Assign one owner to each resource or field. | -| Namespace isolation | Use separate project namespaces and tool-side tenants. Separate build, delivery, and workload namespaces and restrict access to administrator namespaces. | -| Execution policy | Use platform ClusterPolicies and project Policies. Restrict direct TaskRun and CustomRun creation. Optionally require approved resolver types and coordinates. Move from Audit to Enforce after checking intended workflows. | -| Cleanup | Select one cleanup mechanism. For event-based Pruner, review global defaults and namespace override settings before enabling deletion. A global first sync can affect every namespace, with successful and failed history counted separately; inventory tenants and notify owners before changing the defaults. | -| Audit history | Store run records and logs in external Results storage with an explicit retention period. Export approval evidence separately before deleting its objects. | -| Scheduled work | Use ScheduledTrigger or CronJob for reconciliation and rotation, with failure alerts and no overlapping runs for a tenant. The sample uses an atomic per-project lock; operators must inspect failed or cancelled runs before clearing a retained lock. | - -Results Watcher scope deserves a separate rollout decision. Its default watch scope can affect finished runs across the cluster and add `results.tekton.dev/*` finalizers. Scope it before enabling it. Disabling archiving does not remove existing finalizers; follow the product recovery procedure before cleanup. After Pruner removes cluster objects, the external Results store is the audit source. +As more teams join, repeated administration becomes an ownership problem. The +platform must decide which definitions, tool resources, credentials, policies, +and retention settings each project receives. Standardize those decisions before +onboarding, then let projects choose their application-specific workflow within +the established boundary. -The platform repository is the durable, reviewable source for configuration and project boundaries. It makes changes auditable and repeatable, but it is not a backup for sealed-secrets keys, signing keys, external tool data, or the Results store. Keep those dependencies in a separate recovery plan. Reverting a registration commit can prune an ACP project and its same-named namespace, so treat that action as decommissioning rather than an ordinary rollback. +| Platform situation | Approach and reason | Detailed guidance | +| --- | --- | --- | +| Pipelines need tools without distributing administrator credentials. | Use project Connectors and proxy workload tokens; track the remaining long-lived credentials separately. | [Connectors Proxy](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_proxy.html) and [CSI Driver](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors/concepts/connectors_csi.html). | +| Each project needs resources in four external tools. | Orchestrate the four Connectors automatic creation Tasks from a reviewed declaration; assign one reconciliation owner per tenant. | [Onboarding and product Tasks](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md#reuse-the-product-tasks). | +| Platform settings exist only in a running cluster. | Keep component settings, policies, encrypted credentials, and project declarations in a platform repository. | [Platform configuration](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md). | +| Teams must not change one another's environments or credentials. | Separate namespaces and tool tenants, constrain deployment identities, and verify effective access. | [Project boundaries](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md#accept-the-project-boundaries). | +| Projects need restrictions beyond the shared baseline. | Combine platform ClusterPolicies with project Policies; restrict direct TaskRun and CustomRun creation and optionally constrain definition sources. | [Pipeline execution constraints](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/constrain_pipeline_execution_with_kyverno.html). | +| Cluster history must be cleaned while audit evidence remains available. | Select Pruner settings and external Results storage together, with separate approval-evidence retention. | [Audit and recovery](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md#preserve-audit-records-and-plan-recovery). | +| Credentials need periodic reconciliation or renewal. | Schedule the same onboarding workflow with failure alerts and a per-project concurrency lock. | [ScheduledTrigger](https://docs.alauda.io/alauda-devops-pipelines/4.15/triggers/how_to/scheduledtriggers.html) and [reconciliation](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md#reconcile-and-rotate-deliberately). | +| Another system initiates work. | Expose an authenticated Incoming Webhook and define what acceptance and completion mean to the caller. | [Incoming Webhooks](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/incoming_webhooks.html). | + +Proxy workload tokens do not remove every long-lived credential responsibility. +Image pull Secrets, PaC clone credentials, and Chains registry credentials still +need scoped access, rotation, and recovery. A project owner who can create Pods +in a credential-bearing namespace can use those credentials. Administrator, +release, and signing namespaces therefore need explicit run-creation boundaries. + +Cleanup and archiving also have shared consequences. Event-based Pruner defaults +can affect every namespace on the first synchronization, with successful and +failed history counted separately. Inventory tenants and their retention needs +before enabling deletion. Results Watcher can add `results.tekton.dev/*` +finalizers to finished runs across its watch scope; define that scope before +enabling it. Disabling archiving does not remove existing finalizers. Follow the +product recovery procedure before cleanup, and verify access to the external +Results store that remains after live objects are pruned. + +The [platform article](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) +explains the 18 decisions, their repository layers, onboarding outputs, +permissions, rotation, and acceptance. These are administrator-managed settings; +check [Feature Maturity](https://docs.alauda.io/alauda-devops-pipelines/4.15/overview/feature_maturity.html) +and introduce them in stages. The platform repository provides durable +configuration and a review trail. Keys, external tool data, and Results storage +need separate backups. Reverting a project registration can prune the ACP project +and its same-named namespace, so review that operation as decommissioning. + +## Choose an adoption path + +| What the team needs now | Start with | Add when the requirement appears | +| --- | --- | --- | +| A small number of jobs. | Cluster Pipeline and a Run Template. | PaC for source review; central templates when reuse becomes worthwhile. | +| A traceable application delivery path. | Build and scan, pass digests, and choose deployment per environment. | Production Git review, approval, artifact promotion, and admission according to the release boundary. | +| A platform for isolated projects. | Shared baselines, administrator prerequisites, and a reviewed project declaration. | Scheduled reconciliation, deliberate rotation, policy enforcement, and retained audit evidence after acceptance. | -The [platform article](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) explains the 18 platform decisions, repository layout, ownership boundaries, onboarding workflow, rotation, and acceptance checks. +Each row can be adopted incrementally. A team can keep a simple job in the +cluster while its main application uses PaC and a shared catalog Pipeline. Release +controls should answer the environment's authorization and evidence requirements. +Adding a control also adds a platform owner, prerequisites, and an operational +recovery responsibility. ## Operate without internet access -Make the catalog, pinned tool and base images, package dependencies, certificates, and vulnerability databases available through the platform's approved internal services. Runtime pipelines must resolve Tasks and Pipelines from local catalog or cluster sources. The sample CI configuration points Trivy at `registry.example.com/mirror/trivy-db:2`; replace it with a reachable database mirror and grant the scanner access. - -The degree of network isolation, number of teams, and required approval evidence are deployment inputs that the platform team must decide. Capacity sizing, migration and decommissioning automation, platform disaster recovery, customer change-management integration, and offline package distribution procedures are outside this series. Define those procedures for the target environment. +Make the catalog, pinned tool and base images, package dependencies, certificates, +and vulnerability databases available through approved internal services. Runtime +pipelines must resolve Tasks and Pipelines from local catalog or cluster sources. +The sample points Trivy at `registry.example.com/mirror/trivy-db:2`; replace it with +a reachable database mirror and grant the scanner access. + +The degree of network isolation, number of teams, and required approval evidence +are deployment inputs for the platform team. Capacity sizing, migration and +decommissioning automation, platform disaster recovery, customer change-management +integration, and offline package distribution procedures require their own +operating plans. This series provides the choices and configuration boundaries +from which to develop those plans. ## Related information diff --git a/docs/en/solutions/From_Commit_to_Production_on_ACP.md b/docs/en/solutions/From_Commit_to_Production_on_ACP.md index 1d75f89ae..6af872e2a 100644 --- a/docs/en/solutions/From_Commit_to_Production_on_ACP.md +++ b/docs/en/solutions/From_Commit_to_Production_on_ACP.md @@ -10,18 +10,57 @@ ProductsVersion: # From Commit to Production on ACP: Choosing Triggers, Deployment, and Release Controls -Use this article to choose where pipeline definitions live, how deployments consume build outputs, and where approvals, promotion, and signature checks belong. For a guided implementation, follow the [Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html). +After a change is merged, a team usually wants it built, checked, and deployed to +test without someone passing image names between systems. Production introduces +further questions: which tested image should be released, who may authorize it, +and where should the desired production state be recorded? -A build that passes is only the start of delivery. The next questions are which image test actually ran, who authorized that image for production, and how the production deployment can be traced back to the same change. Work through those questions in order: choose the definition and trigger, choose the deployment route for each environment, then add the release controls required by your organization. A small project can start with a manual run; central templates and additional controls can be introduced as the workflow grows. +Use this article when those questions lead to choices about pipeline ownership, +GitOps, approvals, or artifact promotion. It explains the alternatives and their +consequences, then follows the choices made by the companion sample. For a guided +implementation of the individual operations, follow the +[Application CI/CD tutorials](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/tutorials/index.html). + +By the end, you should be able to explain: + +- Where a repository's pipeline definition belongs, who maintains it, and how it is triggered. +- Why each environment uses direct deployment or GitOps, and how it receives the selected image digest. +- Whether production needs artifact copying, human approval, approval-gated credentials, and deployment admission. +- Where each control takes effect and which prerequisites the platform team must supply. + +Work through these decisions in that order. A small project can begin with a +manual run; source-based definitions, central templates, and release controls can +be introduced when their benefits justify their ownership and operating cost. This is the delivery part of [Alauda DevOps Best Practices](./Alauda_DevOps_Best_Practices_Overview.md). Its configuration baseline is Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and GitOps 4.4. Project namespaces, tool-side resources, and Connectors must already exist; the [platform article](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) explains their preparation. The [companion samples](https://github.com/AlaudaDevops/devops-best-practices-samples) use a fictional `shop` project. CI runs in `shop-build`; CD and promotion run in `shop-cd`; workloads run in `shop-test` and `shop-prod`. The sample binds the project owner group in `shop` only; applying the PaC Repository, CD and promotion objects, and starting their runs requires a platform administrator or a separately delegated grant in `shop-build` and `shop-cd`. See the sample application's ["Who applies what"](https://github.com/AlaudaDevops/devops-best-practices-samples/blob/main/sample-app/README.md#who-applies-what) table. Replace every example address, tool image, group, credential placeholder, and policy identity for your environment. Make the vulnerability database, tool images, base images, and dependencies available locally for disconnected execution. +## Prepare the project and its responsibilities + +Start with a working project pipeline and access to its code and registry. The +platform team supplies project namespaces and Connectors; it also adds developers +to the GitLab tenant group. Onboarding creates the group but does not synchronize +its members. A developer without the required group role cannot create the +application repository. + +Keep builds in a separate namespace because the container build Tasks can require +a different Pod Security level from application workloads. The sample uses +`baseline` for `shop-build` and `restricted` for workload namespaces. See +[Built-in Tasks and the Restricted Standard](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/comply_with_pod_security_restricted.html#built-in-tasks-and-the-restricted-standard). +The delivery namespace provides a separate place to control release execution; +creating that namespace alone does not establish who may use its credentials. + +This article keeps the choice, prerequisites, minimal configuration, and evidence +for each handoff in the main text. Complete Task parameters and executable +PipelineRuns live in the catalog and companion sample. Platform-wide settings, +project provisioning, retention, and recovery are described in the platform +article. + ## Follow one image identity through the workflow ```mermaid -flowchart LR +flowchart TD Commit[Merge application change] --> CI[Build and scan] CI --> Source[Source registry image by digest] Source --> Test[Deploy and verify in test] @@ -32,7 +71,23 @@ flowchart LR MR --> Sync[Argo CD sync and rollout] ``` -Use `repository@sha256:` for deployment, promotion sources, signature verification, and audit correlation. Tags help people identify releases; a tag alone does not identify immutable content. A Harbor push event establishes that an image was pushed. It does not establish successful scanning, testing, signing, or approval. +The six decisions along this route are definition ownership, triggering, +deployment per environment, transfer of build results, release authorization, and +enforcement. Choosing a tool for one step does not settle the decisions at the +other steps. + +Use `repository@sha256:` for deployment, promotion sources, signature +verification, and audit correlation. Tags help people identify releases; a tag +alone does not identify immutable content. Record both the repository and digest: +the content identity can remain the same when promotion changes the repository, +but permissions and admission rules depend on that repository. + +A useful starting design is PaC for application events, catalog Tasks for the +work, direct deployment to test, and a reviewed GitOps change for production. +Add approval and artifact copying where the release boundary requires them. The +sample deliberately uses GitOps for test as well; the reason is explained below. +A Harbor push event establishes that an image was pushed. It does not establish +successful scanning, testing, signing, or approval. When a vulnerability is found in an already deployed image, fix it and use the same build, promotion, and deployment workflow to release a new digest. Keep the build evidence and the decision to replace the vulnerable image with the release record. @@ -44,7 +99,70 @@ When a vulnerability is found in an already deployed image, fix it and use the s | Pipelines as Code | PipelineRun files in the application's `.tekton/` directory | Push and merge-request events | Definitions versioned with source and merge-request checks | | Central template | Versioned catalog Pipeline, or a Pipeline in a controlled cluster namespace | Run Template or a PaC PipelineRun with `pipelineRef` | Shared standards across projects | -Start with a cluster Pipeline when that is sufficient. Add PaC when the change history and merge-request experience matter. Use central templates when projects should reuse a standard workflow. The same project can use all three for different jobs. Choose one route, PaC or Triggers, for a given event. +### Start with a cluster Pipeline when the job is simple + +A Pipeline in the project namespace is useful for a few jobs that the platform +operates directly, including work without a code repository. Save parameters, +Workspaces, and Integrations in a Run Template so that a person, an event, or a +schedule can invoke the same execution configuration. If the job needs source, +`git-clone` can obtain it through a Connector. + +The benefit is a straightforward entry and fewer repository resources to prepare. +The tradeoff is that definition review and change history belong to the platform's +configuration process. Keeping the Pipeline only in the live cluster provides no +Git review trail. This route can still build the same source and run the same +Tasks as PaC. + +### Use PaC when the workflow should evolve with source + +PaC keeps PipelineRun definitions in `.tekton/`, selects them for repository +events, and reports results to the code platform. This suits branch-specific +behavior and checks that reviewers should see alongside an application change. +The application team can propose definition changes through its normal review +process. + +That convenience brings repository access, webhook delivery, definition +provenance, and pre-merge credential exposure into scope. Select the definition +source deliberately; a reviewed definition can still execute proposed application +code. The runtime credential boundary is discussed below. + +### Publish a central template when projects share a process + +When many repositories implement the same build and scan sequence, separate +shared workflow maintenance from project inputs. The platform publishes a +versioned catalog Pipeline, while each project supplies its repository, image +location, and execution settings. A controlled cluster Pipeline is another option +when the platform owns its change process. + +The benefit is one maintained workflow with consistent behavior. The cost is an +explicit compatibility contract: the platform maintains versions and Tasks, and +projects review adoption of new versions. Publishing a new version does not +upgrade every consumer. A central template can be used from either a Run Template +or PaC; it is an ownership choice that can accompany either trigger route. + +### Choose one event route for each job + +The Application CI/CD tutorials start with a cluster Pipeline and Triggers. PaC +provides another route when source review is the natural entry. Both ultimately +create PipelineRuns: + +```mermaid +flowchart LR + Event[Git or Harbor event] --> Listener[EventListener and interceptor] + Listener --> Binding[TriggerBinding and Run Template] + Schedule[ScheduledTrigger] --> Binding + Binding --> RunA[PipelineRun] + Code[Git push or merge request] --> PaC[PaC Repository and event selection] + PaC --> RunB[PipelineRun from .tekton] + RunA --> Shared[Controlled Pipeline or catalog version] + RunB --> Shared +``` + +Use Triggers when the platform owns event filtering and parameter mapping. Use +PaC when definitions and checks should follow repository review. Choose one route +for the same event and job, since enabling both can duplicate the work. The +same project can keep simple jobs in the cluster, use PaC for its main application, +and reference a central template from both. For a retry from a merge request, comment `/test branch:` on the merge request. For a push-triggered run, comment the same form on the commit at the head of that branch. Keep the complete syntax in one place by linking to [Trigger PAC Pipelines](https://docs.alauda.io/alauda-devops-pipelines/4.15/pac/how_to/trigger_pipeline.html). @@ -79,7 +197,18 @@ For a self-managed GitLab instance, set `spec.git_provider.url` on every PaC `Re Set `spec.settings.pipelinerun_provenance: default_branch` where pre-merge runs must use reviewed definitions. This selects the definition, not trusted application code: a merge-request run still builds the proposed source with the credentials mounted by that definition. Limit those credentials and keep merge-request image outputs separate from release candidates. -The sample separates the two outputs: +### Why the sample combines PaC and a catalog Pipeline + +The sample keeps trigger annotations and project inputs in the application +repository and references `python-image-build-scan-deploy` version `0.3`. Reviewers +can see when work runs and where it writes, while the platform maintains the +build-and-scan implementation. The version reference makes adoption a reviewed +project change rather than an immediate fleet-wide rollout. + +Merge-request validation also pushes an image because the scanner reads from the +registry. It therefore needs a separate output location and a short retention +policy. Only the trusted post-merge path is intended to produce candidates for +test CD and promotion. The sample separates the two outputs: | Definition | Event | Image repository | Delivery use | | --- | --- | --- | --- | @@ -98,12 +227,43 @@ The platform and GitOps repository validation samples compose `git-clone` and `r ## Choose a deployment route per environment +Choose per environment rather than adopting one route merely because CI already +uses it. The main question is where the desired environment state should live and +who is allowed to change it. + | Route | How the selected digest reaches the environment | Tradeoff | | --- | --- | --- | | Direct deployment | A task such as `workload-apply`, `helm-upgrade`, or `run-script` updates the workload. | Fewer moving parts; Git does not hold a second copy of environment state. | | GitOps | A change updates the environment's image reference in Git; Argo CD reconciles it. | Reviewable environment state and history; requires a repository and a sync path. | -A useful starting point is direct deployment for development or test and GitOps for production. The samples use GitOps for both test and production so that the same reconciliation path can be used in both environments. +### Prefer direct deployment when a cluster update is sufficient + +Direct deployment suits development and test environments that should update +quickly and do not need a separate reviewed history of desired state. The pipeline +can update a workload with `workload-apply`, a Helm release with `helm-upgrade`, or +custom logic with `run-script`. Cross-cluster execution needs the appropriate +Kubernetes Connector and authorization. + +This route has fewer components to operate. Its tradeoff is that Git does not +hold a second copy of the environment's state; retain the run inputs and logs and +define how to recover or repeat a deployment. If someone edits the workload +later, this route alone does not continuously restore the intended state. + +### Prefer GitOps when desired state needs review and reconciliation + +GitOps gives an environment a versioned desired state and a review point before +Argo CD changes the cluster. This is useful for production and pre-production, +where operators need to answer which version was intended, who changed it, and +how to propose a return to an earlier version. + +The tradeoff is another repository and synchronization path to operate. A merged +change can still fail rendering, admission, image pulling, or rollout. Check the +selected revision and workload after synchronization. An emergency live edit can +also be reverted by self-healing; establish the Git-based recovery path before +relying on it. + +A useful starting point is direct deployment for development or test and GitOps +for production. These routes can coexist in one delivery design. For direct deployment, use a digest in `workloadContainers`; use a lowercase `workloadKind`, such as `deployment`. The run identity needs access to read and update the target workload. Consult the catalog Task for its rollout, timeout, rollback, and cross-cluster options. @@ -117,18 +277,95 @@ gitops/ └── prod/kustomization.yaml ``` -In the sample, test CD uses `submitMode: direct` and `kustomize edit set image`, which may rewrite the test overlay. Production CD uses `submitMode: pr` and changes only the `digest:` line with `sed`; the production overlay is seeded with the production registry repository. The first production deployment waits for a promoted image. - Enforce production Git review in the code platform. Use protected branches and, where supported, code-owner approval. Other options include a separately protected production branch or repository, or a reviewed server-side policy. Branch protection alone does not create directory-level protection. Remove direct production write access that would bypass the chosen review path. +### Why the sample uses GitOps for both environments + +The sample uses `envs/test` and `envs/prod`, each with its own Application and +automated synchronization, pruning, and self-healing. Test exercises the same +rendering and synchronization path that production will use. This consistency is +useful when establishing the workflow, even though a direct test deployment +would need fewer components. + +The review requirement differs by environment. Test CD commits directly with +`submitMode: direct`; production CD opens a merge request with `submitMode: pr`. +The production overlay already contains the production registry repository, so a +release changes only its digest. This keeps the release diff focused on the +content being selected. Test uses `kustomize edit set image`, which can rewrite +its overlay; production uses a narrowly validated `sed` change and verifies the +rendered result. + +The sample starts all three handoffs manually: test CD, promotion, and production +CD. This makes the inputs and identities visible while setting up the workflow. +Automatic Harbor event chaining requires additional authentication, mapping, and +CI-evidence checks; it is an extension to this sample. Adding another environment +also requires a project GitOps directory and validation path. Re-running +onboarding does not upgrade a non-empty repository with that directory. + ### Pass results between build, promotion, and GitOps -`gitops-repo-update` is a Pipeline and runs as a separate PipelineRun. Pass the selected digest as an argument to its update script. Use a fresh source Workspace volume per run and an image that contains every command used by the script. `submitMode: pr` also requires the configured GitLab CLI image and a token with API access. +Keep an explicit handoff contract. CI records an image repository and +`IMAGE_DIGEST`; promotion selects that immutable source and reports its destination +digest. The next stage must use the actual preceding result and the destination +repository, without substituting a mutable tag or an unrelated successful run. -The sample updates the production `digest:` line with `sed` and verifies the rendered image with Kustomize. This preserves a one-line production diff. Avoid using a rewrite operation that changes unrelated formatting or image fields in production. The GitOps writer is a group access token with Maintainer access in the sample; that identity can change branch protection and code-owner settings, so use a project-scoped least-privilege token or a separately protected production repository in a hardened deployment. Keep it separate from the application CI token. +| Handoff | Value passed | Check before continuing | +| --- | --- | --- | +| Trusted CI to test | Source repository plus `IMAGE_DIGEST`, as `repository@sha256:`. | The `push` run on `main` succeeded and its scan results match that digest. | +| Test to promotion | The tested source reference and a human-readable release label. | Test ran that exact image and the release record includes its application checks. | +| Copy to production CD | Destination repository plus `dst-IMAGE_DIGEST`; the sample promotion exposes `promoted-digest`. | The copy result and registry digest match the selected candidate. | +| Production CD to GitOps | The bare destination digest in the sample's fixed production overlay. | The merge request changes the intended environment, and Kustomize renders the complete production reference. | + +With direct deployment, pass the full reference into the deployment Task's image +parameter. With GitOps, start `gitops-repo-update` and pass the value to the update +script. It has its own status and identity; a successful build does not imply that +this later run or its merge request succeeded. Pass the selected digest as an +argument to its update script. Use a fresh source Workspace volume per run and an image that contains every command used by the script. `submitMode: pr` also requires the configured GitLab CLI image and a token with API access. + +The update script must validate the input and the rendered image before +committing. The sample's production script requires a SHA-256 digest and exactly +one digest field in its overlay, then changes only that field. Keep the GitOps +writer separate from the application CI token and restrict both its Git rights +and the namespace from which it can be used. The execution-rights section below +explains why the sample's Maintainer token needs additional protection. The sample starts test CD, promotion, and production CD manually from rendered PipelineRun YAML. Replace the template parameters, verify the image mapping, identity, and Workspaces, and submit with `kubectl create`. A console Run Template form exposes Pipeline parameters: in the sample, remove the empty third `updateArgs` entry, use the full `repository@sha256:digest` for test CD, and use the bare `sha256:digest` for production CD because the production repository is fixed by the seeded overlay. Restrict who may create PipelineRuns in the delivery namespace; a console form can expose parameters that are fixed in the rendered YAML path. Use [Run Template](https://docs.alauda.io/alauda-devops-pipelines/4.15/pipelines/how_to/run_template.html) for other execution entries. Harbor event automation is an additional integration, not part of the sample's manual path. +### Assign execution rights separately from repository ownership + +The application team maintains the CD and promotion definitions, but that does +not automatically grant permission to apply or run them. The sample's owner +binding is in `shop`; access in the build and delivery namespaces needs an +administrator or an explicit delegated grant. + +| Object or action | Namespace | Responsible party and permission | +| --- | --- | --- | +| Application PaC Repository and its access Secret | `shop-build` | Platform administrator, or a delegated maintainer with Repository and Secret write access. | +| Inspect CI results | `shop-build` | An authorized reader of PipelineRuns, TaskRuns, and the required logs. | +| Apply CD Run Templates and the promotion Pipeline | `shop-cd` | Administrator or delegated maintainer with the corresponding resource write access. | +| Start CD or promotion and inspect its result | `shop-cd` | Authorized initiator with PipelineRun creation and read access; assess the credentials reachable by that permission. | +| Approve promotion | `shop-cd` | A member of the configured approver group with approval permissions; console access also needs the optional ACP role binding. | +| Grant production Connector access or manage its AccessPolicy | `platform-connectors` | Platform administrator, independently of an application Pipeline editor. | + +The sample uses `shop-delivery` for GitOps updates and `shop-release` for +promotion. CI uses the namespace's default execution identity unless +`TektonConfig` configures another default. When changing that default, update the +sample's Connector bindings to match. A named Connector `get` grant records the +intended relationship but does not narrow wider ACP default read permissions. +Proxy authorization is a separate control when its feature flags are enabled. + +Permission to create a Pod or PipelineRun in `shop-cd` also exposes the same +namespace's GitOps writer Secret. Protect that namespace while the Secret remains +there. To delegate execution without handing over the writer, move the Connector +and its Secret to a protected namespace and authorize a controlled cross-namespace +mount. Production Git protection must also constrain the writer itself: the +sample's Maintainer group token can change protection settings. Use a writer that +cannot change those rules, directly write production, or merge its own production +change, together with a separately controlled production branch or repository +where needed. + +### Return to a previous production version + Production rollback follows the same review boundary. Rebuilds can produce different content, so select a retained, previously promoted digest and use this GitOps path: 1. Find the previous digest in the production Git history. @@ -142,7 +379,42 @@ Database and data migrations need an application-specific rollback plan. Rehears ## Select the release controls -Promotion, approval, credential gating, and image admission solve separate decisions: +A release can require several kinds of authorization. Decide which question each +control must answer before adding it: copying an artifact, approving a release, +releasing a credential, and admitting a workload are separate operations. + +### Decide whether to copy the artifact or update only its reference + +A separate production registry project is useful when build writers must not be +production writers, or when production content needs its own retention and access +policy. Copy the tested digest without rebuilding it, then update the production +Git reference. This adds a release identity, registry permissions, copy evidence, +and signature-storage responsibilities. + +If both environments intentionally consume the same repository, updating the +production Git reference can be sufficient. A repository allow-list then cannot +distinguish a test candidate from an approved production digest. Where that +distinction is required, add admission based on approved-digest evidence rather +than assuming that access to the shared repository represents release approval. + +### Separate release approval from access to its credentials + +An ApprovalTask records a person's decision and can stop the normal workflow +before copying. A production Git review authorizes the desired environment state. +An organization can choose either review point or require both; the sample +requires both so that the release decision and the production change remain +visible as separate records. + +Approval-gated Connector access is useful when the production credential owner +and the Pipeline owner are different teams. It lets the credential owner maintain +an independent approver set. It also introduces installation-wide proxy +permission prerequisites and a rollout sequence. If one trusted platform team +owns both, a dedicated release identity, protected credential namespace, and +promotion admission can provide a simpler starting point. An approval node alone +does not prevent another run from trying to use the same credential. + +Promotion, approval, credential gating, and image admission therefore address the +following decisions: | Decision | Recommended control | | --- | --- | @@ -155,6 +427,19 @@ Use the Core promotion admission rules in [Constrain Artifact Promotion with Kyv Promotion validates the source and destination mapping and copies bytes. It does not query CI to establish that the image passed. The approver must correlate the digest with the trusted build, scan, and test evidence. The sample validates that the image mapping is first, the source uses a SHA-256 digest, and the repositories and release-label format match the configured workflow. Optional signature mappings must refer to the same image digest. +### Why the sample keeps promotion and production CD separate + +The sample has two Harbor projects, `shop` and `shop-prod`. Promotion copies the +selected candidate to the production project after approval; a later production +CD run uses the resulting digest to propose a Git change. Copying makes the image +available for production. Merging and synchronizing the Git change selects it for +the environment. Inspect both outcomes before declaring a release complete. + +The production Connector lives in `platform-connectors`, where project members +must not be able to create Pods or runs. Standing read grants are always required; +the production AccessPolicy and its feature flags are optional until the platform +chooses approval-gated proxy access. Keep that choice explicit during acceptance. + ### Add an approval node Use ACP group resource names in `approvers`, rather than a group's display name or an individual's name. The following is a task fragment; replace the example group with the actual resource name: @@ -200,14 +485,53 @@ Set feature flags through `ConnectorsConfig`. For the full configuration and ver `defaultPermission` policies have authorization prerequisites and are not a substitute for every cross-namespace RBAC grant. A name-scoped `get` Role does not by itself satisfy a policy that requires namespace-wide reads. A `defaultPermission` policy applies only to identities with namespace-wide `get`; other identities can be silently skipped. The sample grants the onboarding identity namespace-wide `get` in the management namespace, while keeping proxy permissions limited to the named administrator Connectors; keep that broader read scoped to the management namespace. A Ready policy alone does not establish a working mount. Verify the actual identity, Pod events, mounted configuration, and AccessRequest result. +### Connect the release steps in their actual order + +The intended order is to identify the candidate and its CI and test evidence, +validate its source and destination, obtain approval, copy it, verify the +production signature and provenance, and propose the production GitOps change. +Where source signature verification is required, perform it before copying. +Waiting for approval must not silently change the candidate; keep the digest in +the run inputs and release record. + +Harbor automation can start a later run after a push or scan event, but it needs +four additional agreements: a reachable listener, authenticated callbacks, +TriggerBinding parameters that match the Run Template, and a check that the +selected digest belongs to a successful trusted CI run. A notification only +alerts an approver; it neither grants permission nor supplies that evidence. + +Incoming Webhooks provide an entry for another system. The caller still needs a +contract for run identification, completion, failure, and the evidence it retains. +Receiving the request and completing the release are separate events. + ## Sign and verify the image +### Choose what the signature should establish + +Use signatures when deployment must verify a statement from a platform-controlled +signer. Decide which statement matters: that the candidate was signed after a +build, that the copied production artifact has release provenance, or that its +digest was approved. A signature check alone establishes the configured signer's +statement. The trust in that statement depends on protection of the key, +execution identities, and allowed definitions. + +Verification before promotion can reject an unsuitable source before it enters +the production project. Verification at workload admission checks the image that +the cluster is being asked to run, including requests outside the normal release +path. Both can be used. The sample supplies deployment verification and separate +cosign acceptance checks; it does not add a source-verification task before its +copy step. + +### Prepare signing and registry access centrally + Configure Chains signing keys centrally and distribute only the public verification key to projects. Protect both the signing Secret and the ability to create Pods in its namespace. Application teams do not need access to the signing Secret. The sample stores signatures and attestations beside the image. Chains first uses matching registry credentials from the run's ServiceAccount. In the sample, the image-pull Secret also has push permission and is used by Chains, so it is not a read-only credential; separate a read-only workload Secret from a signing credential in a hardened deployment. Those credentials must be able to push to the image's registry project. Where they are absent, configure a centrally managed registry credential on the Chains controller through `TektonConfig`. Chains does not use a Connector's CSI proxy Workspace as its registry credential. The central credential is created by an administrator, outside Git. Limit it to the registry projects that Chains must read and sign. When onboarding a project, the platform team must add its source and production registry projects to that credential's access. Rotate it and restrict its Secret and namespace. Apply controller configuration changes in a maintenance window, preserving other container options already configured. +### Verify stored evidence independently + A successful PipelineRun is not proof of stored signatures. Inspect TaskRun signing annotations and events, and independently verify the image and its attestation. The acceptance object here is the image digest and the copy TaskRun's provenance; approval history is separate evidence. ```bash @@ -222,8 +546,22 @@ For an offline installation configured without a transparency log, use the produ If the platform chooses a separate signature repository, configure `storage.oci.repository` centrally and use the corresponding `COSIGN_REPOSITORY` in verification and the repository setting in Kyverno. Review storage and overwrite behavior before combining build and promotion attestations for the same digest. Copying build signature attachments during promotion is optional; the sample uses explicit `.sig` and `.att` mappings. For multi-architecture images, follow the Skopeo Copy Task contract and set `multiArch: all` when copying the index and its attachments; do not assume the single-image mappings cover an index automatically. +### Introduce deployment verification with positive and negative controls + At deployment admission, start the image-verification policy in Audit, with the sample's `mutateDigest: false` setting, and review the matching images and reports before Enforce. Kyverno also needs read credentials for private production registry projects through `imageRegistryCredentials.secrets`; otherwise a valid signature can be unreadable and Enforce will reject every production image. The sample's production image list is explicit: add each newly onboarded production repository to the policy. See [Image Signature Verification](https://docs.alauda.io/alauda-devops-pipelines/4.15/chains/how_to/image_signature_verification.html). +The signature policy selects explicitly listed production image repositories. +Images from another repository can fall outside that selection even in a +production namespace. Pair verification with a production-image-source and digest +restriction when that is the platform's release boundary; the sample does not +install that additional restriction. + +An attestation's Pipeline label can help detect selecting the wrong workflow, but +a name alone is not a unique execution identity: another namespace can contain a +Pipeline with the same name. The sample's SLSA v0.2 provenance does not establish +the namespace through that label. Protect the signing path and admission inputs +rather than treating the label as proof of a particular platform-owned Pipeline. + ## Check the controls at their actual enforcement points | Point in the workflow | What is checked | Remaining responsibility | @@ -240,8 +578,70 @@ Use platform ClusterPolicies for the baseline and project Policies for additiona The sample's resolver and bare-run policies exclude `platform-connectors`, where onboarding uses administrator Connectors. Keep Pod and PipelineRun creation in that namespace limited to platform administrators and the platform ServiceAccount; the policy exclusions do not create that boundary. +Roll policies out with evidence from a disposable project. Audit establishes +which requests would match; Enforce is needed to demonstrate rejection. Verify +allowed requests first, then disallowed ones, so that a missing registry credential +or a Pod Security rejection is not mistaken for a successful signature test. +Prepare high availability and a local recovery procedure before enforcing a +webhook-dependent control. An unreachable webhook with `failurePolicy: Fail` can +block matching writes; an orderly controller shutdown can remove webhook +registration, so do not assume every form of controller unavailability has the +same admission result. + +### Match delivery steps to components and tools + +The workflow combines product capabilities rather than placing every function in +one component. Tool access normally uses project Connectors; PaC's clone Secret, +image-pull configuration, and Chains registry credentials have separate lifecycles. + +| Delivery step | Capability and tool | Configuration or evidence owner | +| --- | --- | --- | +| Receive a push or merge request | PaC or Triggers with GitLab/GitHub. | Platform prepares the route; the project selects events and required merge checks. | +| Obtain source | `git-clone` with a Git Connector, or PaC-provided clone credentials. | Project supplies the repository and revision; platform governs credential scope. | +| Obtain dependencies | Maven, npm, or PyPI configuration through the relevant Connector and Nexus. | Platform supplies the mirror and CA; the build must fail if required mirror configuration is absent. | +| Check code quality | A SonarQube scan Task with its project Connector. | Project chooses the applicable quality gate and retains the scan result. | +| Build, push, and scan an image | Catalog build/scan Tasks with Harbor or another OCI registry. | Platform supplies Tasks, tool images, registry access, and the offline vulnerability database. | +| Hand off a registry event | Harbor EventListener, interceptor, and TriggerBinding. | Platform authenticates and maps the event; the workflow checks the digest's CI evidence. | +| Update an environment | `workload-apply`, `helm-upgrade`, or `run-script`, or Argo CD synchronization. | Execution identity needs the target authorization; project verifies the rollout. | +| Propose a GitOps change | `gitops-repo-update`, using Git and GitLab MR creation for `submitMode: pr`. | Project supplies the update script; platform protects the writer and production review path. | +| Approve and copy | ApprovalTask, `skopeo-copy`, and optional Connector AccessPolicy. | Release approvers authorize the digest; the credential owner controls access. | +| Notify an approver | Optional catalog `send-mail` Task and SMTP. | Local integration supplies recipients, SMTP configuration, and credentials. | +| Sign and verify | Chains, cosign, and Kyverno admission. | Platform controls keys, registry access, trusted image scope, and policy rollout. | +| Retain evidence | Results storage and separate approval-record export. | Platform defines retention; the release owner correlates the records before cleanup. | + +For dependency setup, see +[Using Maven Connector as Maven Registry Mirror](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-maven/how_to/using-as-maven-registry-mirror.html). +Bind the relevant Connector's CA where the Task requires it. Direct workload +operations and custom application logic are covered by +[Manage ACP Application with run-script Task](https://docs.alauda.io/alauda-devops-artifacthub-shim/1.0/catalog/how_to/manage_acp_application_with_run_script.html). +Use the product links in each decision section for complete fields and supported +variants. + ## Accept the delivery workflow +### Identify the expected delivery records + +A completed release should connect a reviewed source revision, successful trusted +CI, scan and test results, an immutable image reference, the approval response, +the copy result, a reviewed production GitOps revision, and the actual production +rollout. Keep identifiers that join those records; a collection of unrelated +successful runs is insufficient. + +| Outcome | Record that demonstrates it | +| --- | --- | +| The candidate came from the intended change. | Source revision and trusted `push` run, including its exact image result. | +| Test exercised that candidate. | Test image reference, rollout, and application checks. | +| Production content was selected and authorized. | Approval responses, source/destination mapping, and promotion results. | +| Production desired state was reviewed. | The merge request, validation result, and merged Git revision. | +| Production runs the selected content. | Argo CD revision, rendered image, Deployment image, and completed rollout. | +| Required controls remain effective. | Independent signature/provenance checks and allowed/denied admission probes. | + +The sample's manual entries are useful during setup, but each operator must copy +the preceding result accurately. When automating those entries, preserve the same +handoff checks and release records rather than equating an event with readiness. + +### Run acceptance and inspect the remaining evidence + Run `acceptance/01-commit-to-production.sh` after the first promotion, production GitOps merge, and completed rollout. Use a disposable acceptance project with its own namespaces and Harbor production repository. `PROJECT` is required; inspect the remaining defaults and replace all environment-specific inputs. The script creates temporary admission probes in that project's production namespace. The table distinguishes its checks from the release evidence that you must inspect separately. | Check | Script coverage | Evidence to inspect separately | @@ -254,6 +654,14 @@ Run `acceptance/01-commit-to-production.sh` after the first promotion, productio | Admission reports or enforces the policy | First checks a signed production image, then an unsigned image; selects reports by probe UID and policy. | Audit reports demonstrate reporting. Claim enforcement only after the same controls pass in Enforce. Distinguish signature denial from Pod Security and registry-access failures. | | Approval is attributable | Checks the approval object's ownership chain and responses, including group members. | Save those responses, the approved digest, and related run identifiers in the release record before cleanup. | +### Adapt the reference to the target environment + +Replace the example project and group resource names, code and registry addresses, +cluster destination, tool images, CA sources, signing key, mirrored dependencies, +and credentials. Register new production repositories in verification policy and +central signing access. Review environment directories, AppProject destinations, +execution identities, and the actual Git platform's review capabilities together. + Before production, address the sample's explicit hardening work: restrict the GitOps writer's namespace, protect production Git changes, install the promotion Core rules, constrain production image sources, and limit direct cluster writes. Do not grant project members Pod creation in namespaces that contain administrator or release credentials. Establish local rollback and policy-service recovery runbooks. Continue with [Managing Platform Configuration and Onboarding Isolated Projects](./Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md) for the platform configuration, project declarations, and operating boundaries behind this workflow. diff --git a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md index a8d4d445d..c64144644 100644 --- a/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md +++ b/docs/en/solutions/Managing_Platform_Configuration_and_Onboarding_Isolated_Projects_with_Argo_CD_and_Tekton_on_ACP.md @@ -10,9 +10,30 @@ ProductsVersion: # Managing Platform Configuration and Onboarding Isolated Projects with Argo CD and Tekton on ACP -Use a platform Git repository to review and preserve shared configuration. This gives the platform team a durable, auditable, repeatable source, but it does not replace backups of sealed-secrets keys, signing keys, external tool data, or Results storage. Use an onboarding pipeline to create a project's namespaces and tool-side resources, then propose its deployment configuration through a merge request. Argo CD reconciles the objects owned by Git; the onboarding tasks reconcile the resources they create in external tools. - -This approach is useful when platform settings exist only in a running cluster, each new project requires repeated administration in four tools, or the team needs to explain who may change a shared boundary. Git provides a recorded starting point and a review trail. For tracked configuration, it also provides a way to inspect and revert a change. Keep the corresponding keys and external data recoverable separately: replaying YAML cannot recover them, and reverting a project registration can delete resources. +A platform team often starts by configuring components in the cluster and +creating tool accounts when a project asks for them. As the number of projects +grows, it becomes harder to answer which settings are intentional, how to create +the next project consistently, and what must be prepared in another environment. + +Use this article when platform configuration needs a durable reviewed source, +when opening a project repeats administration in GitLab, Harbor, Nexus, and +SonarQube, or when deployment and credential boundaries need to be explained +together. A platform Git repository records shared intent. An onboarding pipeline +turns a reviewed project declaration into namespaces and tool resources, then +proposes the project's deployment registration through a merge request. + +The two mechanisms have different ownership. Argo CD reconciles the objects and +fields owned by Git. The onboarding Tasks reconcile the external resources they +create. Neither mechanism replaces backups of decryption keys, signing keys, +external tool data, or Results storage. Reverting a registration can delete +resources, so recovery needs more than replaying or reverting YAML. + +By the end, you should be able to: + +- Separate platform configuration, shared templates, and project application state. +- Explain the 18 platform decisions, their owners, and their repository layers. +- Prepare the prerequisites, review a project declaration, and follow onboarding through its registration merge request. +- Verify the resulting project boundaries and plan reconciliation, credential rotation, and configuration recovery. This article is the platform part of [Alauda DevOps Best Practices](./Alauda_DevOps_Best_Practices_Overview.md). It assumes one ACP project and separate tool-side tenants per project. Its configuration baseline is Pipelines 4.15, Artifact Hub Shim 1.0, Connectors 1.14, and GitOps 4.4 on a compatible ACP installation. Teams sharing one namespace and one tool tenant need a different boundary model. @@ -28,7 +49,17 @@ Keep three kinds of repository separate: | Template repository and catalog | Versioned shared Pipelines and Tasks; initial project GitOps repository content | Platform team | | Project repositories | Application source and `.tekton/`; environment manifests and digest references in the project GitOps repository | Project team | -Run Templates are execution settings in project namespaces. They are distinct from versioned catalog Pipeline definitions. Keep application manifests out of the platform repository, and keep platform policy ownership out of application repositories. +This separation keeps three kinds of review understandable. A platform change +can alter many projects' constraints and needs platform review. A catalog version +changes a reusable workflow and needs compatibility and adoption decisions. An +application release changes the project's own desired state and belongs to the +project's release process. + +Run Templates are execution settings in project namespaces, distinct from +versioned catalog Pipeline definitions. They save parameters, Workspaces, and +Integrations for a particular use of a Pipeline. Keep application manifests in +the project GitOps repository. Projects may add namespaced policies, while the +shared baseline remains platform-owned. ```mermaid flowchart TB @@ -47,7 +78,37 @@ flowchart TB Seed --> ProjectArgo[Project Applications] ``` -Assign one owner to each resource or field. Argo CD self-healing applies to objects it tracks; it does not reconcile every object in the cluster or reverse external tool operations. Removing a declaration is not an automated decommissioning procedure. +The diagram describes ownership and handoffs. The post-merge PaC dispatcher +starts onboarding for changed declarations after waiting for their published +configuration; Argo CD does not itself trigger Tekton merely by synchronizing a +ConfigMap. Onboarding later proposes a separate registration change, which must +be reviewed before the project Applications take effect. + +Assign one owner to each resource or field. Argo CD self-healing applies to +objects it tracks; it does not reconcile every object in the cluster or reverse +external tool operations. Removing a declaration is not an automated +decommissioning procedure. + +### Decide what configuration recovery should provide + +Keeping configuration in Git offers a starting point for initialization, a review +trail, and a way to propose a return to earlier settings. It also lets a reviewer +inspect project destinations, owners, and platform policies together. These +benefits depend on explicit field ownership and a recoverable set of prerequisites. + +| Expected benefit | What this design provides | Operating consequence | +| --- | --- | --- | +| Initialize another environment consistently. | Reviewed component settings, policy, templates, and project declarations. | Install compatible controllers and restore credentials before synchronizing. | +| Explain a platform change. | A Git diff, review, and merged revision. | Restrict direct cluster writes and access to the platform repository. | +| Repair tracked drift. | Self-healing in layers where it is enabled and verified. | Operator-owned fields and credential rotations need their own ownership rules. | +| Return to previous configuration. | A reviewed change to earlier Git content. | Inspect pruning and external side effects before treating a revert as safe. | +| Reconcile project resources. | Repeated onboarding with the same supported parameters. | Account for token rotation, partial runs, and the project lock. | + +A registry, database, or identity system can retain state that is absent from +Git. Define whether recovery means restoring that state or creating a new tenant +with the same declared shape. This series provides the configuration and +onboarding path; a complete recovery runbook must be exercised in the target +environment. ## Prepare the platform before bootstrap @@ -76,6 +137,25 @@ For a disconnected environment, prepare local catalog content, pinned tool and b ## Decide the 18 platform baselines +Projects should not each decide how administrator credentials are exposed, where +audit history is kept, or which controllers may bypass a shared policy. Resolve +those matters before handing over the platform. Application-specific inputs and +additional restrictions can remain with the project. + +The sample groups the decisions into four Git ownership layers: + +| Layer | Configuration it owns | Reason for a separate layer | +| --- | --- | --- | +| Component | TektonConfig fragments, Chains options, Pruner, Results, and monitoring settings. | Shared controllers and singleton fields need operator-aware synchronization. | +| Policy | ClusterPolicies and the shared admission boundary. | Platform restrictions need controlled rollout and protection from project changes. | +| Credential | Administrator Connectors, encrypted Secret source, and the platform PaC registration. | Credential access and rotation have a different lifecycle from application state. | +| Project | Declarations and reviewed registration output: Applications, bindings, and destination patches. | Each project's requested resources and deployment scope can be reviewed together. | + +The onboarding installation is a separate Application because its execution +permissions and release lifecycle also need review. The layer column below +identifies responsibility; the four automatic creation Tasks still manage their +own external resources. + The table identifies the platform decision and its repository layer. Values and detailed steps belong in the linked product documentation. These are administrator-managed recommendations, with maturity and prerequisites determined by the selected release. | # | When this matters | Decision | Why it is shared | Layer and guidance | @@ -99,24 +179,61 @@ The table identifies the platform decision and its repository layer. Values and | 17 | Before live Tekton objects are pruned. | Define Results storage, watch scope, authentication, and retention. | Audit records need storage independent of live objects. | Component: [Results retention](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/retention_policy_configuration.html). | | 18 | Before operating the platform without controller health evidence. | Enable metrics and operational monitoring. | Operators need evidence of controller and execution health. | Component: [Metrics Collection](https://docs.alauda.io/alauda-devops-pipelines/4.15/configure/metrics_collection.html). | -The following matrix summarizes the six roles and the grants supplied or required by this reference. Kubernetes grants are additive: an absent sample binding does not revoke permissions inherited from ACP or another role. Git permissions and production protection must be configured in GitLab separately. - -| Role | Platform repo | Project declaration | Project namespace | Delivery namespace | GitOps `envs/prod` | Production | Connectors | -| --- | --- | --- | --- | --- | --- | --- | --- | -| Platform administrator | Write, prerequisite | Write, prerequisite | Administrator, prerequisite | Apply definitions and start runs | Configure protection separately | Administrator, prerequisite | Administrator Connectors | -| Application developer / project owner | No Git grant supplied | No Git grant supplied | Owners group gets `edit` in `shop` | No sample binding | No Git grant supplied | No workload binding supplied | Can use credentials in `shop` through Pods | -| Release approver | No grant supplied | No grant supplied | No binding supplied | ApprovalTask response; optional console read role | No Git review role supplied | No workload binding supplied | No mount grant supplied | -| CI ServiceAccount | No access supplied | No access supplied | Runs in `shop-build` | No binding supplied | No writer supplied | No workload binding supplied | Mount/proxy grants for project Harbor, SonarQube, Nexus | -| Delivery ServiceAccount | No access supplied | No access supplied | No binding supplied | Executes CD; no run-creation grant supplied | Maintainer group token through writer Connector | Changes Git references, reconciled by Argo CD | GitOps writer mount/proxy grant | -| Promotion ServiceAccount | No access supplied | No access supplied | No binding supplied | Executes promotion; no run-creation grant supplied | No writer supplied | Writes production registry through Connector | Mount both registry Connectors; direct proxy grant only for source | - -The release approver is not automatically a Git code owner. Configure the Git review role separately if both approvals are required. The supplied Maintainer writer can change GitLab protection settings; a production deployment needs a separately protected branch or repository and a narrower writer identity. Verify effective permissions with the actual users and ServiceAccounts before delegating delivery-namespace run creation. - -Name-scoped Connector `get` grants record the mounts this workflow needs; they do not remove ACP's broader default Connector reads within a project. Identity-specific proxy restrictions require the proxy permission features and grants described in the [delivery article](./From_Commit_to_Production_on_ACP.md#gate-the-production-connector-after-approval). Keep the distinction between read authorization, proxy authorization, and access to raw Secrets when reviewing this matrix. +### Choose shared defaults without hiding their costs + +For repository access and triggers, centralize webhook administration so that +projects do not need a webhook-management token. Keep the lower-privilege +Repository token, reviewed definition source, and explicit provider URL with the +repository configuration. A standard retry command helps operators, but a release +workflow can intentionally prohibit comment retries: the application sample's +`ci` requires a real `push` on `main`. Use its separate merge-request definition +for comment-triggered checks. + +For tool access, an administrator prepares the creation Tasks and their +credentials once; projects receive scoped resources and Connectors. Proxy +workload tokens reduce routine exposure to tool credentials. They do not remove +rotation of the tool's backing credential, PaC clone credentials, image-pull +Secrets, or Chains registry access. Build namespace separation and restrictions +on Pod creation are part of this choice, not merely labels on the resources. + +For release trust, the platform owns the signing key, verification policy, and +protected production credential path. A project chooses which environment needs +approval and can add restrictions. Installing approval components alone does not +protect the credential. Production promotion also needs the Core admission rules +and restrictions on deployed image sources described in the delivery article; +those additions are not installed by this sample. + +For shared assets, publish catalog Tasks and versioned Pipelines through local +services. A pinned version makes adoption reviewable; the platform must also +retain the referenced content and define whether published coordinates can change. +The initial GitOps template is a seed, so upgrading that template requires +separate changes to repositories that already contain content. + +For operations, choose cleanup, archiving, and monitoring together. Cleanup saves +cluster capacity but removes live evidence. Results requires external storage and +its own read authorization, while approval responses need separate preservation. +The first Pruner synchronization and Results watch scope can affect existing +tenants, so introduce them with an inventory and agreed retention limits. + +These settings are platform decisions, not a promise that a product default has +already enabled the intended control. Check +[Feature Maturity](https://docs.alauda.io/alauda-devops-pipelines/4.15/overview/feature_maturity.html) +for the selected release and introduce each control in stages. ### Separate policy scope from definition trust -Use platform ClusterPolicies for shared controls and namespaced Policies for project-specific restrictions. Restrict direct TaskRun and CustomRun creation by the actual submitting identity; client-supplied owner references alone are not an authorization boundary. Enumerate controller identities and review their permissions. +Use platform ClusterPolicies for shared controls and namespaced Policies for +project-specific restrictions. A project can narrow the permitted template, +version, image repositories, or parameters without weakening the platform +baseline. That depends on preventing the project from granting itself an +exception or changing the platform policy. + +Restrict direct TaskRun and CustomRun creation by the actual submitting identity; +client-supplied owner references alone are not an authorization boundary. +Enumerate controller identities and review their permissions. Allowing an +automation identity to create direct runs opens that entry for anyone who can use +that identity. The sample excludes the management namespace from its resolver and +direct-run policies, so that namespace's execution permissions remain critical. Optionally restrict PipelineRun references to approved resolver types. This abbreviated policy illustrates only the source-type decision; adapt namespace exclusions for every platform system namespace: @@ -151,12 +268,53 @@ spec: This does not constrain a resolver's URL, catalog, namespace, version, definition content, or runtime credential scope. It also rejects inline `pipelineSpec` when enforced. Convert the sample's inline validation flow to a catalog Pipeline before enabling that restriction for project repositories. Restrict PolicyException creation or disable exceptions according to platform policy. -The sample's `argocd-project-namespace-boundary` policy constrains writes of Role, RoleBinding, ResourceQuota, LimitRange, NetworkPolicy, Kyverno PolicyException, and Connectors AccessPolicy by the configured Argo CD identities in labelled project namespaces. It allows the platform-owned registration path and requires `allowExistingViolations: false`. Adapt both Argo CD identities for the target cluster and verify the intended allow and deny cases for the installed APIs before relying on the boundary. The exception for the platform `projects` Application depends on Argo CD's `argocd.argoproj.io/tracking-id` annotation; label-only resource tracking is not covered. Restrict exception and AccessPolicy creation by other identities as well, and keep any platform exception namespace outside project destinations. Install and verify the policy on every destination cluster to which the boundary should apply. +### Constrain project deployment without replacing ACP ownership + +The sample's `argocd-project-namespace-boundary` policy constrains writes of Role, +RoleBinding, ResourceQuota, LimitRange, NetworkPolicy, Kyverno PolicyException, +and Connectors AccessPolicy by the configured Argo CD identities in labelled +project namespaces. It allows the platform-owned registration path and requires `allowExistingViolations: false`. Adapt both Argo CD identities for the target cluster and verify the intended allow and deny cases for the installed APIs before relying on the boundary. The exception for the platform `projects` Application depends on Argo CD's `argocd.argoproj.io/tracking-id` annotation; label-only resource tracking is not covered. Restrict exception and AccessPolicy creation by other identities as well, and keep any platform exception namespace outside project destinations. Install and verify the policy on every destination cluster to which the boundary should apply. + +The policy denies a protected write when all three conditions hold: + +1. The writer is one of the actual Argo CD identities: the registered cluster identity or the application controller's identity. +2. The destination is a namespace labelled `cpaas.io/project`, excluding the platform system project. +3. The object's tracking annotation does not identify the platform-owned `projects` Application. + +This gives the platform registration a controlled path while preventing a project +Application from using deployment rights to rewrite its own permissions, quota, +network boundary, policy exceptions, or Connector access. It covers both new and +existing objects, so test create, update, and delete requests with the actual +identities. The reports controller also needs read access to the matched APIs; +the sample includes the aggregate read role for PolicyException and AccessPolicy. + +A rejected object makes the Application's synchronization fail with the policy +message. Retry settings determine subsequent attempts. Remove the forbidden +object from the project source and synchronize the corrected revision. This +admission policy has no background scan, so an Audit trial needs an actual matching +write before there is a report to inspect. Kyverno availability and recovery are platform operating concerns. For policies used as admission gates, keep the resource webhooks registered with `failurePolicy: Fail`; when Kyverno is unreachable, matching writes are then rejected. An absent webhook provides no enforcement, and an orderly scale-to-zero can remove resource webhook registrations. Operate Kyverno highly available and keep a controlled recovery procedure, including the conditions for temporarily removing and later restoring webhook configurations. A signing result proves the signing key, not the pipeline that produced an image. Restrict ClusterPolicy, resolver, webhook, and PolicyException changes to platform administrators. See [Pipeline Policy Constraints with Tekton and Kyverno](./Pipeline_Policy_Constraints_with_Tekton_and_Kyverno.md) for the policy design. ## Organize and synchronize the platform repository +### Connect the repository, cluster, and ownership layers + +Argo CD needs three independent agreements: repository read access, a registered +destination cluster, and an AppProject that permits the source and destination. +Complete all three before debugging the workload manifests. Use a read-only +platform repository credential and a repository credential template covering the +approved project GitOps URL prefixes. Onboarding creates Applications, not those +Argo CD credentials. + +Use GitLab URLs ending in `.git` in Application and AppProject references to avoid +a redirect that Argo CD does not follow. Use `destination.name: ` +from Alauda GitOps registration, and the same name in AppProject destinations. +The default `in-cluster` entry can be limited to the Argo CD namespace; it is not +an interchangeable name for the ACP-registered cluster. When migrating a +manifest that used `destination.server`, remove that field rather than leaving +both `name` and `server` on the live object. + Use `platform-repo/` as the tree root below; the repository itself can have any name. The sample is split into four ownership layers and the onboarding installation: ```text @@ -171,6 +329,12 @@ platform-repo/ └── .tekton/ # Validation and changed-project dispatch ``` +One Application per layer lets a reviewer understand its path and synchronization +policy. A rendering failure in one layer can stop all objects in that Application; +separate Applications reduce the affected scope, while prerequisites between +controllers, credentials, and runs still need to be respected. The sample has the +root, four ownership layers, and a fifth child for onboarding. + Copy `onboarding/` from the sample to the platform repository root as described by its README. Register the repository read-only with Argo CD and use one child Application per layer. The root Application manages those Application definitions. After replacing placeholders and preparing the credentials and CRDs above, bootstrap once from that repository's root: ```bash @@ -179,18 +343,61 @@ kubectl apply -k argocd/apps/ Declarations already present in `projects/` are not picked up by the merge dispatcher until a later `main` change. After the Applications synchronize, run the onboarding Run Template once for each existing declaration. Later changes follow the reviewed Git and dispatcher path. +### Choose synchronization behavior by resource lifecycle + +Automatic synchronization applies a new Git revision. Self-healing repairs a live +change without a new revision. Pruning deletes tracked resources removed from +Git. These are separate decisions: enabling one does not establish that all +three are appropriate for a layer. + +| Application or layer | Sample choice | Reason and condition | +| --- | --- | --- | +| Root Application definitions | Automated sync, prune, and self-heal. | Git owns the child Application configuration. Review removals and finalizers before deleting an Application. | +| Components | Automated sync; `prune: false`, `selfHeal: false`. | Partial singleton TektonConfig fields coexist with an operator; a removed fragment must not delete the singleton. | +| Credentials | Automated sync with self-healing disabled. | Coordinate sealed source, controller-managed fields, and live token rotation. | +| Policies, projects, and onboarding | Automated sync, prune, and self-heal in the sample. | Git owns these resources; registration deletion has project-level consequences. | +| Project environment Applications | Automated sync, prune, and self-heal. | Git is the desired application state; test and production differ in Git review requirements. | + For partial singleton component configuration, use server-side apply so the platform repository owns only the specified fields. The samples also set `argocd.argoproj.io/compare-options: ServerSideDiff=true`. Review the generated diff and the operator's field ownership before synchronization. The component layer uses `prune: false` so that removing a fragment does not delete the singleton TektonConfig, and starts with `selfHeal: false`. Observe the operator's write-back behavior and effective ownership in the target environment before enabling ongoing self-healing. Automatic synchronization of a new Git revision and self-healing of a live edit are separate decisions. The credential layer does not enable self-healing: seal credentials for the target controller and plan how rotations update live Secrets and encrypted source. Other layers can use prune and self-heal according to their ownership. Namespaces are created by onboarding; do not add a second namespace creator with `CreateNamespace`. +### Review the effective AppProject and its alternatives + ACP maintains the AppProject associated with an ACP project. The registration patch manages only the intended repository and destination fields. Reconcile the effective destinations with the platform-managed entries, and use the admission boundary for object-kind restrictions. Do not rely on an Application's sync status alone to prove the deployment boundary. +The sample patches only `sourceRepos` and `destinations` on the ACP-managed +AppProject. ACP can also maintain destination entries for the project's own +namespaces, including build and delivery. Inspect the effective object; the +registration patch is not an exclusive list of reachable namespaces. Keep the +object-kind boundary in platform admission instead of relying on fields that ACP +may reconcile. + +Another option is a separately named deployment AppProject, such as +`-deploy`, with its own source, destination, and resource restrictions. +It gives Argo CD ownership of that project's restrictions, but Applications using +it do not appear in the ACP project's GitOps list when that list selects +`spec.project` equal to the ACP project name. Operators would use Argo CD to view +them. Choose based on ownership and the required console experience. + +The sample's platform AppProject permits broad registration operations. Narrow +its sources, destinations, and allowed kinds for a production platform; do not +hand its registration rights to project application writers. Decommissioning also +needs to inspect and remove the ACP-associated AppProject separately. + +### Treat registration rollback as a lifecycle operation + Review deletion carefully. Reverting a registration commit, or removing `projects//` together with its `resources` entry, can make Argo CD prune the registered ACP project. That can delete its same-named namespace and the Secrets, Connectors, and Repository objects there. It does not reverse tool-side creation or automatically remove every build, delivery, or environment namespace. Deleting only `projects/.yaml` is different: it does not decommission the project, and leaving its `configMapGenerator` entry in place makes the declaration layer fail to render. Use a separate approved decommissioning procedure covering workloads, retained evidence, external tenants, credentials, webhooks, and each namespace. ### Validate changes before merging +Render each directory an Application consumes before a change is merged. A bad +Kustomize reference prevents generation of that Application's manifest set, so +unrelated objects in the same layer can also stop synchronizing. Finding the +problem in merge-request validation keeps it close to the author and the diff. + Both the platform and project GitOps templates include `.tekton/validate.yaml`, which composes catalog `git-clone` and `run-script` Tasks. Replace its tool image with a locally available image containing the required shell and Kustomize commands. Establish its webhook before the first merge-request run. With default-branch definition provenance, a change to the validation definition is first exercised after it has entered that branch. Validate every directory owned by an Application, and add new directories to the validation paths. Re-run a merge-request check with `/test validate branch:`. YAML parsing and Kustomize rendering do not validate CRD admission behavior; use the target schemas and an isolated project for that acceptance step. @@ -203,6 +410,17 @@ The sample onboarding dispatcher compares the pushed revision with its first par ## Declare a project and run onboarding +The declaration records what the project requests: ownership groups, +environments, namespaces, and tool tenants. It contains no credentials or +runtime-generated tool IDs. The platform merge request also supplies the named +permissions and schedule that make the request executable. + +The sample intentionally supports one naming convention and all four tools. +Explicit derived names let a reviewer see the requested shape without memorizing +that convention; the onboarding validator checks they agree. A declaration is +not a general-purpose provisioning API, and changing a name can change identity +or credential behavior in an external tool. + The declaration is a sample schema, not an ACP API. The pipeline validates its supported naming conventions. A complete `shop` declaration includes the following fields; use the sample's full file: ```yaml @@ -242,12 +460,37 @@ sonarqube: The declaration's `approvers` field is a reviewed record only: onboarding validates its shape but does not apply or compare it. The approvers that take effect are the promotion Pipeline's `ApprovalTask` list and the production `AccessPolicy`; nothing keeps those three sources in sync, so update them together in one reviewed change. Keep the production `AccessPolicy` under platform-administrator control because a person who can edit the Pipeline can otherwise change its `ApprovalTask` list while the AccessPolicy continues to enforce its own approver set. +### Review all inputs in the project merge request + +Include the declaration, its `configMapGenerator` entry, the project's named +onboarding permissions, its ScheduledTrigger and Kustomize entries, and its +production image-verification repository entry. These belong together because a +declaration alone cannot grant the onboarding identity permission to manage new +production Connectors or expand the verification policy. + +Per-project permission files are intentional: they keep release Connector writes +limited to listed names. Generating arbitrary RoleBindings from any declaration +would require broader `bind` or `escalate` rights. Review that security cost before +replacing the explicit files with more automation. Separately, add the new +registry projects to the platform-owned central Chains credential's access. + Submit the declaration, project-specific onboarding permissions, and schedule through a platform merge request. Confirm that the declaration and onboarding Applications have synchronized; synchronize them manually when using a manual sync policy. Then examine the dispatcher result and every child onboarding run. A successful merge or dispatcher does not by itself establish complete project readiness. The sample dispatches changed declarations after a merge. A path filter can also admit a run with no relevant declaration changes; the dispatcher is responsible for selecting actual declarations. A deleted declaration is skipped, rather than interpreted as an instruction to delete a project. ### Understand the seven onboarding stages +Before these stages, the Pipeline atomically acquires the project's ConfigMap +lock. It then validates the declaration before changing tenant resources. Among +other checks, the sample requires the four supported tools, group-shaped +approvers, matching derived names, and namespaces outside the platform's reserved +names. It refuses to adopt another project's namespace. The exception is the +same-named inner namespace created with the ACP project, which onboarding labels +and adopts explicitly. + +The seven stages describe responsibilities; several tool Tasks run in parallel +after namespaces are ready, and the registration step starts its own child run. + | Stage | Work | Owner or mechanism | | --- | --- | --- | | 1 | Export the existing ACP project without runtime metadata; create or reconcile labelled namespaces and delivery identities. Workloads use restricted Pod Security; the build namespace is separate. | Sample orchestration and Kubernetes APIs | @@ -258,7 +501,27 @@ The sample dispatches changed declarations after a merge. A path filter can also | 6 | Seed the empty project GitOps repository and create its PaC Repository and access Secret. Complete and verify webhook configuration separately. | Sample template seeding and PaC resources | | 7 | Render project Applications, AppProject patch, owner and Connector bindings, and the exported ACP project. Open a platform registration merge request. | A separate `gitops-repo-update` PipelineRun | -Onboarding creates the project's GitLab group but does not add people to it. Before handoff, a platform administrator must add the project's developers with a role that allows them to create repositories in that group, according to the platform's access policy. +The first and last stages are sample orchestration. The four external tools use +product automatic creation Tasks, while repository seeding copies a reviewed +template once. This separation keeps tool-specific account and rotation behavior +in the product Tasks and keeps ordering and registration in the sample Pipeline. + +The exported ACP project omits status, server-generated metadata, and GitOps +tracking markers. Keeping runtime tracking out of Git avoids a registration merge +request whose only purpose is to copy the previous synchronization's metadata. +Namespaces and delivery identities are created before the tool Tasks need them. + +Harbor pull configuration is distributed to workload and build namespaces. The +sample delivery namespace runs tool images and does not receive that project pull +Secret. The project's four Connectors live in the project namespace; the GitOps +writer lives in delivery; production release Connectors live in management. Check +the permissions actually held by each credential rather than inferring them from +a name such as "pull" or "readonly". + +Onboarding creates the project's GitLab group but does not add people to it. +Before handoff, a platform administrator must add the project's developers with a +role that allows them to create repositories in that group, according to the +platform's access policy. The registration step proposes Git changes instead of directly applying Argo CD resources. Review the entire diff, including RBAC and destination changes. Submit one platform merge request that adds the declaration, its `configMapGenerator` entry in `projects/kustomization.yaml`, the project's onboarding RBAC and ScheduledTrigger files together with their entries in `onboarding/kustomization.yaml`, and the project's production repository in the image-verification policy. Without the `configMapGenerator` entry the dispatcher waits for the project's ConfigMap and then skips the project. Keep sufficient execution capacity for both the parent and child run: the waiting parent still reserves its Pod resources. Cancellation and cleanup must account for the independently created child run. @@ -266,6 +529,12 @@ The pipeline runs in the management namespace with administrator Connectors. Per ### Reuse the product Tasks +Use the product Tasks for each tool's tenant resources, accounts, tokens, and +Connector output. The sample adds declaration parsing, namespace ownership, +sequencing, locking, seeding, and platform registration. Reimplementing the four +tools' account logic inside the onboarding script would duplicate their contracts +and rotation behavior. + The four Tasks have different parameters and rotation behavior. Read their installation, access, and prerequisite instructions: - [GitLab tenant group and Connector](https://docs.alauda.io/alauda-devops-connectors/1.14/connectors-gitlab/how_to/using_gitlab_connector_automatic_creation_task.html). @@ -277,6 +546,13 @@ Set the target `connector` as `namespace/name`. Set GitLab access levels explici The sample writes only to the current cluster using an explicitly authorized ServiceAccount. For cross-cluster application, follow each Task's Kubernetes Connector and `kubeconfig` Workspace instructions. Do not assume that the management namespace's default ServiceAccount has the required write permissions. +Choosing the current-cluster ServiceAccount avoids distributing another +administrator kubeconfig, at the cost of explicit RBAC for every required write. +A TaskRun or PipelineRun submitted with `generateName` uses `kubectl create`. +Install Tasks in the namespace where they run and check the run identity before +submission; an unspecified default ServiceAccount is not an implicit +administrator. + An alternative product workflow creates project Connectors from namespace events: [Automatically Create Connectors for a Project](https://docs.alauda.io/alauda-devops-connectors/1.14/how_to/automatically_create_connectors_for_a_project.html). If choosing that path, assign one creator and rotation owner per tenant and resource. Avoid two independent reconcilers managing the same credentials. ## Register repositories and verify events @@ -306,8 +582,35 @@ Set the provider URL explicitly so completion requests and the Repository token The project's PaC Repository uses the Developer-level group access token. When automatic webhook registration is enabled, the controller uses the centralized webhook-registration credential configured by platform administrators to create and delete the webhook; verify a fresh event and its resulting run. The separate GitOps writer uses the Maintainer-level group access token. Keep the webhook-registration credential available through repository decommissioning, and check for remaining code-platform webhooks afterward. The application's Repository belongs in the build namespace, where CI executes. Application CI objects belong in the build namespace; CD and promotion Run Templates and PipelineRuns belong in the delivery namespace and require an administrator or an explicit delegated grant. +### Separate platform parameters from application inputs + +The onboarding flow writes platform-provided values such as the registry address +and Connector names into the generated Repository's `spec.params`. A PaC +definition can consume those values while the application supplies its own source +revision and build inputs. This gives a platform-owned address a clear owner and +avoids repeating it in every definition that consumes the Repository parameter. + +That parameter layer does not rewrite all existing configuration. Registry and +Git addresses also appear in seeded GitOps content, application CI/CD, promotion +mappings, Argo CD references, and verification policy. An address migration must +update those consumers through their own reviewed changes. Re-running onboarding +leaves a non-empty GitOps repository intact. + +Repository access and webhook administration also have different lifecycles. +PaC generates per-run clone credentials from the Repository access token; those +credentials remain sensitive even though the run did not receive the webhook +administrator token. Test registration, delivery of a fresh event, the intended +run namespace, and completion status separately. + ## Reconcile and rotate deliberately +### Distinguish reconciliation from template upgrades + +Repeated onboarding is useful for a supported declaration change, scheduled tool +reconciliation, and rebuilding declared configuration after prerequisites have +been restored. It is not a blanket rollback of external state. A Task can rotate +or recreate a credential as part of an otherwise idempotent resource operation. + Use repeated runs of the same supported parameters for reconciliation. Do not equate Connector `Ready=True` with an unexpired, usable tool token; perform an authorized tool request or workload operation before and after rotation. | Change | Required operating action | @@ -320,14 +623,93 @@ Use repeated runs of the same supported parameters for reconciliation. Do not eq The sample's supported GitLab subgroups are `apps` and `gitops`. Changing that convention requires a reviewed pipeline change, rather than only a declaration edit. Tool-side identity changes can recreate accounts or credentials; inspect the product Task's contract before changing names or project sets. +For example, adding `staging` requires a reviewed declaration change and a +project GitOps change that supplies `envs/staging` and its validation path. The +registration output can then point at a real directory. Updating only the +declaration can create an Application that cannot render. Similarly, changing the +seed template is an upgrade proposal for existing projects, not a reason to +overwrite their repositories. + +Registration is regenerated from reviewed templates. A rerun with no registration +diff should not create another merge request. Inspect the resulting Git change or +absence of one; a generic update-step result alone is not proof that a +registration was needed or merged. + +### Serialize tenant changes and recover a retained lock + Do not overlap runs for the same tenant. The three sample entry points—scheduled reconciliation, merge dispatch, and the manual Run Template—use the same onboarding Pipeline. Its first task atomically creates a per-project ConfigMap lock before any tenant changes; a competing run fails. A successful run releases its own lock in a final task. A failure, cancellation, or interrupted cleanup can leave the lock in place. Inspect its recorded PipelineRun and any independently started child runs, confirm that all work has stopped, then remove the lock before retrying. Never clear a lock just because the run is taking longer than expected. +Lock release is tied to the owning run's UID, so a competing run cannot clean up +another run's lock. There is no timeout-based takeover: a slow external API or +independent registration child can still be working after the parent looks +stalled. Record the owner, stop and inspect all related runs, and only then remove +a retained lock. A retry uses the same Pipeline and must acquire that lock again. + +### Plan rotation before credentials expire + Arrange rotation outside active pipeline windows and monitor failed runs before the current credential expires. CSI workload-token issuance is separate from rotating the tool's long-lived credential. Consult the Task guide for expiry recovery; define a local recovery procedure rather than depending on an unobserved natural-expiry path. If enabling Connector approval, prepare grants for every affected workload, enable `enable-connectors-approval`, immediately create the required AccessPolicies, and then enable `enable-connector-proxy-permissions`. The sample's optional administrator AccessPolicy is excluded from the default Kustomization and is applied only after the feature is enabled. Use RBAC for the documented cross-namespace grants and keep approval-gated production Connectors free of direct proxy grants. See the [delivery article](./From_Commit_to_Production_on_ACP.md#gate-the-production-connector-after-approval) for the complete rollout boundary. +## Understand the six project boundaries + +Onboarding creates resources from which isolation can be established. Verify the +effective rights as well as the resources: a separate account can still have a +broad role, and a project label does not constrain every writer. + +| Boundary | Result and remaining condition | +| --- | --- | +| Namespace and Pod Security | Workloads, build, and delivery have distinct namespaces and security levels. The sample owner binding grants `edit` in the project namespace only. | +| Deployment authorization | Effective AppProject destinations constrain location; the platform admission boundary constrains sensitive kinds written by the actual Argo CD identities. Direct Kubernetes writers still need RBAC restrictions. | +| Pipeline routing | Repositories and triggers route events to the intended project. Definition-source and direct-run policies constrain execution only where installed and enforced. | +| External tool tenancy | GitLab group, private Harbor projects, Nexus repositories/user, and SonarQube permissions are scoped per project. Verify instance-wide anonymous and visibility settings as prerequisites. | +| Credential access | Project users cannot create workloads in administrator or signing namespaces. Proxy grants, raw Secret access, PaC credentials, and Chains credentials are reviewed separately. | +| Execution history | Live cleanup and namespace overrides coexist with retained external Results records. Approval responses have a separate preservation path. | + +The production release credential in this sample can read the staging project as +well as write production because the copy Task uses the destination proxy +configuration for both ends. The mount named source-readonly supplies the source +CA; it does not establish a separate authentication path. The delivery article +explains this copy-specific behavior and its approval boundary. + +Cross-project pull access needs a separately reviewed grant. The declaration's +reserved `imagePullFrom` field only emits a warning in the sample; it does not +create a Harbor permission or distribute a new pull credential. Keep projects +private and demonstrate denial before treating an omitted grant as isolation. + +### Check the six roles against the actual grants + +The following matrix summarizes the six roles and the grants supplied or required by this reference. Kubernetes grants are additive: an absent sample binding does not revoke permissions inherited from ACP or another role. Git permissions and production protection must be configured in GitLab separately. + +| Role | Platform repo | Project declaration | Project namespace | Delivery namespace | GitOps `envs/prod` | Production | Connectors | +| --- | --- | --- | --- | --- | --- | --- | --- | +| Platform administrator | Write, prerequisite | Write, prerequisite | Administrator, prerequisite | Apply definitions and start runs | Configure protection separately | Administrator, prerequisite | Administrator Connectors | +| Application developer / project owner | No Git grant supplied | No Git grant supplied | Owners group gets `edit` in `shop` | No sample binding | No Git grant supplied | No workload binding supplied | Can use credentials in `shop` through Pods | +| Release approver | No grant supplied | No grant supplied | No binding supplied | ApprovalTask response; optional console read role | No Git review role supplied | No workload binding supplied | No mount grant supplied | +| CI ServiceAccount | No access supplied | No access supplied | Runs in `shop-build` | No binding supplied | No writer supplied | No workload binding supplied | Mount/proxy grants for project Harbor, SonarQube, Nexus | +| Delivery ServiceAccount | No access supplied | No access supplied | No binding supplied | Executes CD; no run-creation grant supplied | Maintainer group token through writer Connector | Changes Git references, reconciled by Argo CD | GitOps writer mount/proxy grant | +| Promotion ServiceAccount | No access supplied | No access supplied | No binding supplied | Executes promotion; no run-creation grant supplied | No writer supplied | Writes production registry through Connector | Mount both registry Connectors; direct proxy grant only for source | + +The release approver is not automatically a Git code owner. Configure the Git review role separately if both approvals are required. The supplied Maintainer writer can change GitLab protection settings; a production deployment needs a separately protected branch or repository and a narrower writer identity. Verify effective permissions with the actual users and ServiceAccounts before delegating delivery-namespace run creation. + +Name-scoped Connector `get` grants record the mounts this workflow needs; they do not remove ACP's broader default Connector reads within a project. Identity-specific proxy restrictions require the proxy permission features and grants described in the [delivery article](./From_Commit_to_Production_on_ACP.md#gate-the-production-connector-after-approval). Keep the distinction between read authorization, proxy authorization, and access to raw Secrets when reviewing this matrix. + +Keep CI, GitOps delivery, and artifact promotion identities separate. A runtime +ServiceAccount having execution credentials does not mean it can create its own +PipelineRuns. Likewise, being an application repository maintainer does not +establish a Kubernetes grant in the build or delivery namespace. Check both sides +before delegating a Run Template to project owners. + +The optional ACP console approver role supplies the read access needed to find +approval work without granting run creation or Secret reads. It does not assign a +Git code-owner role or prove that an approver cannot initiate a release through +some other inherited permission. Review actual group membership and effective +permissions, and keep release initiators and approvers separate. + ## Accept the project boundaries +### Inspect each onboarding result before handoff + | Boundary | What must be demonstrated | | --- | --- | | Namespaces and Pod Security | Workload namespaces have the intended labels; builds and delivery use their separate namespaces and identities. | @@ -343,8 +725,39 @@ For cross-project image denial, first verify that the exact image exists and can Test self-healing only on a disposable resource in the acceptance project. The sample's default check deletes the project owners RoleBinding, so the project owner can temporarily lose access while the object is restored. Record its UID and recovery manifest, perform the approved change, and confirm recreation with a different UID. Restore it if the check fails. This is separate from proving ongoing automatic reconciliation of the operator-managed component layer. +### Know which outputs should exist + +After reviewing and merging the registration, the expected outputs are: + +| Location | Expected output | +| --- | --- | +| Platform Git | One declaration per project, its registered Kustomize entries, and reviewed project registration files. | +| Argo CD | Root and layer Applications, onboarding Application, and one Application per project environment. | +| GitLab | Project group, `apps` and `gitops` subgroups, two group access tokens, and the seeded `gitops/config` repository. | +| Harbor | Private source and production projects, a project robot and the two release-path robots, with the required pull configuration. | +| Nexus | The declared repositories and their scoped user. | +| SonarQube | A user, token, and project permission template; the first scan creates the application project. | +| Project Kubernetes resources | Labelled namespaces, delivery identities, four project Connectors, PaC registration, and reviewed owner/mount bindings. | +| Management Kubernetes resources | Administrator and project release Connectors, named onboarding grants, and the reconciliation schedule. | + +Do not require every Application to report the same status immediately. +Placeholder workload digests are not runnable images; environment health follows +the first real deployment. The `projects` AppProject patch can remain OutOfSync +because ACP also maintains fields. The component layer's ongoing self-healing is +disabled pending ownership verification. Read the effective object, revision, +policy decision, and workload state that correspond to each acceptance claim. + +Project readiness means its child onboarding run succeeded, registration was +reviewed and synchronized, tool access works within the intended tenant, a fresh +repository event reaches the right definition, and forbidden access is rejected +for the intended reason. Delivery readiness additionally requires the first real +build and release described in the delivery article. A successful dispatcher or +four Ready Connectors alone does not meet those conditions. + ## Preserve audit records and plan recovery +### Choose cleanup and archiving as one operating decision + Choose one cleanup mechanism and review its scope before enabling it. Use event-based Pruner according to the global and namespace configuration guides; disable the older periodic mechanism when migrating. Verify each selected retention mode in the target environment before it can remove shared run history. Global values apply cluster-wide after the first synchronization, including other tenants' namespaces unless a namespace override applies, and successful and failed runs are counted separately per namespace. On a shared cluster, count each namespace's finished PipelineRuns and standalone TaskRuns before enabling cleanup, choose limits above the highest count or agree lower limits with the tenants, and notify owners first. @@ -353,6 +766,8 @@ Use external PostgreSQL and the documented log storage for Results. Define watch Scope Results Watcher before enabling it: default watching can affect finished runs across namespaces and add `results.tekton.dev/*` finalizers. Disabling archiving does not remove finalizers already present. Follow the product recovery procedure before deletion and treat external Results storage as the audit source after cluster objects are pruned. See [S3 storage](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/s3_storage_configuration.html) and [Results retention](https://docs.alauda.io/alauda-devops-pipelines/4.15/results/configure/retention_policy_configuration.html). +### Restore prerequisites before replaying configuration + Git does not replace backups of credentials and external data. Before a platform rebuild, the platform team must prepare and rehearse recovery of these dependencies: 1. Compatible platform components, Kyverno, and sealed-secrets CRDs and controllers. @@ -363,6 +778,22 @@ Git does not replace backups of credentials and external data. Before a platform After preparing those dependencies, synchronize the platform layers, run onboarding for the declared projects, review registration changes, and complete the acceptance checks. This is a dependency outline; a complete empty-cluster or second-cluster recovery runbook must be exercised in the target environment. +During a rebuild, registration objects can be present before all project +namespaces exist. Allow declarations to synchronize before namespace-dependent +bindings so onboarding can read them; do not place the declaration ConfigMaps in +a later sync wave blocked by those bindings. Inspect failed objects and retries +rather than treating a temporarily incomplete layer as proof that onboarding +cannot proceed. This dependency sequence needs its own recovery acceptance. + +### Adapt and operate the platform deliberately + +Replace tool addresses, management namespace, project/group naming conventions, +environment list, Argo CD cluster name, both boundary-policy identities, signing +public key, encrypted credentials, local images, token lifetimes, and rotation +intervals. Verify every target-cluster boundary after adapting the identities. +Address migration also needs changes to repositories already seeded and to policy +and promotion inputs; Repository parameters are only one part of that operation. + Before production, complete the sample's hardening work: isolate pre-merge validation from administrator credentials, narrow the AppProject used for project registration, protect the GitOps writer's namespace, install promotion Core rules, and restrict production image sources. Restrict direct cluster writes alongside Git review. Template upgrades, decommissioning, capacity sizing, offline package distribution, and customer change-management integration require separate operating procedures. Continue with [From Commit to Production](./From_Commit_to_Production_on_ACP.md) to use the onboarded project's build, test, promotion, and production GitOps paths.