Feature: Allow workloadKind: Deployment

CHANGELOG.md

@@ -15,9 +15,15 @@ All notable changes to this project will be documented in this file.
   The cache was previously unbounded, which a caller could exploit to exhaust the memory limit of
   the sidecar, as cache keys are built from caller-supplied parameters. Entries beyond the limit are now
   evicted least-recently-used first ([#863])
+- The `servers` role can now run as a `Deployment` instead of a `DaemonSet`, selected via
+  `spec.servers.roleConfig.workloadKind` ([#873]).
+- A `PodDisruptionBudget` is now written out for the `servers` role when it runs as a `Deployment`,
+  with `maxUnavailable: 1`. Configurable via `spec.servers.roleConfig.podDisruptionBudget` ([#873]).
 
 ### Changed
 
+- OPA Pods now default to a soft anti-affinity that spreads them across nodes. This is a no-op for a
+  `DaemonSet`, which already runs one Pod per node, but keeps a `Deployment`'s replicas from being deployed together ([#873]).
 - Internal operator refactoring: introduce a build() step in the reconciler that
   assembles all relevant Kubernetes resources before anything is applied ([#852]).
 - Bump `stackable-operator` to 0.114.0 ([#867]).
@@ -74,6 +80,7 @@ All notable changes to this project will be documented in this file.
 - The reconciler now applies resources and derives the cluster status in discrete
   apply and update_status steps for the `opa_controller` ([#872]).
 
+[#873]: https://github.com/stackabletech/opa-operator/pull/873
 [#852]: https://github.com/stackabletech/opa-operator/pull/852
 [#861]: https://github.com/stackabletech/opa-operator/pull/861
 [#863]: https://github.com/stackabletech/opa-operator/pull/863

deploy/helm/opa-operator/templates/clusterrole-operator.yaml

@@ -55,12 +55,26 @@ rules:
       - bind
     resourceNames:
       - {{ include "opa-operator.name" . }}-clusterrole
-  # DaemonSet created per role group. Applied via SSA, tracked for orphan cleanup, and
-  # owned by the controller.
+  # DaemonSet or Deployment created per role group, depending on the role's `workloadKind`.
+  # Applied via SSA, tracked for orphan cleanup, and owned by the controller.
   - apiGroups:
       - apps
     resources:
       - daemonsets
+      - deployments
+    verbs:
+      - create
+      - delete
+      - get
+      - list
+      - patch
+      - watch
+  # PodDisruptionBudget created per role, when the role has it enabled. Also needs `delete`, because
+  # disabling it (or switching to a DaemonSet) must clean the existing budget up.
+  - apiGroups:
+      - policy
+    resources:
+      - poddisruptionbudgets
     verbs:
       - create
       - delete

docs/modules/opa/pages/usage-guide/operations/pod-disruptions.adoc

@@ -1,5 +1,39 @@
 = Allowed Pod disruptions
+:description: Whether the operator creates a PodDisruptionBudget for OPA depends on the workload kind, and how to configure or disable that budget.
 
-For OPA clusters, the operator does not deploy any PodDisruptionBudgets (PDBs), as there is one instance per Kubernetes node running (Daemonset).
-When a Kubernetes node gets drained to gracefully shut it down, the OPA Pod get's evicted - there is no point in blocking the eviction.
-In case the OPA Pod terminated before the products depending on OPA (e.g. Trino coordinator) on the same node, the products can still use the OPA Service, as it routes to OPA Pods running on other Kubernetes nodes.
+You can configure the allowed Pod disruptions as described in xref:concepts:operations/pod_disruptions.adoc[].
+
+What the operator creates depends on the xref:usage-guide/workload-kind.adoc[workload kind] of the role.
+
+== DaemonSet
+
+No PodDisruptionBudget is created, and you should not enable one.
+
+`kubectl drain` skips DaemonSet Pods, a budget is never consulted during a node drain.
+Moreover, PodDisruptionBudgets over DaemonSets can never be evaluated: DaemonSet does not implement the scale subresource.
+Therefore the budget stays at `disruptionsAllowed: 0` with a `SyncFailed` condition and refuses every direct eviction.
+
+Because the role Service routes node-locally in this mode, products on a drained node cannot reach OPA on another node.
+Drain nodes together with the products that query OPA on them.
+
+== Deployment
+
+The operator creates one PodDisruptionBudget per role, allowing one Pod to be unavailable at a time by default:
+
+[source,yaml]
+----
+spec:
+  servers:
+    roleConfig:
+      podDisruptionBudget:
+        enabled: true # <1>
+        maxUnavailable: 1 # <2>
+----
+<1> Defaults to true for a Deployment. Set it to false to create no budget.
+<2> Defaults to 1. Raise it to allow more Pods to be unavailable at once.
+
+Keep `maxUnavailable` at 1 unless you have measured that OPA tolerates more.
+Products query OPA on every request, budgets that drain too many Pods at once turn a node rotation into a platform-wide slowdown.
+
+NOTE: A budget only helps when there is another Pod to fall back to.
+With `replicas` set to 1, the single Pod can still be evicted.

docs/modules/opa/pages/usage-guide/operations/pod-placement.adoc

@@ -0,0 +1,33 @@
+= Pod placement
+:description: The default affinity spreads OPA Pods across Kubernetes nodes, which matters only for a Deployment.
+
+You can configure Pod placement for OPA Pods as described in xref:concepts:operations/pod_placement.adoc[].
+
+== Defaults
+
+The default affinity created by the operator is:
+
+* Distribute all Pods of the `servers` role across nodes, so that multiple Pods don't end up on the same Kubernetes node (weight 70)
+
+This constrains nothing for a DaemonSet, which already places exactly one Pod per node.
+It matters for a Deployment, where several replicas would otherwise be free to share the same node.
+See xref:usage-guide/workload-kind.adoc[].
+
+[source,yaml]
+----
+affinity:
+  podAntiAffinity:
+    preferredDuringSchedulingIgnoredDuringExecution:
+    - podAffinityTerm:
+        labelSelector:
+          matchLabels:
+            app.kubernetes.io/component: server
+            app.kubernetes.io/instance: cluster-name
+            app.kubernetes.io/name: opa
+        topologyKey: kubernetes.io/hostname
+      weight: 70
+----
+
+NOTE: The default affinity is only preferred and not enforced, because not every setup has multiple Kubernetes nodes.
+To enforce it, set your own `requiredDuringSchedulingIgnoredDuringExecution` affinity.
+Be aware that a Deployment with more replicas than nodes then leaves Pods unschedulable.

docs/modules/opa/pages/usage-guide/workload-kind.adoc

@@ -0,0 +1,84 @@
+= Workload kind
+:description: Run the OPA servers as a DaemonSet or as a Deployment, and learn how that choice affects the Pod count, Service routing and changing it later.
+
+By default the `servers` role runs as a DaemonSet, which places one OPA Pod on every Kubernetes node.
+Set `workloadKind` to Deployment to run a fixed number of Pods instead.
+
+[source,yaml]
+----
+spec:
+  servers:
+    roleConfig:
+      workloadKind: Deployment # <1>
+    roleGroups:
+      default:
+        replicas: 3 # <2>
+----
+<1> Either DaemonSet (default) or Deployment.
+<2> Only used by Deployment, DaemonSet derives Pod count from the number of nodes.
+
+The workload kind also decides whether the operator creates a xref:usage-guide/operations/pod-disruptions.adoc[PodDisruptionBudget], and it changes what the default xref:usage-guide/operations/pod-placement.adoc[Pod placement] achieves.
+
+== Choosing a workload kind
+
+Use a DaemonSet when every node runs products that query OPA.
+Each product then queries the OPA Pod on its own node, so no policy query crosses the network.
+The number of OPA Pods grows and shrinks with the node count.
+
+Use a Deployment when the number of OPA Pods should be fixed.
+You set the count with `replicas` and queries are spread across all Pods.
+This fits large clusters, and clusters where only a few nodes run products that query OPA.
+
+== Service routing
+
+The operator derives the role Service's `internalTrafficPolicy` from the workload kind.
+
+* A DaemonSet gets Local, so a query only reaches the OPA Pod on the client's own node.
+  This avoids the network hop, and a DaemonSet covers every node, thus such a Pod always exists.
+* A Deployment gets Cluster, so a query reaches any OPA Pod of the role.
+  A Deployment's Pods do not cover every node necessarily, so node-local routing would leave products on the remaining nodes unable to reach OPA at all.
+
+If the derived value doesn't suit your cluster, you can override it as described in <<override-traffic-policy>>.
+
+[#override-traffic-policy]
+== Override traffic policy
+
+In edge cases (e.g. node autoscaling under load), it is useful to use a DaemonSet with `internalTrafficPolicy: Cluster`.
+This can be achieved using xref:concepts:overrides.adoc#object-overrides[object overrides]:
+
+[source,yaml]
+----
+apiVersion: opa.stackable.tech/v1alpha2
+kind: OpaCluster
+metadata:
+  name: simple-opa
+  namespace: default
+spec:
+  image:
+    productVersion: 1.16.2
+  objectOverrides:
+    - apiVersion: v1
+      kind: Service
+      metadata:
+        name: simple-opa-server
+        namespace: default
+      spec:
+        internalTrafficPolicy: Cluster # <1>
+  servers:
+    roleGroups:
+      default: {}
+----
+<1> Changes `internalTrafficPolicy` from Local to Cluster.
+
+== Changing the workload kind
+
+Changing `workloadKind` replaces the workload object, so policy queries can fail while the new Pods start up.
+Products usually treat a failed policy query as a denied request.
+
+Changing to DaemonSet is the more disruptive direction.
+The role Service narrows to Local as soon as you apply the change, while the running Pods still belong to the outgoing Deployment and cover only some nodes.
+Products on the remaining nodes fail until the DaemonSet has rolled out everywhere.
+
+To ease the interruption, pin the traffic policy to Cluster as described in <<override-traffic-policy>>, and remove the override once the rollout has finished.
+This keeps every product able to reach any OPA Pod throughout the change.
+A short window in which no Pod is ready can still occur, because the outgoing workload is removed as the new one starts.

docs/modules/opa/partials/nav.adoc

@@ -3,6 +3,7 @@
 ** xref:opa:getting_started/first_steps.adoc[]
 * xref:opa:usage-guide/index.adoc[]
 ** xref:opa:usage-guide/listenerclass.adoc[]
+** xref:opa:usage-guide/workload-kind.adoc[]
 ** xref:opa:usage-guide/policies.adoc[]
 ** xref:opa:usage-guide/user-info-fetcher.adoc[]
 ** xref:opa:usage-guide/resource-info-fetcher.adoc[]
@@ -14,7 +15,7 @@
 ** xref:opa:usage-guide/tls.adoc[]
 ** xref:opa:usage-guide/operations/index.adoc[]
 *** xref:opa:usage-guide/operations/cluster-operations.adoc[]
-// *** xref:hdfs:usage-guide/operations/pod-placement.adoc[] Missing
+*** xref:opa:usage-guide/operations/pod-placement.adoc[]
 *** xref:opa:usage-guide/operations/pod-disruptions.adoc[]
 *** xref:opa:usage-guide/operations/graceful-shutdown.adoc[]
 * xref:opa:reference/index.adoc[]

extra/crds.yaml

@@ -1493,10 +1493,54 @@ spec:
                     type: object
                     x-kubernetes-preserve-unknown-fields: true
                   roleConfig:
-                    default: {}
-                    description: |-
-                      This is a product-agnostic RoleConfig, with nothing in it. It is used e.g. by products that have
-                      nothing configurable at role level.
+                    default:
+                      podDisruptionBudget:
+                        enabled: null
+                        maxUnavailable: null
+                      workloadKind: DaemonSet
+                    description: Role-level configuration for the OPA servers.
+                    properties:
+                      podDisruptionBudget:
+                        default:
+                          enabled: null
+                          maxUnavailable: null
+                        description: |-
+                          This struct is used to configure:
+
+                          1. If PodDisruptionBudgets are created by the operator
+                          2. The allowed number of Pods to be unavailable (`maxUnavailable`)
+
+                          Documentation:
+                          [allowed Pod disruptions documentation](https://docs.stackable.tech/home/nightly/concepts/operations/pod_disruptions).
+                        properties:
+                          enabled:
+                            description: |-
+                              Whether a PodDisruptionBudget should be written out for this role.
+
+                              Defaults to `true` when `workloadKind` is `Deployment` and to `false` when it is
+                              `DaemonSet`, since a PodDisruptionBudget doesn't make sense for a DaemonSet.
+                            nullable: true
+                            type: boolean
+                          maxUnavailable:
+                            description: The number of Pods that are allowed to be down simultaneous.
+                            format: uint16
+                            maximum: 65535.0
+                            minimum: 0.0
+                            nullable: true
+                            type: integer
+                        type: object
+                      workloadKind:
+                        default: DaemonSet
+                        description: |-
+                          The Kubernetes workload the OPA servers run as.
+
+                          * `DaemonSet`: one Pod per node. `replicas` is ignored.
+
+                          * `Deployment`: fixed number of Pods, configured by `replicas`.
+                        enum:
+                        - DaemonSet
+                        - Deployment
+                        type: string
                     type: object
                   roleGroups:
                     additionalProperties:
@@ -3754,10 +3798,54 @@ spec:
                     type: object
                     x-kubernetes-preserve-unknown-fields: true
                   roleConfig:
-                    default: {}
-                    description: |-
-                      This is a product-agnostic RoleConfig, with nothing in it. It is used e.g. by products that have
-                      nothing configurable at role level.
+                    default:
+                      podDisruptionBudget:
+                        enabled: null
+                        maxUnavailable: null
+                      workloadKind: DaemonSet
+                    description: Role-level configuration for the OPA servers.
+                    properties:
+                      podDisruptionBudget:
+                        default:
+                          enabled: null
+                          maxUnavailable: null
+                        description: |-
+                          This struct is used to configure:
+
+                          1. If PodDisruptionBudgets are created by the operator
+                          2. The allowed number of Pods to be unavailable (`maxUnavailable`)
+
+                          Documentation:
+                          [allowed Pod disruptions documentation](https://docs.stackable.tech/home/nightly/concepts/operations/pod_disruptions).
+                        properties:
+                          enabled:
+                            description: |-
+                              Whether a PodDisruptionBudget should be written out for this role.
+
+                              Defaults to `true` when `workloadKind` is `Deployment` and to `false` when it is
+                              `DaemonSet`, since a PodDisruptionBudget doesn't make sense for a DaemonSet.
+                            nullable: true
+                            type: boolean
+                          maxUnavailable:
+                            description: The number of Pods that are allowed to be down simultaneous.
+                            format: uint16
+                            maximum: 65535.0
+                            minimum: 0.0
+                            nullable: true
+                            type: integer
+                        type: object
+                      workloadKind:
+                        default: DaemonSet
+                        description: |-
+                          The Kubernetes workload the OPA servers run as.
+
+                          * `DaemonSet`: one Pod per node. `replicas` is ignored.
+
+                          * `Deployment`: fixed number of Pods, configured by `replicas`.
+                        enum:
+                        - DaemonSet
+                        - Deployment
+                        type: string
                     type: object
                   roleGroups:
                     additionalProperties:

rust/operator-binary/src/controller/apply.rs

@@ -83,21 +83,29 @@ impl<'a> Applier<'a> {
         // compile here instead of silently never being applied.
         let KubernetesResources {
             daemon_sets,
+            deployments,
             services,
             config_maps,
             service_accounts,
             role_bindings,
+            pod_disruption_budgets,
             status: _,
         } = resources;
 
-        // Apply order is: DaemonSets last (a changed mounted ConfigMap must exist first, else the
-        // Pods restart a second time -- commons-operator#111). The ServiceAccount comes first
-        // because the Pods reference it at creation time.
+        // Apply order is: the workload objects last (a changed mounted ConfigMap must exist first,
+        // else the Pods restart a second time -- commons-operator#111). The ServiceAccount comes
+        // first because the Pods reference it at creation time.
+        //
+        // A role group runs as either a DaemonSet or a Deployment, so for any given role group only
+        // one of the two collections has an entry; applying both is how the other one gets cleaned
+        // up as orphaned after a `workloadKind` switch.
         let service_accounts = self.add_resources(service_accounts).await?;
         let role_bindings = self.add_resources(role_bindings).await?;
         let services = self.add_resources(services).await?;
         let config_maps = self.add_resources(config_maps).await?;
+        let pod_disruption_budgets = self.add_resources(pod_disruption_budgets).await?;
         let daemon_sets = self.add_resources(daemon_sets).await?;
+        let deployments = self.add_resources(deployments).await?;
 
         self.remove_legacy_field_manager_scope(&daemon_sets).await?;
 
@@ -108,10 +116,12 @@ impl<'a> Applier<'a> {
 
         Ok(KubernetesResources {
             daemon_sets,
+            deployments,
             services,
             config_maps,
             service_accounts,
             role_bindings,
+            pod_disruption_budgets,
             status: PhantomData,
         })
     }