Operators become ready before the CRD has been installed

[NickLarsenNZ]

This leads to errors as seen in getting_started scripts (described in a slack thread):

Creating Airflow cluster
error: resource mapping not found for name: "airflow" namespace: "" from "airflow.yaml": no matches for kind "AirflowCluster" in version "airflow.stackable.tech/v1alpha1"
ensure CRDs are installed first

The workaround is to add a sleep 20

Why now

Previously, Helm installed the CRDs, and did so before deploying the operators.

Now Helm deploys the operators without CRDs (by default), and the CRDs are created after the operator pod is up. An apply of a CR that happens straight after that but before the maintainer has created the CRD will fail.

Possible solution

Add a startupProbe endpoint which only gives 200 OK when the CRD has been deployed.

[lfrancke]

Scope

1. Core: make an operator pod stay not-Ready until its CRD is established. The operator already
   computes this (signal::crd_established) apparently so my hope is this is not too hard.
   Add a health route reflecting crd_established and a startup probe in the shared Deployment template.
2. Follow-on: remove the CRD-wait workarounds in the getting-started scripts once readiness stuff works.

Done when

• On a fresh install, the operator Deployment reports Ready only after its CRD is established, and
  a CR applied immediately after readiness succeeds with no sleep or poll.
• The getting-started scripts no longer need their CRD workarounds (each operator seems to use a different one...)

Out of scope

• Any other probes (liveness, readiness, ...)
• Other health checks

Expected Effort

2-3 days for the endpoint, the probe, a test on one operator, assuming the endpoint can reuse the existing webhook server.
If that is not the case stop and let me know.
The getting-started cleanup and the rollout are not part of that effort.

[xeniape]

PRs related to this issue

• feat: Add /ready endpoint to WebhookServer operator-rs#1272
• operator rollout
• add startup probe in operator deployments
• getting started cleanup