From 6145e26f0172b95650de9804e2ac6fca68007c19 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:23:24 -0700 Subject: [PATCH 01/43] docs: add kubernetes-agent solution design spec Co-Authored-By: Claude Sonnet 4.6 --- ...-08-25-kubernetes-agent-solution-design.md | 162 ++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md diff --git a/docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md b/docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md new file mode 100644 index 00000000..6f57e283 --- /dev/null +++ b/docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md @@ -0,0 +1,162 @@ +# Kubernetes Agent Solution Design + +**Date:** 2026-08-25 +**Status:** Approved +**Linear:** [CX-14](https://linear.app/cortexio/issue/CX-14/cli-solution-kubernetes) + +## Overview + +A Cortex CLI solution bundle that demonstrates the Kubernetes agent integration end-to-end. Users run the solution inside a GitHub Codespace with a pre-configured kind cluster. One command installs the k8s-agent (via public helm chart), deploys sample workloads (Deployment, StatefulSet, CronJob, Argo Rollout), and configures a demo entity in Cortex — making it easy to show how the integration works without any local setup. + +**Audience:** Internal testers initially; broadens when the k8s-agent image is made public. + +--- + +## Architecture + +### Two-Phase Setup + +**Phase 1 — Devcontainer (one-time Codespace initialization)** + +`.devcontainer/kubernetes-agent/devcontainer.json` configures the Codespace environment: + +- Base image: `mcr.microsoft.com/devcontainers/base:ubuntu` with Docker-in-Docker feature +- `onCreate` script installs: `kind`, `kubectl`, `helm` +- `onCreate` script creates the kind cluster: `kind create cluster --name cortex-demo` + +Users pre-set two Codespace secrets before opening the Codespace: +- `CORTEX_API_KEY` — their Cortex API key +- `GHCR_TOKEN` — GitHub PAT with `read:packages` scope (requested from Cortex support until image is public) + +Both secrets are injected as environment variables automatically when the Codespace starts. + +**Phase 2 — Solution post-install (user-initiated)** + +```bash +cortex solutions install -s kubernetes-agent +cortex solutions post-install -s kubernetes-agent +``` + +The post-install script (`setup.py`) runs the following steps in order, with idempotency via `already_done()` / `mark_done()` state: + +1. **Prompt for inputs** — `CORTEX_API_KEY`, `GHCR_TOKEN`, `CORTEX_BASE_URL` (default: `https://api.getcortexapp.com`), cluster name (default: `demo`) +2. **Create k8s image pull secret** — `kubectl create secret docker-registry cortex-ghcr-secret --docker-server=ghcr.io --docker-username= --docker-password=` +3. **Helm install k8s-agent** — from the public helm chart repo; passes `CORTEX_API_KEY`, `CORTEX_BASE_URL`, cluster name, and pull secret name as values +4. **Wait for agent readiness** — polls `kubectl rollout status deployment/k8s-agent` with timeout +5. **Install Argo Rollouts CRD** — `kubectl apply -f ` +6. **Apply demo k8s manifests** — `kubectl apply -f manifests/` (Deployment, StatefulSet, CronJob, Rollout — all annotated `cortex.io/tag: demo-kubernetes`) +7. **Create demo Cortex entity** — `cortex catalog create -f catalog/demo-kubernetes.yaml` + +Agent auto-registers with Cortex on connect using the API key — no explicit Cortex-side integration configuration step required. + +--- + +## Solution Bundle Structure + +``` +cortexapps_cli/solutions/kubernetes-agent/ +├── README.md +├── setup.py +├── catalog/ +│ └── demo-kubernetes.yaml # demo service entity, tag: demo-kubernetes +└── manifests/ # k8s demo workloads (sourced from internal/k8s/manifests) + ├── deployment.yaml # nginx Deployment, annotated cortex.io/tag: demo-kubernetes + ├── statefulset.yaml # StatefulSet, annotated cortex.io/tag: demo-kubernetes + ├── cronjob.yaml # CronJob, annotated cortex.io/tag: demo-kubernetes + └── rollout.yaml # Argo Rollout (workloadRef → deployment), annotated demo-kubernetes +``` + +``` +.devcontainer/kubernetes-agent/ +├── devcontainer.json +└── onCreate.sh # kind cluster creation + tool install script +``` + +### Demo Entity (`catalog/demo-kubernetes.yaml`) + +```yaml +openapi: 3.0.0 +info: + title: Demo Kubernetes + description: Demo entity for the Kubernetes agent integration + x-cortex-tag: demo-kubernetes + x-cortex-type: service +``` + +### Demo Manifests + +All four manifests are adapted from `internal/k8s/manifests/` with the cortex tag updated from `k8s-test-annotation` to `demo-kubernetes`. The Rollout uses `workloadRef` pointing to the Deployment (same pattern as `k8s-test-rollout.yaml`). + +--- + +## Helm Chart + +The k8s-agent helm chart is public. The solution uses `helm repo add` + `helm install` — the chart URL needs to be confirmed and hardcoded before implementation begins. + +Key helm values passed by the setup script: +- `cortexApiKey` — from `CORTEX_API_KEY` +- `cortexBaseUrl` — from `CORTEX_BASE_URL` +- `clusterName` — from user prompt (default: `demo`) +- `image.pullSecrets[0].name` — `cortex-ghcr-secret` + +--- + +## `setup.py` Design + +Follows the `SolutionSetup` base class pattern (same as workday): + +```python +class KubernetesAgentSetup(SolutionSetup): + solution_tag = "kubernetes-agent" + + def collect_prompts(self): + # Prompt for CORTEX_API_KEY, GHCR_TOKEN, CORTEX_BASE_URL, cluster name + # Reads from env vars first (Codespace secrets auto-inject them) + + def steps(self): + return [ + ("Create image pull secret", self._create_pull_secret), + ("Install k8s-agent via helm", self._helm_install), + ("Wait for agent readiness", self._wait_for_readiness), + ("Install Argo Rollouts CRD", self._install_argo_crds), + ("Apply demo k8s manifests", self._apply_manifests), + ("Create demo Cortex entity", self._create_entity), + ] + + def post_steps(self): + # Print success message + link to Cortex entity k8s tab +``` + +State persistence via `~/.cortex/solutions/kubernetes-agent.json` ensures re-runs skip completed steps. + +--- + +## Devcontainer + +`.devcontainer/kubernetes-agent/devcontainer.json`: +- Uses Docker-in-Docker feature so kind can run containers inside the Codespace container +- `onCreate` installs kind, kubectl, helm via apt/curl/brew and creates the `cortex-demo` kind cluster +- Codespace secrets (`CORTEX_API_KEY`, `GHCR_TOKEN`) are automatically available as env vars in the terminal + +--- + +## Image Credential Note + +The k8s-agent container image (`ghcr.io/cortexapps/k8s-agent/k8s-agent`) is currently private on GHCR. Users need a `GHCR_TOKEN` (GitHub PAT, `read:packages` scope) obtained from Cortex support. This requirement goes away once the image is made public — the pull secret creation step will be removed at that point. + +--- + +## Out of Scope (v1) + +- Scorecard +- Making the GHCR image public (tracked separately) +- Playwright UI verification (internal tooling only) +- Local machine support (Codespace only for v1) + +--- + +## Open Questions + +1. **Helm chart URL** — needs to be confirmed before implementation. Is it hosted at a `cortexapps` GitHub Pages repo? +2. **Argo Rollouts CRD URL** — standard upstream URL (`https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml`) or a pinned version? +3. **`GHCR_TOKEN` username** — does the pull secret need a real GitHub username or can it be a placeholder (some GHCR PATs work with any username)? From 3e30d71968500aa8c55435b801422a912a525239 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:25:07 -0700 Subject: [PATCH 02/43] docs: resolve open questions in kubernetes-agent solution spec Co-Authored-By: Claude Sonnet 4.6 --- .../specs/2026-08-25-kubernetes-agent-solution-design.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md b/docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md index 6f57e283..9d6e987d 100644 --- a/docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md +++ b/docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md @@ -155,8 +155,8 @@ The k8s-agent container image (`ghcr.io/cortexapps/k8s-agent/k8s-agent`) is curr --- -## Open Questions +## Resolved Decisions -1. **Helm chart URL** — needs to be confirmed before implementation. Is it hosted at a `cortexapps` GitHub Pages repo? -2. **Argo Rollouts CRD URL** — standard upstream URL (`https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml`) or a pinned version? -3. **`GHCR_TOKEN` username** — does the pull secret need a real GitHub username or can it be a placeholder (some GHCR PATs work with any username)? +1. **Helm chart** — bundle from `internal/k8s/helm-chart/` into the solution directory; no public helm repo exists yet. +2. **Argo Rollouts CRD URL** — `https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/crds/rollout-crd.yaml` (just the CRD, not the full controller; `stable` channel). +3. **GHCR pull secret username** — use placeholder `cortex`; GHCR PATs authenticate by token, not username. From 9f684e75feb7fd14b82a3dcce3c05d1a16f699de Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:29:25 -0700 Subject: [PATCH 03/43] docs: add kubernetes-agent solution implementation plan Co-Authored-By: Claude Sonnet 4.6 --- .../2026-08-25-kubernetes-agent-solution.md | 754 ++++++++++++++++++ 1 file changed, 754 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-25-kubernetes-agent-solution.md diff --git a/docs/superpowers/plans/2026-08-25-kubernetes-agent-solution.md b/docs/superpowers/plans/2026-08-25-kubernetes-agent-solution.md new file mode 100644 index 00000000..93af123d --- /dev/null +++ b/docs/superpowers/plans/2026-08-25-kubernetes-agent-solution.md @@ -0,0 +1,754 @@ +# Kubernetes Agent Solution Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a `kubernetes-agent` Cortex CLI solution bundle that deploys the k8s-agent + demo workloads in a GitHub Codespace kind cluster, demonstrating the Cortex k8s integration end-to-end. + +**Architecture:** A devcontainer configures a kind cluster on Codespace open; `cortex solutions post-install -s kubernetes-agent` creates k8s secrets, helm-installs the agent from the bundled chart, applies demo workloads, and registers one Cortex entity. The agent auto-registers with Cortex on connect. + +**Tech Stack:** Python 3.11+, Typer (CLI), `subprocess` for kubectl/helm, `requests` for GHCR tag fetch + Cortex API, kind (k8s in Docker), helm 3, GitHub Codespaces devcontainer. + +**Spec:** `docs/superpowers/specs/2026-08-25-kubernetes-agent-solution-design.md` + +## Global Constraints + +- Python 3.11+; follow patterns in `cortexapps_cli/solutions/workday/setup.py` exactly +- `SolutionSetup` base class: `cortexapps_cli/solutions/_lib/setup_base.py` — read it before coding `setup.py` +- Solution tag: `kubernetes-agent`; Cortex entity tag: `demo-kubernetes` +- All kubectl/helm calls: `subprocess.run([...], check=True)` +- k8s secret names are fixed: API key secret = `cortex-key` (key field = `api-key`); image pull secret = `cortex-docker-registry-secret` +- Helm chart lives at `cortexapps_cli/solutions/kubernetes-agent/helm-chart/` (bundled, not a remote repo) +- Argo Rollouts CRD URL: `https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/crds/rollout-crd.yaml` +- GHCR image: `ghcr.io/cortexapps/k8s-agent/k8s-agent`; tag fetched from GitHub API at setup time +- All manifests annotated with `cortex.io/tag: demo-kubernetes` +- Workload names: `demo-deployment`, `demo-statefulset`, `demo-cronjob`, `demo-rollout` + +--- + +## File Map + +**Create:** +- `.devcontainer/kubernetes-agent/devcontainer.json` — Codespace config with Docker-in-Docker + tool install +- `.devcontainer/kubernetes-agent/onCreate.sh` — installs kind/kubectl/helm, creates kind cluster +- `cortexapps_cli/solutions/kubernetes-agent/catalog/demo-kubernetes.yaml` — Cortex service entity +- `cortexapps_cli/solutions/kubernetes-agent/manifests/deployment.yaml` — nginx Deployment +- `cortexapps_cli/solutions/kubernetes-agent/manifests/statefulset.yaml` — nginx StatefulSet +- `cortexapps_cli/solutions/kubernetes-agent/manifests/cronjob.yaml` — busybox CronJob +- `cortexapps_cli/solutions/kubernetes-agent/manifests/rollout.yaml` — Argo Rollout (workloadRef → deployment) +- `cortexapps_cli/solutions/kubernetes-agent/helm-chart/` — copied verbatim from `internal/k8s/helm-chart/` +- `cortexapps_cli/solutions/kubernetes-agent/setup.py` — post-install automation script +- `cortexapps_cli/solutions/kubernetes-agent/README.md` — user-facing docs + +--- + +### Task 1: Catalog entity + demo k8s manifests + +**Files:** +- Create: `cortexapps_cli/solutions/kubernetes-agent/catalog/demo-kubernetes.yaml` +- Create: `cortexapps_cli/solutions/kubernetes-agent/manifests/deployment.yaml` +- Create: `cortexapps_cli/solutions/kubernetes-agent/manifests/statefulset.yaml` +- Create: `cortexapps_cli/solutions/kubernetes-agent/manifests/cronjob.yaml` +- Create: `cortexapps_cli/solutions/kubernetes-agent/manifests/rollout.yaml` + +**Interfaces:** +- Produces: `catalog/demo-kubernetes.yaml` (consumed by Task 3 setup.py entity creation step), manifests dir (consumed by Task 3 manifest apply step) + +- [ ] **Step 1: Create the solution directory structure** + +```bash +mkdir -p cortexapps_cli/solutions/kubernetes-agent/catalog +mkdir -p cortexapps_cli/solutions/kubernetes-agent/manifests +``` + +- [ ] **Step 2: Create `catalog/demo-kubernetes.yaml`** + +```yaml +openapi: 3.0.0 +info: + title: Demo Kubernetes + description: Demo entity for the Kubernetes agent integration + x-cortex-tag: demo-kubernetes + x-cortex-type: service +``` + +- [ ] **Step 3: Create `manifests/deployment.yaml`** + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: demo-deployment + labels: + app: demo-k8s-label + annotations: + cortex.io/tag: demo-kubernetes +spec: + replicas: 1 + selector: + matchLabels: + app: demo-k8s + template: + metadata: + labels: + app: demo-k8s + spec: + containers: + - name: hello + image: nginx:alpine + ports: + - containerPort: 80 +``` + +- [ ] **Step 4: Create `manifests/statefulset.yaml`** + +```yaml +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: demo-statefulset + labels: + app: demo-k8s-label + annotations: + cortex.io/tag: demo-kubernetes +spec: + serviceName: demo-k8s + replicas: 1 + selector: + matchLabels: + app: demo-k8s-ss + template: + metadata: + labels: + app: demo-k8s-ss + spec: + containers: + - name: hello + image: nginx:alpine + ports: + - containerPort: 80 +``` + +- [ ] **Step 5: Create `manifests/cronjob.yaml`** + +```yaml +apiVersion: batch/v1 +kind: CronJob +metadata: + name: demo-cronjob + labels: + app: demo-k8s-label + annotations: + cortex.io/tag: demo-kubernetes +spec: + schedule: "*/10 * * * *" + jobTemplate: + spec: + template: + spec: + containers: + - name: hello + image: busybox:latest + command: + - /bin/sh + - -c + - echo "$(date '+%Y-%m-%d %H:%M:%S') - Hello from demo-kubernetes cronjob" >> /tmp/hello-world.txt + restartPolicy: OnFailure +``` + +- [ ] **Step 6: Create `manifests/rollout.yaml`** + +The Rollout references `demo-deployment` via `workloadRef` — Cortex resolves containers from the referenced Deployment. + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Rollout +metadata: + name: demo-rollout + labels: + app: demo-k8s-label + annotations: + cortex.io/tag: demo-kubernetes +spec: + replicas: 1 + selector: + matchLabels: + app: demo-k8s-rollout + workloadRef: + apiVersion: apps/v1 + kind: Deployment + name: demo-deployment + scaleDown: onsuccess + strategy: + canary: + steps: + - setWeight: 100 +``` + +- [ ] **Step 7: Validate all YAML files parse correctly** + +```bash +python -c " +import yaml, pathlib +for f in pathlib.Path('cortexapps_cli/solutions/kubernetes-agent').rglob('*.yaml'): + try: + yaml.safe_load(f.read_text()) + print(f'OK: {f}') + except yaml.YAMLError as e: + print(f'FAIL: {f}: {e}') + exit(1) +" +``` + +Expected: `OK:` line for each `.yaml` file, no FAIL lines. + +- [ ] **Step 8: Commit** + +```bash +git add cortexapps_cli/solutions/kubernetes-agent/catalog/ \ + cortexapps_cli/solutions/kubernetes-agent/manifests/ +git commit -m "feat: add kubernetes-agent solution catalog entity and demo manifests" +``` + +--- + +### Task 2: Bundle helm chart + +**Files:** +- Create: `cortexapps_cli/solutions/kubernetes-agent/helm-chart/` (copy of `internal/k8s/helm-chart/`) + +**Interfaces:** +- Produces: `helm-chart/` directory (consumed by Task 3 helm install step — path is `Path(__file__).parent / "helm-chart"`) + +- [ ] **Step 1: Copy the helm chart from internal** + +```bash +cp -r internal/k8s/helm-chart cortexapps_cli/solutions/kubernetes-agent/helm-chart +``` + +- [ ] **Step 2: Remove the dev-only comment from the deployment template** + +The template at `helm-chart/templates/deployment.yaml` has a commented minikube host alias block that is confusing in a public-facing solution. Remove lines 26–31: + +``` + ######### remove before deploy - used for local testing ########### + # hostAliases: + # - ip: "192.168.64.1" + # hostnames: + # - "host.minikube.internal" + ################################################################### +``` + +Open `cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/deployment.yaml` and delete those 6 lines. + +- [ ] **Step 3: Add a warning comment to `helm-chart/Chart.yaml`** + +Open `cortexapps_cli/solutions/kubernetes-agent/helm-chart/Chart.yaml` and add a comment at the top: + +```yaml +# Bundled copy of the Cortex k8s-agent helm chart for demo purposes. +# This copy is not kept up-to-date. Once the chart is published to a +# public helm repo, this bundle will be replaced with a helm repo reference. +apiVersion: v2 +name: cortex-k8s-agent +description: A Helm chart for deploying Cortex K8s agent in your cluster +type: application +version: 0.1.0 +appVersion: 1.16.0 +``` + +- [ ] **Step 4: Verify helm can render the chart (requires helm installed locally)** + +```bash +helm template test-release cortexapps_cli/solutions/kubernetes-agent/helm-chart \ + --set image.tag=test \ + --set app.keySecret=cortex-key \ + --set app.baseUrl=https://api.getcortexapp.com \ + --set app.clusterName=demo \ + > /dev/null && echo "Helm template OK" +``` + +Expected: `Helm template OK` with no errors. If helm is not installed locally, skip this step — it will be verified in the Codespace. + +- [ ] **Step 5: Commit** + +```bash +git add cortexapps_cli/solutions/kubernetes-agent/helm-chart/ +git commit -m "feat: bundle k8s-agent helm chart in kubernetes-agent solution" +``` + +--- + +### Task 3: setup.py + +**Files:** +- Create: `cortexapps_cli/solutions/kubernetes-agent/setup.py` + +**Interfaces:** +- Consumes: `catalog/demo-kubernetes.yaml` (`Path(__file__).parent / "catalog" / "demo-kubernetes.yaml"`), `manifests/` dir, `helm-chart/` dir +- Consumes: `SolutionSetup` base from `cortexapps_cli/solutions/_lib/setup_base.py` — read this file before writing setup.py to understand all available methods +- Produces: `main(cortex_api_key, cortex_base_url, no_prompt, **kwargs)` entry point (called by `cortex solutions post-install`) + +**Before coding:** Read `cortexapps_cli/solutions/_lib/setup_base.py` in full to understand `prompt()`, `confirm()`, `mark_done()`, `already_done()`, `mark_undone()`, and how `steps()` returns `list[tuple[str, callable]]`. + +Also read `cortexapps_cli/solutions/workday/setup.py` for the exact class pattern to follow. + +- [ ] **Step 1: Create `setup.py` with imports and constants** + +```python +""" +Post-install setup script for the kubernetes-agent solution. +Deploys the Cortex k8s-agent to a kind cluster and creates a demo entity. +Run via: cortex solutions post-install -s kubernetes-agent +""" + +SETUP_DESCRIPTION = ( + "This solution deploys the Cortex Kubernetes agent to a local kind cluster " + "and creates a demo entity to demonstrate the k8s integration." +) + +import subprocess +import sys +from pathlib import Path + +import requests + +try: + from cortexapps_cli.solutions._lib.setup_base import SolutionSetup +except ImportError: + sys.path.insert(0, str(Path(__file__).parent.parent.parent)) + from _lib.setup_base import SolutionSetup + +SOLUTION_DIR = Path(__file__).parent +CATALOG_FILE = SOLUTION_DIR / "catalog" / "demo-kubernetes.yaml" +MANIFESTS_DIR = SOLUTION_DIR / "manifests" +HELM_CHART_DIR = SOLUTION_DIR / "helm-chart" + +GHCR_IMAGE = "ghcr.io/cortexapps/k8s-agent/k8s-agent" +ARGO_CRD_URL = "https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/crds/rollout-crd.yaml" +``` + +- [ ] **Step 2: Create the `KubernetesAgentSetup` class with `__init__` and `collect_prompts`** + +```python +class KubernetesAgentSetup(SolutionSetup): + solution_tag = "kubernetes-agent" + + def __init__( + self, + cortex_api_key: str = None, + cortex_base_url: str = None, + no_prompt: bool = False, + **kwargs, + ): + super().__init__(no_prompt=no_prompt, **kwargs) + self._api_key = cortex_api_key or "" + self._base_url = (cortex_base_url or "https://api.getcortexapp.com").rstrip("/") + self._ghcr_token = "" + self._cluster_name = "" + + def _cortex_headers(self) -> dict: + return { + "Authorization": f"Bearer {self._api_key}", + "Content-Type": "application/yaml", + } + + def collect_prompts(self) -> None: + self._ghcr_token = self.prompt( + "GHCR_TOKEN", + description="GitHub PAT with read:packages scope for pulling the k8s-agent image", + env_var="GHCR_TOKEN", + secret=True, + ) + self._cluster_name = self.prompt( + "cluster_name", + description="Name for this cluster as it will appear in Cortex", + default="demo", + ) +``` + +- [ ] **Step 3: Add `_fetch_image_tag` helper** + +```python + def _fetch_image_tag(self) -> str: + """Fetch the latest k8s-agent image tag from the GitHub API.""" + r = requests.get( + "https://api.github.com/orgs/cortexapps/packages/container/k8s-agent%2Fk8s-agent/versions", + headers={ + "Authorization": f"Bearer {self._ghcr_token}", + "Accept": "application/vnd.github+json", + }, + ) + r.raise_for_status() + versions = r.json() + if not versions: + raise RuntimeError("No k8s-agent versions found in GHCR — is GHCR_TOKEN valid?") + tags = versions[0].get("metadata", {}).get("container", {}).get("tags", []) + tag = tags[0] if tags else "" + if not tag: + raise RuntimeError("Could not determine k8s-agent image tag from GHCR API response") + print(f" Using image tag: {tag}") + return tag +``` + +- [ ] **Step 4: Add `_create_secrets` step** + +```python + def _create_secrets(self) -> None: + if self.already_done("create_secrets"): + return + print(" Creating cortex-docker-registry-secret...") + subprocess.run( + [ + "kubectl", "create", "secret", "docker-registry", + "cortex-docker-registry-secret", + "--docker-server=ghcr.io", + "--docker-username=cortex", + f"--docker-password={self._ghcr_token}", + "--dry-run=client", "-o", "yaml", + ], + check=True, + capture_output=True, + ) + # Pipe output to apply (dry-run=client means we need to apply separately) + result = subprocess.run( + [ + "kubectl", "create", "secret", "docker-registry", + "cortex-docker-registry-secret", + "--docker-server=ghcr.io", + "--docker-username=cortex", + f"--docker-password={self._ghcr_token}", + "--dry-run=client", "-o", "yaml", + ], + check=True, + capture_output=True, + ) + subprocess.run(["kubectl", "apply", "-f", "-"], input=result.stdout, check=True) + + print(" Creating cortex-key secret...") + result = subprocess.run( + [ + "kubectl", "create", "secret", "generic", "cortex-key", + f"--from-literal=api-key={self._api_key}", + "--dry-run=client", "-o", "yaml", + ], + check=True, + capture_output=True, + ) + subprocess.run(["kubectl", "apply", "-f", "-"], input=result.stdout, check=True) + self.mark_done("create_secrets") +``` + +- [ ] **Step 5: Add `_helm_install` step** + +```python + def _helm_install(self) -> None: + if self.already_done("helm_install"): + return + image_tag = self._fetch_image_tag() + print(f" Installing k8s-agent via helm (chart: {HELM_CHART_DIR})...") + subprocess.run( + [ + "helm", "upgrade", "--install", "cortex-k8s-agent", + str(HELM_CHART_DIR), + "--set", f"image.tag={image_tag}", + "--set", f"app.baseUrl={self._base_url}", + "--set", f"app.clusterName={self._cluster_name}", + ], + check=True, + ) + # Restart to ensure secrets/configmaps are picked up + subprocess.run( + ["kubectl", "rollout", "restart", "deployment", + "-l", "app.kubernetes.io/name=cortex-k8s-agent"], + check=True, + ) + self.mark_done("helm_install") +``` + +- [ ] **Step 6: Add `_wait_for_readiness` step** + +```python + def _wait_for_readiness(self) -> None: + if self.already_done("wait_for_readiness"): + return + print(" Waiting for k8s-agent pod to be ready (timeout: 120s)...") + subprocess.run( + [ + "kubectl", "rollout", "status", "deployment", + "-l", "app.kubernetes.io/name=cortex-k8s-agent", + "--timeout=120s", + ], + check=True, + ) + self.mark_done("wait_for_readiness") +``` + +- [ ] **Step 7: Add `_install_argo_crd` step** + +```python + def _install_argo_crd(self) -> None: + if self.already_done("install_argo_crd"): + return + print(f" Installing Argo Rollouts CRD...") + subprocess.run( + ["kubectl", "apply", "-f", ARGO_CRD_URL], + check=True, + ) + self.mark_done("install_argo_crd") +``` + +- [ ] **Step 8: Add `_apply_manifests` step** + +```python + def _apply_manifests(self) -> None: + if self.already_done("apply_manifests"): + return + print(f" Applying demo k8s manifests from {MANIFESTS_DIR}...") + subprocess.run( + ["kubectl", "apply", "-f", str(MANIFESTS_DIR)], + check=True, + ) + self.mark_done("apply_manifests") +``` + +- [ ] **Step 9: Add `_create_entity` step** + +```python + def _create_entity(self) -> None: + if self.already_done("create_entity"): + return + print(f" Creating demo-kubernetes Cortex entity...") + yaml_content = CATALOG_FILE.read_bytes() + r = requests.post( + f"{self._base_url}/api/v1/open-api", + headers={ + "Authorization": f"Bearer {self._api_key}", + "Content-Type": "application/openapi;charset=UTF-8", + }, + data=yaml_content, + ) + if not r.ok: + raise RuntimeError( + f"Failed to create Cortex entity: {r.status_code} {r.text}" + ) + self.mark_done("create_entity") +``` + +- [ ] **Step 10: Add `steps`, `post_steps`, and `main`** + +```python + def steps(self) -> list: + return [ + ("Create k8s secrets", self._create_secrets), + ("Install k8s-agent via helm", self._helm_install), + ("Wait for agent readiness", self._wait_for_readiness), + ("Install Argo Rollouts CRD", self._install_argo_crd), + ("Apply demo k8s manifests", self._apply_manifests), + ("Create demo Cortex entity", self._create_entity), + ] + + def post_steps(self) -> None: + print("\n✓ Kubernetes agent deployed and demo workloads running.\n") + print("The agent syncs every 5 minutes. After the first sync, visit:") + print(f" {self._base_url.replace('api.', 'app.')}/catalog/demo-kubernetes/k8s") + print("\nYou should see: demo-deployment, demo-statefulset, demo-cronjob, demo-rollout") + print("\nNote: GHCR_TOKEN requirement goes away once the k8s-agent image is made public.") + + +def main(cortex_api_key=None, cortex_base_url=None, no_prompt=False, **kwargs): + KubernetesAgentSetup( + cortex_api_key=cortex_api_key, + cortex_base_url=cortex_base_url, + no_prompt=no_prompt, + ).run() + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 11: Verify the script imports cleanly (no syntax errors)** + +```bash +python -c "import cortexapps_cli.solutions.kubernetes_agent.setup as s; print('OK')" +``` + +If that path doesn't work (no `__init__.py`), try: + +```bash +cd cortexapps_cli/solutions/kubernetes-agent && python -c "import setup; print('OK')" +``` + +Expected: `OK` + +- [ ] **Step 12: Commit** + +```bash +git add cortexapps_cli/solutions/kubernetes-agent/setup.py +git commit -m "feat: add kubernetes-agent solution post-install setup script" +``` + +--- + +### Task 4: Devcontainer + +**Files:** +- Create: `.devcontainer/kubernetes-agent/devcontainer.json` +- Create: `.devcontainer/kubernetes-agent/onCreate.sh` + +**Interfaces:** +- Produces: a working Codespace environment with kind cluster running, `CORTEX_API_KEY` and `GHCR_TOKEN` available as env vars from Codespace secrets + +- [ ] **Step 1: Create `devcontainer.json`** + +```json +{ + "name": "Cortex Kubernetes Agent Demo", + "image": "mcr.microsoft.com/devcontainers/base:ubuntu-24.04", + "features": { + "ghcr.io/devcontainers/features/docker-in-docker:2": {}, + "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": { + "version": "latest", + "helm": "latest", + "minikube": "none" + } + }, + "onCreateCommand": "bash .devcontainer/kubernetes-agent/onCreate.sh", + "remoteEnv": { + "CORTEX_API_KEY": "${localEnv:CORTEX_API_KEY}", + "GHCR_TOKEN": "${localEnv:GHCR_TOKEN}" + }, + "postCreateMessage": "Run: cortex solutions install -s kubernetes-agent && cortex solutions post-install -s kubernetes-agent" +} +``` + +Note: `kubectl-helm-minikube` feature installs kubectl + helm without minikube (set to `"none"`). kind is installed separately in `onCreate.sh`. + +- [ ] **Step 2: Create `onCreate.sh`** + +```bash +#!/usr/bin/env bash +set -euo pipefail + +echo "==> Installing kind..." +curl -Lo /usr/local/bin/kind \ + https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64 +chmod +x /usr/local/bin/kind + +echo "==> Creating kind cluster 'cortex-demo'..." +kind create cluster --name cortex-demo --wait 60s + +echo "==> Verifying cluster..." +kubectl cluster-info --context kind-cortex-demo + +echo "==> Installing cortexapps-cli..." +pip install cortexapps-cli --quiet + +echo "==> Done. Run: cortex solutions post-install -s kubernetes-agent" +``` + +- [ ] **Step 3: Verify `devcontainer.json` is valid JSON** + +```bash +python -c "import json; json.load(open('.devcontainer/kubernetes-agent/devcontainer.json')); print('OK')" +``` + +Expected: `OK` + +- [ ] **Step 4: Commit** + +```bash +git add .devcontainer/kubernetes-agent/ +git commit -m "feat: add kubernetes-agent Codespace devcontainer" +``` + +--- + +### Task 5: README + +**Files:** +- Create: `cortexapps_cli/solutions/kubernetes-agent/README.md` + +- [ ] **Step 1: Create `README.md`** + +```markdown +# Kubernetes Agent Solution + +Demonstrates the [Cortex Kubernetes agent](https://docs.cortex.io/docs/reference/integrations/kubernetes) integration using a GitHub Codespace with a local kind cluster. + +## What this installs + +- **Cortex k8s-agent** — connects your cluster to Cortex and syncs workload metadata +- **Demo workloads** — Deployment, StatefulSet, CronJob, and Argo Rollout, all tagged `demo-kubernetes` +- **demo-kubernetes** — a Cortex service entity that the workloads annotate to + +After setup, visit your entity's K8s tab to see live workload data synced from the cluster. + +## Prerequisites + +- A [GitHub Codespace](https://github.com/features/codespaces) opened from this repository +- A Cortex API key (`CORTEX_API_KEY`) — get from Cortex Settings → API Keys +- A GitHub PAT with `read:packages` scope (`GHCR_TOKEN`) — request from Cortex support + +Set both as [Codespace secrets](https://docs.github.com/en/codespaces/managing-your-codespaces/managing-secrets-for-your-codespaces) before opening the Codespace. + +## Quick start + +1. Open a Codespace from this repository (select the `kubernetes-agent` devcontainer configuration) +2. Wait for `onCreate` to finish (installs tools + creates kind cluster, ~2 min) +3. Run the solution: + +```bash +cortex solutions install -s kubernetes-agent +cortex solutions post-install -s kubernetes-agent +``` + +4. Wait ~5 minutes for the agent's first sync, then visit: + `https://app.getcortexapp.com/catalog/demo-kubernetes/k8s` + +## What you should see + +- `demo-deployment` (Deployment) +- `demo-statefulset` (StatefulSet) +- `demo-cronjob` (CronJob) +- `demo-rollout` (Argo Rollout — containers resolved from `demo-deployment`) + +## Re-running setup + +The setup script is idempotent — re-run `cortex solutions post-install -s kubernetes-agent` to retry any failed step. Completed steps are skipped. + +## Temporary limitation + +The k8s-agent image is currently private on GHCR, requiring `GHCR_TOKEN`. This requirement will be removed once the image is made public. +``` + +- [ ] **Step 2: Commit** + +```bash +git add cortexapps_cli/solutions/kubernetes-agent/README.md +git commit -m "docs: add kubernetes-agent solution README" +``` + +--- + +## Self-Review + +**Spec coverage check:** +- ✓ Devcontainer with Docker-in-Docker, kind install, onCreate → Task 4 +- ✓ `cortex solutions install` + `cortex solutions post-install` flow → Tasks 1–3 (install picks up catalog/; post-install runs setup.py) +- ✓ Create k8s image pull secret (`cortex-docker-registry-secret`) → Task 3 Step 4 +- ✓ Create API key secret (`cortex-key`, key `api-key`) → Task 3 Step 4 +- ✓ Fetch image tag from GHCR API → Task 3 Step 3 +- ✓ Helm install from bundled chart → Task 3 Step 5 +- ✓ Wait for agent readiness → Task 3 Step 6 +- ✓ Install Argo Rollouts CRD → Task 3 Step 7 +- ✓ Apply 4 demo manifests (Deployment, StatefulSet, CronJob, Rollout) → Tasks 1 + 3 Step 8 +- ✓ Rollout `workloadRef` points to `demo-deployment` → Task 1 Step 6 +- ✓ Create `demo-kubernetes` Cortex entity → Task 3 Step 9 +- ✓ Warning about GHCR_TOKEN requirement → Task 3 Step 10 (post_steps), Task 5 README +- ✓ Dev comment removed from helm chart deployment template → Task 2 Step 2 +- ✓ Helm chart bundle warning comment → Task 2 Step 3 + +**No placeholders found.** + +**Type consistency:** All step method names referenced in `steps()` (Task 3 Step 10) match the method definitions in Steps 4–9. `CATALOG_FILE`, `MANIFESTS_DIR`, `HELM_CHART_DIR` constants defined once in Step 1 and used consistently throughout. From b728ceac7898b456761df317ce7374cb2292bbed Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:33:29 -0700 Subject: [PATCH 04/43] feat: add kubernetes-agent solution catalog entity and demo manifests --- .../catalog/demo-kubernetes.yaml | 6 +++++ .../kubernetes-agent/manifests/cronjob.yaml | 22 +++++++++++++++++ .../manifests/deployment.yaml | 23 ++++++++++++++++++ .../kubernetes-agent/manifests/rollout.yaml | 22 +++++++++++++++++ .../manifests/statefulset.yaml | 24 +++++++++++++++++++ 5 files changed, 97 insertions(+) create mode 100644 cortexapps_cli/solutions/kubernetes-agent/catalog/demo-kubernetes.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/manifests/cronjob.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/manifests/deployment.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/manifests/rollout.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/manifests/statefulset.yaml diff --git a/cortexapps_cli/solutions/kubernetes-agent/catalog/demo-kubernetes.yaml b/cortexapps_cli/solutions/kubernetes-agent/catalog/demo-kubernetes.yaml new file mode 100644 index 00000000..b53abc82 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/catalog/demo-kubernetes.yaml @@ -0,0 +1,6 @@ +openapi: 3.0.0 +info: + title: Demo Kubernetes + description: Demo entity for the Kubernetes agent integration + x-cortex-tag: demo-kubernetes + x-cortex-type: service diff --git a/cortexapps_cli/solutions/kubernetes-agent/manifests/cronjob.yaml b/cortexapps_cli/solutions/kubernetes-agent/manifests/cronjob.yaml new file mode 100644 index 00000000..87016de3 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/manifests/cronjob.yaml @@ -0,0 +1,22 @@ +apiVersion: batch/v1 +kind: CronJob +metadata: + name: demo-cronjob + labels: + app: demo-k8s-label + annotations: + cortex.io/tag: demo-kubernetes +spec: + schedule: "*/10 * * * *" + jobTemplate: + spec: + template: + spec: + containers: + - name: hello + image: busybox:latest + command: + - /bin/sh + - -c + - echo "$(date '+%Y-%m-%d %H:%M:%S') - Hello from demo-kubernetes cronjob" >> /tmp/hello-world.txt + restartPolicy: OnFailure diff --git a/cortexapps_cli/solutions/kubernetes-agent/manifests/deployment.yaml b/cortexapps_cli/solutions/kubernetes-agent/manifests/deployment.yaml new file mode 100644 index 00000000..3d9fa182 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/manifests/deployment.yaml @@ -0,0 +1,23 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: demo-deployment + labels: + app: demo-k8s-label + annotations: + cortex.io/tag: demo-kubernetes +spec: + replicas: 1 + selector: + matchLabels: + app: demo-k8s + template: + metadata: + labels: + app: demo-k8s + spec: + containers: + - name: hello + image: nginx:alpine + ports: + - containerPort: 80 diff --git a/cortexapps_cli/solutions/kubernetes-agent/manifests/rollout.yaml b/cortexapps_cli/solutions/kubernetes-agent/manifests/rollout.yaml new file mode 100644 index 00000000..903826d2 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/manifests/rollout.yaml @@ -0,0 +1,22 @@ +apiVersion: argoproj.io/v1alpha1 +kind: Rollout +metadata: + name: demo-rollout + labels: + app: demo-k8s-label + annotations: + cortex.io/tag: demo-kubernetes +spec: + replicas: 1 + selector: + matchLabels: + app: demo-k8s-rollout + workloadRef: + apiVersion: apps/v1 + kind: Deployment + name: demo-deployment + scaleDown: onsuccess + strategy: + canary: + steps: + - setWeight: 100 diff --git a/cortexapps_cli/solutions/kubernetes-agent/manifests/statefulset.yaml b/cortexapps_cli/solutions/kubernetes-agent/manifests/statefulset.yaml new file mode 100644 index 00000000..7f410fad --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/manifests/statefulset.yaml @@ -0,0 +1,24 @@ +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: demo-statefulset + labels: + app: demo-k8s-label + annotations: + cortex.io/tag: demo-kubernetes +spec: + serviceName: demo-k8s + replicas: 1 + selector: + matchLabels: + app: demo-k8s-ss + template: + metadata: + labels: + app: demo-k8s-ss + spec: + containers: + - name: hello + image: nginx:alpine + ports: + - containerPort: 80 From 6b92211a7ab0ff3af6aa47542a29a8460ba2fb63 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:35:23 -0700 Subject: [PATCH 05/43] feat: bundle k8s-agent helm chart in kubernetes-agent solution --- .../kubernetes-agent/helm-chart/.helmignore | 23 ++++++ .../kubernetes-agent/helm-chart/Chart.yaml | 9 +++ .../kubernetes-agent/helm-chart/README.md | 28 +++++++ .../helm-chart/templates/_helpers.tpl | 73 +++++++++++++++++++ .../helm-chart/templates/clusterrole.yaml | 10 +++ .../templates/clusterrolebinding.yaml | 14 ++++ .../helm-chart/templates/configmap.yaml | 9 +++ .../helm-chart/templates/deployment.yaml | 51 +++++++++++++ .../helm-chart/templates/service.yaml | 15 ++++ .../helm-chart/templates/serviceaccount.yaml | 13 ++++ .../kubernetes-agent/helm-chart/values.yaml | 33 +++++++++ 11 files changed, 278 insertions(+) create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/.helmignore create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/Chart.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/README.md create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/_helpers.tpl create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/clusterrole.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/clusterrolebinding.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/configmap.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/deployment.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/service.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/serviceaccount.yaml create mode 100644 cortexapps_cli/solutions/kubernetes-agent/helm-chart/values.yaml diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/.helmignore b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/.helmignore new file mode 100644 index 00000000..0e8a0eb3 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/.helmignore @@ -0,0 +1,23 @@ +# Patterns to ignore when building packages. +# This supports shell glob matching, relative path matching, and +# negation (prefixed with !). Only one pattern per line. +.DS_Store +# Common VCS dirs +.git/ +.gitignore +.bzr/ +.bzrignore +.hg/ +.hgignore +.svn/ +# Common backup files +*.swp +*.bak +*.tmp +*.orig +*~ +# Various IDEs +.project +.idea/ +*.tmproj +.vscode/ diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/Chart.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/Chart.yaml new file mode 100644 index 00000000..03d7e458 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/Chart.yaml @@ -0,0 +1,9 @@ +# Bundled copy of the Cortex k8s-agent helm chart for demo purposes. +# This copy is not kept up-to-date. Once the chart is published to a +# public helm repo, this bundle will be replaced with a helm repo reference. +apiVersion: v2 +name: cortex-k8s-agent +description: A Helm chart for deploying Cortex K8s agent in your cluster +type: application +version: 0.1.0 +appVersion: 1.16.0 diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/README.md b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/README.md new file mode 100644 index 00000000..a6dfa9a4 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/README.md @@ -0,0 +1,28 @@ +# Cortex k8s Helm Chart + +## Requirements +* [Helm](https://helm.sh/docs/intro/install/) +* A token for our package registry + +## Process +1. Generate a new Cortex API Key on the [API Keys Settings tab](https://app.getcortexapp.com/admin/settings/api-keys) in Cortex. + - This will be used for the Cortex Kubernetes agent to communicate and push service information to Cortex backend without exposing your public API Key. +2. Inside your Kubernetes cluster, run the following command to generate a Kubernetes secret for the Cortex API Key. + `kubectl create secret generic cortex-key --from-literal api-key=YOUR_API_KEY` +3. Run `kubectl create secret docker-registry cortex-docker-registry-secret --docker-server=ghcr.io --docker-username=$GITHUB_USERNAME --docker-password=$GITHUB_PASSWORD --docker-email=` +4. Download the helm chart and inside the repository run the following command to install the agent in your cluster. + `helm install YOUR_SELECTED_CHART_NAME .` + +## Customization +The helm chart make installation quick and simple, but if you want to customize any of the installation features for the Cortex agent you can do so by changing the following information in the `values.yaml` of the helm chart. +### Service Account +To authenticate the Cortex agent in your cluster and grant it access to service information, the agent needs its own service account. The helm chart by default creates a Service Account `cortex-service-account`, but you can customize the `name` and `namespace` of this Service Account. If you already have a Service Account that you want the Cortex agent to use, set `create: false` under `serviceAccount` and enter the `name` and `namespace` of the Service Account you wish to use. +### Service +The service type and port can be customized as well. For security, the agent uses a default `ClusterIP` service type that only allows the service to be accessed from within the cluster. +### Resources +By default, no resources are specified. While the Cortex Kubernetes agent is designed to be lightweight and minimize resource utilization, you have the option to add custom CPU limits and requests. +### Base URL +The Base URL defaults to that for the hosted version of Cortex. If you are using the on-prem version of Cortex, you should change the `app/baseUrl` value to the correct URL for your on-prem Cortex. + +# Usage +After installation, usage is very simple as no additional steps are required. The next time you go to create a new service in your Service Directory Homepage, you should see all of your Kubernetes services already added, ready for you to use in Cortex. If you do not want to import all of your Kubernetes discovered services, you can simply remove the ones you do not want to add. Removed services will still show up in the Kubernetes tab of Discovered Services if you want to go back and add them later. diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/_helpers.tpl b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/_helpers.tpl new file mode 100644 index 00000000..d8ec2543 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/_helpers.tpl @@ -0,0 +1,73 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "helm-chart.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). +If release name contains chart name it will be used as a full name. +*/}} +{{- define "helm-chart.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Create chart name and version as used by the chart label. +*/}} +{{- define "helm-chart.chart" -}} +{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Common labels +*/}} +{{- define "helm-chart.labels" -}} +helm.sh/chart: {{ include "helm-chart.chart" . }} +{{ include "helm-chart.selectorLabels" . }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Selector labels +*/}} +{{- define "helm-chart.selectorLabels" -}} +app.kubernetes.io/name: {{ include "helm-chart.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end }} + +{{/* +Create the name of the service account to use +*/}} +{{- define "helm-chart.serviceAccountName" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "helm-chart.fullname" .) .Values.serviceAccount.name }} +{{- else }} +{{- default "default" .Values.serviceAccount.name }} +{{- end }} +{{- end }} + +{{/* +Create the namespace of the service account to use +*/}} +{{- define "helm-chart.serviceAccountNamespace" -}} +{{- if .Values.serviceAccount.create }} +{{- default (include "helm-chart.fullname" .) .Values.serviceAccount.namespace }} +{{- else }} +{{- default "default" .Values.serviceAccount.namespace }} +{{- end }} +{{- end }} diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/clusterrole.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/clusterrole.yaml new file mode 100644 index 00000000..b12f0782 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/clusterrole.yaml @@ -0,0 +1,10 @@ +{{- if .Values.clusterRole.create -}} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "helm-chart.serviceAccountName" . }} +rules: + - apiGroups: ["apps", "argoproj.io", "batch"] + resources: ["deployments", "services", "pods", "replicationcontrollers", "statefulsets", "rollouts", "cronjobs"] + verbs: ["get", "watch", "list"] + {{- end -}} diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/clusterrolebinding.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/clusterrolebinding.yaml new file mode 100644 index 00000000..a6874acb --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/clusterrolebinding.yaml @@ -0,0 +1,14 @@ +{{- if .Values.clusterRoleBinding.create -}} +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "helm-chart.serviceAccountName" . }} +subjects: + - kind: ServiceAccount + name: {{ include "helm-chart.serviceAccountName" . }} + namespace: {{ include "helm-chart.serviceAccountNamespace" . }} +roleRef: + kind: ClusterRole + name: {{ include "helm-chart.serviceAccountName" . }} + apiGroup: rbac.authorization.k8s.io + {{- end -}} diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/configmap.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/configmap.yaml new file mode 100644 index 00000000..a1843703 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/configmap.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ template "helm-chart.fullname" . }}-configmap +data: + SPRING_PROFILES_ACTIVE: prod + BASE_URL: {{ required "Base URL must be defined." .Values.app.baseUrl }} + {{ if .Values.app.clusterName }}CORTEX_CLUSTER: {{ .Values.app.clusterName }}{{ end }} + {{ if .Values.app.namespace }}SELECTED_NAMESPACE: {{ .Values.app.namespace }}{{ end }} diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/deployment.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/deployment.yaml new file mode 100644 index 00000000..21071c30 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/deployment.yaml @@ -0,0 +1,51 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "helm-chart.fullname" . }} + labels: + {{- include "helm-chart.labels" . | nindent 4 }} +spec: + replicas: 1 + selector: + matchLabels: + {{- include "helm-chart.selectorLabels" . | nindent 6 }} + template: + metadata: + annotations: + checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + {{- with .Values.podAnnotations }} + {{- toYaml . | nindent 8 }} + {{- end }} + labels: + {{- include "helm-chart.selectorLabels" . | nindent 8 }} + spec: + {{- with .Values.image.secrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} + serviceAccountName: {{ include "helm-chart.serviceAccountName" . }} + containers: + - name: {{ .Chart.Name }} + image: "{{ required "Image repository must be defined" .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" + imagePullPolicy: {{ .Values.image.pullPolicy }} + ports: + - name: http-api + containerPort: 80 + protocol: TCP + readinessProbe: + initialDelaySeconds: 30 + periodSeconds: 5 + httpGet: + path: /actuator/health + port: 8080 + resources: + {{- toYaml .Values.resources | nindent 12 }} + env: + - name: CORTEX_API_KEY + valueFrom: + secretKeyRef: + name: {{ required "A secret containing api-key=CORTEX_API_KEY is required" .Values.app.keySecret }} + key: api-key + envFrom: + - configMapRef: + name: {{ template "helm-chart.fullname" . }}-configmap diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/service.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/service.yaml new file mode 100644 index 00000000..64fefc2e --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/service.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Service +metadata: + name: {{ include "helm-chart.fullname" . }} + labels: + {{- include "helm-chart.labels" . | nindent 4 }} +spec: + type: {{ .Values.service.type }} + ports: + - port: {{ .Values.service.port }} + targetPort: http-api + protocol: TCP + name: http-{{- include "helm-chart.fullname" . }} + selector: + {{- include "helm-chart.selectorLabels" . | nindent 4 }} diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/serviceaccount.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/serviceaccount.yaml new file mode 100644 index 00000000..7d0532d0 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/templates/serviceaccount.yaml @@ -0,0 +1,13 @@ +{{- if .Values.serviceAccount.create -}} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ include "helm-chart.serviceAccountName" . }} + namespace: {{ include "helm-chart.serviceAccountNamespace" . }} + labels: + {{- include "helm-chart.labels" . | nindent 4 }} + {{- with .Values.serviceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/values.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/values.yaml new file mode 100644 index 00000000..4578dae7 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/values.yaml @@ -0,0 +1,33 @@ +image: + repository: ghcr.io/cortexapps/k8s-agent/k8s-agent + pullPolicy: IfNotPresent + tag: "" + secrets: + - name: cortex-docker-registry-secret + +imagePullSecrets: [] +nameOverride: "" +fullnameOverride: cortex-k8s-agent + +serviceAccount: + create: true + annotations: {} + name: cortex-service-account + namespace: default + +clusterRole: + create: true + +clusterRoleBinding: + create: true + +service: + type: ClusterIP + port: 80 + +resources: {} + +app: + baseUrl: https://api.getcortexapp.com + keySecret: cortex-key + clusterName: Dev Cluster From 7ce61864d60ed657a3c6408e201db3a7b2f6da2d Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:38:08 -0700 Subject: [PATCH 06/43] feat: add kubernetes-agent solution post-install setup script Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/setup.py | 216 ++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 cortexapps_cli/solutions/kubernetes-agent/setup.py diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py new file mode 100644 index 00000000..18ffafef --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -0,0 +1,216 @@ +""" +Post-install setup script for the kubernetes-agent solution. +Deploys the Cortex k8s-agent to a kind cluster and creates a demo entity. +Run via: cortex solutions post-install -s kubernetes-agent +""" + +SETUP_DESCRIPTION = ( + "This solution deploys the Cortex Kubernetes agent to a local kind cluster " + "and creates a demo entity to demonstrate the k8s integration." +) + +import subprocess +import sys +from pathlib import Path + +import requests + +try: + from cortexapps_cli.solutions._lib.setup_base import SolutionSetup +except ImportError: + sys.path.insert(0, str(Path(__file__).parent.parent.parent)) + from _lib.setup_base import SolutionSetup + +SOLUTION_DIR = Path(__file__).parent +CATALOG_FILE = SOLUTION_DIR / "catalog" / "demo-kubernetes.yaml" +MANIFESTS_DIR = SOLUTION_DIR / "manifests" +HELM_CHART_DIR = SOLUTION_DIR / "helm-chart" + +GHCR_IMAGE = "ghcr.io/cortexapps/k8s-agent/k8s-agent" +ARGO_CRD_URL = "https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/crds/rollout-crd.yaml" + + +class KubernetesAgentSetup(SolutionSetup): + solution_tag = "kubernetes-agent" + + def __init__( + self, + cortex_api_key: str = None, + cortex_base_url: str = None, + no_prompt: bool = False, + **kwargs, + ): + super().__init__(no_prompt=no_prompt, **kwargs) + self._api_key = cortex_api_key or "" + self._base_url = (cortex_base_url or "https://api.getcortexapp.com").rstrip("/") + self._ghcr_token = "" + self._cluster_name = "" + + def collect_prompts(self) -> None: + self._ghcr_token = self.prompt( + "GHCR_TOKEN", + "GitHub PAT with read:packages scope for pulling the k8s-agent image", + env_var="GHCR_TOKEN", + secret=True, + ) + self._cluster_name = self.prompt( + "cluster_name", + "Name for this cluster as it will appear in Cortex", + default="demo", + ) + + def _fetch_image_tag(self) -> str: + """Fetch the latest k8s-agent image tag from the GitHub API.""" + r = requests.get( + "https://api.github.com/orgs/cortexapps/packages/container/k8s-agent%2Fk8s-agent/versions", + headers={ + "Authorization": f"Bearer {self._ghcr_token}", + "Accept": "application/vnd.github+json", + }, + ) + r.raise_for_status() + versions = r.json() + if not versions: + raise RuntimeError("No k8s-agent versions found in GHCR — is GHCR_TOKEN valid?") + tags = versions[0].get("metadata", {}).get("container", {}).get("tags", []) + tag = tags[0] if tags else "" + if not tag: + raise RuntimeError("Could not determine k8s-agent image tag from GHCR API response") + print(f" Using image tag: {tag}") + return tag + + def _create_secrets(self) -> None: + if self.already_done("create_secrets"): + return + print(" Creating cortex-docker-registry-secret...") + result = subprocess.run( + [ + "kubectl", "create", "secret", "docker-registry", + "cortex-docker-registry-secret", + "--docker-server=ghcr.io", + "--docker-username=cortex", + f"--docker-password={self._ghcr_token}", + "--dry-run=client", "-o", "yaml", + ], + check=True, + capture_output=True, + ) + subprocess.run(["kubectl", "apply", "-f", "-"], input=result.stdout, check=True) + + print(" Creating cortex-key secret...") + result = subprocess.run( + [ + "kubectl", "create", "secret", "generic", "cortex-key", + f"--from-literal=api-key={self._api_key}", + "--dry-run=client", "-o", "yaml", + ], + check=True, + capture_output=True, + ) + subprocess.run(["kubectl", "apply", "-f", "-"], input=result.stdout, check=True) + self.mark_done("create_secrets") + + def _helm_install(self) -> None: + if self.already_done("helm_install"): + return + image_tag = self._fetch_image_tag() + print(f" Installing k8s-agent via helm (chart: {HELM_CHART_DIR})...") + subprocess.run( + [ + "helm", "upgrade", "--install", "cortex-k8s-agent", + str(HELM_CHART_DIR), + "--set", f"image.tag={image_tag}", + "--set", f"app.baseUrl={self._base_url}", + "--set", f"app.clusterName={self._cluster_name}", + ], + check=True, + ) + # Restart to ensure secrets/configmaps are picked up + subprocess.run( + ["kubectl", "rollout", "restart", "deployment", + "-l", "app.kubernetes.io/name=cortex-k8s-agent"], + check=True, + ) + self.mark_done("helm_install") + + def _wait_for_readiness(self) -> None: + if self.already_done("wait_for_readiness"): + return + print(" Waiting for k8s-agent pod to be ready (timeout: 120s)...") + subprocess.run( + [ + "kubectl", "rollout", "status", "deployment", + "-l", "app.kubernetes.io/name=cortex-k8s-agent", + "--timeout=120s", + ], + check=True, + ) + self.mark_done("wait_for_readiness") + + def _install_argo_crd(self) -> None: + if self.already_done("install_argo_crd"): + return + print(f" Installing Argo Rollouts CRD...") + subprocess.run( + ["kubectl", "apply", "-f", ARGO_CRD_URL], + check=True, + ) + self.mark_done("install_argo_crd") + + def _apply_manifests(self) -> None: + if self.already_done("apply_manifests"): + return + print(f" Applying demo k8s manifests from {MANIFESTS_DIR}...") + subprocess.run( + ["kubectl", "apply", "-f", str(MANIFESTS_DIR)], + check=True, + ) + self.mark_done("apply_manifests") + + def _create_entity(self) -> None: + if self.already_done("create_entity"): + return + print(f" Creating demo-kubernetes Cortex entity...") + yaml_content = CATALOG_FILE.read_bytes() + r = requests.post( + f"{self._base_url}/api/v1/open-api", + headers={ + "Authorization": f"Bearer {self._api_key}", + "Content-Type": "application/openapi;charset=UTF-8", + }, + data=yaml_content, + ) + if not r.ok: + raise RuntimeError( + f"Failed to create Cortex entity: {r.status_code} {r.text}" + ) + self.mark_done("create_entity") + + def steps(self) -> list: + return [ + ("Create k8s secrets", self._create_secrets), + ("Install k8s-agent via helm", self._helm_install), + ("Wait for agent readiness", self._wait_for_readiness), + ("Install Argo Rollouts CRD", self._install_argo_crd), + ("Apply demo k8s manifests", self._apply_manifests), + ("Create demo Cortex entity", self._create_entity), + ] + + def post_steps(self) -> None: + print("\n✓ Kubernetes agent deployed and demo workloads running.\n") + print("The agent syncs every 5 minutes. After the first sync, visit:") + print(f" {self._base_url.replace('api.', 'app.')}/catalog/demo-kubernetes/k8s") + print("\nYou should see: demo-deployment, demo-statefulset, demo-cronjob, demo-rollout") + print("\nNote: GHCR_TOKEN requirement goes away once the k8s-agent image is made public.") + + +def main(cortex_api_key=None, cortex_base_url=None, no_prompt=False, **kwargs): + KubernetesAgentSetup( + cortex_api_key=cortex_api_key, + cortex_base_url=cortex_base_url, + no_prompt=no_prompt, + ).run() + + +if __name__ == "__main__": + main() From 9999d2eefa6e4c9e82412c92f1d533e538a7aaf1 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:40:10 -0700 Subject: [PATCH 07/43] feat: add kubernetes-agent Codespace devcontainer --- .../kubernetes-agent/devcontainer.json | 18 ++++++++++++++++++ .devcontainer/kubernetes-agent/onCreate.sh | 18 ++++++++++++++++++ 2 files changed, 36 insertions(+) create mode 100644 .devcontainer/kubernetes-agent/devcontainer.json create mode 100644 .devcontainer/kubernetes-agent/onCreate.sh diff --git a/.devcontainer/kubernetes-agent/devcontainer.json b/.devcontainer/kubernetes-agent/devcontainer.json new file mode 100644 index 00000000..524dfe0f --- /dev/null +++ b/.devcontainer/kubernetes-agent/devcontainer.json @@ -0,0 +1,18 @@ +{ + "name": "Cortex Kubernetes Agent Demo", + "image": "mcr.microsoft.com/devcontainers/base:ubuntu-24.04", + "features": { + "ghcr.io/devcontainers/features/docker-in-docker:2": {}, + "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": { + "version": "latest", + "helm": "latest", + "minikube": "none" + } + }, + "onCreateCommand": "bash .devcontainer/kubernetes-agent/onCreate.sh", + "remoteEnv": { + "CORTEX_API_KEY": "${localEnv:CORTEX_API_KEY}", + "GHCR_TOKEN": "${localEnv:GHCR_TOKEN}" + }, + "postCreateMessage": "Run: cortex solutions install -s kubernetes-agent && cortex solutions post-install -s kubernetes-agent" +} diff --git a/.devcontainer/kubernetes-agent/onCreate.sh b/.devcontainer/kubernetes-agent/onCreate.sh new file mode 100644 index 00000000..defc5d17 --- /dev/null +++ b/.devcontainer/kubernetes-agent/onCreate.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash +set -euo pipefail + +echo "==> Installing kind..." +curl -Lo /usr/local/bin/kind \ + https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64 +chmod +x /usr/local/bin/kind + +echo "==> Creating kind cluster 'cortex-demo'..." +kind create cluster --name cortex-demo --wait 60s + +echo "==> Verifying cluster..." +kubectl cluster-info --context kind-cortex-demo + +echo "==> Installing cortexapps-cli..." +pip install cortexapps-cli --quiet + +echo "==> Done. Run: cortex solutions post-install -s kubernetes-agent" From b1656cd6df2e962fbb3a1412316665759b87c4f2 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:41:04 -0700 Subject: [PATCH 08/43] docs: add kubernetes-agent solution README --- .../solutions/kubernetes-agent/README.md | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 cortexapps_cli/solutions/kubernetes-agent/README.md diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md new file mode 100644 index 00000000..97552258 --- /dev/null +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -0,0 +1,48 @@ +# Kubernetes Agent Solution + +Demonstrates the [Cortex Kubernetes agent](https://docs.cortex.io/docs/reference/integrations/kubernetes) integration using a GitHub Codespace with a local kind cluster. + +## What this installs + +- **Cortex k8s-agent** — connects your cluster to Cortex and syncs workload metadata +- **Demo workloads** — Deployment, StatefulSet, CronJob, and Argo Rollout, all tagged `demo-kubernetes` +- **demo-kubernetes** — a Cortex service entity that the workloads annotate to + +After setup, visit your entity's K8s tab to see live workload data synced from the cluster. + +## Prerequisites + +- A [GitHub Codespace](https://github.com/features/codespaces) opened from this repository +- A Cortex API key (`CORTEX_API_KEY`) — get from Cortex Settings → API Keys +- A GitHub PAT with `read:packages` scope (`GHCR_TOKEN`) — request from Cortex support + +Set both as [Codespace secrets](https://docs.github.com/en/codespaces/managing-your-codespaces/managing-secrets-for-your-codespaces) before opening the Codespace. + +## Quick start + +1. Open a Codespace from this repository (select the `kubernetes-agent` devcontainer configuration) +2. Wait for `onCreate` to finish (installs tools + creates kind cluster, ~2 min) +3. Run the solution: + +```bash +cortex solutions install -s kubernetes-agent +cortex solutions post-install -s kubernetes-agent +``` + +4. Wait ~5 minutes for the agent's first sync, then visit: + `https://app.getcortexapp.com/catalog/demo-kubernetes/k8s` + +## What you should see + +- `demo-deployment` (Deployment) +- `demo-statefulset` (StatefulSet) +- `demo-cronjob` (CronJob) +- `demo-rollout` (Argo Rollout — containers resolved from `demo-deployment`) + +## Re-running setup + +The setup script is idempotent — re-run `cortex solutions post-install -s kubernetes-agent` to retry any failed step. Completed steps are skipped. + +## Temporary limitation + +The k8s-agent image is currently private on GHCR, requiring `GHCR_TOKEN`. This requirement will be removed once the image is made public. From e110b769431a9ffe17f5619cf22ef5a78bbf118b Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 09:44:47 -0700 Subject: [PATCH 09/43] fix: use explicit deployment name for kubectl rollout commands --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 18ffafef..3482a76b 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -127,8 +127,7 @@ def _helm_install(self) -> None: ) # Restart to ensure secrets/configmaps are picked up subprocess.run( - ["kubectl", "rollout", "restart", "deployment", - "-l", "app.kubernetes.io/name=cortex-k8s-agent"], + ["kubectl", "rollout", "restart", "deployment/cortex-k8s-agent"], check=True, ) self.mark_done("helm_install") @@ -139,8 +138,7 @@ def _wait_for_readiness(self) -> None: print(" Waiting for k8s-agent pod to be ready (timeout: 120s)...") subprocess.run( [ - "kubectl", "rollout", "status", "deployment", - "-l", "app.kubernetes.io/name=cortex-k8s-agent", + "kubectl", "rollout", "status", "deployment/cortex-k8s-agent", "--timeout=120s", ], check=True, From 450624349f4e7e83f36f1cad1cb4b31403ec630e Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 10:52:58 -0700 Subject: [PATCH 10/43] chore: add README frontmatter with solution name and description Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index 97552258..bfb19c72 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -1,3 +1,8 @@ +--- +name: Kubernetes Agent +description: Deploy the Cortex Kubernetes agent in a GitHub Codespace with a local kind cluster to demonstrate live workload discovery and k8s integration. +--- + # Kubernetes Agent Solution Demonstrates the [Cortex Kubernetes agent](https://docs.cortex.io/docs/reference/integrations/kubernetes) integration using a GitHub Codespace with a local kind cluster. From 4713f007d5da5ed4068c5babd9048b9f13dec1fb Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 10:54:05 -0700 Subject: [PATCH 11/43] chore: set default cluster name to cortex-demo in helm values Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/helm-chart/values.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/values.yaml b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/values.yaml index 4578dae7..639ef2dc 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/helm-chart/values.yaml +++ b/cortexapps_cli/solutions/kubernetes-agent/helm-chart/values.yaml @@ -30,4 +30,4 @@ resources: {} app: baseUrl: https://api.getcortexapp.com keySecret: cortex-key - clusterName: Dev Cluster + clusterName: cortex-demo From 3397bd40756d280268d8242dcee129f4231fa28a Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 10:56:36 -0700 Subject: [PATCH 12/43] chore: sync cluster name default to cortex-demo in setup prompt Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 3482a76b..2d4a2d4a 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -56,7 +56,7 @@ def collect_prompts(self) -> None: self._cluster_name = self.prompt( "cluster_name", "Name for this cluster as it will appear in Cortex", - default="demo", + default="cortex-demo", ) def _fetch_image_tag(self) -> str: From 63041eca1d34eb6fdb1637a4b9e00cba01d39fdc Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 10:59:21 -0700 Subject: [PATCH 13/43] chore: add k8s prerequisites doc link to GHCR_TOKEN prompt and README Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/README.md | 2 +- cortexapps_cli/solutions/kubernetes-agent/setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index bfb19c72..5a617f11 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -19,7 +19,7 @@ After setup, visit your entity's K8s tab to see live workload data synced from t - A [GitHub Codespace](https://github.com/features/codespaces) opened from this repository - A Cortex API key (`CORTEX_API_KEY`) — get from Cortex Settings → API Keys -- A GitHub PAT with `read:packages` scope (`GHCR_TOKEN`) — request from Cortex support +- A GitHub PAT with `read:packages` scope (`GHCR_TOKEN`) — see [Kubernetes prerequisites](https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites) Set both as [Codespace secrets](https://docs.github.com/en/codespaces/managing-your-codespaces/managing-secrets-for-your-codespaces) before opening the Codespace. diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 2d4a2d4a..2ee6becc 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -49,7 +49,7 @@ def __init__( def collect_prompts(self) -> None: self._ghcr_token = self.prompt( "GHCR_TOKEN", - "GitHub PAT with read:packages scope for pulling the k8s-agent image", + "GitHub PAT with read:packages scope (see https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites)", env_var="GHCR_TOKEN", secret=True, ) From adf9edda363c090c0f19e38b5f8b967e51d1a7bd Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 11:02:22 -0700 Subject: [PATCH 14/43] chore: clarify GHCR_TOKEN is provided by Cortex Customer Engineering Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/README.md | 2 +- cortexapps_cli/solutions/kubernetes-agent/setup.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index 5a617f11..a51ab979 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -19,7 +19,7 @@ After setup, visit your entity's K8s tab to see live workload data synced from t - A [GitHub Codespace](https://github.com/features/codespaces) opened from this repository - A Cortex API key (`CORTEX_API_KEY`) — get from Cortex Settings → API Keys -- A GitHub PAT with `read:packages` scope (`GHCR_TOKEN`) — see [Kubernetes prerequisites](https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites) +- A GitHub PAT provided by Cortex Customer Engineering (`GHCR_TOKEN`) — required to pull the k8s-agent image; see [Kubernetes prerequisites](https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites) Set both as [Codespace secrets](https://docs.github.com/en/codespaces/managing-your-codespaces/managing-secrets-for-your-codespaces) before opening the Codespace. diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 2ee6becc..011f8ea8 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -49,7 +49,7 @@ def __init__( def collect_prompts(self) -> None: self._ghcr_token = self.prompt( "GHCR_TOKEN", - "GitHub PAT with read:packages scope (see https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites)", + "GitHub PAT provided by Cortex Customer Engineering for pulling the k8s-agent image (see https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites)", env_var="GHCR_TOKEN", secret=True, ) From 2658b437f63b69e94ed20220d5a89c821c08809b Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 11:14:04 -0700 Subject: [PATCH 15/43] chore: clarify kind runs in GitHub Codespace, add kind docs link Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 011f8ea8..89b29aca 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -1,11 +1,12 @@ """ Post-install setup script for the kubernetes-agent solution. -Deploys the Cortex k8s-agent to a kind cluster and creates a demo entity. +Deploys the Cortex k8s-agent to a kind cluster in a GitHub Codespace and creates a demo entity. Run via: cortex solutions post-install -s kubernetes-agent """ SETUP_DESCRIPTION = ( - "This solution deploys the Cortex Kubernetes agent to a local kind cluster " + "This solution deploys the Cortex Kubernetes agent to a kind cluster " + "(https://kind.sigs.k8s.io) running in your GitHub Codespace, " "and creates a demo entity to demonstrate the k8s integration." ) From e5fbfe5cf0d87e4678e54b3589435bc760d1ad5a Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 11:15:16 -0700 Subject: [PATCH 16/43] fix: add preflight cluster connectivity check with Codespace guidance Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/setup.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 89b29aca..4fbc98a1 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -80,6 +80,22 @@ def _fetch_image_tag(self) -> str: print(f" Using image tag: {tag}") return tag + def _check_cluster(self) -> None: + """Verify kubectl can reach a running cluster before attempting any k8s operations.""" + result = subprocess.run( + ["kubectl", "cluster-info"], + capture_output=True, + ) + if result.returncode != 0: + raise RuntimeError( + "kubectl cannot reach a cluster.\n\n" + "This solution must be run inside a GitHub Codespace — the kind cluster\n" + "(https://kind.sigs.k8s.io) is created automatically when the Codespace\n" + "opens. Open a Codespace from this repository using the 'kubernetes-agent'\n" + "devcontainer configuration, wait for setup to complete, then re-run:\n\n" + " cortex solutions post-install -s kubernetes-agent" + ) + def _create_secrets(self) -> None: if self.already_done("create_secrets"): return @@ -187,6 +203,7 @@ def _create_entity(self) -> None: def steps(self) -> list: return [ + ("Check cluster connectivity", self._check_cluster), ("Create k8s secrets", self._create_secrets), ("Install k8s-agent via helm", self._helm_install), ("Wait for agent readiness", self._wait_for_readiness), From e95bdac286c7d9571ed8af423bb194c63c9413fc Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Tue, 25 Aug 2026 11:37:32 -0700 Subject: [PATCH 17/43] feat: support both GitHub Codespace and existing cluster deployment paths setup.py now prompts the user to choose between: - Spinning up a GitHub Codespace with a kind cluster automatically (via gh codespace create + gh codespace ssh for all k8s/helm commands) - Deploying to any existing Kubernetes cluster configured in kubectl context (runs kubectl/helm locally as before) Idempotent: Codespace name is persisted in state so re-runs reconnect rather than creating a new Codespace. Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/README.md | 43 ++- .../solutions/kubernetes-agent/setup.py | 307 +++++++++++++----- 2 files changed, 267 insertions(+), 83 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index a51ab979..3c7f4f4c 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -1,11 +1,14 @@ --- name: Kubernetes Agent -description: Deploy the Cortex Kubernetes agent in a GitHub Codespace with a local kind cluster to demonstrate live workload discovery and k8s integration. +description: Deploy the Cortex Kubernetes agent to a kind cluster in a GitHub Codespace, or to any existing Kubernetes cluster, to demonstrate live workload discovery and k8s integration. --- # Kubernetes Agent Solution -Demonstrates the [Cortex Kubernetes agent](https://docs.cortex.io/docs/reference/integrations/kubernetes) integration using a GitHub Codespace with a local kind cluster. +Demonstrates the [Cortex Kubernetes agent](https://docs.cortex.io/docs/reference/integrations/kubernetes) integration. Supports two deployment paths: + +- **GitHub Codespace** — creates a Codespace with a local [kind](https://kind.sigs.k8s.io) cluster automatically (no cluster setup required) +- **Existing cluster** — deploys to any Kubernetes cluster already configured in your `kubectl` context ## What this installs @@ -17,28 +20,46 @@ After setup, visit your entity's K8s tab to see live workload data synced from t ## Prerequisites -- A [GitHub Codespace](https://github.com/features/codespaces) opened from this repository - A Cortex API key (`CORTEX_API_KEY`) — get from Cortex Settings → API Keys - A GitHub PAT provided by Cortex Customer Engineering (`GHCR_TOKEN`) — required to pull the k8s-agent image; see [Kubernetes prerequisites](https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites) -Set both as [Codespace secrets](https://docs.github.com/en/codespaces/managing-your-codespaces/managing-secrets-for-your-codespaces) before opening the Codespace. +**For the GitHub Codespace path only:** +- The [gh CLI](https://cli.github.com) installed and authenticated (`gh auth login`) -## Quick start +**For the existing cluster path only:** +- `kubectl`, `helm` installed and configured to reach your cluster -1. Open a Codespace from this repository (select the `kubernetes-agent` devcontainer configuration) -2. Wait for `onCreate` to finish (installs tools + creates kind cluster, ~2 min) -3. Run the solution: +## Quick start ```bash cortex solutions install -s kubernetes-agent cortex solutions post-install -s kubernetes-agent ``` -4. Wait ~5 minutes for the agent's first sync, then visit: - `https://app.getcortexapp.com/catalog/demo-kubernetes/k8s` +The setup script will ask which path you want: + +``` +Create a new GitHub Codespace with a kind cluster? (yes = spin up Codespace, no = use an existing configured cluster) [yes]: +``` + +### GitHub Codespace path + +The script will: +1. Create a Codespace from the `cortexapps/cli` repository using the `kubernetes-agent` devcontainer +2. Wait for the Codespace to start and the kind cluster to initialize (~2-4 min) +3. Deploy the k8s-agent and demo workloads inside the Codespace via `gh codespace ssh` + +After setup, the Codespace URL and `gh codespace ssh` command are printed. + +### Existing cluster path + +Requires `kubectl` pointed at a running cluster. The script deploys the k8s-agent and demo workloads into whatever namespace your current context targets. ## What you should see +After the agent's first sync (~5 min), visit: +`https://app.getcortexapp.com/catalog/demo-kubernetes/k8s` + - `demo-deployment` (Deployment) - `demo-statefulset` (StatefulSet) - `demo-cronjob` (CronJob) @@ -48,6 +69,8 @@ cortex solutions post-install -s kubernetes-agent The setup script is idempotent — re-run `cortex solutions post-install -s kubernetes-agent` to retry any failed step. Completed steps are skipped. +For the Codespace path, the Codespace name is saved locally so re-runs reconnect to the same Codespace rather than creating a new one. + ## Temporary limitation The k8s-agent image is currently private on GHCR, requiring `GHCR_TOKEN`. This requirement will be removed once the image is made public. diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 4fbc98a1..cf81271f 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -1,17 +1,21 @@ """ Post-install setup script for the kubernetes-agent solution. -Deploys the Cortex k8s-agent to a kind cluster in a GitHub Codespace and creates a demo entity. +Deploys the Cortex k8s-agent either by creating a GitHub Codespace with a kind +cluster, or against an existing Kubernetes cluster. Run via: cortex solutions post-install -s kubernetes-agent """ SETUP_DESCRIPTION = ( - "This solution deploys the Cortex Kubernetes agent to a kind cluster " - "(https://kind.sigs.k8s.io) running in your GitHub Codespace, " - "and creates a demo entity to demonstrate the k8s integration." + "This solution deploys the Cortex Kubernetes agent to a Kubernetes cluster " + "and creates a demo entity to demonstrate the k8s integration. " + "It can spin up a GitHub Codespace with a kind cluster " + "(https://kind.sigs.k8s.io) automatically, or deploy to any existing cluster." ) +import shlex import subprocess import sys +import time from pathlib import Path import requests @@ -27,9 +31,11 @@ MANIFESTS_DIR = SOLUTION_DIR / "manifests" HELM_CHART_DIR = SOLUTION_DIR / "helm-chart" -GHCR_IMAGE = "ghcr.io/cortexapps/k8s-agent/k8s-agent" ARGO_CRD_URL = "https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/crds/rollout-crd.yaml" +CODESPACE_READY_TIMEOUT = 600 # 10 minutes for Codespace + kind cluster startup +CODESPACE_POLL_INTERVAL = 20 # seconds between readiness checks + class KubernetesAgentSetup(SolutionSetup): solution_tag = "kubernetes-agent" @@ -46,11 +52,34 @@ def __init__( self._base_url = (cortex_base_url or "https://api.getcortexapp.com").rstrip("/") self._ghcr_token = "" self._cluster_name = "" + self._github_repo = "" + # Recover codespace name from previous run; its presence means codespace mode + self._codespace_name = self._state.get("codespace_name", "") + self._use_codespace = bool(self._codespace_name) def collect_prompts(self) -> None: + # If a Codespace was already created in a prior run, stay in codespace mode. + # Otherwise ask the user which path they want. + if not self._codespace_name: + use_cs_raw = self.prompt( + "use_codespace", + "Create a new GitHub Codespace with a kind cluster?" + " (yes = spin up Codespace, no = use an existing configured cluster)", + default="yes", + ) + self._use_codespace = use_cs_raw.lower() in ("yes", "y", "true", "1") + + if self._use_codespace: + self._github_repo = self.prompt( + "github_repo", + "GitHub repository to create the Codespace from (org/repo)", + default="cortexapps/cli", + ) + self._ghcr_token = self.prompt( "GHCR_TOKEN", - "GitHub PAT provided by Cortex Customer Engineering for pulling the k8s-agent image (see https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites)", + "GitHub PAT provided by Cortex Customer Engineering for pulling the k8s-agent image" + " (see https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites)", env_var="GHCR_TOKEN", secret=True, ) @@ -60,6 +89,26 @@ def collect_prompts(self) -> None: default="cortex-demo", ) + # ------------------------------------------------------------------------- + # Helpers + # ------------------------------------------------------------------------- + + def _remote_solution_dir(self) -> str: + """Path to the solution directory inside the Codespace.""" + repo_name = self._github_repo.split("/")[-1] + return f"/workspaces/{repo_name}/cortexapps_cli/solutions/kubernetes-agent" + + def _run_remote(self, bash_cmd: str) -> None: + """Run a bash command inside the Codespace via gh codespace ssh.""" + subprocess.run( + [ + "gh", "codespace", "ssh", + "-c", self._codespace_name, + "--", "bash", "-c", bash_cmd, + ], + check=True, + ) + def _fetch_image_tag(self) -> str: """Fetch the latest k8s-agent image tag from the GitHub API.""" r = requests.get( @@ -80,112 +129,214 @@ def _fetch_image_tag(self) -> str: print(f" Using image tag: {tag}") return tag - def _check_cluster(self) -> None: - """Verify kubectl can reach a running cluster before attempting any k8s operations.""" + # ------------------------------------------------------------------------- + # Step: Create GitHub Codespace (codespace mode only) + # ------------------------------------------------------------------------- + + def _check_gh_cli(self) -> None: + """Verify gh CLI is installed and authenticated.""" + if subprocess.run(["gh", "--version"], capture_output=True).returncode != 0: + raise RuntimeError( + "The 'gh' CLI is required but not found.\n" + "Install it from https://cli.github.com and run 'gh auth login' first." + ) + if subprocess.run(["gh", "auth", "status"], capture_output=True).returncode != 0: + raise RuntimeError( + "The 'gh' CLI is not authenticated.\n" + "Run 'gh auth login' and try again." + ) + + def _create_codespace(self) -> None: + """Create a GitHub Codespace and wait for the kind cluster to be ready.""" + # Always check gh CLI — also needed for SSH steps that follow + self._check_gh_cli() + + if self._codespace_name: + print(f" Using existing Codespace: {self._codespace_name}") + return + + print(f" Creating GitHub Codespace from {self._github_repo}...") result = subprocess.run( - ["kubectl", "cluster-info"], + [ + "gh", "codespace", "create", + "--repo", self._github_repo, + "--devcontainer-path", ".devcontainer/kubernetes-agent/devcontainer.json", + ], + check=True, capture_output=True, + text=True, ) + self._codespace_name = result.stdout.strip() + if not self._codespace_name: + raise RuntimeError("gh codespace create did not return a codespace name") + print(f" Codespace created: {self._codespace_name}") + + # Persist the name so re-runs find the existing Codespace + self._state["codespace_name"] = self._codespace_name + self._save_file() + + print( + f" Waiting for Codespace and kind cluster to initialize " + f"(up to {CODESPACE_READY_TIMEOUT // 60} min)..." + ) + deadline = time.time() + CODESPACE_READY_TIMEOUT + while time.time() < deadline: + probe = subprocess.run( + [ + "gh", "codespace", "ssh", + "-c", self._codespace_name, + "--", "kubectl", "cluster-info", + ], + capture_output=True, + ) + if probe.returncode == 0: + return + time.sleep(CODESPACE_POLL_INTERVAL) + + raise RuntimeError( + f"Timed out waiting for the kind cluster in Codespace '{self._codespace_name}'.\n" + "The Codespace may still be initializing. Re-run this command to retry." + ) + + # ------------------------------------------------------------------------- + # Step: Check existing cluster (existing-cluster mode only) + # ------------------------------------------------------------------------- + + def _check_cluster(self) -> None: + """Verify kubectl can reach a running cluster.""" + result = subprocess.run(["kubectl", "cluster-info"], capture_output=True) if result.returncode != 0: raise RuntimeError( - "kubectl cannot reach a cluster.\n\n" - "This solution must be run inside a GitHub Codespace — the kind cluster\n" - "(https://kind.sigs.k8s.io) is created automatically when the Codespace\n" - "opens. Open a Codespace from this repository using the 'kubernetes-agent'\n" - "devcontainer configuration, wait for setup to complete, then re-run:\n\n" - " cortex solutions post-install -s kubernetes-agent" + "kubectl cannot reach a cluster.\n" + "Ensure your kubectl context points to a running Kubernetes cluster and try again." ) + # ------------------------------------------------------------------------- + # Steps: shared between both modes + # ------------------------------------------------------------------------- + def _create_secrets(self) -> None: if self.already_done("create_secrets"): return print(" Creating cortex-docker-registry-secret...") - result = subprocess.run( - [ - "kubectl", "create", "secret", "docker-registry", - "cortex-docker-registry-secret", - "--docker-server=ghcr.io", - "--docker-username=cortex", - f"--docker-password={self._ghcr_token}", - "--dry-run=client", "-o", "yaml", - ], - check=True, - capture_output=True, - ) - subprocess.run(["kubectl", "apply", "-f", "-"], input=result.stdout, check=True) + if self._use_codespace: + self._run_remote( + f"kubectl create secret docker-registry cortex-docker-registry-secret " + f"--docker-server=ghcr.io --docker-username=cortex " + f"--docker-password={shlex.quote(self._ghcr_token)} " + f"--dry-run=client -o yaml | kubectl apply -f -" + ) + else: + result = subprocess.run( + [ + "kubectl", "create", "secret", "docker-registry", + "cortex-docker-registry-secret", + "--docker-server=ghcr.io", + "--docker-username=cortex", + f"--docker-password={self._ghcr_token}", + "--dry-run=client", "-o", "yaml", + ], + check=True, + capture_output=True, + ) + subprocess.run(["kubectl", "apply", "-f", "-"], input=result.stdout, check=True) print(" Creating cortex-key secret...") - result = subprocess.run( - [ - "kubectl", "create", "secret", "generic", "cortex-key", - f"--from-literal=api-key={self._api_key}", - "--dry-run=client", "-o", "yaml", - ], - check=True, - capture_output=True, - ) - subprocess.run(["kubectl", "apply", "-f", "-"], input=result.stdout, check=True) + if self._use_codespace: + self._run_remote( + f"kubectl create secret generic cortex-key " + f"--from-literal=api-key={shlex.quote(self._api_key)} " + f"--dry-run=client -o yaml | kubectl apply -f -" + ) + else: + result = subprocess.run( + [ + "kubectl", "create", "secret", "generic", "cortex-key", + f"--from-literal=api-key={self._api_key}", + "--dry-run=client", "-o", "yaml", + ], + check=True, + capture_output=True, + ) + subprocess.run(["kubectl", "apply", "-f", "-"], input=result.stdout, check=True) + self.mark_done("create_secrets") def _helm_install(self) -> None: if self.already_done("helm_install"): return image_tag = self._fetch_image_tag() - print(f" Installing k8s-agent via helm (chart: {HELM_CHART_DIR})...") - subprocess.run( - [ - "helm", "upgrade", "--install", "cortex-k8s-agent", - str(HELM_CHART_DIR), - "--set", f"image.tag={image_tag}", - "--set", f"app.baseUrl={self._base_url}", - "--set", f"app.clusterName={self._cluster_name}", - ], - check=True, - ) - # Restart to ensure secrets/configmaps are picked up - subprocess.run( - ["kubectl", "rollout", "restart", "deployment/cortex-k8s-agent"], - check=True, - ) + print(" Installing k8s-agent via helm...") + if self._use_codespace: + helm_chart = f"{self._remote_solution_dir()}/helm-chart" + self._run_remote( + f"helm upgrade --install cortex-k8s-agent {helm_chart} " + f"--set image.tag={shlex.quote(image_tag)} " + f"--set app.baseUrl={shlex.quote(self._base_url)} " + f"--set app.clusterName={shlex.quote(self._cluster_name)}" + ) + self._run_remote("kubectl rollout restart deployment/cortex-k8s-agent") + else: + subprocess.run( + [ + "helm", "upgrade", "--install", "cortex-k8s-agent", + str(HELM_CHART_DIR), + "--set", f"image.tag={image_tag}", + "--set", f"app.baseUrl={self._base_url}", + "--set", f"app.clusterName={self._cluster_name}", + ], + check=True, + ) + # Restart to ensure secrets/configmaps are picked up + subprocess.run( + ["kubectl", "rollout", "restart", "deployment/cortex-k8s-agent"], + check=True, + ) self.mark_done("helm_install") def _wait_for_readiness(self) -> None: if self.already_done("wait_for_readiness"): return print(" Waiting for k8s-agent pod to be ready (timeout: 120s)...") - subprocess.run( - [ - "kubectl", "rollout", "status", "deployment/cortex-k8s-agent", - "--timeout=120s", - ], - check=True, - ) + if self._use_codespace: + self._run_remote( + "kubectl rollout status deployment/cortex-k8s-agent --timeout=120s" + ) + else: + subprocess.run( + [ + "kubectl", "rollout", "status", "deployment/cortex-k8s-agent", + "--timeout=120s", + ], + check=True, + ) self.mark_done("wait_for_readiness") def _install_argo_crd(self) -> None: if self.already_done("install_argo_crd"): return - print(f" Installing Argo Rollouts CRD...") - subprocess.run( - ["kubectl", "apply", "-f", ARGO_CRD_URL], - check=True, - ) + print(" Installing Argo Rollouts CRD...") + if self._use_codespace: + self._run_remote(f"kubectl apply -f {ARGO_CRD_URL}") + else: + subprocess.run(["kubectl", "apply", "-f", ARGO_CRD_URL], check=True) self.mark_done("install_argo_crd") def _apply_manifests(self) -> None: if self.already_done("apply_manifests"): return - print(f" Applying demo k8s manifests from {MANIFESTS_DIR}...") - subprocess.run( - ["kubectl", "apply", "-f", str(MANIFESTS_DIR)], - check=True, - ) + print(" Applying demo k8s manifests...") + if self._use_codespace: + manifests = f"{self._remote_solution_dir()}/manifests" + self._run_remote(f"kubectl apply -f {manifests}") + else: + subprocess.run(["kubectl", "apply", "-f", str(MANIFESTS_DIR)], check=True) self.mark_done("apply_manifests") def _create_entity(self) -> None: if self.already_done("create_entity"): return - print(f" Creating demo-kubernetes Cortex entity...") + print(" Creating demo-kubernetes Cortex entity...") yaml_content = CATALOG_FILE.read_bytes() r = requests.post( f"{self._base_url}/api/v1/open-api", @@ -202,8 +353,13 @@ def _create_entity(self) -> None: self.mark_done("create_entity") def steps(self) -> list: + first_step = ( + ("Create GitHub Codespace", self._create_codespace) + if self._use_codespace + else ("Check cluster connectivity", self._check_cluster) + ) return [ - ("Check cluster connectivity", self._check_cluster), + first_step, ("Create k8s secrets", self._create_secrets), ("Install k8s-agent via helm", self._helm_install), ("Wait for agent readiness", self._wait_for_readiness), @@ -214,6 +370,11 @@ def steps(self) -> list: def post_steps(self) -> None: print("\n✓ Kubernetes agent deployed and demo workloads running.\n") + if self._codespace_name: + print(f"Codespace: {self._codespace_name}") + print(f" Open terminal: gh codespace ssh -c {self._codespace_name}") + print(f" Stop Codespace: gh codespace stop -c {self._codespace_name}") + print() print("The agent syncs every 5 minutes. After the first sync, visit:") print(f" {self._base_url.replace('api.', 'app.')}/catalog/demo-kubernetes/k8s") print("\nYou should see: demo-deployment, demo-statefulset, demo-cronjob, demo-rollout") From 4c894413f7c9ab421a458e6a9330657ba026d195 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 26 Aug 2026 15:33:30 -0700 Subject: [PATCH 18/43] chore: pin Codespace creation to current feature branch Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index cf81271f..18d0deb2 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -33,6 +33,7 @@ ARGO_CRD_URL = "https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/crds/rollout-crd.yaml" +CODESPACE_BRANCH = "worktree-kubernetes-agent-solution" CODESPACE_READY_TIMEOUT = 600 # 10 minutes for Codespace + kind cluster startup CODESPACE_POLL_INTERVAL = 20 # seconds between readiness checks @@ -160,6 +161,7 @@ def _create_codespace(self) -> None: [ "gh", "codespace", "create", "--repo", self._github_repo, + "--branch", CODESPACE_BRANCH, "--devcontainer-path", ".devcontainer/kubernetes-agent/devcontainer.json", ], check=True, From 0286e0eacd9b0357dd35afba81df652a0a478f31 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 12:37:19 -0700 Subject: [PATCH 19/43] fix: specify machine type for gh codespace create to avoid no-terminal error --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 1 + 1 file changed, 1 insertion(+) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 18d0deb2..0af6a499 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -163,6 +163,7 @@ def _create_codespace(self) -> None: "--repo", self._github_repo, "--branch", CODESPACE_BRANCH, "--devcontainer-path", ".devcontainer/kubernetes-agent/devcontainer.json", + "--machine", "basicLinux32gb", ], check=True, capture_output=True, From caae054be5b60731d9fbf81c4ee54e0d6cfdac82 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 13:21:20 -0700 Subject: [PATCH 20/43] fix: use sudo+arch-detect for kind install in onCreate.sh; clarify Codespace wait message --- .devcontainer/kubernetes-agent/onCreate.sh | 8 +++++--- cortexapps_cli/solutions/kubernetes-agent/setup.py | 4 ++-- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/.devcontainer/kubernetes-agent/onCreate.sh b/.devcontainer/kubernetes-agent/onCreate.sh index defc5d17..86d8c1fd 100644 --- a/.devcontainer/kubernetes-agent/onCreate.sh +++ b/.devcontainer/kubernetes-agent/onCreate.sh @@ -2,9 +2,11 @@ set -euo pipefail echo "==> Installing kind..." -curl -Lo /usr/local/bin/kind \ - https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64 -chmod +x /usr/local/bin/kind +ARCH=$(uname -m) +KIND_ARCH="amd64" +[ "$ARCH" = "aarch64" ] && KIND_ARCH="arm64" +curl -Lo /tmp/kind "https://kind.sigs.k8s.io/dl/latest/kind-linux-${KIND_ARCH}" +sudo install -o root -g root -m 0755 /tmp/kind /usr/local/bin/kind echo "==> Creating kind cluster 'cortex-demo'..." kind create cluster --name cortex-demo --wait 60s diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 0af6a499..67cb2bf8 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -179,8 +179,8 @@ def _create_codespace(self) -> None: self._save_file() print( - f" Waiting for Codespace and kind cluster to initialize " - f"(up to {CODESPACE_READY_TIMEOUT // 60} min)..." + f" Codespace is up. Waiting for onCreate.sh to finish installing kind " + f"and creating the cluster (up to {CODESPACE_READY_TIMEOUT // 60} min)..." ) deadline = time.time() + CODESPACE_READY_TIMEOUT while time.time() < deadline: From a4aee9a85296adefb648039f9e766bd06f59dd82 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 13:22:19 -0700 Subject: [PATCH 21/43] chore: add phased status messages during Codespace/kind cluster wait --- .../solutions/kubernetes-agent/setup.py | 30 +++++++++++++++++-- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 67cb2bf8..8bd0a08b 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -178,11 +178,34 @@ def _create_codespace(self) -> None: self._state["codespace_name"] = self._codespace_name self._save_file() + deadline = time.time() + CODESPACE_READY_TIMEOUT + + # Phase 1: wait for Codespace to reach Available state + print(" Waiting for Codespace to start...") + while time.time() < deadline: + state_result = subprocess.run( + ["gh", "codespace", "view", "-c", self._codespace_name, "--json", "state"], + capture_output=True, + text=True, + ) + if state_result.returncode == 0: + import json as _json + state = _json.loads(state_result.stdout).get("state", "") + if state == "Available": + print(" Codespace is up.") + break + time.sleep(CODESPACE_POLL_INTERVAL) + else: + raise RuntimeError( + f"Timed out waiting for Codespace '{self._codespace_name}' to start.\n" + "Re-run this command to retry." + ) + + # Phase 2: wait for onCreate.sh to finish (kind cluster ready) print( - f" Codespace is up. Waiting for onCreate.sh to finish installing kind " - f"and creating the cluster (up to {CODESPACE_READY_TIMEOUT // 60} min)..." + f" Waiting for kind cluster to initialize inside Codespace " + f"(onCreate.sh installs kind and creates the cluster, ~2-4 min)..." ) - deadline = time.time() + CODESPACE_READY_TIMEOUT while time.time() < deadline: probe = subprocess.run( [ @@ -193,6 +216,7 @@ def _create_codespace(self) -> None: capture_output=True, ) if probe.returncode == 0: + print(" Kind cluster is ready.") return time.sleep(CODESPACE_POLL_INTERVAL) From 48096c69b76b287f88ab8453a4fc92e48ba97c04 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 13:25:21 -0700 Subject: [PATCH 22/43] fix: move kubectl/helm install into onCreate.sh, remove kubectl-helm-minikube feature --- .devcontainer/kubernetes-agent/devcontainer.json | 7 +------ .devcontainer/kubernetes-agent/onCreate.sh | 16 ++++++++++++---- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/.devcontainer/kubernetes-agent/devcontainer.json b/.devcontainer/kubernetes-agent/devcontainer.json index 524dfe0f..0c1bfcab 100644 --- a/.devcontainer/kubernetes-agent/devcontainer.json +++ b/.devcontainer/kubernetes-agent/devcontainer.json @@ -2,12 +2,7 @@ "name": "Cortex Kubernetes Agent Demo", "image": "mcr.microsoft.com/devcontainers/base:ubuntu-24.04", "features": { - "ghcr.io/devcontainers/features/docker-in-docker:2": {}, - "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": { - "version": "latest", - "helm": "latest", - "minikube": "none" - } + "ghcr.io/devcontainers/features/docker-in-docker:2": {} }, "onCreateCommand": "bash .devcontainer/kubernetes-agent/onCreate.sh", "remoteEnv": { diff --git a/.devcontainer/kubernetes-agent/onCreate.sh b/.devcontainer/kubernetes-agent/onCreate.sh index 86d8c1fd..75e0ad3c 100644 --- a/.devcontainer/kubernetes-agent/onCreate.sh +++ b/.devcontainer/kubernetes-agent/onCreate.sh @@ -1,11 +1,19 @@ #!/usr/bin/env bash set -euo pipefail -echo "==> Installing kind..." ARCH=$(uname -m) -KIND_ARCH="amd64" -[ "$ARCH" = "aarch64" ] && KIND_ARCH="arm64" -curl -Lo /tmp/kind "https://kind.sigs.k8s.io/dl/latest/kind-linux-${KIND_ARCH}" +BIN_ARCH="amd64" +[ "$ARCH" = "aarch64" ] && BIN_ARCH="arm64" + +echo "==> Installing kubectl..." +curl -Lo /tmp/kubectl "https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable.txt)/bin/linux/${BIN_ARCH}/kubectl" +sudo install -o root -g root -m 0755 /tmp/kubectl /usr/local/bin/kubectl + +echo "==> Installing helm..." +curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash + +echo "==> Installing kind..." +curl -Lo /tmp/kind "https://kind.sigs.k8s.io/dl/latest/kind-linux-${BIN_ARCH}" sudo install -o root -g root -m 0755 /tmp/kind /usr/local/bin/kind echo "==> Creating kind cluster 'cortex-demo'..." From dbf8acd81bee753bc0cf20c9162f9349f1d07c8e Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 13:38:01 -0700 Subject: [PATCH 23/43] fix: verify saved Codespace exists before reusing, create new one if not found --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 8bd0a08b..06d1626a 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -153,8 +153,17 @@ def _create_codespace(self) -> None: self._check_gh_cli() if self._codespace_name: - print(f" Using existing Codespace: {self._codespace_name}") - return + # Verify the saved Codespace still exists; if not, create a fresh one + probe = subprocess.run( + ["gh", "codespace", "view", "-c", self._codespace_name, "--json", "name"], + capture_output=True, + ) + if probe.returncode == 0: + print(f" Using existing Codespace: {self._codespace_name}") + return + print(f" Saved Codespace '{self._codespace_name}' no longer exists — creating a new one...") + self._codespace_name = "" + self._state.pop("codespace_name", None) print(f" Creating GitHub Codespace from {self._github_repo}...") result = subprocess.run( From af160ed8727fd232162c0f8ad4ff78483bf0bf32 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 13:46:32 -0700 Subject: [PATCH 24/43] chore: save GHCR_TOKEN between runs using hidden=True --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 06d1626a..d26a65b9 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -82,7 +82,7 @@ def collect_prompts(self) -> None: "GitHub PAT provided by Cortex Customer Engineering for pulling the k8s-agent image" " (see https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#prerequisites)", env_var="GHCR_TOKEN", - secret=True, + hidden=True, ) self._cluster_name = self.prompt( "cluster_name", From 4ac61c8aac37d0b8c06843726f4894999ee083be Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 14:09:17 -0700 Subject: [PATCH 25/43] chore: increase Codespace timeout to 15 min, update wait message to reflect longer onCreate --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index d26a65b9..744a5a5e 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -34,7 +34,7 @@ ARGO_CRD_URL = "https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/crds/rollout-crd.yaml" CODESPACE_BRANCH = "worktree-kubernetes-agent-solution" -CODESPACE_READY_TIMEOUT = 600 # 10 minutes for Codespace + kind cluster startup +CODESPACE_READY_TIMEOUT = 900 # 15 minutes for Codespace + kind cluster startup CODESPACE_POLL_INTERVAL = 20 # seconds between readiness checks @@ -213,7 +213,7 @@ def _create_codespace(self) -> None: # Phase 2: wait for onCreate.sh to finish (kind cluster ready) print( f" Waiting for kind cluster to initialize inside Codespace " - f"(onCreate.sh installs kind and creates the cluster, ~2-4 min)..." + f"(onCreate.sh installs kubectl/helm/kind and creates the cluster, ~5-10 min)..." ) while time.time() < deadline: probe = subprocess.run( From dffa45fea68a53c88fa689337f2ef96db4eef177 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 14:15:55 -0700 Subject: [PATCH 26/43] fix: poll kubectl readiness when reusing existing Codespace Previously the existing-Codespace path returned immediately after verifying the Codespace exists, without waiting for kubectl/kind to be ready. Now both the new and reuse paths converge on the same kubectl readiness poll. Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/setup.py | 93 ++++++++++--------- 1 file changed, 49 insertions(+), 44 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 744a5a5e..cabd4844 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -152,6 +152,7 @@ def _create_codespace(self) -> None: # Always check gh CLI — also needed for SSH steps that follow self._check_gh_cli() + existing = bool(self._codespace_name) if self._codespace_name: # Verify the saved Codespace still exists; if not, create a fresh one probe = subprocess.run( @@ -160,60 +161,64 @@ def _create_codespace(self) -> None: ) if probe.returncode == 0: print(f" Using existing Codespace: {self._codespace_name}") - return - print(f" Saved Codespace '{self._codespace_name}' no longer exists — creating a new one...") - self._codespace_name = "" - self._state.pop("codespace_name", None) + else: + print(f" Saved Codespace '{self._codespace_name}' no longer exists — creating a new one...") + self._codespace_name = "" + self._state.pop("codespace_name", None) + existing = False - print(f" Creating GitHub Codespace from {self._github_repo}...") - result = subprocess.run( - [ - "gh", "codespace", "create", - "--repo", self._github_repo, - "--branch", CODESPACE_BRANCH, - "--devcontainer-path", ".devcontainer/kubernetes-agent/devcontainer.json", - "--machine", "basicLinux32gb", - ], - check=True, - capture_output=True, - text=True, - ) - self._codespace_name = result.stdout.strip() if not self._codespace_name: - raise RuntimeError("gh codespace create did not return a codespace name") - print(f" Codespace created: {self._codespace_name}") + print(f" Creating GitHub Codespace from {self._github_repo}...") + result = subprocess.run( + [ + "gh", "codespace", "create", + "--repo", self._github_repo, + "--branch", CODESPACE_BRANCH, + "--devcontainer-path", ".devcontainer/kubernetes-agent/devcontainer.json", + "--machine", "basicLinux32gb", + ], + check=True, + capture_output=True, + text=True, + ) + self._codespace_name = result.stdout.strip() + if not self._codespace_name: + raise RuntimeError("gh codespace create did not return a codespace name") + print(f" Codespace created: {self._codespace_name}") - # Persist the name so re-runs find the existing Codespace - self._state["codespace_name"] = self._codespace_name - self._save_file() + # Persist the name so re-runs find the existing Codespace + self._state["codespace_name"] = self._codespace_name + self._save_file() deadline = time.time() + CODESPACE_READY_TIMEOUT # Phase 1: wait for Codespace to reach Available state - print(" Waiting for Codespace to start...") - while time.time() < deadline: - state_result = subprocess.run( - ["gh", "codespace", "view", "-c", self._codespace_name, "--json", "state"], - capture_output=True, - text=True, - ) - if state_result.returncode == 0: - import json as _json - state = _json.loads(state_result.stdout).get("state", "") - if state == "Available": - print(" Codespace is up.") - break - time.sleep(CODESPACE_POLL_INTERVAL) - else: - raise RuntimeError( - f"Timed out waiting for Codespace '{self._codespace_name}' to start.\n" - "Re-run this command to retry." - ) + if not existing: + print(" Waiting for Codespace to start...") + while time.time() < deadline: + state_result = subprocess.run( + ["gh", "codespace", "view", "-c", self._codespace_name, "--json", "state"], + capture_output=True, + text=True, + ) + if state_result.returncode == 0: + import json as _json + state = _json.loads(state_result.stdout).get("state", "") + if state == "Available": + print(" Codespace is up.") + break + time.sleep(CODESPACE_POLL_INTERVAL) + else: + raise RuntimeError( + f"Timed out waiting for Codespace '{self._codespace_name}' to start.\n" + "Re-run this command to retry." + ) # Phase 2: wait for onCreate.sh to finish (kind cluster ready) + # Always poll — even for existing Codespaces that may still be initializing. print( - f" Waiting for kind cluster to initialize inside Codespace " - f"(onCreate.sh installs kubectl/helm/kind and creates the cluster, ~5-10 min)..." + " Waiting for kind cluster to be ready" + + (" (onCreate.sh installs kubectl/helm/kind and creates the cluster, ~5-10 min)..." if not existing else "...") ) while time.time() < deadline: probe = subprocess.run( From 9860560c7cc3767a112486904b2c12a7d0c7f866 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 14:39:34 -0700 Subject: [PATCH 27/43] fix: log onCreate.sh output and improve timeout error message - Tee all onCreate.sh output to /tmp/onCreate.log for diagnosability - Use python3 -m pip instead of pip (more reliable on Ubuntu 24.04) - Timeout error now shows the SSH + log command to diagnose failures Co-Authored-By: Claude Sonnet 4.6 --- .devcontainer/kubernetes-agent/onCreate.sh | 5 ++++- cortexapps_cli/solutions/kubernetes-agent/setup.py | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/.devcontainer/kubernetes-agent/onCreate.sh b/.devcontainer/kubernetes-agent/onCreate.sh index 75e0ad3c..52ae9d09 100644 --- a/.devcontainer/kubernetes-agent/onCreate.sh +++ b/.devcontainer/kubernetes-agent/onCreate.sh @@ -1,6 +1,9 @@ #!/usr/bin/env bash set -euo pipefail +# Log all output so failures are diagnosable: cat /tmp/onCreate.log +exec > >(tee /tmp/onCreate.log) 2>&1 + ARCH=$(uname -m) BIN_ARCH="amd64" [ "$ARCH" = "aarch64" ] && BIN_ARCH="arm64" @@ -23,6 +26,6 @@ echo "==> Verifying cluster..." kubectl cluster-info --context kind-cortex-demo echo "==> Installing cortexapps-cli..." -pip install cortexapps-cli --quiet +python3 -m pip install cortexapps-cli --quiet echo "==> Done. Run: cortex solutions post-install -s kubernetes-agent" diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index cabd4844..76567a07 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -236,7 +236,10 @@ def _create_codespace(self) -> None: raise RuntimeError( f"Timed out waiting for the kind cluster in Codespace '{self._codespace_name}'.\n" - "The Codespace may still be initializing. Re-run this command to retry." + "To diagnose, SSH into the Codespace and check the log:\n" + f" gh codespace ssh -c {self._codespace_name}\n" + " cat /tmp/onCreate.log\n" + "Re-run this command to retry once the cluster is ready." ) # ------------------------------------------------------------------------- From d087cd67a3e4c02428faf2030bfbed75bdca9d19 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 14:56:43 -0700 Subject: [PATCH 28/43] fix: detect onCreate.sh failure immediately instead of timing out - onCreate.sh writes /tmp/onCreate.failed on non-zero exit (EXIT trap) - Poll loop checks failure sentinel on each iteration via a single SSH call, failing fast instead of waiting out the full 15-min timeout - Combined kubectl check + sentinel check into one SSH round-trip - Corrected wait message to "may take 15-20 min" (was "~5-10 min") Co-Authored-By: Claude Sonnet 4.6 --- .devcontainer/kubernetes-agent/onCreate.sh | 9 +++++++ .../solutions/kubernetes-agent/setup.py | 26 +++++++++++++------ 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/.devcontainer/kubernetes-agent/onCreate.sh b/.devcontainer/kubernetes-agent/onCreate.sh index 52ae9d09..921a2244 100644 --- a/.devcontainer/kubernetes-agent/onCreate.sh +++ b/.devcontainer/kubernetes-agent/onCreate.sh @@ -4,6 +4,15 @@ set -euo pipefail # Log all output so failures are diagnosable: cat /tmp/onCreate.log exec > >(tee /tmp/onCreate.log) 2>&1 +# Write a failure sentinel on non-zero exit so setup.py can detect it fast +_on_exit() { + local rc=$? + if [[ $rc -ne 0 ]]; then + echo "$rc" > /tmp/onCreate.failed + fi +} +trap _on_exit EXIT + ARCH=$(uname -m) BIN_ARCH="amd64" [ "$ARCH" = "aarch64" ] && BIN_ARCH="arm64" diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 76567a07..72969964 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -216,27 +216,37 @@ def _create_codespace(self) -> None: # Phase 2: wait for onCreate.sh to finish (kind cluster ready) # Always poll — even for existing Codespaces that may still be initializing. - print( - " Waiting for kind cluster to be ready" - + (" (onCreate.sh installs kubectl/helm/kind and creates the cluster, ~5-10 min)..." if not existing else "...") - ) + print(" Waiting for kind cluster to be ready (may take 15-20 min on first run)...") while time.time() < deadline: - probe = subprocess.run( + # Single SSH call: check failure sentinel and cluster readiness together + status = subprocess.run( [ "gh", "codespace", "ssh", "-c", self._codespace_name, - "--", "kubectl", "cluster-info", + "--", "bash", "-c", + "if [ -f /tmp/onCreate.failed ]; then echo FAILED; cat /tmp/onCreate.failed; " + "elif kubectl cluster-info > /dev/null 2>&1; then echo READY; " + "else echo WAITING; fi", ], capture_output=True, + text=True, ) - if probe.returncode == 0: + output = status.stdout.strip() + if output.startswith("READY"): print(" Kind cluster is ready.") return + if output.startswith("FAILED"): + raise RuntimeError( + f"onCreate.sh failed in Codespace '{self._codespace_name}'.\n" + "Check the log for details:\n" + f" gh codespace ssh -c {self._codespace_name}\n" + " cat /tmp/onCreate.log" + ) time.sleep(CODESPACE_POLL_INTERVAL) raise RuntimeError( f"Timed out waiting for the kind cluster in Codespace '{self._codespace_name}'.\n" - "To diagnose, SSH into the Codespace and check the log:\n" + "Check the log for details:\n" f" gh codespace ssh -c {self._codespace_name}\n" " cat /tmp/onCreate.log\n" "Re-run this command to retry once the cluster is ready." From 7c5d6738b48214210d3c0e69041add614e12d483 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 15:21:10 -0700 Subject: [PATCH 29/43] fix: remove cortexapps-cli install from onCreate.sh (python3 not found) The CLI runs locally; installing it in the Codespace is unnecessary and was causing onCreate.sh to fail (Ubuntu 24.04 base has no python3 in PATH by default), which triggered Alpine recovery mode. Co-Authored-By: Claude Sonnet 4.6 --- .devcontainer/kubernetes-agent/devcontainer.json | 2 +- .devcontainer/kubernetes-agent/onCreate.sh | 5 +---- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/.devcontainer/kubernetes-agent/devcontainer.json b/.devcontainer/kubernetes-agent/devcontainer.json index 0c1bfcab..1d042ab6 100644 --- a/.devcontainer/kubernetes-agent/devcontainer.json +++ b/.devcontainer/kubernetes-agent/devcontainer.json @@ -9,5 +9,5 @@ "CORTEX_API_KEY": "${localEnv:CORTEX_API_KEY}", "GHCR_TOKEN": "${localEnv:GHCR_TOKEN}" }, - "postCreateMessage": "Run: cortex solutions install -s kubernetes-agent && cortex solutions post-install -s kubernetes-agent" + "postCreateMessage": "kind cluster is ready. Run cortex solutions post-install -s kubernetes-agent from your local machine." } diff --git a/.devcontainer/kubernetes-agent/onCreate.sh b/.devcontainer/kubernetes-agent/onCreate.sh index 921a2244..b0d56252 100644 --- a/.devcontainer/kubernetes-agent/onCreate.sh +++ b/.devcontainer/kubernetes-agent/onCreate.sh @@ -34,7 +34,4 @@ kind create cluster --name cortex-demo --wait 60s echo "==> Verifying cluster..." kubectl cluster-info --context kind-cortex-demo -echo "==> Installing cortexapps-cli..." -python3 -m pip install cortexapps-cli --quiet - -echo "==> Done. Run: cortex solutions post-install -s kubernetes-agent" +echo "==> Done." From 37feda0a2b99968a9e09ead3021e52ef8b868934 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 16:08:06 -0700 Subject: [PATCH 30/43] fix: auto-fetch and print onCreate.log on failure instead of asking user Both the FAILED sentinel and timeout error paths now SSH in and fetch /tmp/onCreate.log automatically, printing it inline so the user sees the failure reason immediately without manual steps. Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/setup.py | 23 ++++++++++++------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 72969964..3db25f90 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -134,6 +134,15 @@ def _fetch_image_tag(self) -> str: # Step: Create GitHub Codespace (codespace mode only) # ------------------------------------------------------------------------- + def _fetch_codespace_log(self) -> str: + """Fetch /tmp/onCreate.log from the Codespace, or a placeholder if unavailable.""" + result = subprocess.run( + ["gh", "codespace", "ssh", "-c", self._codespace_name, "--", "cat", "/tmp/onCreate.log"], + capture_output=True, + text=True, + ) + return result.stdout.strip() if result.stdout.strip() else "(log not available)" + def _check_gh_cli(self) -> None: """Verify gh CLI is installed and authenticated.""" if subprocess.run(["gh", "--version"], capture_output=True).returncode != 0: @@ -236,19 +245,17 @@ def _create_codespace(self) -> None: print(" Kind cluster is ready.") return if output.startswith("FAILED"): + log = self._fetch_codespace_log() raise RuntimeError( - f"onCreate.sh failed in Codespace '{self._codespace_name}'.\n" - "Check the log for details:\n" - f" gh codespace ssh -c {self._codespace_name}\n" - " cat /tmp/onCreate.log" + f"onCreate.sh failed in Codespace '{self._codespace_name}'.\n\n" + f"--- /tmp/onCreate.log ---\n{log}\n---" ) time.sleep(CODESPACE_POLL_INTERVAL) + log = self._fetch_codespace_log() raise RuntimeError( - f"Timed out waiting for the kind cluster in Codespace '{self._codespace_name}'.\n" - "Check the log for details:\n" - f" gh codespace ssh -c {self._codespace_name}\n" - " cat /tmp/onCreate.log\n" + f"Timed out waiting for the kind cluster in Codespace '{self._codespace_name}'.\n\n" + f"--- /tmp/onCreate.log ---\n{log}\n---\n\n" "Re-run this command to retry once the cluster is ready." ) From 501a2fd7a8353ba4f11d504402da5b5a37a94e22 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Wed, 2 Sep 2026 16:41:24 -0700 Subject: [PATCH 31/43] fix: give Phase 2 its own deadline; add sshd feature for early SSH access MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Phase 1 (Codespace Available) and Phase 2 (kubectl ready) now each get their own full CODESPACE_READY_TIMEOUT budget instead of sharing one — Phase 1 taking 5+ min was consuming kubectl polling time - Add sshd devcontainer feature so SSH is available immediately when the Codespace starts, before onCreateCommand completes Co-Authored-By: Claude Sonnet 4.6 --- .devcontainer/kubernetes-agent/devcontainer.json | 3 ++- cortexapps_cli/solutions/kubernetes-agent/setup.py | 9 +++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/.devcontainer/kubernetes-agent/devcontainer.json b/.devcontainer/kubernetes-agent/devcontainer.json index 1d042ab6..b7ae4a62 100644 --- a/.devcontainer/kubernetes-agent/devcontainer.json +++ b/.devcontainer/kubernetes-agent/devcontainer.json @@ -2,7 +2,8 @@ "name": "Cortex Kubernetes Agent Demo", "image": "mcr.microsoft.com/devcontainers/base:ubuntu-24.04", "features": { - "ghcr.io/devcontainers/features/docker-in-docker:2": {} + "ghcr.io/devcontainers/features/docker-in-docker:2": {}, + "ghcr.io/devcontainers/features/sshd:1": {} }, "onCreateCommand": "bash .devcontainer/kubernetes-agent/onCreate.sh", "remoteEnv": { diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 3db25f90..dcff2828 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -199,12 +199,11 @@ def _create_codespace(self) -> None: self._state["codespace_name"] = self._codespace_name self._save_file() - deadline = time.time() + CODESPACE_READY_TIMEOUT - - # Phase 1: wait for Codespace to reach Available state + # Phase 1: wait for Codespace to reach Available state (own deadline) if not existing: print(" Waiting for Codespace to start...") - while time.time() < deadline: + phase1_deadline = time.time() + CODESPACE_READY_TIMEOUT + while time.time() < phase1_deadline: state_result = subprocess.run( ["gh", "codespace", "view", "-c", self._codespace_name, "--json", "state"], capture_output=True, @@ -224,7 +223,9 @@ def _create_codespace(self) -> None: ) # Phase 2: wait for onCreate.sh to finish (kind cluster ready) + # Fresh deadline — Phase 1 timing does not eat into this budget. # Always poll — even for existing Codespaces that may still be initializing. + deadline = time.time() + CODESPACE_READY_TIMEOUT print(" Waiting for kind cluster to be ready (may take 15-20 min on first run)...") while time.time() < deadline: # Single SSH call: check failure sentinel and cluster readiness together From 86de5163a2bb7a806ccf844f5863e809d9ae57f7 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 14:34:59 -0700 Subject: [PATCH 32/43] fix: use explicit kubeconfig path in poll; use login shell for _run_remote Non-interactive SSH sessions don't load .bashrc, so kubectl doesn't find ~/.kube/config via KUBECONFIG. The readiness poll now uses the explicit binary path and kubeconfig. _run_remote uses bash -lc (login shell) so PATH and env are set correctly for kubectl/helm calls. Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/setup.py | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index dcff2828..31fa31db 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -100,12 +100,16 @@ def _remote_solution_dir(self) -> str: return f"/workspaces/{repo_name}/cortexapps_cli/solutions/kubernetes-agent" def _run_remote(self, bash_cmd: str) -> None: - """Run a bash command inside the Codespace via gh codespace ssh.""" + """Run a bash command inside the Codespace via gh codespace ssh. + + Sources /etc/profile and ~/.bashrc so PATH and KUBECONFIG are set + correctly in the non-interactive SSH session. + """ subprocess.run( [ "gh", "codespace", "ssh", "-c", self._codespace_name, - "--", "bash", "-c", bash_cmd, + "--", "bash", "-lc", bash_cmd, ], check=True, ) @@ -228,14 +232,17 @@ def _create_codespace(self) -> None: deadline = time.time() + CODESPACE_READY_TIMEOUT print(" Waiting for kind cluster to be ready (may take 15-20 min on first run)...") while time.time() < deadline: - # Single SSH call: check failure sentinel and cluster readiness together + # Single SSH call: check failure sentinel and cluster readiness together. + # Use explicit kubeconfig path — non-interactive SSH sessions don't load + # .bashrc so KUBECONFIG env var and PATH additions are not set. status = subprocess.run( [ "gh", "codespace", "ssh", "-c", self._codespace_name, "--", "bash", "-c", "if [ -f /tmp/onCreate.failed ]; then echo FAILED; cat /tmp/onCreate.failed; " - "elif kubectl cluster-info > /dev/null 2>&1; then echo READY; " + "elif /usr/local/bin/kubectl --kubeconfig /home/vscode/.kube/config " + "cluster-info > /dev/null 2>&1; then echo READY; " "else echo WAITING; fi", ], capture_output=True, From 31a4938416039fba191867137706ff4b6d129423 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 14:57:32 -0700 Subject: [PATCH 33/43] fix: replace bash -c poll script with simple SSH commands MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit gh codespace ssh joins post-'--' args with spaces before the remote shell sees them, so shell metacharacters (;, >, |) in our bash -c script were interpreted by the remote shell, not bash. Replace with two simple SSH calls: 'test -f /tmp/onCreate.failed' for the sentinel and direct kubectl invocation for readiness — no shell metacharacters. Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/setup.py | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 31fa31db..b65ec939 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -232,32 +232,32 @@ def _create_codespace(self) -> None: deadline = time.time() + CODESPACE_READY_TIMEOUT print(" Waiting for kind cluster to be ready (may take 15-20 min on first run)...") while time.time() < deadline: - # Single SSH call: check failure sentinel and cluster readiness together. - # Use explicit kubeconfig path — non-interactive SSH sessions don't load - # .bashrc so KUBECONFIG env var and PATH additions are not set. - status = subprocess.run( - [ - "gh", "codespace", "ssh", - "-c", self._codespace_name, - "--", "bash", "-c", - "if [ -f /tmp/onCreate.failed ]; then echo FAILED; cat /tmp/onCreate.failed; " - "elif /usr/local/bin/kubectl --kubeconfig /home/vscode/.kube/config " - "cluster-info > /dev/null 2>&1; then echo READY; " - "else echo WAITING; fi", - ], + # Check failure sentinel — simple command, no shell metacharacters. + fail_check = subprocess.run( + ["gh", "codespace", "ssh", "-c", self._codespace_name, + "--", "test", "-f", "/tmp/onCreate.failed"], capture_output=True, - text=True, ) - output = status.stdout.strip() - if output.startswith("READY"): - print(" Kind cluster is ready.") - return - if output.startswith("FAILED"): + if fail_check.returncode == 0: log = self._fetch_codespace_log() raise RuntimeError( f"onCreate.sh failed in Codespace '{self._codespace_name}'.\n\n" f"--- /tmp/onCreate.log ---\n{log}\n---" ) + + # Check cluster readiness with explicit binary + kubeconfig paths. + # Non-interactive SSH sessions don't load .bashrc, so PATH and + # KUBECONFIG are not set from the user's shell configuration. + ready_check = subprocess.run( + ["gh", "codespace", "ssh", "-c", self._codespace_name, + "--", "/usr/local/bin/kubectl", + "--kubeconfig", "/home/vscode/.kube/config", + "cluster-info"], + capture_output=True, + ) + if ready_check.returncode == 0: + print(" Kind cluster is ready.") + return time.sleep(CODESPACE_POLL_INTERVAL) log = self._fetch_codespace_log() From e22aa9fccba0ba9ae622e99e6a32a0f7f4781872 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 15:21:11 -0700 Subject: [PATCH 34/43] fix: use gh codespace cp to upload scripts instead of bash -c via SSH gh codespace ssh joins post-'--' args with spaces before the remote shell sees them, making it impossible to safely pass scripts with shell metacharacters (|, >, ;). bash -lc also sources profile scripts that can corrupt stdout of piped commands (e.g. kubectl apply -f -). _run_remote now writes the script to a local temp file, copies it to the Codespace via gh codespace cp, then executes it with a clean env. Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/setup.py | 49 ++++++++++++++----- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index b65ec939..b54e20a6 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -12,9 +12,11 @@ "(https://kind.sigs.k8s.io) automatically, or deploy to any existing cluster." ) +import os import shlex import subprocess import sys +import tempfile import time from pathlib import Path @@ -100,19 +102,44 @@ def _remote_solution_dir(self) -> str: return f"/workspaces/{repo_name}/cortexapps_cli/solutions/kubernetes-agent" def _run_remote(self, bash_cmd: str) -> None: - """Run a bash command inside the Codespace via gh codespace ssh. - - Sources /etc/profile and ~/.bashrc so PATH and KUBECONFIG are set - correctly in the non-interactive SSH session. + """Run a bash command inside the Codespace. + + Writes the command to a temp script file and copies it via + 'gh codespace cp', then executes it. This avoids two pitfalls + of passing complex commands via 'gh codespace ssh -- bash -c ...': + 1. gh joins post-'--' args with spaces before the remote shell + sees them, so shell metacharacters (|, >, ;) are interpreted + by the remote shell rather than bash. + 2. 'bash -lc' sources profile scripts that can print to stdout, + corrupting piped commands (e.g. kubectl apply -f -). """ - subprocess.run( - [ - "gh", "codespace", "ssh", - "-c", self._codespace_name, - "--", "bash", "-lc", bash_cmd, - ], - check=True, + script = ( + "#!/bin/bash\n" + "set -euo pipefail\n" + "export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin\n" + "export KUBECONFIG=/home/vscode/.kube/config\n" + f"{bash_cmd}\n" ) + with tempfile.NamedTemporaryFile( + mode="w", suffix=".sh", delete=False + ) as f: + f.write(script) + tmp_path = f.name + try: + subprocess.run( + ["gh", "codespace", "cp", + tmp_path, "remote:/tmp/cortex-remote-cmd.sh", + "-c", self._codespace_name], + check=True, + capture_output=True, + ) + subprocess.run( + ["gh", "codespace", "ssh", "-c", self._codespace_name, + "--", "bash", "/tmp/cortex-remote-cmd.sh"], + check=True, + ) + finally: + os.unlink(tmp_path) def _fetch_image_tag(self) -> str: """Fetch the latest k8s-agent image tag from the GitHub API.""" From 019da9162298069a004485ec8415580303507c1d Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 15:50:36 -0700 Subject: [PATCH 35/43] fix: pipe script via SSH stdin to tee instead of using gh codespace cp MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit gh codespace cp wraps remote paths in single quotes (scp behavior), making the destination filename include literal quote characters. Instead, pipe the script to 'tee /home/vscode/cortex-run.sh' via SSH stdin — no file path quoting issues, no shell metacharacters. Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/setup.py | 48 ++++++++----------- 1 file changed, 19 insertions(+), 29 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index b54e20a6..449b59a0 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -12,11 +12,9 @@ "(https://kind.sigs.k8s.io) automatically, or deploy to any existing cluster." ) -import os import shlex import subprocess import sys -import tempfile import time from pathlib import Path @@ -104,14 +102,13 @@ def _remote_solution_dir(self) -> str: def _run_remote(self, bash_cmd: str) -> None: """Run a bash command inside the Codespace. - Writes the command to a temp script file and copies it via - 'gh codespace cp', then executes it. This avoids two pitfalls - of passing complex commands via 'gh codespace ssh -- bash -c ...': + Pipes the script to 'tee' via SSH stdin, then executes it. + This avoids two pitfalls of 'gh codespace ssh -- bash -c SCRIPT': 1. gh joins post-'--' args with spaces before the remote shell - sees them, so shell metacharacters (|, >, ;) are interpreted - by the remote shell rather than bash. - 2. 'bash -lc' sources profile scripts that can print to stdout, - corrupting piped commands (e.g. kubectl apply -f -). + sees them, so metacharacters (|, >, ;) in the script are + interpreted by the remote shell instead of bash. + 2. 'bash -lc' sources profile scripts that print to stdout, + corrupting piped commands (e.g. kubectl create | kubectl apply). """ script = ( "#!/bin/bash\n" @@ -119,27 +116,20 @@ def _run_remote(self, bash_cmd: str) -> None: "export PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin\n" "export KUBECONFIG=/home/vscode/.kube/config\n" f"{bash_cmd}\n" + ).encode() + remote_script = "/home/vscode/cortex-run.sh" + subprocess.run( + ["gh", "codespace", "ssh", "-c", self._codespace_name, + "--", "tee", remote_script], + input=script, + check=True, + capture_output=True, + ) + subprocess.run( + ["gh", "codespace", "ssh", "-c", self._codespace_name, + "--", "bash", remote_script], + check=True, ) - with tempfile.NamedTemporaryFile( - mode="w", suffix=".sh", delete=False - ) as f: - f.write(script) - tmp_path = f.name - try: - subprocess.run( - ["gh", "codespace", "cp", - tmp_path, "remote:/tmp/cortex-remote-cmd.sh", - "-c", self._codespace_name], - check=True, - capture_output=True, - ) - subprocess.run( - ["gh", "codespace", "ssh", "-c", self._codespace_name, - "--", "bash", "/tmp/cortex-remote-cmd.sh"], - check=True, - ) - finally: - os.unlink(tmp_path) def _fetch_image_tag(self) -> str: """Fetch the latest k8s-agent image tag from the GitHub API.""" From 514a151167c92fbb608bff81493605aa72e4e6b7 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 16:17:46 -0700 Subject: [PATCH 36/43] fix: chmod helm world-executable after install (get-helm-3 sets 750) The helm install script installs with -rwxr-xr-- (750), which the vscode user (non-root, not in root group) cannot execute. Co-Authored-By: Claude Sonnet 4.6 --- .devcontainer/kubernetes-agent/onCreate.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/.devcontainer/kubernetes-agent/onCreate.sh b/.devcontainer/kubernetes-agent/onCreate.sh index b0d56252..2a2d05a8 100644 --- a/.devcontainer/kubernetes-agent/onCreate.sh +++ b/.devcontainer/kubernetes-agent/onCreate.sh @@ -23,6 +23,7 @@ sudo install -o root -g root -m 0755 /tmp/kubectl /usr/local/bin/kubectl echo "==> Installing helm..." curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash +sudo chmod a+rx /usr/local/bin/helm echo "==> Installing kind..." curl -Lo /tmp/kind "https://kind.sigs.k8s.io/dl/latest/kind-linux-${BIN_ARCH}" From 92d9b9345f0204242037831b23fda48d09cd7ccc Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 16:29:38 -0700 Subject: [PATCH 37/43] fix: use --server-side apply for Argo Rollouts CRD Client-side kubectl apply stores the full manifest in a last-applied-configuration annotation, which exceeds the 262144-byte limit for the large Argo Rollouts CRD. Server-side apply avoids this. Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index 449b59a0..ceca027c 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -402,10 +402,14 @@ def _install_argo_crd(self) -> None: if self.already_done("install_argo_crd"): return print(" Installing Argo Rollouts CRD...") + # --server-side avoids the 262144-byte annotation limit that + # kubectl apply (client-side) hits with large CRDs like Argo Rollouts. if self._use_codespace: - self._run_remote(f"kubectl apply -f {ARGO_CRD_URL}") + self._run_remote(f"kubectl apply --server-side -f {ARGO_CRD_URL}") else: - subprocess.run(["kubectl", "apply", "-f", ARGO_CRD_URL], check=True) + subprocess.run( + ["kubectl", "apply", "--server-side", "-f", ARGO_CRD_URL], check=True + ) self.mark_done("install_argo_crd") def _apply_manifests(self) -> None: From a88d3ea991fcc79a4707ad5aea468c00851924cd Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 16:32:21 -0700 Subject: [PATCH 38/43] fix: correct post-install URL to /admin/resources?tag=demo-kubernetes Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/setup.py b/cortexapps_cli/solutions/kubernetes-agent/setup.py index ceca027c..7a967fe3 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/setup.py +++ b/cortexapps_cli/solutions/kubernetes-agent/setup.py @@ -466,7 +466,7 @@ def post_steps(self) -> None: print(f" Stop Codespace: gh codespace stop -c {self._codespace_name}") print() print("The agent syncs every 5 minutes. After the first sync, visit:") - print(f" {self._base_url.replace('api.', 'app.')}/catalog/demo-kubernetes/k8s") + print(f" {self._base_url.replace('api.', 'app.')}/admin/resources?tag=demo-kubernetes") print("\nYou should see: demo-deployment, demo-statefulset, demo-cronjob, demo-rollout") print("\nNote: GHCR_TOKEN requirement goes away once the k8s-agent image is made public.") From dab81f0b732b5bc1cf2c9fa6f73e1ee41374434e Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 16:42:10 -0700 Subject: [PATCH 39/43] docs: add architecture diagram; fix timing and URL in README Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/README.md | 26 +++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index 3c7f4f4c..205e459b 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -18,6 +18,28 @@ Demonstrates the [Cortex Kubernetes agent](https://docs.cortex.io/docs/reference After setup, visit your entity's K8s tab to see live workload data synced from the cluster. +## Architecture + +```mermaid +flowchart TD + subgraph CS["GitHub Codespace"] + subgraph kind["kind cluster — cortex-demo"] + K8S["Kubernetes API Server"] + subgraph workloads["demo workloads"] + DW["Deployment / StatefulSet / CronJob / Rollout"] + end + AGENT["cortex-k8s-agent pod"] + end + end + + CORTEX["Cortex Platform"] + + AGENT -->|"reads resources every 5 min"| K8S + AGENT -->|"pushes data via HTTPS"| CORTEX +``` + +The agent runs inside the kind cluster as a Kubernetes Deployment. Every 5 minutes it queries the Kubernetes API for workload resources (Deployments, StatefulSets, CronJobs, Argo Rollouts, etc.) and pushes the collected metadata to the Cortex API over HTTPS. Cortex stores the data and surfaces it on each entity's K8s tab. + ## Prerequisites - A Cortex API key (`CORTEX_API_KEY`) — get from Cortex Settings → API Keys @@ -46,7 +68,7 @@ Create a new GitHub Codespace with a kind cluster? (yes = spin up Codespace, no The script will: 1. Create a Codespace from the `cortexapps/cli` repository using the `kubernetes-agent` devcontainer -2. Wait for the Codespace to start and the kind cluster to initialize (~2-4 min) +2. Wait for the Codespace to start and the kind cluster to initialize (~15-20 min on first run) 3. Deploy the k8s-agent and demo workloads inside the Codespace via `gh codespace ssh` After setup, the Codespace URL and `gh codespace ssh` command are printed. @@ -58,7 +80,7 @@ Requires `kubectl` pointed at a running cluster. The script deploys the k8s-agen ## What you should see After the agent's first sync (~5 min), visit: -`https://app.getcortexapp.com/catalog/demo-kubernetes/k8s` +`https://app.getcortexapp.com/admin/resources?tag=demo-kubernetes` - `demo-deployment` (Deployment) - `demo-statefulset` (StatefulSet) From b0741fa8981c8d0520d36aa7ffcc4ca3b2e6c4dc Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 16:42:28 -0700 Subject: [PATCH 40/43] docs: add Next Steps section for rolling out to real clusters Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/README.md | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index 205e459b..c250051a 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -93,6 +93,36 @@ The setup script is idempotent — re-run `cortex solutions post-install -s kube For the Codespace path, the Codespace name is saved locally so re-runs reconnect to the same Codespace rather than creating a new one. +## Next steps + +Once you have verified the agent working against the demo cluster, roll it out to your real clusters: + +1. **Install the Helm chart into each cluster** you want Cortex to ingest. For each cluster, run: + + ```bash + helm upgrade --install cortex-k8s-agent \ + oci://ghcr.io/cortexapps/k8s-agent/helm/cortex-k8s-agent \ + --set app.apiKey= \ + --set app.clusterName= \ + --set app.baseUrl=https://api.getcortexapp.com + ``` + + Give each cluster a distinct `clusterName` — this is how Cortex identifies which cluster a workload belongs to. + +2. **Annotate your workloads** with your Cortex entity tag so the agent can map resources to catalog entities: + + ```yaml + metadata: + annotations: + cortex.io/service: my-service-tag + ``` + +3. **Wait for the first sync** (~5 min) then visit each entity's K8s tab in Cortex to confirm workload data is flowing. + +4. **Set up additional clusters** by repeating step 1 with a different `clusterName` for each environment (e.g., `prod-us-east`, `staging`, `dev`). + +See the [Cortex Kubernetes integration docs](https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes) for full configuration options including namespace filtering, custom resource types, and RBAC setup. + ## Temporary limitation The k8s-agent image is currently private on GHCR, requiring `GHCR_TOKEN`. This requirement will be removed once the image is made public. From e94379025412d3e05f3d63095b1992e1c64ac5a2 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Thu, 3 Sep 2026 19:30:12 -0700 Subject: [PATCH 41/43] docs: add troubleshooting note for K8s metadata label customization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When metadata labels are configured, Cortex bypasses cortex.io/tag annotation lookup entirely — the two strategies are mutually exclusive. Co-Authored-By: Claude Sonnet 4.6 --- cortexapps_cli/solutions/kubernetes-agent/README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index c250051a..ab197594 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -87,6 +87,15 @@ After the agent's first sync (~5 min), visit: - `demo-cronjob` (CronJob) - `demo-rollout` (Argo Rollout — containers resolved from `demo-deployment`) +## Troubleshooting + +**No K8s details showing on the entity page after the first sync** + +If you have configured [K8s metadata label customization](https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes#auto-mapping-customization) in your Cortex settings (Settings → Kubernetes → Metadata labels), Cortex uses *only* those labels for resource mapping and ignores the `cortex.io/tag` annotation used by the demo manifests. Either: + +- Remove the metadata label customization to use the default annotation-based mapping, or +- Add a matching label (e.g., `app: demo-kubernetes`) to the demo manifests + ## Re-running setup The setup script is idempotent — re-run `cortex solutions post-install -s kubernetes-agent` to retry any failed step. Completed steps are skipped. From dd27b241731f3c7279250cfc62026e4f6a3e3ff0 Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Mon, 7 Sep 2026 14:22:20 -0700 Subject: [PATCH 42/43] fix: replace Mermaid with ASCII art; rename sections for CLI menu - Data Model: ASCII art renders in the CLI 'What next?' menu - Architecture -> Data Model (matches _extract_first_codeblock path) - Next steps -> After Installing (matches _extract_section lookup) - Updated After Installing content per product guidance Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/README.md | 62 +++++++++---------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index ab197594..5c251a89 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -18,27 +18,34 @@ Demonstrates the [Cortex Kubernetes agent](https://docs.cortex.io/docs/reference After setup, visit your entity's K8s tab to see live workload data synced from the cluster. -## Architecture - -```mermaid -flowchart TD - subgraph CS["GitHub Codespace"] - subgraph kind["kind cluster — cortex-demo"] - K8S["Kubernetes API Server"] - subgraph workloads["demo workloads"] - DW["Deployment / StatefulSet / CronJob / Rollout"] - end - AGENT["cortex-k8s-agent pod"] - end - end - - CORTEX["Cortex Platform"] - - AGENT -->|"reads resources every 5 min"| K8S - AGENT -->|"pushes data via HTTPS"| CORTEX +## Data Model + +``` + GitHub Codespace + ┌──────────────────────────────────────────────────────────┐ + │ kind cluster (cortex-demo) │ + │ ┌────────────────────────────────────────────────────┐ │ + │ │ │ │ + │ │ ┌─────────────────┐ ┌──────────────────────┐ │ │ + │ │ │ Kubernetes API │◀──│ cortex-k8s-agent │ │ │ + │ │ │ Server │ │ (polls every 5 min) │ │ │ + │ │ └─────────────────┘ └──────────┬───────────┘ │ │ + │ │ │ HTTPS push │ │ + │ │ ┌──────────────────────────────┐ │ │ │ + │ │ │ demo workloads │ │ │ │ + │ │ │ Deployment · StatefulSet │ │ │ │ + │ │ │ CronJob · Argo Rollout │ │ │ │ + │ │ └──────────────────────────────┘ │ │ │ + │ └────────────────────────────────────────────────────┘ │ + └──────────────────────────────────┬───────────────────────┘ + ▼ + ┌─────────────────┐ + │ Cortex │ + │ Platform │ + └─────────────────┘ ``` -The agent runs inside the kind cluster as a Kubernetes Deployment. Every 5 minutes it queries the Kubernetes API for workload resources (Deployments, StatefulSets, CronJobs, Argo Rollouts, etc.) and pushes the collected metadata to the Cortex API over HTTPS. Cortex stores the data and surfaces it on each entity's K8s tab. +The agent runs inside the cluster as a Deployment. Every 5 minutes it queries the Kubernetes API for workload resources and pushes the metadata to Cortex over HTTPS. Cortex surfaces the data on each entity's K8s tab. ## Prerequisites @@ -102,11 +109,11 @@ The setup script is idempotent — re-run `cortex solutions post-install -s kube For the Codespace path, the Codespace name is saved locally so re-runs reconnect to the same Codespace rather than creating a new one. -## Next steps +## After Installing Once you have verified the agent working against the demo cluster, roll it out to your real clusters: -1. **Install the Helm chart into each cluster** you want Cortex to ingest. For each cluster, run: +1. **Install the Cortex Kubernetes agent Helm chart into each cluster** for which you want data associated with your Cortex entities. Give each cluster a distinct `clusterName` — this is how Cortex identifies which cluster a workload belongs to: ```bash helm upgrade --install cortex-k8s-agent \ @@ -116,20 +123,13 @@ Once you have verified the agent working against the demo cluster, roll it out t --set app.baseUrl=https://api.getcortexapp.com ``` - Give each cluster a distinct `clusterName` — this is how Cortex identifies which cluster a workload belongs to. +2. **Ensure your Kubernetes workloads have the appropriate annotation or label** so Cortex can map them to your catalog entities: -2. **Annotate your workloads** with your Cortex entity tag so the agent can map resources to catalog entities: - - ```yaml - metadata: - annotations: - cortex.io/service: my-service-tag - ``` + - **Annotation** (default): add `cortex.io/tag: ` to each workload's metadata + - **Label** (if you have configured K8s metadata label customization in Cortex Settings): add the configured label key with the entity tag as the value, e.g. `app: ` 3. **Wait for the first sync** (~5 min) then visit each entity's K8s tab in Cortex to confirm workload data is flowing. -4. **Set up additional clusters** by repeating step 1 with a different `clusterName` for each environment (e.g., `prod-us-east`, `staging`, `dev`). - See the [Cortex Kubernetes integration docs](https://docs.cortex.io/ingesting-data-into-cortex/integrations/kubernetes) for full configuration options including namespace filtering, custom resource types, and RBAC setup. ## Temporary limitation From de6f218f71e347e2da5a40a8c8b0fc56068ea04e Mon Sep 17 00:00:00 2001 From: Jeff Schnitter Date: Mon, 7 Sep 2026 14:56:31 -0700 Subject: [PATCH 43/43] docs: fix ASCII art spacing in kubernetes-agent data model diagram Co-Authored-By: Claude Sonnet 4.6 --- .../solutions/kubernetes-agent/README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/cortexapps_cli/solutions/kubernetes-agent/README.md b/cortexapps_cli/solutions/kubernetes-agent/README.md index 5c251a89..089cba8e 100644 --- a/cortexapps_cli/solutions/kubernetes-agent/README.md +++ b/cortexapps_cli/solutions/kubernetes-agent/README.md @@ -26,16 +26,16 @@ After setup, visit your entity's K8s tab to see live workload data synced from t │ kind cluster (cortex-demo) │ │ ┌────────────────────────────────────────────────────┐ │ │ │ │ │ - │ │ ┌─────────────────┐ ┌──────────────────────┐ │ │ - │ │ │ Kubernetes API │◀──│ cortex-k8s-agent │ │ │ - │ │ │ Server │ │ (polls every 5 min) │ │ │ - │ │ └─────────────────┘ └──────────┬───────────┘ │ │ + │ │ ┌─────────────────┐ ┌──────────────────────┐ │ │ + │ │ │ Kubernetes API │◀──│ cortex-k8s-agent │ │ │ + │ │ │ Server │ │ (polls every 5 min) │ │ │ + │ │ └─────────────────┘ └──────────┬───────────┘ │ │ │ │ │ HTTPS push │ │ - │ │ ┌──────────────────────────────┐ │ │ │ - │ │ │ demo workloads │ │ │ │ - │ │ │ Deployment · StatefulSet │ │ │ │ - │ │ │ CronJob · Argo Rollout │ │ │ │ - │ │ └──────────────────────────────┘ │ │ │ + │ │ ┌──────────────────────────────┐ │ │ │ + │ │ │ demo workloads │ │ │ │ + │ │ │ Deployment · StatefulSet │ │ │ │ + │ │ │ CronJob · Argo Rollout │ │ │ │ + │ │ └──────────────────────────────┘ │ │ │ │ └────────────────────────────────────────────────────┘ │ └──────────────────────────────────┬───────────────────────┘ ▼