docs: Wave 1 fixes (KI-059 to KI-207, KI-236 to KI-246)

platform/control-plane/catalogs/create-catalog.md

@@ -38,7 +38,7 @@ The catalog's page opens. Continue with [Catalog detail](./catalog-detail.md) to
 | **TPC-DS** | Generated benchmark data. | **Scale**. |
 | **Generic** | Whatever properties you write in **Advanced properties**. | The engines it is valid for (**trino** needs `connector.name`; **starrocks** needs `type`) and the property pairs. |
 
-The **Storage connection** list is the platform's [storage connections](../storage/index.md). **Password secret** lists [secrets](../secrets.md), so a credential is never typed into the catalog. The list shows every secret, including the ones egret creates for engines and catalog clients (`engine-internal-…`, `engine-monitor-…`, `engine-tls-…`, `catalog-client-…`); pick one of your own.
+The **Storage connection** list is the platform's [storage connections](../storage/index.md). **Password secret** lists [secrets](../secrets.md), so a credential is never typed into the catalog. The list leaves out the **System** secrets egret creates for engines and catalog clients (`engine-internal-…`, `engine-monitor-…`, `engine-tls-…`, `catalog-client-…`).
 
 ## Advanced properties
 

platform/control-plane/secrets.md

@@ -14,7 +14,15 @@ You need **secret:read** to open the page; creating, rotating and deleting need
 
 A strip counts the **secrets**.
 
-The list also shows the secrets egret creates and manages itself: `engine-internal-<engine id>-…` (the Trino internal-communication secret), `engine-monitor-…` (the egret-monitor password for the native Trino monitor), `engine-tls-…` (the coordinator keystore) for every provisioned engine, and `catalog-client-…` (a Catalog Client credential an engine presents to egret's catalog). Leave them alone: rotating or deleting one breaks the engine or client that uses it. On a 1440-pixel-wide screen the **Used by** and **Created** columns are cut off at the right; scroll the table sideways.
+The list also shows the secrets egret creates and manages itself: `engine-internal-<engine id>-…` (the Trino internal-communication secret), `engine-monitor-…` (the egret-monitor password for the native Trino monitor), `engine-tls-…` (the coordinator keystore) for every provisioned engine, the StarRocks root and Trino delegator passwords, and `catalog-client-…` (a Catalog Client credential an engine presents to egret's catalog). The page says *Secrets marked System are created by egret for engines and catalogs and are managed with them.*
+
+### System secrets
+
+A secret egret manages carries a **System** badge next to its name, in the list and on its page. Egret rotates and deletes it together with the engine or catalog that created it, so you cannot change it by hand:
+
+- **Edit**, **Rotate value** and **Delete** are disabled, with the reason *Managed by egret: it is rotated and deleted with the engine or catalog that created it.*
+- The API refuses a rename, a rotation or a delete with 409: *«name» is managed by egret: it is rotated and deleted with the engine or catalog that created it, not here*. The secret's `system_managed` field is `true`.
+- No secret picker offers it: the **Secret** fields of storage and database connections, the catalog **Password secret** pickers and the engine create and security forms leave it out.
 
 | Column | Meaning |
 |---|---|
@@ -67,7 +75,7 @@ Rotate when a credential changes, or on a schedule. Everything that references t
 
 ## Delete a secret
 
-Press **Delete** on the list or the secret page. A secret that is in use cannot be deleted: the **Used by** count must be zero. Detach it from every connection first.
+Press **Delete** on the list or the secret page. A secret that is in use cannot be deleted: the **Used by** count must be zero. Detach it from every connection first. A **System** secret cannot be deleted here at all ([System secrets](#system-secrets)).
 
 :::warning
 A secret is a platform-wide object. Rotating it changes the credential for every connection that references it.
@@ -83,6 +91,7 @@ A secret is a platform-wide object. Rotating it changes the credential for every
 
 - Values cannot be read back, only replaced.
 - Names are global across the platform.
+- Secrets that existed before the **System** flag was introduced are marked from the engines and catalogs that point at them, and from the name and description of a minted secret. A minted secret that nothing points at any more and whose name or description was changed by hand is not marked.
 
 ## Further reading
 

platform/control-plane/storage/index.md

@@ -61,9 +61,20 @@ The header shows the name, provider and the **BUCKET** and **ENDPOINT**, with **
 
 ## Browse a connection
 
-The **Browse** tab lists the bucket's folders and files, so you can confirm a connection works and see what egret stored: for example folders such as `apps`, `query-results`, `workflow` and the warehouse folders named by id. Select a folder to open it; files show their size. An empty folder says *This folder is empty.*
+The **Browse** tab lists the connection's folders and files, so you can confirm a connection works and see what egret stored: for example folders such as `apps`, `query-results`, `workflow` and the warehouse folders named by id. Select a folder to open it; files show their size and can be downloaded. An empty folder says *This folder is empty.*
 
-The demo bucket is shared by everything egret stores, so `query-results` holds the stored result files of every user's SQL Lab queries. Anyone with **storage:read** can list and download them, including the built-in `data_steward` role; these files are what each person was allowed to see, after their masks and row filters. Keep **storage:read** for administrators.
+Browse and download start at the connection's **Key prefix**, not at the bucket root. Keys are relative to that prefix, and the first crumb of the path shows `bucket/prefix`: a connection on bucket `warehouse` with Key prefix `sandbox` opens at `warehouse/sandbox`. A connection without a Key prefix opens at the bucket root and its crumb shows the bucket name.
+
+### Folders only a platform admin can open
+
+In egret's own bucket (the `[storage] bucket` of the server configuration), egret writes other users' data under three prefixes: stored SQL Lab results (`engine.results_prefix`, `query-results` by default), Data App sources (`apps.storage.prefix`) and workflow files (`workflow.storage.prefix`). A stored SQL Lab result is what one person was allowed to see, after their masks and row filters. These prefixes are closed to everyone who is not a platform admin, whatever storage permission they hold:
+
+- Browse leaves them out and says so under the list, for example *apps, query-results, workflow are not shown: egret stores its users' SQL Lab results, Data App sources and workflow files there. Only a platform admin can open them.*
+- Listing or downloading a key under them answers 403 with the reason: *… holds data egret writes for its users (stored SQL Lab results, Data App sources, workflow files), so it cannot be browsed or downloaded with storage:read; only a platform admin can open it*. The match ignores case and also applies after percent-decoding the key.
+
+A platform admin sees and downloads them as before.
+
+Keys that try to leave the folder are refused with 400 for everyone, platform admins included: a `.` or `..` segment, an empty segment (a leading `/`, a trailing `/` on a file, or `//`), a backslash or a control character, also once percent-decoded.
 
 ![The Browse tab listing the top-level folders of the warehouse bucket](/img/platform/admin/cp-storage-browse.png)
 
@@ -74,7 +85,7 @@ Press **Delete**. The dialog *Delete «\<connection\>»?* says *This storage con
 ## Limits and known issues
 
 - Only S3-compatible stores work; Google Cloud Storage and Azure Blob are not available yet.
-- The **Browse** tab lists the bucket root even for a connection with a **Key prefix** (checked with `sandbox-lake`, prefix `sandbox/`).
+- **storage:read** still lets a holder (for example the built-in `data_steward` role) download the raw Iceberg data files of every warehouse in the bucket, which bypasses column masks and row filters (KI-236). Keep **storage:read** for administrators.
 - A key prefix does not keep two egret warehouses apart: a warehouse on `sandbox-lake` was refused with *Storage profile overlaps with existing warehouse demo-warehouse*. Use a separate bucket for each warehouse.
 
 ## Further reading

platform/deploy-operate/authentication.md

@@ -61,7 +61,8 @@ The decoded access token for `tester` in this realm contains `"groups": ["data-t
 
 :::danger The realm file is for development only
 It disables TLS (`sslRequired: none`), accepts `*` as a redirect URI, enables the password grant, contains a client secret
-and three users with known passwords. Do not import it into a shared Keycloak. Create your own realm.
+and three users with known passwords. Do not import it into a shared Keycloak. Create your own realm. The Helm chart
+bundles this realm with `oidc.enabled=true` only when you also set `oidc.devRealm=true` (see [Kubernetes](./kubernetes)).
 :::
 
 ### Set up your own realm
@@ -114,7 +115,8 @@ misconfigured, or has not been connected yet.
 EGRET_ADMIN_EMAIL=ops@example.com EGRET_ADMIN_PASSWORD='a long random password' egret init
 ```
 
-- Without `EGRET_ADMIN_PASSWORD`, `init` generates a password and prints it once. Store it in a vault.
+- Without `EGRET_ADMIN_PASSWORD` (or with it empty), `init` generates a password and prints it once. Store it in a vault. A
+  password you supply is never printed: `init` prints `Password: taken from EGRET_ADMIN_PASSWORD (not shown)`.
 - `init` does nothing when a platform administrator already exists. `egret init --force` adds another one.
 - On the sign-in page the account is behind **Admin sign-in (break-glass)** when SSO is on.
 - Local sign-in is rate limited (`rate_limit.login_*`: 5 attempts per 5 minutes, successful attempts included), which also

platform/deploy-operate/configuration.md

@@ -35,7 +35,9 @@ does not: run `set -a; source .env; set +a` first. The server does not read `.en
 
 The server validates the result before it listens. A missing or malformed required value stops the process with one line
 naming the key; see [Startup checks](../reference/configuration-keys#startup-checks). Unknown keys are ignored, so a
-misspelled variable leaves the default in place without any message.
+misspelled variable leaves the default in place, but the server logs a `WARN` for each one at start, with the closest real
+key (`ignored environment variable EGRET_SERVER__PORTT: … (did you mean EGRET_SERVER__PORT?)`). The warning never stops the
+start.
 
 ## The keys you set first
 

platform/deploy-operate/kubernetes.md

@@ -40,7 +40,7 @@ Optional objects appear with these values:
 
 | When | Extra objects |
 |---|---|
-| `oidc.enabled=true` | `Deployment/keycloak`, `Service/keycloak`, `ConfigMap/keycloak-realm` (the realm file). |
+| `oidc.enabled=true` and `oidc.devRealm=true` | `Deployment/keycloak`, `Service/keycloak`, `ConfigMap/keycloak-realm` (the realm file). |
 | `ingress.enabled=true` | `Ingress/egret` (routes `/auth`, `/data-api`, `/oauth`, `/.well-known/*` and `/`) and `Ingress/egret-catalog`. `/service` (the policy API) is not routed. |
 | `sharedStore.enabled=true` | `Deployment/valkey` and `Service/valkey`. |
 | `workflow.containerRunner=kubernetes` | `Namespace/egret-tasks` (Pod Security `restricted`), `Role` and `RoleBinding` `egret-task-runner`, `NetworkPolicy/egret-task-isolation`. |
@@ -61,7 +61,8 @@ just deploy-kind     # builds egret-backend, -frontend, -renderer and -apps imag
 10m` and then restarts the pods, because a rebuilt `:dev` image with an unchanged tag is otherwise ignored. The web app is
 at [http://localhost:30080](http://localhost:30080); sign in as `admin@egret.local` with the password from `values.yaml` (`admin.password`).
 
-For single sign-on and an ingress, `just deploy-kind-oidc` adds `--set oidc.enabled=true --set ingress.enabled=true`. It
+For single sign-on and an ingress, `just deploy-kind-oidc` adds `--set oidc.enabled=true --set oidc.devRealm=true --set
+ingress.enabled=true` (`just deploy-kind-governance` sets the same two OIDC values). It
 needs the ingress controller and the DNS rewrite recipes (`just ingress-install`, `just coredns-rewrite`) first.
 
 Tear down with `just kind-undeploy` (the release only) or `just k8s-down` (the cluster).
@@ -128,7 +129,8 @@ with more than one replica and no shared store, so the guard holds outside the c
 | Value | Default | Meaning |
 |---|---|---|
 | `admin.email`, `admin.password` | `admin@egret.local`, `Admin12345!` | The break-glass administrator `egret init` creates. **Change the password.** |
-| `oidc.enabled` | `false` | Turn on SSO against the **bundled Keycloak** (`oidc.realm`, `clientId`, `clientSecret`, `keycloakImage`, `keycloakAdmin`, `keycloakAdminPassword`). The bundled Keycloak runs in development mode with default credentials. |
+| `oidc.enabled` | `false` | Turn on SSO against the **bundled Keycloak** (`oidc.realm`, `clientId`, `clientSecret`, `keycloakImage`, `keycloakAdmin`, `keycloakAdminPassword`). The bundled Keycloak runs in development mode with default credentials. Requires `oidc.devRealm=true`. |
+| `oidc.devRealm` | `false` | Confirms that you want the bundled Keycloak with the only realm the chart ships, the **development** realm (`files/egret-realm.json`: no TLS requirement, a `*` redirect URI, the password grant, users with published passwords, client secret `egret-secret`). With `oidc.enabled=true` and this left `false`, the chart refuses to render and names those settings. With it on, the install notes print a warning that anyone who reaches the Keycloak can sign in as a platform admin. Never set it on a cluster other people can reach. |
 
 To use your own identity provider, leave `oidc.enabled=false` (so no Keycloak is deployed) and set the `EGRET_OIDC__*`
 variables on the Deployment yourself, as in [Authentication setup](./authentication). The chart has no values for an

platform/deploy-operate/local.md

@@ -49,11 +49,11 @@ keys](../reference/configuration-keys); the ones that matter for the compose sta
 | Need | Variable | Value for the compose stack |
 |---|---|---|
 | Metadata database | `EGRET_DATABASE__URL` | `postgres://egret:egret@localhost:5432/egret` |
-| Object storage | `EGRET_STORAGE__ENDPOINT`, `__BUCKET`, `__ACCESS_KEY`, `__SECRET_KEY` | `http://localhost:9000`, a bucket you create, `egret`, `egretsecret` |
+| Object storage | `EGRET_STORAGE__ENDPOINT`, `__BUCKET`, `__ACCESS_KEY`, `__SECRET_KEY` | `http://localhost:9000`, `warehouse` (create it yourself), `egret`, `egretsecret` |
 | Single sign-on | `EGRET_OIDC__ISSUER_URL`, `__CLIENT_ID`, `__CLIENT_SECRET`, `__REDIRECT_URL` | `http://localhost:8081/realms/egret`, `egret`, `egret-secret`, `http://localhost:5173/api/auth/oidc/callback` |
 
-`.env.example` does not include the `EGRET_STORAGE__*` variables, and the server will not start without them; the Quick
-start shows the lines to add.
+`.env.example` sets all of these, and a test in the repository keeps it passing the server's startup checks. The bucket
+itself is not created for you; the Quick start shows how.
 
 The OIDC redirect URL must be on the origin the browser uses. With the Vite dev server that is port 5173, so the session
 cookie is set for the app and not for port 8080.
@@ -73,9 +73,9 @@ EGRET_STORAGE__ACCESS_KEY=egret EGRET_STORAGE__SECRET_KEY=egretsecret \
 ```
 
 If you also enable Data Apps (`apps.origin`), give each instance its own `apps.listen`: that listener binds
-`0.0.0.0:8081` by default. Changing the port does not change the catalog's advertised address: with the catalog on, also
-set `EGRET_CATALOG__PUBLIC_URL` to the new origin, or the token issuer in `/.well-known/openid-configuration` keeps
-saying `http://localhost:8080` (observed on an instance started on port 18080).
+`0.0.0.0:8081` by default. When `EGRET_CATALOG__PUBLIC_URL` is unset, the catalog's advertised address (and with it the token issuer in
+`/.well-known/openid-configuration`) follows the port: `http://localhost:18080` here. Set it yourself when clients reach the
+instance under another name.
 
 ## StarRocks and Ranger in the compose stack
 

platform/deploy-operate/security-hardening.md

@@ -30,8 +30,8 @@ that means. Each item says how to check it. Tick them in order.
 
 ## 2. The break-glass administrator
 
-- [ ] **Set the password yourself** with `EGRET_ADMIN_PASSWORD` at `egret init` (or let `init` generate one and store it in a
-  vault). Never keep the Helm default `Admin12345!` (`admin.password`).
+- [ ] **Set the password yourself** with `EGRET_ADMIN_PASSWORD` at `egret init`; `init` does not print a supplied password
+  (or let `init` generate one, printed once, and store it in a vault). Never keep the Helm default `Admin12345!` (`admin.password`).
 - [ ] **Move day-to-day administration to IdP groups.** Grant `platform_admin` to a group in **Control Plane → Groups**, and
   keep the local account for recovery.
 - [ ] **Keep the local sign-in rate limit.** Five attempts per five minutes (`rate_limit.login_*`) is what stands in front of
@@ -54,6 +54,8 @@ Egret speaks plain HTTP; terminate TLS in front of it.
   which is the safe direction.
 - [ ] **Use HTTPS to the identity provider**, and register only the exact redirect URI. The development realm uses
   `sslRequired: none` and `*` as a redirect URI.
+- [ ] **Do not set `oidc.devRealm=true` in the Helm chart** outside a cluster only you can reach. It is the opt-in that lets
+  `oidc.enabled=true` install the bundled Keycloak with the development realm; without it the chart refuses to render.
 - [ ] **Enable TLS on the engines.** Trino enforces JWT only over HTTPS (over HTTP it accepts a valid token but also trusts a
   bare `X-Trino-User`). StarRocks' per-user JWT login sends the token in clear text with `mysql_clear_password` unless the
   front end has TLS.

platform/deploy-operate/storage-and-catalog.md

@@ -19,6 +19,8 @@ The `[storage]` section is the object store the server itself uses. One bucket h
 | Data Apps drafts and versions | `apps.storage.prefix` | `apps/` |
 | The default catalog warehouse | `catalog.warehouse` (`s3://<bucket>`) | `s3://warehouse` |
 
+The first three prefixes hold other users' data, so **Control Plane → Storage → Browse** and its download close them to everyone except a platform admin ([Storage](../control-plane/storage/index.md#folders-only-a-platform-admin-can-open)).
+
 ```bash
 EGRET_STORAGE__ENDPOINT=http://localhost:9000   # omit for AWS S3
 EGRET_STORAGE__REGION=us-east-1
@@ -59,7 +61,7 @@ EGRET_CATALOG__PUBLIC_URL=http://egret.example.internal:8080
 | `catalog.enabled` | Off by default. When on, `warehouse`, `auth_token` and the storage credentials are required. The server logs lakekeeper's own migrations on first start, in the same database. |
 | `catalog.auth_token` | A static token for catalog administration. Treat it as a secret. |
 | `catalog.warehouse` | The default catalog's location. |
-| `catalog.public_url` | The address clients and engines use to reach this server, without `/catalog`. The default is `http://localhost:8080`, **whatever `server.port` is**: change it when the server is reachable under another name, port or scheme. It is also where the machine identity provider's issuer falls back to. |
+| `catalog.public_url` | The address clients and engines use to reach this server, without `/catalog`. Unset, it is `http://localhost:<server.port>`: change it when the server is reachable under another name or scheme. It is also where the machine identity provider's issuer falls back to. |
 | `catalog.system_user` | The stable name (`egret-system`) of the principal that bootstraps the catalog's default project. Keep it stable across deploys. |
 
 ### What an engine sees

platform/deploy-operate/troubleshooting/cannot-sign-in.md

@@ -15,7 +15,7 @@ when the server answers `401 {"error":"unauthorized"}`.
 **Causes**
 
 - The email or password is wrong. The break-glass account is the one `egret init` created; `.env.example` and the Helm chart
-  default it to `admin@egret.local`, but its password is the one `init` printed, or `EGRET_ADMIN_PASSWORD`, or the chart's
+  default it to `admin@egret.local`, but its password is the one `init` generated and printed, or the `EGRET_ADMIN_PASSWORD` it was given (never printed), or the chart's
   `admin.password`.
 - The account was never created: a fresh database has no administrator until `egret init` runs. In Kubernetes the
   `egret-init` hook job creates it after install; check `kubectl get job -n egret-system egret-init`.