diff --git a/platform/control-plane/access-control/access-explorer.md b/platform/control-plane/access-control/access-explorer.md index 0e0f6fb..bee8e5a 100644 --- a/platform/control-plane/access-control/access-explorer.md +++ b/platform/control-plane/access-control/access-explorer.md @@ -6,7 +6,7 @@ sidebar_position: 5 The **Access Explorer** answers "what can this person reach?" and "who can reach this object?" with full provenance: for every line it says whether the access is direct, comes through a group, or comes through workspace membership. Use it before you grant access, and when someone reports they can see (or cannot see) something. -![The Access Explorer for member-sso: workspaces and shared objects, each with how the access arises](/img/platform/admin/cp-explorer-user.png) +![The Access Explorer for alice.nguyen@egret.local: workspaces and shared objects, each with how the access arises](/img/platform/admin/cp-explorer-user.png) You need **role:read** to open the page. Two tabs switch the direction. @@ -23,7 +23,7 @@ The result has two sections: | **Workspaces** | Each workspace the principal belongs to, with the workspace role and where it comes from. | | **Shared objects** | Each object shared with the principal: the kind (**dashboard**, **folder**, and so on), the name, the workspace, the role, the **allow** or deny effect and the source. | -The source reads **direct** for access given to the person, or **via \** when it comes through a group. For example, member-sso holds `ws_viewer` on *Docs sandbox* directly and `ws_editor` through the group `data-team`, and shows the shared dashboard *Finance KPIs* via `data-team`. +The source reads **direct** for access given to the person, or **via \** when it comes through a group. For example, alice.nguyen@egret.local holds `ws_admin` on *Docs sandbox* and *Finance analytics* directly, `ws_editor` on *Finance analytics* and `finance_reporter` on *Docs sandbox* via `finance-analysts`, and sees an app shared with that group. ## Object → who can access @@ -33,7 +33,7 @@ The source reads **direct** for access given to the person, or **via \** ![Who can access the Finance KPIs dashboard, with the source of each line](/img/platform/admin/cp-explorer-object.png) -The result lists every principal that can reach the object. The header shows the kind, name and workspace. A filter, *All sources (5)*, narrows by source. The columns are: +The result lists every principal that can reach the object. The header shows the kind, name and workspace. A filter, *All sources (10)* for *Finance KPIs* on the demo stack, narrows by source. The columns are: | Column | Meaning | |---|---| diff --git a/platform/control-plane/access-control/groups.md b/platform/control-plane/access-control/groups.md index b9f8c26..763a115 100644 --- a/platform/control-plane/access-control/groups.md +++ b/platform/control-plane/access-control/groups.md @@ -6,7 +6,7 @@ sidebar_position: 2 A **group** is a set of users you can give roles and grants to once instead of person by person. **Control Plane → Identity → Groups** lists local groups you create in egret and groups synced from your identity provider. -![The Groups list showing two groups synced from OIDC](/img/platform/admin/cp-groups.png) +![The Groups list: local groups and groups synced from OIDC](/img/platform/admin/cp-groups.png) You need **group:read** to open the page. @@ -34,12 +34,12 @@ The dialog says *Create a local group.* After creating it, open the group to add ## The group page -![A synced group with members and roles](/img/platform/admin/cp-group-detail.png) +![The local group finance-analysts: details, three members and no roles](/img/platform/admin/cp-group-detail.png) | Section | What it holds | |---|---| | **Details** | The **Name** (editable for local groups). | -| **Members** | *Members are synced from your identity provider* for a synced group, with **Member**, **Email** and a remove action for local groups. | +| **Members** | For a local group: *Users in this group. Search to add someone, or remove with ×.*, a **User** search with **Add**, and the **Member** and **Email** of each member with a remove button. For a synced group: *Members are synced from your identity provider.* | | **Roles** | *Roles granted to every member of this group.* Choose **Scope** (**Platform** or **Workspace**) and **Role**, then **Add**. | When a group holds a role, every member gets it. In the platform a role held through a group works as expected. In the query engine, however, a group name never reaches Trino or StarRocks: engines count only roles a user holds directly, which the **Applied on engines** column of [People & groups](../../data-governance/people-and-groups/index.md) shows as *egret only — not applied on engines: held through a group*. diff --git a/platform/control-plane/access-control/index.md b/platform/control-plane/access-control/index.md index 450296a..271c58d 100644 --- a/platform/control-plane/access-control/index.md +++ b/platform/control-plane/access-control/index.md @@ -28,18 +28,18 @@ Access overview needs the **user:read** permission, which platform admins hold. | **USERS** | Total accounts, with how many are active. | | **GROUPS** | Total groups, split into *local* and *synced* (from the identity provider). | | **ROLES** | Total roles, split into platform and workspace roles. | -| **PLATFORM ADMINS** | Users holding the `platform_admin` role, and their share of all users (for example *25% of users — review*). | +| **PLATFORM ADMINS** | Users holding the `platform_admin` role, and their share of all users (for example *15% of users — review*). | ### Needs attention Findings about the way access is set up. Each has a title, a sentence and a **Review** button. Examples on the demo instance: -- **5 roles have no assignees**: *They grant nothing until assigned to a user or group.* -- **1 platform admins · 25% of users**: *Review who holds full-platform access and trim if it's too broad.* +- **6 roles have no assignees**: *They grant nothing until assigned to a user or group.* Roles that are only used as workspace roles count here. +- **2 platform admins · 15% of users**: *Review who holds full-platform access and trim if it's too broad.* ### Recent changes -The latest audit entries with the action, resource, the actor, and the time. **View audit log** opens the full [Audit log](../audit-log.md). +The latest audit entries with the action (for example `perm_grant:create`), the resource type, the actor and the time. The actor is shown as a short user id (`by 8a66d837`), not a name. **View audit log** opens the full [Audit log](../audit-log.md). ### How access works diff --git a/platform/control-plane/access-control/roles.md b/platform/control-plane/access-control/roles.md index 7d794ac..0903e1c 100644 --- a/platform/control-plane/access-control/roles.md +++ b/platform/control-plane/access-control/roles.md @@ -33,7 +33,7 @@ A strip counts **roles**, **platform** and **workspace**. **Scope** filters **Al | **Permissions** | How many permissions the role grants, including inherited ones. | | **Assignees** | How many users or groups hold it. | -![The tree view groups roles by scope](/img/platform/admin/cp-roles-tree.png) +![The tree view groups roles by scope and shows inheritance](/img/platform/admin/cp-roles-tree.png) ## Create a custom role @@ -50,7 +50,7 @@ The new role has no permissions. Open it to add some. Select a role to open its page. The header says *Workspace role · 8 permissions · 0 assignees* and offers **Delete role**. A role with assignees or child roles shows who holds it; with none it says *No assignees or inheriting roles.* -![A custom workspace role inheriting from ws_viewer with two direct permissions](/img/platform/admin/cp-role-detail.png) +![The custom workspace role finance_reporter: inherits from ws_viewer and adds two permissions](/img/platform/admin/cp-role-detail.png) ### Details diff --git a/platform/control-plane/access-control/users.md b/platform/control-plane/access-control/users.md index 756bec3..6210ecc 100644 --- a/platform/control-plane/access-control/users.md +++ b/platform/control-plane/access-control/users.md @@ -27,6 +27,8 @@ A summary strip counts **users**, **active** and **disabled**. Two filters narro | **Roles** | Platform roles the user holds, or a dash. | | **Groups** | How many groups the user belongs to. | +The **⋯** menu on a row offers **View access** (the [Access Explorer](./access-explorer.md) for that person), **Assign roles**, **Edit**, **Impersonate** and **Delete**. + ## Create a local account Use this for break-glass accounts and for installations without SSO. @@ -45,13 +47,13 @@ Prefer SSO for people. A local account has its own password and does not follow **Bulk select** adds checkboxes. After you tick users, a bar shows *N selected* with **Disable**, **Enable**, **Assign role…** and **Clear**. **Done** leaves bulk mode. -![Bulk select with one user ticked](/img/platform/admin/cp-users-bulk.png) +![Bulk select with two users ticked and the actions bar](/img/platform/admin/cp-users-bulk.png) ## The user page -Select a user to open their page. The header shows the avatar, name, status, provider and email, with **Disable** (or **Enable**) and **Delete**. +Select a user to open their page. The header shows the avatar, name, status, provider and email, with **Disable** (or **Enable**) and **Delete**. A local account also has **Reset password**: *Set a new password for this local account. Their existing sessions are signed out immediately.* -![A user page with details, roles and groups](/img/platform/admin/cp-user-detail.png) +![The page of Farid Haddad: details, two platform roles and one group](/img/platform/admin/cp-user-detail.png) ### Details @@ -73,7 +75,7 @@ The table lists **Role** and **Scope** with a remove button. With none it reads ## Disable and delete -**Disable** blocks sign-in without removing anything; the status becomes **disabled** and **Enable** reverses it. **Delete** removes the account. Prefer disabling: grants and shares that name a deleted user become unmatched names (see [People & groups](../../data-governance/people-and-groups/index.md)). +**Disable** blocks sign-in without removing anything and signs the user out; it applies at once, without a confirmation. The status becomes **disabled** and **Enable** reverses it. **Delete** asks *Delete "\"?* (*This user, their sessions, group memberships, and role grants will be permanently removed. This cannot be undone.*) and needs you to type `delete`. Prefer disabling: grants and shares that name a deleted user become unmatched names (see [People & groups](../../data-governance/people-and-groups/index.md)). ## Limits and known issues diff --git a/platform/control-plane/audit-log.md b/platform/control-plane/audit-log.md index 1ffec8d..a94084f 100644 --- a/platform/control-plane/audit-log.md +++ b/platform/control-plane/audit-log.md @@ -30,7 +30,7 @@ Every page loads 100 rows. **Load more** fetches the next 100 using a keyset cur | **Decision** | An authorization check: the actor asked to do something and the policy engine answered. | **allow** or **deny**, with the grants that decided. | `secret:read` on `platform`, **allow**, `grant-0` | | **Mutation** | A change that was made. | A dash. | `member:add`, `object_grant:put`, `perm_rule:create`, `tag:assign` | -Decisions are recorded for the platform's role-based checks. Governance changes also appear on the Governance overview's **Recent changes**; the audit log keeps the raw entries. +Decisions are recorded for the platform's role-based checks. Impersonation is recorded as the mutations `impersonation.start` (with the reason) and `impersonation.end`, with the admin as the actor; the decisions made while impersonating are recorded under the impersonated user, with nothing that names the admin. Governance changes also appear on the Governance overview's **Recent changes**; the audit log keeps the raw entries. ## Filter the log diff --git a/platform/control-plane/catalogs/catalog-detail.md b/platform/control-plane/catalogs/catalog-detail.md index 887e0f7..ea1ec7c 100644 --- a/platform/control-plane/catalogs/catalog-detail.md +++ b/platform/control-plane/catalogs/catalog-detail.md @@ -13,25 +13,25 @@ Select a catalog on **Control Plane → Catalogs** to open its page. The header ## Configuration -![The Configuration tab of the TPC-H catalog tpch_bench](/img/platform/admin/cp-catalog-config.png) +![The Configuration tab of the TPC-H catalog tpch_bench: Generated data with the Scale field, and the description](/img/platform/admin/cp-catalog-config.png) -*What every attached engine is told about this catalog. Saving re-applies it to each of them.* **Edit configuration** opens the same fields as the create form. Below them the tab shows the **Description** and **Stored configuration (JSON)**. +*What every attached engine is told about this catalog. Saving re-applies it to each of them.* **Edit configuration** opens the same fields as the create form. Below them the tab shows the result of egret's own checks (*Nothing here fails the rules this build checks — but it checks only the rules it knows, and egret validates the whole configuration when you save.*), the **Description** and **Stored configuration (JSON)**. For an egret warehouse the tab says *egret manages this catalog's configuration with its warehouse — the catalog is the warehouse, and its storage is the warehouse's.* There is nothing to edit. ## Engines -![The Engines tab after attaching tpch_bench to docs-trino](/img/platform/admin/cp-catalog-engines.png) +![The Engines tab of tpch_bench: applied on the provisioned docs-trino, declared on the external trino-demo-ext](/img/platform/admin/cp-catalog-engines.png) *Attaching a catalog makes it a catalog on that engine. Trino applies it to the cluster and restarts the coordinator; StarRocks creates it on the FE in place.* ### Attach a catalog to an engine 1. Under **Engine to attach**, choose an engine (*Select an engine*). -2. Optionally press **Test** to check the catalog first. A catalog with no network endpoint egret can dial, such as TPC-H, says *No test ran: egret had neither an engine that could run the query nor a host it could dial* and suggests attaching it to a Trino engine to test it. +2. Optionally press **Test** to check the catalog first. For a catalog with a host, egret only checks that it can open a TCP connection, for example *Reachability check — egret could not open a TCP connection to finance-db.internal:5432. Even a connection that had succeeded would not verify credentials, TLS or permissions.* Once the catalog is attached to a provisioned Trino engine, **Test** runs a query there instead. 3. Press **Attach**. -The table lists the engine, its type, whether it is **Primary**, and the **Apply** and **Applied** states, with **Retry** and **Detach**. An external engine shows the state *declared*: egret recorded that the engine serves the catalog and applied nothing. A provisioned engine shows whether the apply succeeded. +The table lists the engine with its status badge, its type, whether it is **Primary**, and the **Apply** and **Applied** states, with **Retry** and **Detach**. An external engine shows the state *declared*: egret recorded that the engine serves the catalog and applied nothing. A provisioned engine shows whether the apply succeeded (*applied*). :::note **Apply** is this catalog's state on that engine, not the engine's own health. An engine that has since failed still reads applied here, because the catalog did apply before it broke. @@ -43,7 +43,7 @@ With none attached the tab says *This catalog is not attached to any engine, so ## Workspaces -![The Workspaces tab: a checkbox per workspace](/img/platform/admin/cp-catalog-workspaces.png) +![The Workspaces tab of demo_warehouse: Default is assigned](/img/platform/admin/cp-catalog-workspaces.png) Tick the workspaces that should see the catalog and press **Save assignments** (**Reset** discards changes). A toast says *Workspace assignments saved*. @@ -55,42 +55,42 @@ Assigning a catalog to a workspace also writes a grant in the data-plane model: ## Roles & grants (warehouse) -![Catalog roles and their grants](/img/platform/admin/cp-catalog-roles.png) +![The catalog role engine, created with the engine, with its catalog-wide grant and one client](/img/platform/admin/cp-catalog-roles.png) *Catalog roles, keyed by consumer type: engine, notebook or external. Their grants gate reads and writes on the warehouse.* | Control | Does | |---|---| -| **Add role** | A **Name**, a **Consumer type** (**engine**, **notebook**, **external**) and a **Description**. | -| **Add grant** | On a role: a **Namespace** (or **All namespaces**), a **Table** (or **All tables**) and a **Privilege**, **read** or **write**. *A write/manage privilege permits writes; any grant permits reads.* | +| **Add role** | *A catalog role for a consumer type.* A **Name**, a **Consumer type** (**engine**, **notebook**, **external**) and a **Description**. | +| **Add grant** | On a role: a **Namespace** (or **All namespaces**, or one of the warehouse's namespaces such as `analytics`), a **Table** (or **All tables**) and a **Privilege**, **read** or **write**. *A write/manage privilege permits writes; any grant permits reads.* | -The table shows each role's **Consumer**, **Grants** (for example `catalog · write` or `demo_ns.orders · read`) and the number of **Clients**. +The table shows each role's **Consumer**, **Grants** (for example `catalog · manage` or `analytics.orders · read`) and the number of **Clients**. Provisioning an engine creates the role `engine` (*Auto-created role for query engines*) with a catalog-wide `manage` grant, and a client for the engine. ## Catalog Clients (warehouse) *Machine credentials (client id + secret) an engine or notebook uses to authenticate.* -1. Press **Add client**. -2. Type a **Name**, choose the **Role** (for example *demo-engine (engine)*) and the **Workspace**. +1. Press **Add client**. The dialog says *Issue machine credentials bound to a catalog role.* +2. Type a **Name**, choose the **Role** (for example *engine (engine)*) and the **Workspace**. 3. Press **Create client**. Note the client id and secret that egret shows; the list afterwards shows only the **Client ID**. -The list shows **Name**, **Consumer**, **Role**, **Client ID** and **Revoke**. +The list shows **Name**, **Consumer**, **Role**, **Client ID** and **Revoke**. A provisioned engine has its own client, for example `engine-docs-trino` with the role `engine`; revoking it cuts the engine off from the warehouse. ## Browse (warehouse) -![Browsing the analytics namespace and the sessions table](/img/platform/admin/cp-catalog-browse.png) +![Browsing the analytics namespace of demo_warehouse: the sessions table with its columns](/img/platform/admin/cp-catalog-browse.png) A tree of **Namespaces** (with a count) and a **Recycle bin** for dropped tables, with a search field (*Search tables & views…*). Select a table to see its **Format** version, **Columns**, **Snapshots**, **Updated** time and **Location**, with the tabs **Schema**, **Properties** and **Snapshots**. Each table and column has a tag control, which assigns a [classification tag](../../data-governance/classifications/tagging.md). ## Connect (warehouse) -![The Connect tab with the REST catalog URI and the PyIceberg snippet](/img/platform/admin/cp-catalog-connect.png) +![The Connect tab of demo_warehouse with the REST catalog URI and the PyIceberg snippet](/img/platform/admin/cp-catalog-connect.png) -*egret exposes this warehouse as an Iceberg REST catalog. Point any Iceberg-compatible engine at it using a Catalog Client's credentials.* The tab shows the **CATALOG URI**, **WAREHOUSE** and **OAUTH TOKEN URL**, and ready-made snippets for **PyIceberg**, **Trino** and **StarRocks**. Replace the `:` placeholders with a client's values. If the engine runs in Docker, use the egret host reachable from that container instead of `localhost`. The snippets ask egret to vend short-lived, table-scoped S3 credentials, so no static object-store keys are needed. +*egret exposes this warehouse as an Iceberg REST catalog. Point any Iceberg-compatible engine at it using a Catalog Client's credentials.* The tab shows the **CATALOG URI**, **WAREHOUSE** and **OAUTH TOKEN URL**, and ready-made snippets for **PyIceberg**, **Trino** and **StarRocks**. Replace the `:` placeholders with a client's values, and the example table `demo_ns.events` with one of yours. If the engine runs in Docker, use the egret host reachable from that container instead of the public address. On the demo stack the snippet's `s3.endpoint` is `http://rustfs:9000`, an address that resolves only inside the Docker network. The snippets ask egret to vend short-lived, table-scoped S3 credentials, so no static object-store keys are needed. ## Settings (warehouse) {#settings-warehouse} -![The warehouse settings](/img/platform/admin/cp-catalog-settings.png) +![The Settings tab of demo_warehouse: status, retention, format version, storage connection and two scheduled background tasks](/img/platform/admin/cp-catalog-settings.png) **Edit** changes the editable settings. @@ -101,11 +101,11 @@ A tree of **Namespaces** (with a count) and a **Recycle bin** for dropped tables | **Format version** | Which Iceberg table format versions clients may create. Default *Allow v1 and v2*. | | **Storage connection** | The connection this warehouse's data lives on. Its S3 key is managed there. | -**Background tasks** lists the catalog's garbage-collection queues (tabular expiry, purge and task-log cleanup) that reclaim storage from dropped tables, or *No pending tasks.* +**Background tasks** lists the catalog's garbage-collection queues (tabular expiry, purge and task-log cleanup) that reclaim storage from dropped tables, with the **Queue**, **Entity** and **Status** (for example `tabular_expiration` of a dropped table, **SCHEDULED**). ## Limits and known issues -- A TPC-H catalog cannot be tested before it is attached. +- A catalog test before attaching checks only that the host answers on its port; a TPC-H catalog has no host to check. - Attach and detach on Trino restart the coordinator. ## Further reading diff --git a/platform/control-plane/catalogs/create-catalog.md b/platform/control-plane/catalogs/create-catalog.md index ef9d46b..088017a 100644 --- a/platform/control-plane/catalogs/create-catalog.md +++ b/platform/control-plane/catalogs/create-catalog.md @@ -6,7 +6,7 @@ sidebar_position: 1 Press **New catalog** on **Control Plane → Catalogs** to register a catalog. You choose what the catalog reads, name it, and fill the fields that kind needs. The catalog is stored in the registry; attach it to an engine afterwards. -![The New catalog form for the Iceberg kind](/img/platform/admin/cp-catalog-new.png) +![The New catalog form for the TPC-H kind, with the name tpch_large](/img/platform/admin/cp-catalog-new.png) You need **catalog_registry:create**. A name is required: the button stays disabled and the form says *A name is required.* @@ -32,13 +32,13 @@ The catalog's page opens. Continue with [Catalog detail](./catalog-detail.md) to | **Hive** | Hive tables. Both the metastore and storage are required. | **Metastore** (**Hive Metastore** or **AWS Glue**), **Metastore URI**, **Storage connection**. | | **Delta Lake** | Delta tables. Both required. | **Metastore** (**Hive Metastore** or **AWS Glue**), **Metastore URI**, **Storage connection**. | | **Hudi** | Hudi tables. Both required. | **Metastore** (**Hive Metastore**), **Metastore URI**, **Storage connection**. | -| **JDBC** | A relational database: PostgreSQL, MySQL, MariaDB, Oracle, SQL Server or ClickHouse. | **Dialect** (`postgresql`, `mysql`, `mariadb`, `oracle`, `sqlserver`, `clickhouse`), **Host**, **Port**, **User**, **Password secret**, **TLS** (`disable`, `require`, `verify_full`; default `require`). | +| **JDBC** | A relational database: PostgreSQL, MySQL, MariaDB, Oracle, SQL Server or ClickHouse. | **Dialect** (`postgresql`, `mysql`, `mariadb`, `oracle`, `sqlserver`, `clickhouse`), **Host**, **Port**, **Database** (shown for the dialects that need one, such as `postgresql`; required there), **User**, **Password secret**, **TLS** (`disable`, `require`, `verify_full`; default `require`). | | **Elasticsearch** | Indices exposed as tables. The engines connect directly to the hosts. | **Hosts**, **Port**, **TLS**, **User**, **Password secret**. | | **TPC-H** | Generated benchmark data. No external system. | **Scale**. | | **TPC-DS** | Generated benchmark data. | **Scale**. | | **Generic** | Whatever properties you write in **Advanced properties**. | The engines it is valid for (**trino** needs `connector.name`; **starrocks** needs `type`) and the property pairs. | -The **Storage connection** list is the platform's [storage connections](../storage/index.md). **Password secret** lists [secrets](../secrets.md), so a credential is never typed into the catalog. +The **Storage connection** list is the platform's [storage connections](../storage/index.md). **Password secret** lists [secrets](../secrets.md), so a credential is never typed into the catalog. The list shows every secret, including the ones egret creates for engines and catalog clients (`engine-internal-…`, `engine-monitor-…`, `engine-tls-…`, `catalog-client-…`); pick one of your own. ## Advanced properties @@ -54,10 +54,19 @@ For **Generic** catalogs, Advanced is the whole configuration, per engine: for T The catalog page opens with its **Configuration** tab. Egret says *Nothing here fails the rules this build checks — but it checks only the rules it knows, and egret validates the whole configuration when you save.* +## Example: a JDBC catalog for a Postgres replica + +1. **New catalog**, **Kind** = **JDBC**. +2. **Name** `finance_pg`, **Description** *Finance Postgres read replica*. +3. **Dialect** `postgresql`, **Host** `finance-db.internal`, **Port** `5432`, **Database** `finance`, **User** `finance_ro`, **Password secret** `finance-postgres`. Leave **TLS** at `require`. +4. Press **Create catalog**. **Create catalog** stays disabled until the database is filled in. + +The catalog page opens. Before you attach it, **Engines → Test** only checks that egret can open a TCP connection to the host; on the demo stack, where the host does not exist, it says *Could not reach finance-db.internal:5432: failed to lookup address information: Name or service not known.* + ## Limits and known issues - The form checks the rules egret knows; the engine validates the rest when the catalog is applied. -- Creating an egret warehouse needs a workspace and a storage connection that egret can reach. +- Creating an egret warehouse needs a workspace and a storage connection that egret can reach. Two warehouses cannot share a storage location: on the demo stack a second warehouse on the seeded bucket was refused with *Storage profile overlaps with existing warehouse demo-warehouse*, also with another key prefix. ## Further reading diff --git a/platform/control-plane/catalogs/index.md b/platform/control-plane/catalogs/index.md index ae3dea2..6bec0ca 100644 --- a/platform/control-plane/catalogs/index.md +++ b/platform/control-plane/catalogs/index.md @@ -6,7 +6,7 @@ sidebar_position: 5 A **catalog** is a set of tables an engine can query. **Control Plane → Infrastructure → Catalogs** is the registry of every catalog the platform knows about: Iceberg warehouses that egret manages itself, and external sources such as a Hive metastore, a JDBC database or generated TPC-H data. From here you attach a catalog to an engine so it can be queried, and assign it to workspaces so they can see it. -![The Catalogs list with an egret warehouse and a TPC-H catalog](/img/platform/admin/cp-catalogs.png) +![The Catalogs list: the egret warehouse demo_warehouse, the JDBC catalog finance_pg and the TPC-H catalog tpch_bench](/img/platform/admin/cp-catalogs.png) You need **catalog_registry:read** to open the page. Creating, editing, attaching and assigning need the matching `catalog_registry` permissions (see the [permissions catalog](../access-control/permissions.md)). @@ -18,8 +18,8 @@ A strip counts **catalogs**, how many are **attached to an engine** and how many |---|---| | **Name** | The catalog name, as it is addressed in SQL on every engine. | | **Kind** | `egret warehouse`, `iceberg`, `hive`, `delta`, `hudi`, `jdbc`, `elasticsearch`, `tpch`, `tpcds` or `generic`. | -| **Engines** | How many engines it is attached to. | -| **Workspaces** | The workspaces it is assigned to, or a dash. | +| **Engines** | How many engines it is attached to, and their names. | +| **Workspaces** | How many workspaces it is assigned to, and their names. | | **Status** | Shows a state when the catalog needs attention, for example an attachment that could not be applied; a dash otherwise. | | Actions | **Delete**. | @@ -35,7 +35,7 @@ The Governance area uses step 2: grants and rules on a catalog reach only the po ## Delete a catalog -Press **Delete** on a row and confirm. Check the **Engines** and **Workspaces** columns first: a catalog that is attached or assigned is in use. For an egret warehouse the **Settings** tab holds a soft-delete retention (see [Catalog detail](./catalog-detail.md#settings-warehouse)). +Press **Delete** on a row. The dialog *Delete «\»?* asks you to type `delete`. A catalog that is still attached to an engine or assigned to a workspace is refused: the dialog says *The server refuses a catalog that anything still uses, so this delete cannot go ahead yet. Detach it from every engine and unassign it from every workspace first.* and lists what it still has, for example *Attached to engine trino-demo-ext*. For an egret warehouse the **Settings** tab holds a soft-delete retention (see [Catalog detail](./catalog-detail.md#settings-warehouse)). ## Egret warehouses and external catalogs diff --git a/platform/control-plane/query-engines/engine-detail.md b/platform/control-plane/query-engines/engine-detail.md index 83ae839..0eeff89 100644 --- a/platform/control-plane/query-engines/engine-detail.md +++ b/platform/control-plane/query-engines/engine-detail.md @@ -4,11 +4,95 @@ sidebar_position: 3 # Engine detail -Select **Details** on an engine in **Control Plane → Engines** to open its page. The tabs depend on whether egret runs the engine (a provisioned engine) or you do (an external engine). +Select an engine in **Control Plane → Engines** to open its page. The tabs depend on whether egret runs the engine (a provisioned engine) or you do (an external engine). + +## Provisioned engine + +![The Overview tab of the provisioned engine docs-trino: phase running, one pod, two catalogs, one workspace](/img/platform/admin/cp-engine-overview.png) + +The header shows the engine name, its namespace (`egret-engines`), the image, the coordinator address, the authentication (*JWT auth*) and the Ranger plugin version, with the type and status badges. **Open in Engine & policy** opens the engine in Governance, **Restart** rolls the engine, and the **More engine actions** menu (**⋯**) holds **Delete engine**. The tabs are **Overview**, **Catalogs**, **Workspaces**, **Settings** and **Monitor**. + +### Overview + +A strip shows the **phase**, **pods** (ready/total), **replicas** (for example *Coord 1 · Worker 0*), **catalogs**, **workspaces** and **auth** (*JWT auth*, impersonation, password auth or no delegation). For StarRocks, **FE replicas** and **CN replicas** replace *replicas*. + +**Status** lists each pod with its **Phase** and **Ready** state and the **Logs** and **Exec** actions. With the Docker provisioner a "pod" is a container, for example `egret-engine-egret-engines-docs-trino-coordinator`. + +If the engine cannot start the page shows *The engine failed to start.* with the reason. While starting it reads *Starting up — the engine isn't reachable yet.* + +### Catalogs + +![The Catalogs tab: demo_warehouse is primary and applied, tpch_bench is applied](/img/platform/admin/cp-engine-catalogs.png) + +*Data sources this engine can query. Attaching or detaching a catalog applies it to the cluster and restarts the Trino coordinator, so queries in flight are lost.* Each catalog shows its **Kind**, **Primary**, the **Apply** state and when it was **Applied**, with **Test** (runs `SHOW SCHEMAS` on the engine) and **Detach**. The primary catalog cannot be detached: *demo_warehouse is this engine's primary catalog. Detach it by deleting the engine.* + +To attach another, choose it in **Catalog to attach** and press **Attach**. With nothing left the form says *Every catalog egret can show you is already on this engine.* + +*Apply is this catalog's state on this engine, not a check that it still works*: a catalog that reached the engine before the engine broke is still recorded as **applied**. + +### Workspaces + +![The Workspaces tab with Docs sandbox assigned](/img/platform/admin/cp-engine-workspaces.png) + +*Workspaces assigned to this engine can query it with per-user delegation.* + +1. Press **Add workspace**. The dialog says *Assign this engine to an active workspace.* and lists the workspaces not yet assigned. +2. Select one. A toast says *Workspace assigned*. + +The engine then appears in that workspace's SQL Lab as a database named after the engine (`docs-trino (trino)`) and under **Workspace settings → Engines → Provisioned by egret**. **Remove** unassigns it. **Add workspace** is disabled until **JWT authentication** is on in the engine's **Security & Exposure** settings. + +### Settings + +![The Settings tab with the configuration sections; Worker and Security & Exposure carry values](/img/platform/admin/cp-engine-settings.png) + +*Cluster-wide, coordinator, and worker Trino config plus security and exposure. Saving re-provisions the engine.* The sections are the ones of the create form (see [Provision an engine](./provision.md#the-configuration-sections-trino)). + +1. Change a value, for example **Server → config.properties → Add property** `query.max-run-time` = `2h`. +2. Press **Review changes** to see the diff. The dialog says, for example, *2 fields will change.* with each field's old and new value, and *Saving re-provisions (rolls) the engine pods.* It only shows the diff; close it. +3. Press **Save changes**. There is no further confirmation. + +On the demo stack the save recreated the coordinator container within a minute while the header kept showing **running**, and the new value was in effect: `SHOW SESSION LIKE 'query_max_run_time'` reported the default `2.00h`. + +### Monitor + +![The Monitor tab on the demo stack: Engine not reachable, and no live queries](/img/platform/admin/cp-engine-monitor.png) + +The tab has a strip with the **running**, **queued** and **blocked** query counts and the **nodes**, and the **Queries** table (*Live queries on the coordinator, refreshed every 3s*) with **Query**, **User**, **State**, **SQL**, **Progress**, **Elapsed**, **Memory** and **CPU**, a state filter (**All states**) and **Search user or SQL…**. + +On Trino 483 engines the tab shows *Engine not reachable — is a coordinator running?* and *Coordinator unreachable — no live queries.* even while the engine is running and answering queries. This is a product issue, not a Docker limitation: the cluster summary asks Trino for `GET /v1/node`, which Trino 483 answers with 404, and an engine with JWT delegation gives egret's monitor account no sign-in. + +### Restart + +**Restart** asks *Restart engine?* (*This rolls the coordinator and worker pods; running queries are interrupted.*). On the demo stack the status went to **provisioning** and back to **running** in 38 seconds, and the container was recreated. + +### Delete + +![The Delete dialog asks you to type the engine name](/img/platform/admin/cp-engine-delete.png) + +**⋯ → Delete engine** opens *Delete «\»?*: + +- *Query history and notebook engine access on its workspace connections are removed.* +- *Saved queries that use it lose their connection.* +- *Data API endpoints (including trashed ones), alerts and datasets that use it block the delete — you will be shown where they are.* + +Type the engine name and press **Delete**. You return to the Engines list. With the Docker provisioner the engine's containers are removed (measured: 7 seconds). The dialog says *This destroys the engine's Kubernetes cluster* whatever the provisioner is. The engine's policy profile is kept and appears under **Profiles without an engine** in Governance. + +### Kubernetes-only features + +**Logs** and **Exec** work only with the Kubernetes provisioner: + +| Action | Kubernetes | Docker provisioner | +|---|---|---| +| **Logs** | Container logs for the pod, with **Last 1000 lines**, **Previous container**, **Follow** and **Refresh**. | The dialog opens and says *bad request: not available with the Docker engine provisioner*. | +| **Exec** | *Open shell into \?* opens an interactive shell (bash) inside the running pod. | The shell opens and closes at once with *[connection closed]*, without saying why. | + +![The Logs dialog on the Docker demo stack](/img/platform/admin/cp-engine-logs.png) + +With the Docker provisioner, read the log on the Docker host with `docker logs egret-engine-egret-engines--coordinator`. A failed engine's status message carries its last log lines. ## External engine -![The Overview tab of the external engine docs-trino](/img/platform/admin/cp-engine-overview.png) +![The Overview tab of the external engine trino-demo-ext](/img/platform/admin/cp-engine-ext-overview.png) The header shows the engine name, its type, an **External** badge and **Open in Engine & policy**, which opens the engine's page in Governance. A banner says *External engine: egret holds a record of it and does not monitor it, so there is no health status here.* A **More engine actions** menu holds the delete action. The tabs are **Overview**, **Catalogs**, **Plugin config** and **Settings**. @@ -17,7 +101,7 @@ The header shows the engine name, its type, an **External** badge and **Open in | Field | Meaning | |---|---| | **Type** | Trino or StarRocks. | -| **Address** | `host:port`. | +| **Address** | `host:port`, for example `trino-demo:8080`. | | **Ranger plugin version** | The version you declared. | | **Policy profile** | The bound profile, or *None bound. Bind one on the Settings tab.* | @@ -25,29 +109,29 @@ The header shows the engine name, its type, an **External** badge and **Open in *These catalogs are declared as served by this engine, so grants on them reach its policy profile. egret does not create these catalogs on your engine — create them there with the same names.* -1. Under **Catalog to declare**, choose a catalog. With none left the field reads *No catalogs left to declare*. +1. Under **Catalog to declare**, choose a catalog (*Choose a catalog*). 2. Press **Declare attached**. A toast says *Declared attached.* -![The Catalogs tab with demo_catalog declared on docs-trino](/img/platform/admin/cp-engine-catalogs.png) - -The table lists each catalog with **Kind**, **Status** (*declared*) and a **Detach** action. A tooltip on the badge says *Recorded by egret; nothing was applied to the engine.* Detaching asks *Detach "\" from "\"?* and explains that egret stops treating the engine as serving the catalog and recompiles the bound profile. Nothing changes on your cluster, and the catalog itself is kept. +The table lists each catalog with **Kind**, **Status** (*declared*) and a **Detach** action. Detaching asks *Detach «\» from «\»?*: egret stops treating the engine as serving the catalog and recompiles the bound profile. Nothing changes on your cluster, and the catalog itself is kept. Declaring a catalog is what makes governance work: grants on a catalog reach only the profiles of engines that serve it. The Data page then shows the engine in the object header. ### Plugin config -The installation files and warnings for the Ranger plugin. With no profile bound it says *engine … is bound to no policy profile: bind a profile first. Bind a policy profile on the Settings tab, then come back.* See [Ranger plugin](../../data-governance/engines-and-policy/ranger-plugin.md), which describes the same content. +![The Plugin config tab: service name, download URL, warnings and the two configuration files](/img/platform/admin/cp-engine-plugin.png) + +The installation files and warnings for the Ranger plugin: **Ranger service name**, **Policy download URL**, **Authentication**, **Declared plugin version**, *You must supply*, *What egret assumes* and the files `access-control.properties` and `ranger-trino-security.xml` with **Copy** and **Download**. With no profile bound it says *engine … is bound to no policy profile: bind a profile first. Bind a policy profile on the Settings tab, then come back.* See [Ranger plugin](../../data-governance/engines-and-policy/ranger-plugin.md). ### Settings -![The Settings tab with address, Ranger plugin version and policy profile](/img/platform/admin/cp-engine-settings.png) +![The Settings tab of the external engine: address, Ranger plugin version and policy profile](/img/platform/admin/cp-engine-ext-settings.png) | Field | Meaning | |---|---| | **Host** | A host name or IP address: no scheme, port or path. | | **Port** | The engine's port. | | **Plugin version** | The Ranger plugin version (**2.5.0**, **2.8.0** or **Other…**). | -| **Policy profile** | The profile the engine downloads, or **No profile**. | +| **Policy profile** | The profile the engine downloads, or **No profile**. Profiles bound elsewhere are marked, for example *docs-trino — bound to docs-trino*. | Changing or removing the profile recompiles both the old and the new profile. If that would leave two mask or row-filter rules deciding one column differently, egret refuses and changes nothing. **Save changes** applies; before any edit the form says *Nothing has changed yet.* @@ -55,38 +139,24 @@ A Trino profile without a tag service is refused: *policy profile … has no usa ### Remove the record -The **More engine actions** menu has **Delete**. The dialog says *egret removes its record; your cluster is not touched.* and lists what happens: +**More engine actions → Delete engine** opens *Delete «\»?*, which says *egret removes its record; your cluster is not touched.* and lists what happens: - The bound policy profile is unbound, not deleted. - Declared catalog attachments are removed; the catalogs themselves are kept. - The Ranger plugin you installed keeps running until you remove it from your engine. -If something still depends on the engine, egret refuses with *"\" is still in use. Nothing was deleted. Remove or move these first, then delete the engine again.* and lists the dependents. - -## Provisioned engine - -This section is from the product source; the demo instance has no Kubernetes to provision one. - -The header shows the engine name, type and phase, with a primary **Restart** button (rolls the pods) and a **More engine actions** menu with **Delete** (*This destroys the engine's Kubernetes cluster. It cannot be undone.*). Tabs: **Overview**, **Catalogs**, **Workspaces**, **Settings** and **Monitor**. - -| Tab | What it shows | -|---|---| -| **Overview** | A strip with **phase**, **pods** (ready/total), **replicas**, **catalogs**, **workspaces** and **auth** (*JWT delegation*, *Impersonation*, *Password auth* or *No delegation*). For StarRocks, **FE replicas** and **CN replicas** replace *replicas*. | -| **Catalogs** | Attach and detach registry catalogs, with an **Apply** state per catalog and **Retry**. Attaching or detaching on Trino applies the catalog config and restarts the coordinator, so queries in flight are lost. On StarRocks it creates or drops the catalog on the FE in place. | -| **Workspaces** | Assign the engine to workspaces. It says *Enable JWT authentication on the Security & Exposure tab before assigning* until JWT is on, and offers a search (*Search workspaces…*). | -| **Settings** | The configuration form used when you [provision](./provision.md) the engine, with a **Save changes** footer and a review of the diff. | -| **Monitor** | Live health: pods, status and logs. | - -If the engine cannot start the page shows *The engine failed to start.* with the reason. While starting it reads *Starting up — the engine isn't reachable yet.* +Type the engine name and press **Delete**. If something still depends on the engine, egret refuses with *"\" is still in use. Nothing was deleted. Remove or move these first, then delete the engine again.* and lists the dependents. -### Catalog apply states +## Catalog apply states -A Trino apply rebuilds the engine's entire catalog ConfigMap. When one catalog has a stored configuration egret cannot parse, **Retry** and **Detach** are refused on every catalog of that engine, and so is any save on the Settings tab, until you repair or detach the broken one. StarRocks applies each catalog separately, so only the broken catalogs are affected. +A Trino apply rebuilds the engine's entire catalog configuration. When one catalog has a stored configuration egret cannot parse, **Retry** and **Detach** are refused on every catalog of that engine, and so is any save on the Settings tab, until you repair or detach the broken one. StarRocks applies each catalog separately, so only the broken catalogs are affected. ## Limits and known issues - An external engine's page has no health status or restart. Egret cannot see the cluster. -- The Delete action on a provisioned engine is permanent. +- The monitor lists no queries on Trino 483 engines (see [Monitor](#monitor)). +- **Logs** and **Exec** are offered on Docker engines although they cannot work there. +- Delete is permanent. After a delete the Engines list can keep the row until you reload. ## Further reading diff --git a/platform/control-plane/query-engines/index.md b/platform/control-plane/query-engines/index.md index 2a6adbd..0570996 100644 --- a/platform/control-plane/query-engines/index.md +++ b/platform/control-plane/query-engines/index.md @@ -4,22 +4,22 @@ sidebar_position: 2 # Query engines -**Control Plane → Infrastructure → Engines** is the registry of the query engines of the platform: the Trino and StarRocks clusters that run SQL for SQL Lab, charts and dashboards. An engine here is either **provisioned** by egret on Kubernetes, or **external**: a cluster you run yourself and have registered so egret can manage its access policy. +**Control Plane → Infrastructure → Engines** is the registry of the query engines of the platform: the Trino and StarRocks clusters that run SQL for SQL Lab, charts and dashboards. An engine here is either **provisioned** by egret, or **external**: a cluster you run yourself and have registered so egret can manage its access policy. -![The Engines list with one external engine, docs-trino](/img/platform/admin/cp-engines.png) +![The Engines list with the provisioned engine docs-trino (running) and the external engine trino-demo-ext](/img/platform/admin/cp-engines.png) You need **engine:read** to open the page. Registering and creating engines need **engine:create**. ## The engine list -A summary strip counts **engines**, how many are **running** and **not running**, and how many are **external**. Tabs filter by engine type: **Trino**, **StarRocks** and **RisingWave** (RisingWave arrives in a later phase). +A summary strip counts **engines**, how many are **running** and **not running**, and how many are **external**. Tabs filter by engine type: **Trino**, **StarRocks** and **RisingWave** (RisingWave arrives in a later phase). **Search Trino engines…** filters the list by name. | Column | Meaning | |---|---| -| **Name** | The engine name. | +| **Name** | The engine name. It links to the engine page. | | **Type** | `trino` or `starrocks`. | -| **Endpoint** | Where it listens, for example `localhost:8091`. | -| **Status** | A lifecycle status for a provisioned engine. **External** for a registered one: egret does not monitor it. | +| **Endpoint** | Where it listens: `docs-trino-coordinator.egret-engines` for a provisioned engine, the host and port you entered (for example `trino-demo:8080`) for an external one. | +| **Status** | The lifecycle status of a provisioned engine (**provisioning**, **running**, **failed**). **External** for a registered one: egret does not monitor it. | | Actions | **Details** opens the engine. | ## Add an engine @@ -29,26 +29,28 @@ Two buttons sit at the top right: | Button | Does | |---|---| | **Register external engine** | Records a cluster you run yourself. Egret does not provision, restart or monitor it. See [Register an external engine](./register-external.md). | -| **Add engine** | Provisions a Trino or StarRocks cluster on Kubernetes against catalogs from the registry. See [Provision an engine](./provision.md). | +| **Add engine** | Starts a Trino or StarRocks cluster against catalogs from the registry. See [Provision an engine](./provision.md). | -:::note -The demo instance has no Kubernetes. Provisioning cannot run there (the platform logs *kubernetes unavailable; engine provisioning disabled*), so the provisioning pages of this guide are written from the product source. Registering an external engine works everywhere. +Where a provisioned engine runs depends on how egret is deployed. On Kubernetes egret creates Deployments and Services in the engine namespace. With the Docker provisioner (`[engines] provisioner = "docker"`), egret starts each engine as Docker containers on the same Docker host. + +:::note Demo stack +The Docker demo stack uses the Docker provisioner: **Add engine** starts Trino as containers named `egret-engine-egret-engines--coordinator` (and `…-worker-`). The first engine pulls the `trinodb/trino` image; plan about 1.5 to 2 GB of memory per engine. StarRocks engines, pod logs and the shell are Kubernetes-only (see [Engine detail](./engine-detail.md#kubernetes-only-features)). ::: ## Which one to choose | | Provisioned engine | External engine | |---|---|---| -| Who runs the cluster | Egret, on Kubernetes | You | +| Who runs the cluster | Egret, on Kubernetes or Docker | You | | Lifecycle, monitoring, restart | Egret | You | | Policy profile | Created and bound automatically | You create it and bind it (see [Engines and policy](../../data-governance/engines-and-policy/index.md)) | | Ranger plugin | Configured by egret | You install it from the **Plugin config** tab | -| Assign to workspaces | Yes, when JWT is enabled | The workspace picker lists egret-provisioned engines | -| Delete | Destroys the cluster | Removes the record only | +| Assign to workspaces | Yes, when JWT authentication is on | No: the workspace picker lists provisioned engines only | +| Delete | Destroys the cluster (pods or containers) | Removes the record only | ## Open an engine -Select **Details** on a row. The page depends on the kind: see [Engine detail](./engine-detail.md). +Select the name or **Details** on a row. The page depends on the kind: see [Engine detail](./engine-detail.md). ## How engines relate to catalogs and workspaces @@ -60,6 +62,7 @@ Select **Details** on a row. The page depends on the kind: see [Engine detail](. - RisingWave is listed but not available yet. - Egret cannot see whether an external engine is up or whether it downloads its profile. +- After a delete the list can keep showing the engine until you reload the page. ## Further reading diff --git a/platform/control-plane/query-engines/provision.md b/platform/control-plane/query-engines/provision.md index fd3e112..1573657 100644 --- a/platform/control-plane/query-engines/provision.md +++ b/platform/control-plane/query-engines/provision.md @@ -4,66 +4,100 @@ sidebar_position: 2 # Provision an engine -**Add engine** creates a Trino or StarRocks cluster on Kubernetes, attaches registry catalogs to it, and configures it in one step. Egret then monitors its health, binds a policy profile and the Ranger plugin automatically, and can restart or delete it. +**Add engine** starts a Trino or StarRocks cluster, attaches registry catalogs to it and configures it in one step. Egret then monitors its status, binds a policy profile and the Ranger plugin automatically, and can restart, reconfigure or delete it. -:::warning Not run on the demo instance -Provisioning needs a Kubernetes cluster. The demo instance has none, so this page is written from the product source and the page structure, not from a live provisioning run. Treat the defaults described here as what the form offers, and verify behaviour on your own cluster. -::: - -![The New engine form: engine type, name, catalogs and the configuration sections](/img/platform/admin/cp-engine-new.png) +![The New engine form: engine type, name, two catalogs with demo_warehouse primary, and the Worker section with Replicas set to 0](/img/platform/admin/cp-engine-new.png) -You need **engine:create** and **engine:provision**. The form itself opens without a cluster and can be filled in; creating fails when egret cannot reach Kubernetes. +You need **engine:create** and **engine:provision**. Egret runs the engine where its provisioner points: on Kubernetes (Deployments and Services in the engine namespace) or, with the Docker provisioner, as containers on the Docker host. ## Create an engine 1. Go to **Control Plane → Infrastructure → Engines** and press **Add engine**. -2. Choose the **Engine type** (**Trino** or **StarRocks**) and type a **Name**. -3. Under **Catalogs**, tick the catalogs to attach and mark exactly one **Primary**. The text says *Attach any number of registry catalogs, and mark one primary. The primary is the warehouse the engine is created against; the rest are attached in the same request, so the engine never comes up half-configured.* Until one is primary the form says *Mark one catalog primary to create the engine.* -4. Under **Configuration**, adjust what you need. Leave a field blank to use the chart default. +2. Keep the **Engine type** (**Trino**) or choose **StarRocks**, and type a **Name**: lower-case letters, digits and hyphens. +3. Under **Catalogs**, tick the catalogs to attach and mark exactly one **Primary**. The text says *Attach any number of registry catalogs, and mark one primary. The primary is the warehouse the engine is created against; the rest are attached in the same request, so the engine never comes up half-configured.* Until one is primary the form says *Mark one catalog primary to create the engine.* Only an egret warehouse can be primary; a catalog such as `tpch_bench` says *Can be attached, but not primary*. +4. Under **Configuration**, adjust what you need. Leave a field blank to use the default. 5. Press **Create engine**. +The engine page opens with the status **provisioning**. It turns **running** when the coordinator answers. Measured on the demo stack: 72 seconds for the first engine on a Docker host that already had the image, and 38 seconds for a restart. + +:::tip Single-node engine +For a small engine, open **Worker** and set **Replicas** to `0`: the coordinator then schedules the work itself. Left blank, the engine gets the default worker count. On the demo stack a blank value started one worker container next to the coordinator. +::: + ## The Configuration sections (Trino) -The form shows *Cluster-wide, coordinator, and worker Trino config. Leave fields blank to use the chart defaults.* The sections are listed on the left: +The form shows *Cluster-wide, coordinator, and worker Trino config. Leave fields blank to use the chart defaults.* The sections are listed on the left; a number next to a section counts the values you set in it. | Section | What it configures | |---|---| | **Image** | The container **Image** (default `trinodb/trino:483@sha256:…`), **Image pull policy** and **Image pull secrets** (names of Kubernetes docker-registry secrets in the engine namespace). | -| **Server** | `config.properties`, `node.properties`, `log.properties` and the event listener for the cluster, plus **Fault-tolerant execution** (retry failed tasks and queries, spooling exchange data to S3). Pick a property from the catalog or add a custom key with **Add property**. | -| **Coordinator** | The same property files, for the coordinator only. | -| **Worker** | The same property files, for the workers only. | -| **Workload** | Resource groups and workload settings. | -| **Pod & Containers** | Resources, placement, pod options, security contexts, environment, volume mounts, and init and sidecar containers for the coordinator and workers. | -| **Autoscaling** | A HorizontalPodAutoscaler or a KEDA ScaledObject for workers. | -| **Security & Exposure** | JWT authentication, password users, per-user delegation and ingress exposure. | +| **Server** | `config.properties`, `node.properties`, `log.properties` and the **Event listener** for the whole cluster, plus **Fault-tolerant execution** (retry failed tasks and queries, spooling exchange data to S3). | +| **Coordinator** | **Replicas & resources** (replicas, CPU and memory request and limit), **JVM** (fixed `-Xmx`/`-Xms` or a percent of the memory limit, extra `jvm.config` lines), **Per-node memory**, **Health probes**, **Init & sidecar containers**, **Volume mounts**, **Pod options**, **Coordinator placement** and **Additional exposed ports**. | +| **Worker** | The same as the coordinator, plus **Graceful shutdown** (drain a worker before it stops). **Replicas** is the number of worker pods or containers. | +| **Workload** | **Resource groups** (`resource-groups.json`) and **Session properties** (`session-property-config.json`). | +| **Pod & Containers** | **Service account & environment**, **Security contexts**, **Deployment** (rollout strategy and annotations) and **Environment from sources**. | +| **Autoscaling** | **Off** by default; a HorizontalPodAutoscaler or a KEDA ScaledObject for the workers. | +| **Security & Exposure** | **Password users**, **JWT authentication**, **Per-user delegation** and **Ingress exposure**. | + +**Add property** opens a searchable list of known Trino properties with their area and type, for example `query.max-run-time` (*duration*) or `query.max-memory` (*size*); you can also type a custom key. The form validates known keys before you save. -Each property row shows the **Setting**, **Details** and **Value**, and the form validates known keys before you save. After the engine exists, **Save changes** shows a **Review changes** dialog with the diff before applying. +![The Security & Exposure section: JWT authentication, the four per-user delegation modes and Ingress exposure](/img/platform/admin/cp-engine-security.png) ### Security & Exposure | Setting | Meaning | |---|---| -| **JWT authentication** | Lets the engine accept per-user JWTs from egret. An engine must have JWT enabled before it can be assigned to a workspace. | -| **Password users** | Static users with passwords for clients that cannot use JWT. | -| **Per-user delegation** | The delegation mode: none, **JWT delegation** or **Impersonation**, so queries run as the signed-in person. | -| **Ingress exposure** | Whether and how the engine is exposed outside the cluster. | +| **Password users** | *Each user signs in with a username and the password held in the selected egret secret (hashed server-side).* For clients that cannot use JWT. | +| **JWT authentication** | *Also accept egret-issued JWTs (e.g. from notebooks) as client credentials.* An engine needs JWT before it can be assigned to a workspace. | +| **Per-user delegation** | How queries from SQL Lab, charts and dashboards run as the calling user (below). | +| **Ingress exposure** | *Create a Kubernetes Ingress for the coordinator. TLS terminates at the ingress.* With **Expose via ingress** on you set **Host**, **Ingress class**, **Ingress TLS cert** and **Annotations**. | + +The delegation modes: + +| Mode | What it does | +|---|---| +| **Auto — JWT, else impersonate (recommended)** | The default. Egret uses a per-user JWT when the coordinator can verify it, and otherwise falls back to impersonation. | +| **JWT (per-user)** | Each query carries the caller's own short-lived JWT. No shared credential. OIDC users only. | +| **Impersonate (service user)** | Egret connects as a trusted service account and runs the query as the caller. Works for any signed-in user. | +| **None (open engine)** | Every query runs with the connection's own credential. | + +On the demo stack the default mode ran queries as the signed-in person in both cases: an SSO user (`member-sso`) and the local `admin@egret.local` account. `SELECT current_user` returned `member-sso` for the SSO user, and the grants, masks and row filters written in Governance applied to each of them (see [Masks and row filters](../../data-governance/data/masks-and-filters.md)). ## StarRocks -A StarRocks engine has its own configuration screen with sections such as **Replicas & resources**, **JVM**, **Per-node memory**, **Health probes**, **Graceful shutdown**, **Resource groups** and **Session properties**, plus a Security section for authentication and delegation. Attaching or detaching a catalog on StarRocks creates or drops it on the FE in place, without rolling the pods. +A StarRocks engine has its own configuration screen with sections such as **Replicas & resources**, **JVM**, **Per-node memory**, **Health probes**, **Graceful shutdown**, **Resource groups** and **Session properties**, plus a Security section for authentication and delegation. Attaching or detaching a catalog on StarRocks creates or drops it on the FE in place, without rolling the pods. StarRocks engines need the Kubernetes provisioner. ## What happens next -1. Egret creates the cluster and a **policy profile** for it, and shows a **phase** (for example starting, then running). -2. Attached catalogs are applied to the engine. On Trino this applies the catalog ConfigMap and restarts the coordinator. -3. The Governance area can now grant on the catalogs and see the engine under **Engine & policy**. +1. Egret starts the cluster, creates a **policy profile** with the engine's name (for example `docs-trino`, with the tag service `docs-trino_tag`) and binds it. +2. The attached catalogs are applied to the engine (state **applied** on the **Catalogs** tab). +3. Assign the engine to a workspace on its **Workspaces** tab. It then appears in the workspace's SQL Lab as a database with the engine's name, and in **Workspace settings → Engines** under **Provisioned by egret**. +4. The Governance area lists the engine under **Engine & policy**, and grants on its catalogs reach it. If the engine fails to start, its page shows *The engine failed to start.* with the reason, and the Governance overview raises an **engine failed** finding. +## Kubernetes and Docker + +The Docker provisioner runs the same Trino image with the same configuration files, but some settings have no meaning outside Kubernetes. Egret refuses them instead of ignoring them: creating an engine with an ingress host answers *bad request: engine provisioning failed: not supported by the Docker engine provisioner: ingress*, and nothing is created. The table follows the demo stack guide; the ingress refusal was checked on the demo stack. + +| Setting | Kubernetes | Docker | +|---|---|---| +| Replicas, image, `config.properties` and the other property files, JVM, JWT, password users, delegation | Yes | Yes | +| CPU and memory **limits** | Pod limits | Container limits | +| CPU and memory **requests**, health probes, placement | Yes | Accepted, no effect | +| Ingress, autoscaling, ConfigMap and Secret mounts, init and sidecar containers, image pull secrets, additional exposed ports, `shareProcessNamespace` | Yes | Refused with an error that names the setting | +| StarRocks engines | Yes | Refused | +| Pod logs, log follow and the shell (**Logs**, **Exec**) | Yes | Not available (see [Engine detail](./engine-detail.md#kubernetes-only-features)) | + +:::note Demo stack +The demo's Docker provisioner only starts images from `trinodb/trino`, with no published ports, no mounts and all capabilities dropped, on the network `egret-demo_default`. Read an engine's log with `docker logs egret-engine-egret-engines--coordinator`. +::: + ## Limits and known issues - A Trino catalog attach or detach restarts the coordinator, so queries in flight are lost. -- Without Kubernetes the form cannot create an engine. +- Right after creation the Overview strip can show the default worker count (`Worker 2`) even when you set **Replicas** to `0`; it shows the real count after the first save of the settings. +- The engine monitor lists no queries on Trino 483 engines (see [Engine detail](./engine-detail.md#monitor)). ## Further reading diff --git a/platform/control-plane/query-engines/register-external.md b/platform/control-plane/query-engines/register-external.md index 6c75769..22b63ad 100644 --- a/platform/control-plane/query-engines/register-external.md +++ b/platform/control-plane/query-engines/register-external.md @@ -6,7 +6,7 @@ sidebar_position: 1 Register an external engine when you run Trino or StarRocks yourself and want egret to manage its data-access policy. Egret does not provision, restart or monitor a registered engine. It holds a record, so that a policy profile can be bound to the engine and you can read the Ranger plugin configuration to install on it. -![The Register external engine form](/img/platform/admin/cp-engine-register.png) +![The Register external engine form: Engine, Ranger and Policy profile sections](/img/platform/admin/cp-engine-register.png) You need **engine:create**. Binding a profile also needs **policy:read**, and declaring catalogs needs **catalog_registry:read**. @@ -42,7 +42,7 @@ Egret proves migrations and explains access only for Trino with plugin 2.5.0 or ### Policy profile -*Bind a Trino profile now, or leave it and bind one later from the engine's Settings tab. A profile can belong to one engine.* The list shows **No profile yet**, then each profile, marking those already taken (for example *docs-trino-policy — bound to docs-trino*). A profile already bound to another engine cannot be chosen again. If binding would make two mask or row-filter rules decide one column differently, egret refuses and changes nothing, and the form says *Nothing was changed. Resolve one of the two, then try again.* +*Bind a Trino profile now, or leave it and bind one later from the engine's Settings tab. A profile can belong to one engine.* The list shows **No profile yet**, then each Trino profile, marking those already taken (for example *docs-trino — bound to docs-trino*). It does not say which profiles have a tag service, so a profile you created with **New profile** is offered although it cannot be bound. A profile already bound to another engine cannot be chosen again. If binding would make two mask or row-filter rules decide one column differently, egret refuses and changes nothing, and the form says *Nothing was changed. Resolve one of the two, then try again.* :::note A Trino profile created in the console has no tag service and cannot be bound. Use a profile with a tag service (see [Engines and policy](../../data-governance/engines-and-policy/index.md#create-a-policy-profile)). @@ -54,22 +54,30 @@ A Trino profile created in the console has no tag service and cannot be bound. U ## After you register -You land on the engine's page with the tabs **Overview**, **Catalogs**, **Plugin config** and **Settings**. Then: +You land on the engine's **Plugin config** tab. Without a profile it says *conflict: engine \ is bound to no policy profile: bind a profile first Bind a policy profile on the Settings tab, then come back.* The tabs are **Overview**, **Catalogs**, **Plugin config** and **Settings**. Then: -1. **Plugin config**: copy the files to your engine and restart it (see [Ranger plugin](../../data-governance/engines-and-policy/ranger-plugin.md)). -2. **Catalogs**: declare more catalogs, or detach. -3. In **Governance → Engine & policy**, grant engine permissions and review policies. +1. **Settings**: bind a policy profile that has a tag service, and press **Save changes**. +2. **Plugin config**: copy the files to your engine and restart it (see [Ranger plugin](../../data-governance/engines-and-policy/ranger-plugin.md)). +3. **Catalogs**: declare more catalogs, or detach. +4. In **Governance → Engine & policy**, grant engine permissions and review policies. -If a warning appears under the form, read it before leaving; it names settings, such as an unreachable download URL, that will stop the engine from downloading policy. +If a warning appears on **Plugin config**, read it before leaving; it names settings, such as an unreachable download URL, that will stop the engine from downloading policy. -## Seed the demo engine +## Example: register the demo Trino -On the demo instance, `scripts/seed-governance-ui.mjs` in the docs repository registers the demo Trino (`localhost:8091`) as `docs-trino` the same way, through the UI. +On the Docker demo stack the seeded Trino (`trino-demo`, port `8080`) can be registered as an external engine: + +1. **Register external engine**, **Name** `trino-demo-ext`, **Engine type** Trino, **Host** `trino-demo`, **Port** `8080`. +2. **Plugin version** `2.8.0`, **Policy profile** **No profile yet**, tick `tpch_bench` under **Catalogs**. +3. **Register engine**. + +Binding the console-created Trino profile `docs-external-policy` at this point fails with *bad request: policy profile 5 (docs-external-policy) has no usable tag service …*, so the example binds a profile pair created through the API (a `tag` profile and a Trino profile whose `config.tagService` names it) afterwards on **Settings**. The demo Trino has no Ranger plugin installed, so the record is for policy authoring only: nothing is enforced on that engine. ## Limits and known issues - Egret holds a record only: it does not monitor the engine, so the page shows no health status. - An engine name cannot be changed after registration; remove the record and register again. +- The message after registering keeps the raw `conflict:` prefix of the API error. ## Further reading diff --git a/platform/control-plane/secrets.md b/platform/control-plane/secrets.md index a25b8cf..1e1a13d 100644 --- a/platform/control-plane/secrets.md +++ b/platform/control-plane/secrets.md @@ -6,7 +6,7 @@ sidebar_position: 3 A **secret** is a named, stored credential that other settings reference by name instead of holding the value themselves: the secret access key of a storage connection, the password of a JDBC catalog, the password of a database connection. **Control Plane → Infrastructure → Secrets** manages them. **Values are never displayed** after you save them. -![The Secrets list with two secrets](/img/platform/admin/cp-secrets.png) +![The Secrets list: the secrets you created and the ones egret creates for engines and catalog clients](/img/platform/admin/cp-secrets.png) You need **secret:read** to open the page; creating, rotating and deleting need the matching permissions. @@ -14,6 +14,8 @@ You need **secret:read** to open the page; creating, rotating and deleting need A strip counts the **secrets**. +The list also shows the secrets egret creates and manages itself: `engine-internal--…` (the Trino internal-communication secret), `engine-monitor-…` (the egret-monitor password for the native Trino monitor), `engine-tls-…` (the coordinator keystore) for every provisioned engine, and `catalog-client-…` (a Catalog Client credential an engine presents to egret's catalog). Leave them alone: rotating or deleting one breaks the engine or client that uses it. On a 1440-pixel-wide screen the **Used by** and **Created** columns are cut off at the right; scroll the table sideways. + | Column | Meaning | |---|---| | **Name** | The secret name, with its keys underneath (for example `password`, `username`). Select it to open the secret. | @@ -44,7 +46,7 @@ A strip counts the **secrets**. ## The secret page -![A secret's details: name, type and keys; no values](/img/platform/admin/cp-secret-detail.png) +![A secret's details: name, description, type and keys; no values](/img/platform/admin/cp-secret-detail.png) The header shows the name, type and *USED BY n connection(s)*, with **Rotate value** and **Delete**. Tabs: diff --git a/platform/control-plane/storage/index.md b/platform/control-plane/storage/index.md index da32034..bb086b2 100644 --- a/platform/control-plane/storage/index.md +++ b/platform/control-plane/storage/index.md @@ -61,18 +61,21 @@ The header shows the name, provider and the **BUCKET** and **ENDPOINT**, with ** ## Browse a connection -The **Browse** tab lists the bucket's folders and files, so you can confirm a connection works and see what egret stored: for example folders such as `apps`, `query-results`, `workflow` and the warehouse folders named by id. Select a folder to open it. An empty folder says *This folder is empty.* +The **Browse** tab lists the bucket's folders and files, so you can confirm a connection works and see what egret stored: for example folders such as `apps`, `query-results`, `workflow` and the warehouse folders named by id. Select a folder to open it; files show their size. An empty folder says *This folder is empty.* -![The Browse tab listing the folders of the bucket](/img/platform/admin/cp-storage-browse.png) +The demo bucket is shared by everything egret stores, so `query-results` holds the stored result files of every user's SQL Lab queries. Anyone with **storage:read** can list and download them, including the built-in `data_steward` role; these files are what each person was allowed to see, after their masks and row filters. Keep **storage:read** for administrators. + +![The Browse tab listing the top-level folders of the warehouse bucket](/img/platform/admin/cp-storage-browse.png) ## Delete a connection -Press **Delete** and confirm. A connection that warehouses or other resources still use should be left alone: check the **Warehouses** tab first. +Press **Delete**. The dialog *Delete «\»?* says *This storage connection will be removed. Any catalogs or workspaces that reference it may break.* and asks you to type `delete`. A connection that warehouses or other resources still use should be left alone: check the **Warehouses** tab first (for `demo-rustfs` it lists `demo-warehouse`, workspace Default, **Provisioned**). ## Limits and known issues - Only S3-compatible stores work; Google Cloud Storage and Azure Blob are not available yet. -- On the demo instance the **Browse** tab lists the bucket root even for a connection with a **Key prefix**. Treat the prefix as unverified there. +- The **Browse** tab lists the bucket root even for a connection with a **Key prefix** (checked with `sandbox-lake`, prefix `sandbox/`). +- A key prefix does not keep two egret warehouses apart: a warehouse on `sandbox-lake` was refused with *Storage profile overlaps with existing warehouse demo-warehouse*. Use a separate bucket for each warehouse. ## Further reading diff --git a/platform/control-plane/workspaces/index.md b/platform/control-plane/workspaces/index.md index 88236c6..84021a0 100644 --- a/platform/control-plane/workspaces/index.md +++ b/platform/control-plane/workspaces/index.md @@ -32,6 +32,8 @@ The strip counts **total**, **active** and **archived** workspaces. **Status** f The new workspace appears in the list, owned by you, with no members. As a platform admin you can open it at once; everybody else needs to be added as a member (see [Members](../../workspace-settings/members.md)). +The **⋯** menu on a row offers **Open**, **Rename** and **Archive**. + ## Open a workspace Select a row to open the workspace's page, with the tabs **Overview**, **Members**, **Resources**, **Features**, **Access requests** and **Activity**. See [Workspace details](./workspace-detail.md). diff --git a/platform/control-plane/workspaces/workspace-detail.md b/platform/control-plane/workspaces/workspace-detail.md index f44de59..9471450 100644 --- a/platform/control-plane/workspaces/workspace-detail.md +++ b/platform/control-plane/workspaces/workspace-detail.md @@ -10,7 +10,7 @@ Opening a workspace in **Control Plane → Workspaces** shows its page. The head ## Overview -A summary strip shows the number of **members** and **queries run**, then **OWNER**, **CREATED** and **STATUS**. +A summary strip shows the number of **members** and **queries run** (for example *7 members · 27 queries run*), then **OWNER**, **CREATED** and **STATUS**. ## Members @@ -20,11 +20,11 @@ A note reminds you that workspace roles apply only inside the workspace, and tha ## Resources -![The Resources tab of the default workspace: a warehouse and the assigned engines](/img/platform/admin/cp-workspace-resources.png) +![The Resources tab of Docs sandbox: no warehouse, and the engine docs-trino assigned](/img/platform/admin/cp-workspace-resources.png) ### Warehouses -*Iceberg catalogs owned by this workspace.* The table shows each warehouse's **Name**, **Storage** (the storage connection) and **Status** (for example **Provisioned**). **Manage** opens the Catalogs page. +*Iceberg catalogs owned by this workspace.* The table shows each warehouse's **Name**, **Storage** (the storage connection) and **Status** (for example **Provisioned**); a workspace without one says *No warehouses in this workspace yet.* **Manage** opens the Catalogs page. On the demo stack the seeded warehouse `demo-warehouse` belongs to `default`. ### Assigned engines @@ -36,7 +36,7 @@ A footer says *Storage connections and secrets are managed platform-wide in the *These switches decide which modules this workspace has at all — turning one off hides it from the sidebar and blocks its API for everyone in this workspace, including platform admins. It doesn't delete anything: existing content is kept and comes back exactly as it was when the module is switched back on. Who can read or write within an enabled module is still decided by workspace roles.* -![The Features tab with one switch per module](/img/platform/admin/cp-workspace-features.png) +![The Features tab of Docs sandbox with one switch per module](/img/platform/admin/cp-workspace-features.png) | Module | What it covers | |---|---| @@ -63,7 +63,7 @@ Pending requests to join this workspace, with the same **Approve** and **Deny** ## Activity -The audit entries recorded for this workspace: **Time**, **Action**, **Resource**, **Actor** and **Decision**. It is the workspace-scoped slice of the [Audit log](../audit-log.md), newest first. +The audit entries recorded for this workspace: **Time**, **Action**, **Resource**, **Actor** and **Decision**. It is the workspace-scoped slice of the [Audit log](../audit-log.md), newest first. Most rows are authorization decisions such as `workspace:manage_members` (**allow** or **deny**), `query:run` or `dashboard:read`; changes such as `member:add` show a dash. A member who opens a page they may not use shows up as a **deny**. Actions done while a platform admin impersonated someone are listed under the impersonated person. ## Limits and known issues diff --git a/platform/data-governance/classifications/clearance.md b/platform/data-governance/classifications/clearance.md index 7aee853..7396f7d 100644 --- a/platform/data-governance/classifications/clearance.md +++ b/platform/data-governance/classifications/clearance.md @@ -6,14 +6,14 @@ sidebar_position: 1 A **clearance** makes a classification tag a gate. Once a tag has a clearance, only principals who hold one of the required roles, or belong to one of the required groups, can open objects carrying the tag. A tag with no clearance only labels; it restricts nobody. -![The Clearance section of a tag with the role data_steward required, and the warnings egret shows](/img/platform/admin/gov-tag-clearance.png) +![The Clearance tab of Confidential with the role data_steward required, and the warnings egret shows](/img/platform/admin/gov-tag-clearance.png) ## Add a clearance 1. Open **Governance → Classifications** and select the tag. -2. In **Clearance**, choose the type: **Role** or **Group**. +2. Open the **Clearance** tab and choose the type: **Role** or **Group**. 3. Search and pick the role or group (*Select a role…*, or *Search groups…*). -4. Press **Add**. A toast says *Clearance updated*. The tag now shows a **Gated** badge, and the list's **Clearance** column names the role. +4. Press **Add**. The tag's header now shows a **Clearance** badge with a lock, and the list's **Clearance** column counts the requirement (*1 role*). The page adds the warnings described below. Add several rows to require "any of these". To lift the gate, remove all rows. The toast says *Clearance cleared*, and the section reads *Not gated — anyone can open objects with this tag.* @@ -50,6 +50,10 @@ A person who holds the required role only through a group is *Not cleared* in th A StarRocks clearance expands to one policy per tagged column, counted per profile. If more than 500 of a tag's columns are visible to one StarRocks profile, that profile's recompile fails outright rather than truncating the clearance, and every write that would recompile it fails with it, until the tag is narrowed or untagged back under the limit. +## What the engine does + +On the demo stack the tag *Confidential* is on the table `demo_warehouse.finance.invoices`, and its clearance requires the role `data_steward`. `member-sso` has a grant to read the schema `finance`, but `SELECT count(*) FROM demo_warehouse.finance.invoices` answers *Access Denied: Cannot select from columns [] in table or view invoices*. The platform admin gets the same answer: platform admins are not exempt from a clearance unless they hold the role directly. Farid Haddad, who holds `data_steward` directly, is listed as cleared. + ## Check who is cleared - The tag page's **Clearance** table lists what is required. diff --git a/platform/data-governance/classifications/index.md b/platform/data-governance/classifications/index.md index 015f47b..bdbbc52 100644 --- a/platform/data-governance/classifications/index.md +++ b/platform/data-governance/classifications/index.md @@ -6,78 +6,78 @@ sidebar_position: 5 **Governance → Classifications** defines the **tags** you use to classify data and BI objects, for example *Personal data (PII)* or *Finance*. A tag does three jobs: it labels an object, it can select the columns a mask or row filter applies to, and it can **gate** the objects that carry it behind a role or group. -![The Classifications list with one tag, Personal data (PII), and its key and clearance](/img/platform/admin/gov-classifications.png) +![The Classifications list: four tags with their keys, data and BI counts and clearance](/img/platform/admin/gov-classifications.png) The page needs the **tag:read** permission. Creating, editing and deleting tags needs **tag:manage**, which the built-in `data_steward` role holds. ## The tag list -The header counts **tags**, **gated** tags (those with a clearance) and **tagged BI objects**. **Search tags…** filters the table. With no tags the table says *No tags yet.* +The header counts **classifications**, how many are **with clearance** and how many are **unused** (carried by nothing). **Search classifications** filters the table. | Column | Meaning | |---|---| -| **Tag** | The tag name with its colour. Select it to open the tag page. | +| **Name** | The tag name with its colour. Select it to open the tag page. | | **Key** | The immutable identifier used in rules and policies, for example `personal-data-pii`. | | **Description** | What the tag means, or a dash. | +| **Data** | How many tables and columns carry the tag, for example *0 tables · 4 columns*, or a dash. | | **BI objects** | How many datasets, charts or dashboards carry the tag. | -| **Clearance** | The roles and groups that gate the tag, or a dash when it is not gated. | +| **Clearance** | *None*, or how many roles and groups gate the tag (for example *1 role* with a lock). | + +The **⋯** menu on a row has **Delete**. ## Create a tag -1. Press **New tag**. -2. Type a **Name**. It is required. The **Key** is filled from the name (for example `personal-data-pii`). -3. Optionally pick a **Color** and write a **Description**. +1. Press **New classification**. The dialog says *A tag to classify tables, columns and BI objects with.* +2. Type a **Name**. It is required. The **Key** follows the name (for example `personal-data-pii`) until you edit it. +3. Optionally pick a **Colour** and write a **Description**. 4. Edit the **Key** if you want a different one. -5. Press **Create**. - -The dialog says *Create a classification tag that can be assigned to data and BI objects.* +5. Press **Create classification**. ### Fields | Field | Notes | |---|---| | **Name** | Required. Shown everywhere the tag appears. | -| **Color** | A colour picker. | -| **Key** | Required. *Immutable identifier — auto-filled from the name; edit if you like. Lowercase letters, numbers, -, _, :.* It cannot be changed after creation. | -| **Description** | *What this tag means…* | +| **Colour** | A colour picker. | +| **Key** | Required. *Fixed once created. Follows the name until you edit it. Lowercase letters, digits, -, _ and :.* | +| **Description** | Free text. | ## The tag page -Select a tag to open its page. The header shows the colour, name, key, a **Gated** badge when a clearance is set, and the count of tagged objects. **Delete** is on the right. - -![The tag page: details on the left, clearance on the right, and the exposure and rules sections below](/img/platform/admin/gov-tag-detail.png) +Select a tag to open its page. The header shows the colour, name, key and description, a **Clearance** badge when a clearance is set, and **Edit** and **Delete**. **Edit** turns the header into a form for **Name**, **Colour** and **Description** with **Save** and **Cancel**; *Key personal-data-pii is fixed once created.* -### Details +![The tag page of Personal data (PII): the tables and columns that carry it](/img/platform/admin/gov-tag-detail.png) -**Name**, **Color** and **Description** save automatically when you leave the field (*Tag updated*). The **Key** is read-only. +The page has five tabs: -### Clearance - -The clearance gates objects carrying the tag. See [Clearance](./clearance.md). - -### Existing materialized views, Trino views, and Mask and filter rules +| Tab | What it shows | +|---|---| +| **Where used** | **Data**: per schema, each table that carries the tag and on which columns (for example *customers — Columns: email, name, phone*). **BI**: the datasets, charts and dashboards that carry it. | +| **Clearance** | The roles and groups that gate objects carrying the tag. See [Clearance](./clearance.md). | +| **Mask & filter** | The egret rules that select the tag, with **Rule**, **Kind**, **Reaches this tag** and **Exempt**. See [Tag-based policy](./tagging.md#rules-that-select-a-tag). | +| **View exposure** | Materialized views and Trino views that could read classified data. See [View exposure](./view-exposure.md). | +| **Hand-written** | *Hand-written tag policies whose tag matches this classification's name.* | -These sections show where classified data could leak through views, and which rules select the tag. See [View exposure](./view-exposure.md) and [Tag-based policy](./tagging.md#rules-that-select-a-tag). +![The Mask & filter tab of Personal data (PII): the rule Hash personal data, exempting the group data-stewards](/img/platform/admin/gov-tag-rules.png) ## Delete a tag -1. Open the tag and press **Delete**, or use the row action on the list. -2. Confirm *Delete "\"?* The dialog says *This tag will be permanently removed from the catalog and from every object it's assigned to.* +1. Open the tag and press **Delete**, or use **⋯ → Delete** on the list. +2. The dialog *Delete \?* says what goes with it, for example *Removes it from 0 tables and 4 columns.* and *Removes it from 0 BI objects.* Type the key to confirm. -If a mask or row-filter rule still selects the tag, the delete is refused and a message points to the **Mask and filter rules** section of the tag page, which lists the rules. Delete or change those rules first. +If a mask or row-filter rule still selects the tag, the dialog says so (*1 rule selects it: the delete will be refused until they are deleted or retargeted.*) and the delete is refused. Delete or retarget those rules first; the **Mask & filter** tab lists them. ## Where tags are applied Tags attach to: - **Tables and columns** in egret-managed catalogs, from the Data page. See [Tagging data](./tagging.md). -- **BI objects** (datasets, charts, dashboards), from the object's Tags control. The count appears in **BI objects**. +- **BI objects** (datasets, charts, dashboards), from the object's Tags control. The count appears in **BI objects**; tables and columns are counted in **Data**. ## Limits and known issues - A tag key cannot be changed. Create a new tag and move objects over. - Tags cannot be added to catalogs that egret does not manage (*Tags can be added to egret-managed catalogs only*). -- The tag page's *N tagged objects* counts BI objects; a tag on a table column appears on the Data page and in explain, not in this count. ## Further reading diff --git a/platform/data-governance/classifications/tagging.md b/platform/data-governance/classifications/tagging.md index 371fb98..9cda1a6 100644 --- a/platform/data-governance/classifications/tagging.md +++ b/platform/data-governance/classifications/tagging.md @@ -6,7 +6,7 @@ sidebar_position: 2 A tag does nothing until it is applied. This page covers how to apply a tag to the tables and columns of an egret catalog and to BI objects, and how to write a **tag-based rule** so a mask or row filter follows the tag wherever it is applied. -![The Add tag dialog for the started_at column](/img/platform/admin/gov-add-tag.png) +![The Add tag dialog for the started_at column, with the tag list open](/img/platform/admin/gov-add-tag.png) ## Tag a table or a column @@ -17,7 +17,7 @@ Tags attach to tables and columns in egret-managed catalogs (Iceberg warehouses) 3. In **Add tag**, open **Tag** and choose a tag. Tags already on the object are not offered; if there are none left the dialog says *Every tag is already on this table* (or *column*). With no tags defined it says *No tags are defined yet.* 4. Press **Add tag**. A toast says *Tag added.* -![The sessions table with the tag on user_id and the rules applying to it](/img/platform/admin/gov-table-overview.png) +![The Data page for the sessions table: header, engine docs-trino, and the Columns list with tags and masking rules](/img/platform/admin/gov-table-overview.png) The tag then appears in the **Tags** column of the table's **Columns** list, in the object header for a table-level tag, and in the column panel. You can also tag from **Control plane → Catalogs → \ → Browse**, where each table and column has a tag control. @@ -40,7 +40,9 @@ A rule can select **by tag** instead of by path. It then applies to every table 3. Set **Selects by** to **Tag**, search for the tag, and pick it. 4. Preview, then **Create**. -The rule shows under **By tag** on every tagged table, for example *Redact personal data — tag Personal data (PII) — On column user_id*, and in the engine's **Policies** tab as **Masking (by tag)** with the resource `tag: Personal data (PII)`. The tag page's **Mask and filter rules** section lists every rule selecting the tag, with columns **Rule**, **Kind**, **Reaches this tag** and **Exempt**. +The rule shows under **By tag** on every tagged table, for example *Hash personal data — tag Personal data (PII) — mask · MASK_HASH — On column user_id*, and in the engine's **Policies** tab as a masking policy on `tag: Personal data (PII)`. The tag page's **Mask & filter** tab lists every rule selecting the tag, with columns **Rule**, **Kind**, **Reaches this tag** and **Exempt**. + +On the demo stack the rule *Hash personal data* (Hash, exempt group `data-stewards`) made `member-sso` read `user_id` as a 64-character hash such as `B25935C0F9C4BB9A…`. It took effect after the next plugin download, within a minute of saving. :::warning A tag rule is catalog-unscoped: it recompiles every Trino and StarRocks profile an engine uses. A compile failure on any profile fails the save and rolls the rule back. The preview names the profiles first. @@ -54,8 +56,8 @@ Engines match a rule by its tag's **name**, so a rule written on another tag wit ## Examples -- **Mask every personal column**: tag the columns `PII`, add one mask rule by tag `PII` with type **Hash**, exempt the `compliance` group. -- **Restrict reads**: add a [clearance](./clearance.md) to `PII` so only the `data_steward` role can open objects carrying it. +- **Mask every personal column**: tag the columns *Personal data (PII)*, add one mask rule by that tag with type **Hash**, exempt the `data-stewards` group (the demo setup). +- **Restrict reads**: add a [clearance](./clearance.md) to *Confidential* so only the `data_steward` role can read objects carrying it (the demo setup on `finance.invoices`). ## Limits and known issues diff --git a/platform/data-governance/classifications/view-exposure.md b/platform/data-governance/classifications/view-exposure.md index 4ad0c02..f53203b 100644 --- a/platform/data-governance/classifications/view-exposure.md +++ b/platform/data-governance/classifications/view-exposure.md @@ -6,9 +6,9 @@ sidebar_position: 3 A classification's clearance stops a direct read of a classified column. A **view** reads the same data indirectly, so without more care it could read straight through the clearance. The tag page shows, per classification, the views that read classified data and what the engine does about them. This page explains those sections and what clearance does and does not protect through views. -![The tag page sections for materialized views, Trino views and rules](/img/platform/admin/gov-tag-detail.png) +![The View exposure tab of Personal data (PII): no materialized views, and one DEFINER view whose lineage could not be computed](/img/platform/admin/gov-tag-exposure.png) -The sections need **tag:read** (and **policy:read** for the rules section). They are below **Clearance** on the tag page of **Governance → Classifications**. +The sections need **tag:read**. They are on the **View exposure** tab of a tag's page in **Governance → Classifications**. ## What is protected @@ -74,7 +74,8 @@ A view change on a catalog whose profile has a classified column is held until t ## Limits and known issues -- Not run on the demo instance: it has no view over classified data, so both sections show their empty states. +- On the demo stack the view `demo_warehouse.analytics.revenue_by_country` (a DEFINER view over the tagged `sessions` table) is listed with *clearance unknown* and *may read classified data*, because egret could not compute its lineage: reading the view through the engine was refused (*Access Denied: Cannot access catalog demo_warehouse*). The row says *lineage could not be computed; egret has not judged this view. Run the lineage sweep, whose report names the reason.* The console has no button for the sweep; it is the API call `POST /api/control-plane/classifications/lineage-sweep`. +- A view created before its base table was tagged waited no time; the hold applies only once the catalog has a classified column. - The Trino list has a 15 second deadline. If egret's views cannot all be read, the page shows an error instead of a partial list. ## Further reading diff --git a/platform/data-governance/data/explain-access.md b/platform/data-governance/data/explain-access.md index 756edab..4fd4b99 100644 --- a/platform/data-governance/data/explain-access.md +++ b/platform/data-governance/data/explain-access.md @@ -6,7 +6,7 @@ sidebar_position: 3 **Explain access** answers the question "can this person read this?" using the policies the engine actually downloads, not the policies you think you wrote. It is the quickest way to find out why a user is allowed or refused, and which grant decided it. -![The Who can read dialog: an Allowed verdict for member-sso with the grant that decided it](/img/platform/admin/gov-who-can-read.png) +![The Who can read dialog: select is Allowed for member-sso on docs-trino, with the grant that decided it](/img/platform/admin/gov-who-can-read.png) Explain needs the **policy:read** permission. There are two entry points: diff --git a/platform/data-governance/data/grants.md b/platform/data-governance/data/grants.md index 8532abe..6c2f7f2 100644 --- a/platform/data-governance/data/grants.md +++ b/platform/data-governance/data/grants.md @@ -6,7 +6,7 @@ sidebar_position: 1 A **grant** says who can read or write which part of a catalog. Grants are the main way you give people access to data in the query engines. You create them on the **Permissions** tab of the Data page, from a person's page, or from the grant dialog's **Grant access** button. -![The Permissions tab of the sessions table: the grant written for it and the grant inherited from the workspace](/img/platform/admin/gov-table-permissions.png) +![The Permissions tab of the sessions table: the grant to member-sso here, and two inherited catalog grants (the workspace assignment of Default and the role data_steward)](/img/platform/admin/gov-table-permissions.png) ## Anatomy of a grant @@ -17,7 +17,7 @@ A **grant** says who can read or write which part of a catalog. Grants are the m | **Scope** | A catalog, narrowed to a schema, a table and a column. Each narrower level is optional. | | **Principals** | Who the grant is for: **Include** (users, groups or roles) and **Exclude** (principals removed from the include set). | -Grants are written as sentences in lists, for example `user member-sso can read demo_catalog.analytics.sessions`. A level whose value is a pattern (`*`, `ord*`, `order?`) is quoted in the sentence, and the engine matches it as a glob. +Grants are written as sentences in lists, for example `user member-sso can read demo_warehouse.analytics.sessions`. A level whose value is a pattern (`*`, `ord*`, `order?`) is quoted in the sentence, and the engine matches it as a glob. ## Read the Permissions tab @@ -35,15 +35,19 @@ Each row has **Edit** and **Delete** icon buttons. Both need the **policy:update ## Create a grant -1. Open **Governance → Data** and select the table, for example `demo_catalog.analytics.sessions`. +1. Open **Governance → Data** and select the table, for example `demo_warehouse.analytics.sessions`. 2. Press **Grant access**. The dialog opens as **New grant** with the catalog, schema and table already filled in. 3. Leave **Effect** as **Allow**. Under **Actions**, keep **Read** ticked. 4. Under **Principals → Include**, choose a kind (**user**, **group** or **role**), then search. -5. For a user who has no engine username yet, choose the match and press **Allocate username and add**. Confirm **Allocate and add** in the dialog *Allocate a username to Member SSO?* Egret reports *Member SSO was allocated a username and added, stored as member-sso.* The username is kept even if you close the form without saving. -6. Press **Preview**. The preview names the profiles that would be recompiled, for example `docs-trino-policy`, and how many policies would be created, updated, deleted or left unchanged. +5. A user who already has an engine username is added by selecting the match. For a user who has none yet, the match shows **Allocate username and add**: press it. Confirm **Allocate and add** in the dialog *Allocate a username to Member SSO?* Egret reports *Member SSO was allocated a username and added, stored as member-sso.* The username is kept even if you close the form without saving. +6. Press **Preview**. The preview names the profiles that would be recompiled, for example `docs-trino`, and how many policies would be created, updated, deleted or left unchanged. 7. Press **Create**. You see *Grant created.* -![The New grant dialog after a preview: one principal, and the profile it would recompile](/img/platform/admin/gov-grant-dialog.png) +![The New grant dialog after a preview: read and write on demo_warehouse.analytics.sessions for alice-nguyen, which would recompile docs-trino](/img/platform/admin/gov-grant-dialog.png) + +### Check the grant in SQL Lab + +The engine applies the grant at its next policy download (the plugin polls every 30 seconds). On the demo stack, after this grant `member-sso` read `demo_warehouse.analytics.sessions` through the engine `docs-trino`, and a table with no grant for her answered *Access Denied: Cannot select from columns [] in table or view customers*. ### The dialog fields diff --git a/platform/data-governance/data/index.md b/platform/data-governance/data/index.md index 88f1310..c85c7d8 100644 --- a/platform/data-governance/data/index.md +++ b/platform/data-governance/data/index.md @@ -6,13 +6,13 @@ sidebar_position: 2 **Governance → Data** is the place to look at the access to one piece of data: a catalog, a schema, a table, a view or a column. Open an object to see who can read it, which tags and rules apply, which engines serve it and where it comes from. -![The Data page for the sessions table: header, Engines and Tags, and the Columns list](/img/platform/admin/gov-table-overview.png) +![The Data page for the sessions table: header, engine docs-trino, and the Columns list with tags and masking rules](/img/platform/admin/gov-table-overview.png) The page needs the **policy:read** or **tag:read** permission. Reading grants and rules needs **policy:read**; reading and applying tags needs **tag:read**. ## Browse the catalog tree -The left panel is a tree of the **catalogs** egret knows about. Each catalog shows its kind and source, for example *Iceberg · egret* for a warehouse egret manages. Expand a catalog to its schemas, and a schema to its tables, views and materialized views. +The left panel is a tree of the **catalogs** egret knows about. Each catalog shows its kind and source, for example *Iceberg · egret* for a warehouse egret manages, *TPC-H · via engine* or *JDBC · via engine* for catalogs listed through an engine. Expand a catalog to its schemas, and a schema to its tables, views and materialized views. | Control | What it does | |---|---| @@ -36,7 +36,7 @@ Every object page has the same header: - **Grant access**, which opens the [grant dialog](./grants.md) with this object as the scope. - **Add tag** on a table, view or materialized view, which opens the [tag picker](../classifications/tagging.md). Tags exist only on egret-managed catalogs; on any other the button is disabled with *Tags can be added to egret-managed catalogs only*. - **Who can read?**, which opens [Explain access](./explain-access.md) for this object. -- **Engines**: the engines that serve the catalog, for example `docs-trino · Trino · external`. *No engine attached* means no engine is declared to serve the catalog, so no policy reaches an engine. +- **Engines**: the engines that serve the catalog, for example `docs-trino · Trino`. *No engine attached* means no engine is declared to serve the catalog, so no policy reaches an engine. - **Tags**: the tags on this object. A schema or catalog shows none, because tags attach to tables and columns. The header never waits on the API. Its name and path come from the URL, so one failing call never blanks the page. @@ -57,6 +57,9 @@ The selected tab is kept in the address (`?tab=permissions`, `?tab=masks`, `?tab - **Catalog**: the **Schemas** it contains, the **Attached engines** (or *Not attached to any engine*) and the **Workspaces** it is assigned to. - **Schema**: a table with **Name** and **Kind** for each table, view or materialized view. + +![The schema analytics: three tables and the view revenue_by_country](/img/platform/admin/gov-data-schema.png) + - **Table**: the **Columns** table with **Name**, **Type**, **Tags** and **Masking rules** per column. Select a column to open its panel (see below). ### Permissions @@ -81,6 +84,8 @@ On a table, **Reads from** lists the base tables of a view, and **Read by** list | **Workspaces** | Workspaces the catalog is assigned to. | | **Engines** | Engines that serve it, or *None*. | +![The Details tab of the sessions table](/img/platform/admin/gov-table-details.png) + ## The column panel Selecting a column in the **Columns** table opens a panel on the right and adds `?column=` to the address. It shows, for that column: @@ -92,7 +97,7 @@ Selecting a column in the **Columns** table opens a panel on the right and adds Close the panel with **Close**. -![The column panel for user_id with its tag, masking rules and grants](/img/platform/admin/gov-column-panel.png) +![The column panel for user_id with its tag, the masking rule selected by the tag, and no column grant](/img/platform/admin/gov-column-panel.png) ## Limits and known issues diff --git a/platform/data-governance/data/masks-and-filters.md b/platform/data-governance/data/masks-and-filters.md index a22a9d6..d5cf2db 100644 --- a/platform/data-governance/data/masks-and-filters.md +++ b/platform/data-governance/data/masks-and-filters.md @@ -6,7 +6,7 @@ sidebar_position: 2 A **rule** changes what a person sees in a table without denying access to it. A **mask** replaces the values of a column (hash, redact, show the last four characters). A **row filter** hides the rows that do not match an expression. Rules apply to everyone except the principals you exempt, and they are enforced in the query engine. -![The Masks and filters tab of the sessions table, with a row filter, a mask on a column and a mask selected by a tag](/img/platform/admin/gov-table-masks.png) +![The Masks and filters tab of the sessions table: a row filter, a Nullify mask on revenue and a Hash mask selected by the tag Personal data (PII)](/img/platform/admin/gov-table-masks.png) Reading rules needs **policy:read**. Creating, editing and deleting them needs **policy:update**. @@ -67,7 +67,7 @@ A SQL boolean expression over the table's columns, for example `region = 'us'` o | Option | Fields | Notes | |---|---|---| -| **Path** | **Scope**: catalog (required), schema, table, and column for a mask. | A row filter by path needs catalog, schema and table together, and never a column; the dialog shows no column field and blocks saving until the three are set. A line below the fields reads *Selects demo_catalog.analytics.sessions*. | +| **Path** | **Scope**: catalog (required), schema, table, and column for a mask. | A row filter by path needs catalog, schema and table together, and never a column; the dialog shows no column field and blocks saving until the three are set. A line below the fields reads *Selects demo_warehouse.analytics.sessions*. | | **Tag** | **Tag**: search tags by name and pick one. | The rule applies to every table or column carrying the tag. The tag is stored by its key, shown as *stored as personal-data-pii*. **Change** picks another. A tag rule is catalog-unscoped. | ### Exempt @@ -76,7 +76,7 @@ The principals the rule does not apply to. The picker offers **user** and **grou ### Preview this write -Like grants, a rule needs a current **Preview** before **Create**. For a tag rule the preview text warns: *A tag rule is catalog-unscoped, so this can recompile every Trino profile in the installation — see which, here, before committing.* A result reads, for example, *Would recompile docs-trino-policy. created 1 · updated 0 · deleted 0 · unchanged 4*. +Like grants, a rule needs a current **Preview** before **Create**. For a tag rule the preview text warns: *A tag rule is catalog-unscoped, so this can recompile every Trino profile in the installation — see which, here, before committing.* A result reads, for example, *Would recompile docs-trino. created 1 · updated 0 · deleted 0 · unchanged 12*. ## Edit and delete @@ -90,6 +90,18 @@ Two mask or row-filter rules that select the same column or table at the same pr Masks and filters are enforced by the engine, and **Who can read?** does not report them for Trino. To confirm a rule, query the table as a non-exempt user in SQL Lab, or compare the **Policies** tab of the engine, where each rule appears as a `Masking`, `Masking (by tag)` or `Row filter` policy compiled by egret. +On the demo stack the `sessions` table has three rules: the row filter *EU sessions only* (`country IN ('FRANCE', 'GERMANY', 'UNITED KINGDOM', 'ROMANIA', 'RUSSIA')`, exempt group `data-stewards`), the mask *Hide revenue outside finance* (**Nullify** on `revenue`, exempt group `finance-analysts`) and the tag rule *Hash personal data* (**Hash** on every column tagged *Personal data (PII)*, exempt group `data-stewards`). Queried through `docs-trino` as `member-sso`, who is in none of the exempt groups: + +```sql +SELECT country, count(*) FROM demo_warehouse.analytics.sessions GROUP BY 1; +-- 5 rows: GERMANY 554, FRANCE 375, RUSSIA 484, UNITED KINGDOM 655, ROMANIA 655 (of 15 000 sessions) + +SELECT session_id, user_id, revenue FROM demo_warehouse.analytics.sessions LIMIT 1; +-- 52486 | B25935C0F9C4BB9ABCFA13E4C15E56C16564B1390408CB5E24AC68140387D72C | NULL +``` + +The local platform admin `admin@egret.local` got the same masked values: platform admins have all access, but masks and row filters apply to everyone a rule does not exempt. + ## Limits and known issues - On a Trino profile without a tag service, tag rules compile to nothing. Egret refuses to bind such a profile to an engine. diff --git a/platform/data-governance/engines-and-policy/engine-permissions.md b/platform/data-governance/engines-and-policy/engine-permissions.md index ad87393..9530485 100644 --- a/platform/data-governance/engines-and-policy/engine-permissions.md +++ b/platform/data-governance/engines-and-policy/engine-permissions.md @@ -28,13 +28,15 @@ Values are concrete names. The form refuses *Enter a value.*, leading or trailin 2. On **Engine permissions**, press **Grant**. 3. Under **Capability**, pick one. The fields for its values appear. 4. Fill the values, for example `query_max_run_time` for a system session property. -5. Under **Granted to**, choose **user**, **group** or **role**, search, and add at least one principal (*Add at least one user, group or role.*). A group is expanded to its current members. +5. Under **Granted to**, choose **user**, **group** or **role**, search, and add at least one principal (*Add at least one user, group or role.*). A group is expanded to its current members. The search list stays open after you pick someone and covers the **Grant** button; click elsewhere in the dialog to close it. 6. Press **Grant**. -The permission appears under **Granted**, grouped by capability, with the **Values**, **Granted to**, the compiled **Policy** name (for example `perm-engine-8f0e…`) and edit and delete buttons. Editing opens **Edit engine permission**. Deleting shows *Delete engine permission* and the toast *Engine permission deleted.* +The permission appears under **Granted**, grouped by capability, with the **Values**, **Granted to**, the compiled **Policy** name (for example `perm-engine-33b6526b-…`) and edit and delete buttons. On the demo stack, granting **Set a system session property** `query_max_run_time` to `member-sso` on `docs-trino` added the row *query_max_run_time — user member-sso*, and the Governance overview recorded *Administrator created an engine grant: trino.set_system_session_property (query_max_run_time) on engine docs-trino to user member-sso*. Editing opens **Edit engine permission**. Deleting shows *Delete engine permission* and the toast *Engine permission deleted.* The engine applies the permission at its next plugin download. +![The Engine permissions tab of docs-trino: one granted session property, and the policies managed by egret](/img/platform/admin/gov-engine-permissions.png) + ## Check an engine permission **Check** opens **Check an engine permission**: *Asks what \'s policy set answers for one user. Nothing is changed.* Pick the **User** (the list shows each person's stored username, or *no allocated username yet*), the **Capability** and its values, then press **Check**. The answer is either the policy that grants it, or *No policy grants it.* For some capabilities the answer is *Cannot be decided ahead of time*, because the engine decides it from the request. @@ -50,7 +52,7 @@ Below **Granted**, **Managed by egret** lists policies egret seeds itself on the | **Platform admins: all access** | Platform admins have all access. | | **Group admin: all access (seeded defaults)** | The default policy set that gives the group `admin` all access. | -An external engine's profile shows *egret seeds no engine policy on this profile.* +On the provisioned engine `docs-trino` the section held 13 **Platform admins: all access** policies (the user `admin`, every resource kind) and 3 **Every user's baseline** policies. An external engine's profile shows *egret seeds no engine policy on this profile.* ## Not grantable here diff --git a/platform/data-governance/engines-and-policy/index.md b/platform/data-governance/engines-and-policy/index.md index edabe63..5b16789 100644 --- a/platform/data-governance/engines-and-policy/index.md +++ b/platform/data-governance/engines-and-policy/index.md @@ -6,7 +6,7 @@ sidebar_position: 4 **Governance → Engine & policy** is where data-access policy reaches the query engines. Each engine downloads one **policy profile**, a named bundle of Apache Ranger policies that egret compiles from the grants and rules you write. This page lists the engines and profiles; each engine has its own page for engine permissions, policies, overlaps, the Ranger plugin and advanced policies. -![Engine and policy: one engine bound to a profile, and a profile with no engine](/img/platform/admin/gov-engines-index.png) +![Engine and policy: the provisioned docs-trino and the external trino-demo-ext, and four profiles without an engine](/img/platform/admin/gov-engines-index.png) The area needs the **policy:read** permission. Writes are checked per control. @@ -24,19 +24,19 @@ The header counts **engines**, **profiles without an engine** and **hand-written |---|---| | **Engine** | The engine name. Select it to open the engine page. | | **Type** | Trino or StarRocks. | -| **Origin** | **External** for an engine you registered, or provisioned by egret. | +| **Origin** | **Provisioned** for an engine egret runs, **External** for one you registered. | | **Profile** | The profile the engine downloads, or *No profile* with the hint *Bind a profile in Control plane*. | -| **Plugin** | Whether the engine's plugin is up to date. *Not judged: external*, *Not judged: not running* or *Not judged: no profile* say why egret cannot tell: an external engine is not monitored, a stopped engine cannot download, and an engine with no profile has nothing to download. | +| **Plugin** | Whether the engine's plugin is up to date (**Up to date** for `docs-trino` on the demo stack). *Not judged: external*, *Not judged: not running* or *Not judged: no profile* say why egret cannot tell: an external engine is not monitored, a stopped engine cannot download, and an engine with no profile has nothing to download. | | **Engine grants** | How many engine permissions are granted on the engine. | | **Hand-written policies** | How many policies in the profile were written by hand in Ranger rather than compiled by egret. | ### Profiles without an engine -Profiles that no engine downloads yet, with **Profile**, **Type**, **Origin** and **Policies** columns. When every profile is bound the section says *Every profile is bound to an engine.* A profile can be deleted from here: the confirmation says *The profile and every policy in it are removed. No engine downloads it.* +Profiles that no engine downloads yet, with **Profile**, **Type**, **Origin** (**Manual** for one you created, **Generated** for one provisioning created) and **Policies** columns. The profile of a deleted provisioned engine stays here, with its policies. When every profile is bound the section says *Every profile is bound to an engine.* A profile can be deleted from here: the confirmation says *The profile and every policy in it are removed. No engine downloads it.* ## Create a policy profile -Provisioned engines get their profile automatically. For an engine you run yourself, you create the profile first. +Provisioned engines get their profile automatically: provisioning `docs-trino` created the Trino profile `docs-trino` and its tag service `docs-trino_tag`. For an engine you run yourself, you create the profile first. 1. Press **New profile**. 2. Enter a **Name**, for example `starrocks-prod`. @@ -45,7 +45,7 @@ Provisioned engines get their profile automatically. For an engine you run yours The profile appears under **Profiles without an engine**, with origin **Manual**. Bind it to an engine on the engine's **Settings** tab in the Control Plane (see [Register an external engine](../../control-plane/query-engines/register-external.md)). -![An unbound profile with no policies](/img/platform/admin/gov-profile.png) +![The unbound profile docs-external-policy with the 13 policies added by Seed default policies](/img/platform/admin/gov-profile.png) :::warning A profile you create here cannot be bound to a Trino engine: egret refuses with *policy profile … has no usable tag service … so every tag-based column mask and row filter would silently compile to nothing*. A Trino profile needs a tag service, which the console cannot set. Use a profile created by provisioning, or create the profile pair through the API: a profile of type `tag` and a Trino profile whose `config` names it in `tagService`. @@ -53,7 +53,7 @@ A profile you create here cannot be bound to a Trino engine: egret refuses with ## The engine page -Select an engine to open its page. The header shows the engine name, its **Type** badge, **External** (for an engine egret does not run), the bound **Profile**, the **Ranger plugin** version, and **Open in Control plane**. For an external engine a note reads *egret compiles this engine's policies but cannot see whether the external engine downloads them.* +Select an engine to open its page. The header shows the engine name, its **Type** badge, **Provisioned** or **External**, the bound **Profile**, the **Ranger plugin** version, and **Open in Control plane**. For an external engine a note reads *egret compiles this engine's policies but cannot see whether the external engine downloads them.* The tabs are: @@ -68,17 +68,17 @@ The tabs are: **Migrate** appears only while the profile holds a hand-written policy. An unbound profile has its own page with the frame of an engine page, but without **Engine permissions** and **Plugin**. A bound profile has no page of its own: its URL redirects to its engine. -![The Policies tab of docs-trino](/img/platform/admin/gov-engine-policies.png) +![The Policies tab of docs-trino: 28 policies, 12 compiled by egret and 16 seeded by egret](/img/platform/admin/gov-engine-policies.png) ## Make an engine serve a catalog -Policies reach an engine only for the catalogs declared on it. In **Control plane → Engines → *engine* → Catalogs**, declare the catalog. The Data page's **Engines** line then shows the engine, and the Governance overview stops reporting *catalog is attached to no engine*. See [Engine detail](../../control-plane/query-engines/engine-detail.md). +Policies reach an engine only for the catalogs it serves. In **Control plane → Engines → *engine* → Catalogs**, attach the catalog (provisioned engine) or declare it (external engine). The Data page's **Engines** line then shows the engine, and the Governance overview stops reporting *catalog is attached to no engine*. See [Engine detail](../../control-plane/query-engines/engine-detail.md). ## How a change reaches the engine 1. You save a grant, rule, clearance or engine permission. 2. Egret recompiles every profile it touches, in the same transaction. A compile that fails rolls the write back. -3. The engine's plugin polls egret on an interval and downloads the new policy set. Until then the engine enforces the previous one. +3. The engine's plugin polls egret on an interval (30 seconds by default) and downloads the new policy set. Until then the engine enforces the previous one. The engine's **Plugin** tab shows the version each plugin last reported; confirmation takes two polls, about a minute. Changing the profile bound to an engine recompiles both the old and the new profile. If that would leave two mask or row-filter rules deciding one column differently, egret refuses and changes nothing. diff --git a/platform/data-governance/engines-and-policy/migrate.md b/platform/data-governance/engines-and-policy/migrate.md index 701b19e..f3c54b9 100644 --- a/platform/data-governance/engines-and-policy/migrate.md +++ b/platform/data-governance/engines-and-policy/migrate.md @@ -8,9 +8,7 @@ The **Migrate** tab moves hand-written Ranger policies into egret's permission m Previewing, migrating and undoing all need **policy:update**. A reader sees the list but no controls. -:::note -This page describes the screen from the product source. It was not run on the demo instance, because the demo profile holds no hand-written policy. -::: +![The Migrate tab of trino-demo-ext: one migratable policy selected and a preview that proved it equivalent](/img/platform/admin/gov-engine-migrate.png) ## Why migrate @@ -23,11 +21,11 @@ The tab lists every policy in the profile with: | Column | Meaning | |---|---| | **Policy** | The policy name. *Switched off* appears under a disabled one. | -| **Owner** | What the policy is: who owns it. | +| **Owner** | Who owns the policy: **operator** for a hand-written one, **compiler** for one egret compiled. | | **Verdict** | **Migratable**, **Migrated**, **Locked** or **Blocked**. | | **Why** | The server's sentence explaining the verdict. | -A summary counts how many policies are migratable, migrated, locked and blocked. An empty profile shows *No policies in this profile* with *There is nothing to migrate: this profile holds no policies.* **Search policies…** filters the list. +The tab opens with *Profile \: who owns each policy, whether it can move into the permission model, and a proof before it does.* A summary counts the **policies** and how many are **migratable**, **migrated**, **locked** and **blocked**. An empty profile shows *No policies in this profile* with *There is nothing to migrate: this profile holds no policies.* **Search policies…** filters the list. | Verdict | Meaning | |---|---| @@ -41,10 +39,10 @@ Every sentence explaining a verdict, a blocker, a disagreement or a refusal come ## Migrate policies 1. Tick the migratable policies you want. The line below the table reads *Selected: …* or *No policy selected.* -2. Press **Preview**. Egret runs a dry-run migration and compares the engine's decisions before and after on a set of probe requests. +2. Press **Preview migration**. Egret runs a dry-run migration and compares the engine's decisions before and after on a set of probe requests. 3. Read the result. A proved migration says *Proved equivalent: N probe evaluations agreed on every affected profile. Nothing has been written yet.* -4. Read **Before you migrate** (notices), **Affected profiles** and, if any, **Disagreements**. -5. Press **Migrate**. The migration is written, and appears under **Recent changes** on the Governance overview. +4. Read **Before you migrate** (notices), the profiles it changes and, if any, **Disagreements**. +5. Press **Migrate**. There is no further confirmation. The page says *The migration is written. See it in Recent changes*; the policy's verdict becomes **Migrated**, and the compiled policy that replaces it is listed as **Locked** with the owner **compiler**. **Migrate** is enabled only by a proved, current preview of the exact selection. Otherwise the reason is shown: @@ -54,17 +52,27 @@ Every sentence explaining a verdict, a blocker, a disagreement or a refusal come - *The preview was refused, so this selection cannot be migrated.* - *The preview did not prove this migration, so it cannot be migrated.* -### Affected profiles +### The profiles a migration changes -A grant compiles into every profile that serves its catalog, and a tag rule into every profile sharing the tag service, so a migration can change more than the profile on screen. **Affected profiles** lists each one with its **Plugin versions**, its **Disagreements** count and what it **No longer serves**; profiles other than the current one are marked **Also changed**. +A grant compiles into every profile that serves its catalog, and a tag rule into every profile sharing the tag service, so a migration can change more than the profile on screen. The preview lists them under *This migration changes N profile(s)* with the columns **Profile**, **Plugin versions**, **Probes**, **Disagreements**, **Now serves** and **No longer serves**; profiles other than the current one are marked **Also changed**. **Policies and what they become** sums it up, for example *legacy: analysts read finance_pg: 1 grant, 0 rules*. ### Disagreements A disagreement is a probe request the engine would answer differently after the migration, per plugin version and access type. **Migrate** stays disabled unless the preview proves equivalence. Leave such a policy hand-written. +### Example + +On the demo stack the external engine `trino-demo-ext` downloads the profile `docs-legacy-trino`, which holds the hand-written policy `legacy: analysts read finance_pg` (user `member-sso`, `select` on `finance_pg.*.*.*`). + +- With the permissions `select`, `use` and `show`, the policy was **Blocked**: *Access type `use` has no equivalent in egret's permission model on this engine: egret grants only read and write, each a fixed set of Ranger access types, and spelling is matched exactly (`SELECT` is not `select`). Remove `use` from the policy, or keep the policy hand-written.* +- With `select` only it was **Migratable**: *Becomes 1 grant and 0 rules.* The preview said *Proved equivalent: 1575 probe evaluations agreed on every affected profile. Nothing has been written yet.*, with *This migration changes 1 profile* (`docs-legacy-trino`, Ranger 2.8.0, *1575 of 1875 without pruning* probes, 0 disagreements). +- After **Migrate** the row said *This policy was migrated into egret's permission model on … by admin@egret.local (1 grant(s), 0 rule(s), proved on 1575 probe evaluations) and switched off; the model now enforces what it did.* + +The same policy on the catalog `tpch_bench` could not be proved: `tpch_bench` is also served by the provisioned engine `docs-trino`, so the proof had to cover both profiles, and the preview said *The gate could not prove this migration. Nothing was written. proving this migration needs 48735 probe evaluations across 2 profile(s), 2697570 units of work …, over the limit of 1300000.* A grant compiles into every profile that serves its catalog, so a hand-written policy on a widely served catalog may be too large to prove. + ## Undo a migration -On a migrated policy press **Undo migration of \**. The dialog is *Undo the migration of "\"?*. Undo removes the generated grants and rules and switches the original policy back on. A summary of what was restored is shown. +On a migrated policy press **Undo**. The dialog *Undo the migration of “\”?* says *This deletes the grants and rules the migration created and switches the original policy back on. It is refused if any of them has been edited since.* Press **Undo migration**. The page then says, for example, *Undid “legacy: analysts read finance_pg”: deleted 1 grant and 0 rules, and switched the original back on.* Every profile the migration recompiled stays compiled by egret, so no new hand-written policy can be created in it. If the profiles changed since the preview, a migration reports *Migrating found that the profiles changed since the preview* and writes nothing. diff --git a/platform/data-governance/engines-and-policy/policies-and-overlaps.md b/platform/data-governance/engines-and-policy/policies-and-overlaps.md index 79641e4..c7814e3 100644 --- a/platform/data-governance/engines-and-policy/policies-and-overlaps.md +++ b/platform/data-governance/engines-and-policy/policies-and-overlaps.md @@ -6,11 +6,11 @@ sidebar_position: 2 The **Policies** tab of an engine (or of an unbound profile) shows every policy in the profile the engine downloads, whatever its source. The **Overlaps** tab reports mask and row-filter rules that could decide the same request two ways. Together they are the place to check what egret actually put in front of the engine. -![The Policies tab of docs-trino: five policies compiled by egret](/img/platform/admin/gov-engine-policies.png) +![The Policies tab of docs-trino: 28 policies, 12 compiled by egret and 16 seeded by egret](/img/platform/admin/gov-engine-policies.png) ## The Policies tab -The summary line counts policies by source, for example *5 policies: 5 egret-compiled · 0 egret-seeded · 0 hand-written*. An empty profile says *This profile has no policy.* +The summary line counts policies by source, for example *28 policies: 12 egret-compiled · 16 egret-seeded · 0 hand-written* for the provisioned engine `docs-trino` on the demo stack. The 16 egret-seeded policies give the engine's `admin` user all access (`egret-admin - catalog`, `egret-admin - catalog, schema` and so on). An empty profile says *This profile has no policy.* **Search policies** matches a name, resource or principal. Two filters narrow the list: @@ -26,7 +26,7 @@ An empty result reads *No policy matches these filters.* | **Name** | The Ranger policy name. Compiled policies are named after their source, for example `perm-grant-8c01…` or `perm-rule-ae87…`. | | **Source** | **egret-compiled** (*Compiled by egret; change its source*), **egret-seeded** (*Managed by egret (engine delegation, admin access or query limits)*) or **hand-written** (*Not managed by egret*). | | **Type** | **Access**, **Masking**, **Masking (by tag)** or **Row filter**. | -| **Resource** | The catalog, schema, table and column, for example `demo_catalog.analytics.sessions.session_id`, or `tag: Personal data (PII)` for a tag policy. | +| **Resource** | The catalog, schema, table and column, for example `demo_warehouse.analytics.sessions.session_id`, or `tag: Personal data (PII)` for a tag policy. | | **Principals** | Who the policy names. Egret writes usernames; compiled masks and row filters show `public`. | | **Enabled** | Whether the engine uses it. | @@ -38,13 +38,13 @@ A hand-written policy can be switched on or off (*Policy enabled.* / *Policy dis A mask or row filter selects a column or table. When two of them select the same one at the same priority, the engine cannot tell which should win, so the result depends on evaluation order. Egret calls such a pair an **overlap**. -The tab's badge counts the pairs. An empty tab says *No overlapping rules on this profile.* Each pair is listed as, for example, **Mask** on `demo_catalog.analytics.sessions.user_id`, with links to the two **rule** or **policy** members and, when known, which one the engine serves today. A sentence explains why they collide. +The tab's badge counts the pairs. An empty tab says *No overlapping rules on this profile.* Each pair is listed as, for example, **Mask** on `demo_warehouse.analytics.sessions.user_id`, with links to the two **rule** or **policy** members and, when known, which one the engine serves today. A sentence explains why they collide. **How to fix** says: delete one of the two, or narrow one so they no longer select a common resource. If either rule selects by classification tag, unassigning the tag from the column where they meet also breaks the pair. An edit that narrows a member but still meets the other is refused as a new overlap. For a hand-written member, a **Switch off policy** link takes you to its editor, where the **Enabled** switch turns it off. ### How overlaps are prevented -- A write that would create an overlap is refused with a conflict (409) naming the pair. +- A write that would create an overlap is refused with a conflict (409) naming the pair. The rule dialog's **Preview** already shows it. On the demo stack a path mask *Redact visitor ids* on `sessions.user_id`, a column that carries *Personal data (PII)*, previewed as: *recompiling profile "docs-trino" would emit an order-ambiguous pair: rule "Hash personal data" (a tag mask on tag "Personal data (PII)") and rule "Redact visitor ids" (a resource mask) both decide demo_warehouse.analytics.sessions.user_id … Ranger runs the tag phase first, and a tag mask shuts out every resource mask at the same or a lower policyPriority …* - Pairs that already existed before this check are only reported here, never refused retroactively. - The Governance overview raises a **rule overlaps** finding, and **Open** takes you to the pair. diff --git a/platform/data-governance/engines-and-policy/policy-editor.md b/platform/data-governance/engines-and-policy/policy-editor.md index caad02e..df9e337 100644 --- a/platform/data-governance/engines-and-policy/policy-editor.md +++ b/platform/data-governance/engines-and-policy/policy-editor.md @@ -6,7 +6,7 @@ sidebar_position: 3 Most policies in a profile are compiled by egret from your grants and rules. A **hand-written policy** is a raw Apache Ranger policy you write yourself in the **Advanced** tab. Use it for the few things the permission model cannot express, and prefer [grants](../data/grants.md), [rules](../data/masks-and-filters.md) and [engine permissions](./engine-permissions.md) for everything else. -![The Advanced tab of docs-trino with Create policy and Seed default policies](/img/platform/admin/gov-engine-advanced.png) +![The Advanced tab of docs-trino: egret compiles this profile, so new hand-written policies are refused](/img/platform/admin/gov-engine-advanced.png) The tab opens with a fixed warning: *Hand-written policies are outside egret's model: egret does not recompile them, does not check overlaps with grants, and Data and People pages will mark them hand-written. Prefer Engine permissions or grants on Data.* @@ -22,15 +22,27 @@ Creating, editing and deleting need **policy:create**, **policy:update** and **p Once egret compiles a profile, new hand-written policies on it are refused: *egret compiles this profile, so new hand-written policies are refused. Existing ones can still be edited, enabled, disabled and deleted.* Hand-written policies are therefore mainly for profiles of engines that egret does not compile for, and for policies that existed before. ::: -**Seed default policies** adds egret's default Trino set (group `admin`: all access). Existing policies are not changed. If the profile already has policies, egret says *No default policy was added: the profile already has policies.* +**Seed default policies** adds egret's default Trino set (*group admin: all access*). Existing policies are not changed. If the profile already has policies, egret says *No default policy was added: the profile already has policies.* On an empty unbound Trino profile it added 13 policies (`all - catalog`, `all - function` and so on). They are listed on the **Policies** tab as **egret-seeded**, not under **Hand-written policies**, and the button gives no confirmation, so the Advanced tab looks unchanged. -The **Hand-written policies** table lists each policy with its type, resource and whether it is **Enabled**. +The **Hand-written policies** table lists each policy with its **Name**, **Type**, **Resource** (for example `finance_pg.*.*.*`) and whether it is **Enabled**. Select a name to open it in the editor. + +### Example: a hand-written access policy + +On the demo stack, on the unbound Trino profile `docs-legacy-trino` (created through the API with a tag service): + +1. **Advanced → Create policy → Access**. +2. **Policy name** `legacy: analysts read finance_pg`. +3. In **Resource**, type `finance_pg` in **Catalog** and press Enter. Keep `*` for **Schema**, **Table** and **Column**; the compact path reads `finance_pg.*.*.*`. +4. In **Permissions → Allow conditions**, type `member-sso` under **Users** and press Enter, then **Add permissions**, tick `select` and confirm with the check mark. +5. **Save**. You return to **Advanced**, and the profile now has a **Migrate** tab. + +A policy that also allows `use` and `show` is listed on **Migrate** as **Blocked**, because the permission model has only read and write (see [Migrate policies](./migrate.md)). ## The policy editor The editor is a page with a header and three sections in a side list: **Resource**, **Permissions** (**Masking** or **Row filter** for those types) and **Details**. Each shows a count of what is set. -![The policy editor for a new access policy: policy name, enabled switch and the Resource section](/img/platform/admin/gov-policy-editor.png) +![The policy editor for the hand-written access policy legacy: analysts read finance_pg, Resource section](/img/platform/admin/gov-policy-editor.png) ### Header @@ -62,6 +74,8 @@ For a masking or row-filter policy the section edits the mask or the filter item Roles and groups named in a hand-written policy match nobody on engines egret serves, because egret runs no usersync and serves no roles download. Name users. The group `public` is the exception. See [How permissions work](../concepts.md#what-does-not-reach-the-engine). ::: +![The Permissions section: member-sso allowed select, with Add row, Exclude from allow and Deny conditions](/img/platform/admin/gov-policy-editor-permissions.png) + ### Details | Field | Meaning | diff --git a/platform/data-governance/engines-and-policy/ranger-plugin.md b/platform/data-governance/engines-and-policy/ranger-plugin.md index 5446d3d..ad8c794 100644 --- a/platform/data-governance/engines-and-policy/ranger-plugin.md +++ b/platform/data-governance/engines-and-policy/ranger-plugin.md @@ -4,34 +4,50 @@ sidebar_position: 5 # Ranger plugin -The **Plugin** tab of an engine's page tells you how to connect an engine you run yourself to egret's policies. It shows what to install on the engine, what egret assumes, and what you must supply. For an engine egret provisions, the plugin is configured for you. +Every engine enforces egret's policies through an Apache Ranger plugin that downloads its policy profile from egret. For an engine egret provisions, the plugin is configured for you. For an engine you run yourself, egret gives you the files to install, says what it assumes and what you must supply, and shows which plugins have downloaded. -![The Plugin tab: service name, download URL, settings and the two configuration files](/img/platform/admin/gov-engine-plugin.png) +![The Plugin tab of docs-trino: plugin 2.8.0, both downloads up to date, and the plugins that reported](/img/platform/admin/gov-engine-plugin.png) -The tab shows only for an engine bound to a policy profile. With no profile, the Control Plane's **Plugin config** tab reads *engine … is bound to no policy profile: bind a profile first*. +## The Plugin tab in Governance -## Install the plugin on your engine +The **Plugin** tab of an engine in **Governance → Engine & policy** shows whether the engine's plugins hold the current policy set: -1. On the engine page, select **Plugin**. +- The declared **Ranger plugin** version and whether egret models it (*Modelled by egret*). +- **Download state**: one row each for **Policies** and **Tags**, with the **State** (for example **Up to date**: *Plugins of profile docs-trino report holding policy version 65.*), **Last download**, **Current version**, **Newest reported** and **Served**. *A version counts as held once a plugin reports it; confirmation takes two polls, about a minute at the default 30 s poll.* +- **Downloads reported by plugins**: each plugin instance (for example `trino@4ae60563e301-docs-trino`), the kind, the last known and served version, and when. A provisioned engine that was restarted shows its old container's plugin as well, with an older version. +- **Download Trino config** downloads the plugin files. + +For an external engine the state reads *egret does not judge external engines: it does not run them and cannot tell whether they should be polling.*, and the list says *No plugin has downloaded from this profile in the last 24 hours.* until your engine downloads. + +A provisioned engine needs nothing else: egret installs and configures the plugin. The tab shows only for an engine bound to a policy profile. + +## Install the plugin on an external engine + +The installation files are on the engine's **Plugin config** tab in the Control Plane (**Control Plane → Engines → \ → Plugin config**). With no profile bound it reads *engine … is bound to no policy profile: bind a profile first*. + +![The Plugin config tab of trino-demo-ext: service name, download URL, warnings and the two configuration files](/img/platform/admin/cp-engine-plugin.png) + +1. Open the external engine in the Control Plane and select **Plugin config**. 2. Note the **Ranger service name** (the profile name) and the **Policy download URL**. 3. Copy the files under **Files**: `access-control.properties` and `ranger-trino-security.xml` for Trino. Each has **Copy** and **Download** buttons. 4. Install them on the coordinator, restart it, and make sure the engine can reach the download URL. -5. Check the Governance overview: the plugin liveness checks move from *Not judged* once the engine downloads. +5. Check the Governance **Plugin** tab: the plugin appears under **Downloads reported by plugins** once it has downloaded. ### Install on your engine | Field | Meaning | |---|---| -| **Ranger service name** | The profile name the plugin asks for, for example `docs-trino-policy`. | +| **Ranger service name** | The profile name the plugin asks for, for example `docs-legacy-trino`. | | **Policy download URL** | `/service/plugins/policies/download/`. | -| **Authentication** | Whether egret requires a bearer token on the download. Egret does not show or issue the token. | +| **Authentication** | Whether egret requires a bearer token on the download, for example *The policy download is anonymous: no token is needed.* Egret does not show or issue a token. | | **Declared plugin version** | The Ranger plugin version recorded for the engine. | ### Warnings The tab raises warnings when something will stop the download: -- The download URL is a loopback or unspecified address, so no external engine can reach it. The URL comes from the `[catalog] public_url` setting, which defaults to localhost on a fresh install. Set it to an address the engine can reach. +- The download URL is a loopback, unspecified or in-cluster address, so an engine elsewhere cannot reach it. The URL comes from the `[catalog] public_url` setting. On the demo stack it is `http://egret:8080`: *The download URL (http://egret:8080) is an in-cluster address: an engine outside the cluster cannot reach it.* +- The download is anonymous: *anyone who can reach this URL can download every profile's policies. Restrict it at the network level.* - Egret requires a bearer token on `/service`, and a stock Ranger plugin cannot send it, so the download is refused (401) unless something in front of egret adds the token or exempts the engine. ### You must supply @@ -58,13 +74,13 @@ If any assumption is false on your engine, explain can answer *Allowed* where th Egret proves migrations and explains access only for Trino with plugin 2.5.0 or 2.8.0 and StarRocks with 2.8.0 or 2.9.0. Another version is recorded as you declare it, and those features refuse. The governance overview flags an unmodelled version as a finding. -## Where the same information is in the Control Plane +## Plugin settings in the Control Plane -The Control Plane engine page has a **Plugin config** tab with the same content, and its **Settings** tab sets the plugin version. See [Engine detail](../../control-plane/query-engines/engine-detail.md). +The Control Plane engine page has the installation view on **Plugin config** (external engines only), and its **Settings** tab sets the declared plugin version. See [Engine detail](../../control-plane/query-engines/engine-detail.md). ## Limits and known issues -- On the demo instance the download URL is `http://localhost:8080`, which a remote engine cannot reach, and the bearer-token warning is shown. Both warnings are expected locally. +- On the demo stack the download URL is `http://egret:8080`, which only containers on the demo network can reach, and the download is anonymous. Both warnings are expected there. The demo Trino `trino-demo` has no Ranger plugin, so registering it as `trino-demo-ext` enforces nothing. - Egret does not know whether an external engine ever downloaded its profile. ## Further reading diff --git a/platform/data-governance/lineage.md b/platform/data-governance/lineage.md index 1ba33f5..0672ce8 100644 --- a/platform/data-governance/lineage.md +++ b/platform/data-governance/lineage.md @@ -16,10 +16,10 @@ Lineage tells you which tables a view reads and which views read a table. Egret The working lineage view is the **Lineage** tab on a table, view or materialized view in **Governance → Data**. -![The Lineage tab of the sessions table: it reads from nothing and nothing reads it](/img/platform/admin/gov-table-lineage.png) +![The Lineage tab of the sessions table: no recorded readers, and a warning that one view's lineage could not be computed](/img/platform/admin/gov-table-lineage.png) - **Reads from** lists the base tables of a view. -- **Read by** lists the views and materialized views that read a table. An empty list says *No view or materialized view reads this table*, or *No reader is known* when the lineage is incomplete. +- **Read by** lists the views and materialized views that read a table. An empty list says *No view or materialized view reads this table*. When a view's lineage is missing, the tab warns instead: *This list may be incomplete. 1 view in egret's catalogs has lineage egret does not know (failed or not yet computed); it may read this table* and *No reader is known.* - A table with no record says *egret has no lineage recorded for this object*. Each view carries how its lineage was obtained: @@ -34,7 +34,9 @@ Lineage here is table-level, not column-level: it says which tables a view reads ## Why it matters for governance -When a view is created in an egret catalog, egret uses its lineage to tag the view with the classifications of its base tables. If lineage fails to compute, the Governance overview raises a **view lineage failed** finding. See [View exposure](./classifications/view-exposure.md) and [Data](./data/index.md#lineage). +When a view is created in an egret catalog, egret uses its lineage to tag the view with the classifications of its base tables. If lineage fails to compute, the Governance overview raises a **view lineage failed** finding. + +On the demo stack the view `demo_warehouse.analytics.revenue_by_country` reads `sessions`, but egret could not compute its lineage: reading the view through the provisioned engine was refused with *Access Denied: Cannot access catalog demo_warehouse*. So the **Lineage** tab of `sessions` shows the warning above instead of the view, and the view is listed on the tag's **View exposure** tab as *may read classified data*. See [View exposure](./classifications/view-exposure.md) and [Data](./data/index.md#lineage). ## Further reading diff --git a/platform/data-governance/overview.md b/platform/data-governance/overview.md index 4b1fd1c..9d8eae9 100644 --- a/platform/data-governance/overview.md +++ b/platform/data-governance/overview.md @@ -41,9 +41,14 @@ The checks the server runs are: | Classifications without clearance | Tags that gate nothing (**Info**). | | Catalogs attached to no engine | No engine serves the catalog (**Info**). | -On the demo instance before an engine is declared, the only finding is the last one: `demo_catalog` is attached to no engine, so no engine serves it. Declaring the catalog on an engine (see [Engine page tabs](./engines-and-policy/index.md#the-engine-page)) clears it, and the list then reads *Nothing needs attention.* +On the demo stack, with the engine `docs-trino` serving `demo_warehouse`, the list held two findings: -When your permissions do not allow a check, the page says so with a **Not checked here** line that names what it skipped, for example views that may expose classified data through `DEFINER`, and classification coverage. Those two are covered on the [View exposure](./classifications/view-exposure.md) page. +- **Warning**: *egret could not compute the lineage of 1 view, so the tables it reads are not known.*, with the view and the engine's answer: `demo_warehouse.analytics.revenue_by_country — engine docs-trino: engine: Access Denied: Cannot access catalog demo_warehouse`. Egret reads view definitions through the engine, and the engine's policy did not let egret's own account into the catalog. +- **Info**: *3 classifications have no clearance: they gate nothing.*, listing the tags. + +Before an engine serves a catalog, the finding is that the catalog is attached to no engine. When everything is in order the list reads *Nothing needs attention.* + +A **Not checked here** line names what the page does not check: views that may expose classified data through `DEFINER`, and classification coverage. It appears for platform admins too. Those two are covered on the [View exposure](./classifications/view-exposure.md) page. When your permissions do not allow one of the checks above, it is added to that line. :::note Each check needs its own permissions: engine drift checks need **policy:read** and **engine:read**, the catalog attachment check needs **catalog_registry:read**, and so on. A check you cannot run is listed as not checked, not silently dropped. @@ -51,13 +56,13 @@ Each check needs its own permissions: engine drift checks need **policy:read** a ## Recent changes -**Recent changes** lists governance changes as they are recorded, newest first, with the columns **When**, **Who** and **What**, and a **View** link that opens the object concerned. The text describes each change in a sentence, for example *Administrator created rule Hash session ids: mask MASK_HASH on demo_catalog.analytics.sessions.session_id*, *Administrator set the clearance of classification Personal data (PII): roles data_steward; groups none* or *Administrator created an engine grant: trino.set_system_session_property (query_max_run_time) on engine docs-trino to user member-sso*. +**Recent changes** lists governance changes as they are recorded, newest first, with the columns **When**, **Who** and **What**, and a **View** link that opens the object concerned. The text describes each change in a sentence, for example *Administrator created a grant: allow read on demo_warehouse.finance to group finance-analysts, group data-team*, *Administrator set the clearance of classification Confidential: roles data_steward; groups none*, *Administrator created an engine grant: trino.set_system_session_property (query_max_run_time) on engine docs-trino to user member-sso* or *Administrator migrated legacy: analysts read finance_pg of profile docs-legacy-trino into egret grants and rules*. The link column reads **View**, or *(deleted)* when the object no longer exists. The list is paged: use the button at the bottom to load older changes. Under the list a note says *The oldest change you can see is from \*, and that changes older than 90 days are pruned. A second note warns that role assignments and catalog tags are recorded after they commit and can be missing if the audit writer was overloaded. Recording began when this version of egret was installed, so a fresh instance shows *No governance change has been recorded yet*. A change made by an administrator acting as another user is shown as "Admin as User". A change by a deleted user is shown as *A deleted user*, and a change made by the platform itself, for example a recompile, as *egret*. -![Recent changes on the demo instance after creating a grant, rules, a tag, an engine grant and a clearance](/img/platform/admin/gov-overview.png) +![The Governance overview on the demo stack: two findings and the most recent governance changes](/img/platform/admin/gov-overview.png) ## The Governance sidebar diff --git a/platform/data-governance/people-and-groups/check-access.md b/platform/data-governance/people-and-groups/check-access.md index e055eae..cdcd8d2 100644 --- a/platform/data-governance/people-and-groups/check-access.md +++ b/platform/data-governance/people-and-groups/check-access.md @@ -6,7 +6,7 @@ sidebar_position: 1 The **Check access** tab on a person, group or role page asks the engines a precise question: can this principal perform this action on this target? It uses the same evaluator as [Explain access](../data/explain-access.md), but lets you choose any catalog, schema, table or column and either action. -![Check access for member-sso: the action and the target pickers](/img/platform/admin/gov-person-check.png) +![Check access for Member SSO on demo_warehouse.analytics.customers: No policy matched on docs-trino](/img/platform/admin/gov-person-check.png) Check access needs **policy:read**. A user with no username cannot be checked: the tab says *No username yet, so no engine knows this user.* Grant them something first so a username is allocated. @@ -20,6 +20,8 @@ Check access needs **policy:read**. A user with no username cannot be checked: t The result is the same per-profile answer as the Data page's **Who can read?**: a verdict (**Allowed**, **Refused by a deny item**, **Refused by deny-all-else**, **No policy matched**, **Undecidable**), the deciding grant or policy, and the tags that could decide. See [Explain access](../data/explain-access.md#read-the-result-for-a-user) for what each means. +On the demo stack, **read** on `demo_warehouse.analytics.customers` for `member-sso` answers **No policy matched** on `docs-trino` (*Nobody granted this — not the same as somebody forbidding it.*), while `demo_warehouse.analytics.sessions` answers **Allowed** with **View grant** and the Ranger policy `perm-grant-d7441a5e-…`. A query on `customers` as her is refused by the engine. + ## Check a group or role For a group or role the tab asks about every member. The result counts members allowed by every profile, refused by at least one, undecidable, and masked, with a **Refused only** filter. A group with no members asks nobody. diff --git a/platform/data-governance/people-and-groups/index.md b/platform/data-governance/people-and-groups/index.md index d226628..2b50211 100644 --- a/platform/data-governance/people-and-groups/index.md +++ b/platform/data-governance/people-and-groups/index.md @@ -6,7 +6,7 @@ sidebar_position: 3 **Governance → People & groups** answers "what reaches this person?" It lists the users, groups and roles that egret knows, and for each one shows the grants, rules, clearances and engine permissions that apply to them. -![The Users tab of People and groups with four users and their usernames](/img/platform/admin/gov-people-users.png) +![The Users tab of People and groups: twelve people with their emails and engine usernames](/img/platform/admin/gov-people-users.png) The area needs the **policy:read** permission. A steward who holds only **tag:read** is refused. Editing a grant or rule from a person's page needs **policy:update**. @@ -36,6 +36,8 @@ The page has four tabs, kept in the address as `?kind=`: users (the default), `g A footnote reads *A member with no username gets one when a grant naming the group is compiled.* OIDC group membership changes at login. +![The Groups tab: local groups and two groups synced from OIDC, one with a member who has no username yet](/img/platform/admin/gov-people-groups.png) + ### Roles | Column | Meaning | @@ -44,7 +46,9 @@ A footnote reads *A member with no username gets one when a grant naming the gro | **Scopes** | **platform** or **workspace**. | | **Holders** | How many users hold it, with how many are *not on engines: no username*. | -The footnote says *Holders are users. A holder with no username gets one when a grant naming the role is compiled; a group holding a role is not counted, as engines do not apply it.* +The footnote says *Holders are users. A holder with no username gets one when a grant naming the role is compiled; a group holding a role is not counted, as engines do not apply it.* On the demo stack `data_steward` shows *1 holder*: Farid Haddad, who holds it directly. The group `data-stewards` also holds the role, but its members are not counted. + +![The Roles tab: platform and workspace roles with their holder counts](/img/platform/admin/gov-people-roles.png) ### Unmatched names @@ -63,7 +67,7 @@ Select a row to open its page. The header shows the name, the kind (**USER**, ** - **Grant access** opens the [grant dialog](../data/grants.md) with this principal already included. - **Manage in Control Plane** opens the matching user, group or role page in the Control Plane. -![The page of member-sso: username, status and the Overview tab](/img/platform/admin/gov-person-overview.png) +![The page of Member SSO: her group, her roles and workspaces, and whether each applies on engines](/img/platform/admin/gov-person-overview.png) A user's header also shows the email, **Username** and the status (for example **Active**). A group's header shows its source (*OIDC: memberships change at login*) and member count. A role's header shows *Exists in platform* and the number of holders. @@ -71,7 +75,7 @@ A user, a group and a role each have these tabs. A role has no **Masks & filters ### Overview -- **User**: **Groups** (with an **Applied on engines** column), **Roles**, and **Workspaces**. *No roles.* and *No workspaces.* show when empty. +- **User**: **Groups** (with an **Applied on engines** column), **Roles** and **Workspaces**, each with **Through** (*Direct* or the group), **Expires** and **Applied on engines**. Roles include object-level grants, for example a `ws_viewer` on one dashboard. A membership approved with an expiry shows its date, for example *2027-03-31*. *No roles.* and *No workspaces.* show when empty. - **Group**: **Members** (name and username; a member without one reads *No username yet: a grant naming this group allocates one at compile*), **Roles held** and **Workspaces held**, each with **Scope**, **Expires** and **Applied on engines**. - **Role**: **Holders** and **Groups holding this role**. @@ -81,7 +85,7 @@ A user, a group and a role each have these tabs. A role has no **Masks & filters The egret grants that reach the principal, with filters for **Effect** and **Action**. Grants are grouped by how they reach the principal: **Directly**, or through a group or role they hold. Each row shows the grant, **How it reaches** (for example *Directly · applied on engines*) and the **Effect**. Below them, **Engine permissions** lists engine permissions granted to the principal; *No engine permission reaches this user.* when none. -![The Permissions tab of member-sso: a direct grant that is applied on engines](/img/platform/admin/gov-person-permissions.png) +![The Permissions tab of Member SSO: a direct grant, and two grants that reach her through the group data-team](/img/platform/admin/gov-person-permissions.png) ### Masks & filters @@ -89,13 +93,17 @@ Rules that **exempt** the principal. *No egret rule exempts Member SSO* is follo ### Clearances -For each gated classification, whether the principal is cleared. Without any gated tag the tab says *No classification is gated*. The table columns are **Classification**, **Gate requires** and a clearance verdict: **Cleared**, *Cleared through public*, or **Not cleared** with a reason, such as: +For each gated classification, whether the principal is cleared. Without any gated tag the tab says *No classification is gated*. The table columns are **Classification**, **Gate requires**, and two verdicts: **Engines** (what the query engines enforce) and **egret objects** (what egret enforces on datasets, charts and dashboards). A verdict is **Cleared** (for example *Cleared through role data_steward*), *Cleared through public*, or **Not cleared** with a reason, such as: - *no username yet* - *the required role is held through a group, and engines count only roles held directly* - *the required role has expired. Engines may apply it until the next recompile.* - *engines enforce only the gate's roles, not its groups* +For example, Farid Haddad is **Cleared through role data_steward** for *Confidential*, because the role is given to him directly. + +![The Clearances tab of Farid Haddad: cleared for Confidential through the role data_steward](/img/platform/admin/gov-person-clearances.png) + See [Clearance](../classifications/clearance.md). ### Check access diff --git a/platform/workspace-settings/access-requests.md b/platform/workspace-settings/access-requests.md index d0a6c13..7507ad0 100644 --- a/platform/workspace-settings/access-requests.md +++ b/platform/workspace-settings/access-requests.md @@ -6,7 +6,7 @@ sidebar_position: 6 **Workspace settings → Access → Access requests** is the queue of people asking to join the workspace. A workspace admin approves a request, optionally time-boxed, or denies it. -![The queue with one pending request: who, the requested role and the reason](/img/platform/admin/ws-requests.png) +![The queue of Finance analytics with two pending requests: who, the requested role and the reason](/img/platform/admin/ws-requests.png) You need to be a workspace admin or a platform admin. With no requests waiting the page says *No pending requests.* @@ -21,7 +21,7 @@ A signed-in user who is not a member of a workspace asks from the workspace swit 5. Write a **Reason**. 6. Press **Submit request**. A toast says *Access request sent*. -![The Request access dialog as member-sso](/img/platform/admin/ws-request-dialog.png) +![The Request access dialog as member-sso, with her approved request under Your requests](/img/platform/admin/ws-request-dialog.png) The dialog says *Ask a workspace admin to add you. They'll review and approve or deny the request.* **Your requests** below lists each request with a badge: **Pending**, **Approved**, **Denied** or **Cancelled**. A pending one has **Cancel**. Asking twice, or while a member, shows *You already requested or are a member.* @@ -46,12 +46,18 @@ Press **Deny**. The dialog **Deny request** says *\ will not be added to ## Example -A finance analyst, member-sso, belongs to *Docs sandbox* only. She opens **Request access…**, picks *Finance analytics* with the reason *I build the monthly revenue dashboards for finance and need to create charts there.* The workspace admin opens **Access requests**, presses **Approve**, keeps `ws_viewer` and confirms. *member-sso@egret.local* now appears on Members of Finance analytics with `ws_viewer`. +On the demo stack, member-sso belongs to *Docs sandbox* only. She opens **Request access…**, picks *Finance analytics* with the reason *I need to build dashboards on the orders dataset for the sales review.* The workspace admin opens **Access requests**, presses **Approve**, keeps `ws_viewer`, sets **Access expires** to 31 March 2027 and confirms. *member-sso@egret.local* now appears on Members of Finance analytics with `ws_viewer`, and her **Your requests** list shows **Approved**. On her People page in Governance the membership shows the expiry date. + +![The Members page of Finance analytics after the approval: member-sso is a ws_viewer, added by the admin](/img/platform/admin/ws-members-finance.png) + +No mail is sent for a request or a decision: the requester sees the result in the dialog, and admins see the queue only when they open it. ## Where else requests appear Platform admins also see a workspace's requests on **Control Plane → Workspaces → \ → Access requests**, with the same actions. +![The Access requests tab of Finance analytics in the Control Plane](/img/platform/admin/cp-workspace-requests.png) + ## Limits and known issues - A request can ask for `ws_viewer` or `ws_editor` only. An admin can choose a different role when approving. diff --git a/platform/workspace-settings/databases.md b/platform/workspace-settings/databases.md index 26ecf6c..291e397 100644 --- a/platform/workspace-settings/databases.md +++ b/platform/workspace-settings/databases.md @@ -6,7 +6,7 @@ sidebar_position: 4 A **database connection** (shown in the UI as an *engine connection*) tells the workspace how to reach a query engine. SQL Lab, datasets, charts and dashboards run their queries through these connections. They are configured under **Manage → Workspace settings → Connections → Engines**. -![The Engines page of the Docs sandbox workspace with one connection](/img/platform/admin/ws-engines.png) +![The Engines page of the Docs sandbox workspace: the provisioned engine docs-trino and the connection trino-sandbox](/img/platform/admin/ws-engines.png) *Query engine connections this workspace can use in SQL Lab and charts.* The page needs workspace-admin rights; without them it says *Requires workspace-admin to manage engines.* A new workspace says *No engines connected yet.* @@ -25,7 +25,7 @@ A summary strip counts the **connections**, how many are **ok** and how many are Two icons next to a name flag how queries run: *Queries run as the signed-in user (per-user delegation)* and *Row filters and column masks are applied per user at the engine*. -A connection can show **Show catalogs**: the catalogs of a provisioned engine, with their **Catalog**, **Kind**, **Storage** and **Secret**. +A provisioned engine shows as a card with its name, type and a delegation badge (for example **Per-user**), its status, *Provisioned by egret · read-only*, and **Show catalogs**: the catalogs of the engine, with their **Catalog**, **Kind**, **Storage** and **Secret**. It is added by assigning the engine to the workspace in the Control Plane (see [Engine detail](../control-plane/query-engines/engine-detail.md#workspaces)), not here. ## Add a connection @@ -94,11 +94,12 @@ Trino must accept forwarded headers (`http-server.process-forwarded=true`). With ## Example -On the demo instance, the workspace `default` connects to the demo Trino (`localhost:8091`, catalog `tpch`) as `trino-demo`. To connect another workspace to the same Trino, add a connection with Host `localhost`, Port `8091`, **User** `docs`, **Catalog** `tpch` and **Schema** `tiny`, leave the three switches off, and press **Test**. +On the Docker demo stack, the workspace `default` connects to the demo Trino as `trino-demo`. The workspace *Docs sandbox* connects to the same Trino as `trino-sandbox`: Host `trino-demo` (the container name on the demo network), Port `8080`, **User** `egret`, **HTTP scheme** `http`, **Catalog** `tpch`, the three switches off. It also has the provisioned engine `docs-trino` under **Provisioned by egret**. ## Limits and known issues - Egret validates reachability on save, but a new connection shows **Unknown** until you press **Test**. +- Only workspace admins can list the workspace's connections. Members with `ws_editor`, `ws_viewer` or a custom role get *403* on the list, so the database picker in SQL Lab is empty for them, also for a provisioned engine assigned to the workspace. - Only Trino and StarRocks connections can be added here. External engines for Governance are registered in the Control Plane (see [Register an external engine](../control-plane/query-engines/register-external.md)). ## Further reading diff --git a/platform/workspace-settings/members.md b/platform/workspace-settings/members.md index b1915bd..f95a485 100644 --- a/platform/workspace-settings/members.md +++ b/platform/workspace-settings/members.md @@ -6,7 +6,7 @@ sidebar_position: 2 **Workspace settings → Access → Members** controls who can use the workspace and with which role. Members can be single people or whole groups. A person's role decides what they can do in the workspace: view, build and edit, or administer. -![The Members page of the Docs sandbox workspace: a group and a person with their roles](/img/platform/admin/ws-members.png) +![The Members page of the Docs sandbox workspace: people and groups with their roles](/img/platform/admin/ws-members.png) You need to be a workspace admin (the `ws_admin` role) or a platform admin to open this page. @@ -28,11 +28,11 @@ A note under the title says *Workspace roles apply only inside \ — 1. Press **Add member**. 2. Under **Type**, choose **User** or **Group**. -3. Type the person's email (*User email…*) or the group's name (*Group name…*). Use the full email address. The person must already exist in the platform: an SSO user has signed in once, or a local account has been created. +3. Type the person's email (*User email…*) or the group's name (*Group name…*). A few letters are enough: matching people are suggested with their name and email (typing `emma` suggests *Emma Schmidt emma.schmidt@egret.local*); pick one. The person must already exist in the platform: an SSO user has signed in once, or a local account has been created. 4. Choose the **Role**. The default is `ws_viewer`. 5. Press **Add member**. A toast says *Member added*. -![The Add member dialog with an email typed](/img/platform/admin/ws-add-member.png) +![The Add member dialog with an email typed and the matching user suggested](/img/platform/admin/ws-add-member.png) The dialog is *Give a user or group a role in this workspace.* Adding a group gives every member of the group the role. @@ -58,11 +58,24 @@ The menu also has **Impersonate**, which lets a platform admin see the workspace **Bulk select** adds a checkbox to each row. After you tick rows, a bar shows *N selected* with **Remove**, **Set role…** and **Clear**. **Set role…** opens a dialog to choose one role for all the selected members; **Remove** removes them all. Press **Done** to leave bulk mode. -![Bulk select with the actions bar](/img/platform/admin/ws-members-bulk.png) +![Bulk select with two members ticked and the actions bar](/img/platform/admin/ws-members-bulk.png) ## Impersonate a member -**Impersonate** in the row menu starts an impersonation session as that member, which is a way to check what the person sees. It is an administrator action; the demo instance's audit log records it like any other change. It is not covered further here because it was not exercised on the demo instance. +**Impersonate** in the row menu lets a platform admin act as that member, to check what the person sees. + +1. Open **⋯ → Impersonate** on the member's row. The dialog **Impersonate user** says *You'll act as this user until the session expires or you exit. This is logged and requires a reason.* +2. Type a **Reason** (required). +3. If you sign in with a local account, type your **Password**. *Signing in with SSO? Leave this blank — you'll be redirected to re-authenticate.* +4. Press **Impersonate**. + +The workspace reloads as that person, with a banner at the top: *Impersonating member-sso@egret.local · 29:57 left* and an **Exit** button. The session lasts 30 minutes. Pages and actions the person may not use behave as they would for them (for example **Workspace settings → Members** is refused). Press **Exit** to return to your own session. + +The audit log records `impersonation.start` (with your reason) and `impersonation.end` with you as the actor. What you do in between is recorded under the person you impersonate, not under your name. + +:::caution +Impersonation changes the session you are signed in with. Do not use it from a browser session that someone else, or a script, shares. +::: ## Access requests diff --git a/platform/workspace-settings/notifications.md b/platform/workspace-settings/notifications.md index 518d48e..1c256b2 100644 --- a/platform/workspace-settings/notifications.md +++ b/platform/workspace-settings/notifications.md @@ -6,7 +6,7 @@ sidebar_position: 7 **Workspace settings → Workspace → Notifications** sets how the workspace sends alerts and reports: the SMTP relay and from-address for email, and one Slack bot for Slack messages. Every workspace has its own settings, so two teams never share an identity or a sending reputation. -![The Notifications page: SMTP settings and Slack settings](/img/platform/admin/ws-notifications.png) +![The Notifications page with SMTP settings for the demo mail catcher and the Slack section](/img/platform/admin/ws-notifications.png) *Alerts and reports in this workspace send mail through this relay, as this from-address.* Changing the settings needs the workspace permission to manage SMTP (the API checks it on every save). The destinations themselves (an email recipient list, a Slack channel) are chosen on each alert or report in [Alerts & Reports](../explore/alerts-and-reports). @@ -18,7 +18,7 @@ sidebar_position: 7 |---|---| | **SMTP host** | The relay's host name, for example `smtp.example.com`. Required. | | **Port** | The relay's port. Default `587`. | -| **Encryption** | **STARTTLS (usually port 587)** (default), **TLS (usually port 465)** or **None — trusted internal relay only**. Chosen explicitly rather than guessed from the port: guessing wrong can mean credentials sent in the clear. | +| **Encryption** | **STARTTLS (usually port 587)** (default), **TLS (usually port 465)** or **None — trusted internal relay only**, in a drop-down list. Chosen explicitly rather than guessed from the port: guessing wrong can mean credentials sent in the clear. | | **Username** | The login for the relay. Optional. | | **Password** | The relay password. Never shown again: the field starts empty and shows *Unchanged* when one is stored. | | **From address** | The sender, for example `alerts@example.com`. Required, and must contain `@`. | @@ -27,7 +27,7 @@ sidebar_position: 7 ### Save the settings 1. Fill **SMTP host**, **Port**, **Encryption**, **Username**, **Password**, **From address** and **From name**. -2. Press **Save**. A notice says *Settings saved.* +2. Press **Save**. The **Remove** button appears once settings are stored. An untouched password field is omitted from the save so the stored password is kept; leaving it empty after clearing is how you remove it. The password is stored as a secret, not in the settings row. @@ -46,7 +46,7 @@ When settings exist a **Remove** button deletes them (*Settings removed.*), afte | **Bot token** | The bot's OAuth token, for example `xoxb-…`. It needs the `chat:write` scope, and the bot must be invited to each channel. When one is stored the field shows *Unchanged* and the note *A token is stored. Leave blank to keep it.* | | **Default channel** | Optional, for example `#data-alerts`. Shown as a hint when adding a destination. | -Press **Save Slack settings** (*Slack settings saved.*). **Remove** deletes the token and channel (*Slack settings removed.*). +Press **Save Slack settings** (*Slack settings saved.*). The token field then says *A token is stored. Leave blank to keep it.* **Remove Slack app** deletes the token and channel (*Slack settings removed.*). ### Set up Slack @@ -55,13 +55,23 @@ Press **Save Slack settings** (*Slack settings saved.*). **Remove** deletes the 3. Invite the bot to the channels alerts will post to (`/invite @your-bot`). 4. Save, then create an alert whose destination is the Slack channel. -## Example +## Example: the demo mail catcher + +On the Docker demo stack every mail goes to Mailpit (`http://localhost:8025` on the host). For the workspace *Docs sandbox*: + +1. **SMTP host** `mailpit`, **Port** `1025`, **Encryption** **None — trusted internal relay only**, no username or password. +2. **From address** `docs-sandbox@egret.local`, **From name** *Docs sandbox alerts*. Press **Save**. +3. Press **Send test message**. The page says *Test message sent. Check the inbox for the from-address.*, and Mailpit shows the mail *egret test message* from *Docs sandbox alerts*. + +The demo seed configures the `default` workspace the same way. The demo stack has no Slack: a token saved there is stored, but nothing can be delivered to Slack. + +## Example: a production relay For a finance workspace: SMTP host `smtp.corp.example`, port `587`, **STARTTLS**, username `alerts-bot`, from address `finance-alerts@corp.example`, from name *Finance Alerts*. Slack default channel `#finance-alerts`. ## Limits and known issues -- On the demo instance, saving SMTP settings returned an *internal error* (HTTP 500), so the saved state, **Send test message** and the Slack save could not be verified there. The field list is from the product source. +- Before settings exist the page loads two 404 responses (`/notifications/smtp` and `/notifications/slack`) that show as errors in the browser console. They are harmless. - Passwords and tokens are write-only. You cannot read them back; replace them to change them. ## Further reading diff --git a/platform/workspace-settings/roles-and-permissions.md b/platform/workspace-settings/roles-and-permissions.md index 1c84a23..751b67a 100644 --- a/platform/workspace-settings/roles-and-permissions.md +++ b/platform/workspace-settings/roles-and-permissions.md @@ -6,7 +6,7 @@ sidebar_position: 3 **Workspace settings → Access → Roles & permissions** shows what each workspace role can do in this workspace. It is a read-only view: roles are defined once for the whole platform and apply to every workspace. -![The workspace Roles and permissions page with the three built-in roles and a custom role](/img/platform/admin/ws-roles.png) +![The workspace Roles and permissions page with three built-in roles and two custom roles](/img/platform/admin/ws-roles.png) *Roles are defined platform-wide in Control Plane → Access control and apply to every workspace. This is a read-only view.* @@ -45,7 +45,7 @@ Select a row to expand it. The permissions are grouped by area, each with its de | OTHER | `catalog:read` | View catalogs. | | SQL LAB | `saved_query:read` | View saved queries. | -![A custom role expanded: inherited reads plus chart and dashboard management](/img/platform/admin/ws-roles-expanded.png) +![The custom role finance_reporter expanded: it inherits from ws_viewer and adds chart and dashboard management](/img/platform/admin/ws-roles-expanded.png) A role that inherits from another shows only the permissions it adds itself. `finance_reporter`, which inherits from `ws_viewer` and adds `chart:manage` and `dashboard:manage`, shows **2 permissions** here, while the Control Plane role page counts 8 because it includes the seven inherited reads. Read the parent's row to see the inherited ones. diff --git a/platform/workspace-settings/shared-objects.md b/platform/workspace-settings/shared-objects.md index 70471bd..3686f81 100644 --- a/platform/workspace-settings/shared-objects.md +++ b/platform/workspace-settings/shared-objects.md @@ -6,7 +6,7 @@ sidebar_position: 5 **Workspace settings → Access → Shared objects** is a workspace-wide roll-up of everything that has been shared with someone: dashboards, charts, datasets, folders, notebooks and saved queries, and who holds a grant on each. It is the place to audit sharing and to change it in one spot. -![The Shared objects page: one row per object and principal](/img/platform/admin/ws-shares.png) +![The Shared objects page: five shares on a dashboard, a chart and a dataset, one of them denied](/img/platform/admin/ws-shares.png) Only workspace admins can view shares; others see *Only workspace admins can view shares.* Until something is shared the page shows *Nothing shared in this workspace yet.* diff --git a/scripts/capture-member-request-dialog.mjs b/scripts/capture-member-request-dialog.mjs index c5871b0..1fb12ba 100644 --- a/scripts/capture-member-request-dialog.mjs +++ b/scripts/capture-member-request-dialog.mjs @@ -1,7 +1,7 @@ // Captures the "Request access" dialog as the SSO user member-sso (needs a Playwright storage state for that user). // Usage: MEMBER_STATE=/path/member.json BASE=http://localhost:5180 node scripts/capture-member-request-dialog.mjs import {chromium} from 'playwright'; -const BASE = process.env.BASE ?? 'http://localhost:5180'; +const BASE = process.env.BASE ?? 'http://egret.localtest.me:18080'; const browser = await chromium.launch(); const ctx = await browser.newContext({viewport: {width: 1440, height: 900}, storageState: process.env.MEMBER_STATE}); const page = await ctx.newPage(); @@ -11,7 +11,7 @@ await page.getByText(/Request access/i).first().click(); await page.waitForTimeout(1000); const d = page.getByRole('dialog'); await d.getByRole('combobox', {name: 'Workspace'}).click(); -await page.getByRole('option', {name: 'Default'}).click(); -await d.getByLabel('Reason').fill('I need to build dashboards on the orders dataset for the sales review.'); +await page.getByRole('option', {name: 'Sales operations'}).click(); +await d.getByLabel('Reason').fill('I prepare the weekly pipeline review with the sales leads.'); await page.screenshot({path: new URL('../static/img/platform/admin/ws-request-dialog.png', import.meta.url).pathname}); await browser.close(); diff --git a/scripts/seed-admin.mjs b/scripts/seed-admin.mjs new file mode 100644 index 0000000..d683062 --- /dev/null +++ b/scripts/seed-admin.mjs @@ -0,0 +1,276 @@ +// Seed the admin-area demo data (Control Plane, Workspace settings, Governance) on a running demo stack. +// Idempotent: every object is looked up by name first and only created when missing. +// Usage: BASE=http://egret.localtest.me:18080 STATE= [MEMBER_STATE=] node scripts/seed-admin.mjs +// The objects are listed in scripts/shots/admin-demo-data.md. Passwords below are demo values for a throwaway stack. +import {request} from 'playwright'; +import {api} from './api-helper.mjs'; + +const BASE = process.env.BASE ?? 'http://egret.localtest.me:18080'; +const a = await api(process.env.STATE, BASE); +const DEMO_PASSWORD = 'Docs12345!'; + +const must = (r, what) => { + if (r.status >= 300) throw new Error(`${what}: ${r.status} ${JSON.stringify(r.body).slice(0, 300)}`); + return r.body; +}; +const list = async (url) => { + const b = must(await a.get(url), `GET ${url}`); + return Array.isArray(b) ? b : (b.items ?? b.entries ?? b.data ?? []); +}; + +// ---- Users (local accounts) +const people = [ + ['alice.nguyen@egret.local', 'Alice Nguyen'], + ['bao.tran@egret.local', 'Bao Tran'], + ['carla.mendes@egret.local', 'Carla Mendes'], + ['david.kim@egret.local', 'David Kim'], + ['emma.schmidt@egret.local', 'Emma Schmidt'], + ['farid.haddad@egret.local', 'Farid Haddad'], + ['grace.lee@egret.local', 'Grace Lee'], + ['hiro.tanaka@egret.local', 'Hiro Tanaka'], +]; +let users = await list('/api/control-plane/users'); +for (const [email, name] of people) { + if (!users.find((u) => u.email === email)) { + must(await a.post('/api/control-plane/users', {email, name, password: DEMO_PASSWORD}), `create user ${email}`); + console.log('user', email); + } +} +users = await list('/api/control-plane/users'); +const uid = (email) => users.find((u) => u.email === email)?.id; +// Grace Lee left the company: disabled, kept for the audit trail. +const grace = users.find((u) => u.email === 'grace.lee@egret.local'); +if (grace && grace.status !== 'disabled') must(await a.patch(`/api/control-plane/users/${grace.id}`, {status: 'disabled'}), 'disable grace'); + +// ---- Roles +let roles = await list('/api/control-plane/access/roles'); +const roleId = (n) => roles.find((r) => r.name === n)?.id; +const wantRoles = [ + {scope: 'workspace', name: 'finance_reporter', description: 'Reads finance data and builds charts and dashboards', parent: 'ws_viewer', perms: ['chart:manage', 'dashboard:manage']}, + {scope: 'workspace', name: 'marketing_analyst', description: 'Runs SQL and builds charts on marketing data', parent: 'ws_viewer', perms: ['chart:manage', 'query:run']}, + {scope: 'platform', name: 'security_auditor', description: 'Reviews users, groups, roles and engines', parent: null, perms: ['user:read', 'group:read', 'role:read', 'engine:read']}, +]; +for (const r of wantRoles) { + if (!roleId(r.name)) { + const body = {scope: r.scope, name: r.name, description: r.description}; + if (r.parent) body.parentRoleId = roleId(r.parent); + const created = must(await a.post('/api/control-plane/access/roles', body), `create role ${r.name}`); + const res = await a.put(`/api/control-plane/access/roles/${created.id}/permissions`, {permissions: r.perms}); + if (res.status >= 300) console.log(` permissions for ${r.name}: ${res.status} ${JSON.stringify(res.body).slice(0, 200)}`); + console.log('role', r.name); + roles = await list('/api/control-plane/access/roles'); + } +} + +// ---- Groups (local) +const wantGroups = { + 'finance-analysts': ['alice.nguyen@egret.local', 'bao.tran@egret.local', 'carla.mendes@egret.local'], + marketing: ['david.kim@egret.local', 'emma.schmidt@egret.local'], + 'data-stewards': ['farid.haddad@egret.local', 'grace.lee@egret.local'], + executives: ['hiro.tanaka@egret.local'], +}; +let groups = await list('/api/control-plane/groups'); +for (const [name, members] of Object.entries(wantGroups)) { + let g = groups.find((x) => x.name === name); + if (!g) { + g = must(await a.post('/api/control-plane/groups', {name}), `create group ${name}`); + console.log('group', name); + } + const current = (await list(`/api/control-plane/groups/${g.id}/members`)).map((m) => m.id ?? m.userId); + for (const email of members) { + if (!current.includes(uid(email))) must(await a.post(`/api/control-plane/groups/${g.id}/members`, {userId: uid(email)}), `add ${email} to ${name}`); + } +} +groups = await list('/api/control-plane/groups'); +const gid = (n) => groups.find((g) => g.name === n)?.id; +const stewards = groups.find((g) => g.name === 'data-stewards'); +if (stewards && !stewards.roles?.includes('data_steward')) await a.post(`/api/control-plane/groups/${stewards.id}/roles`, {roleId: roleId('data_steward')}); +const farid = users.find((u) => u.email === 'farid.haddad@egret.local'); +if (farid && !farid.roles?.includes('security_auditor')) await a.post(`/api/control-plane/users/${farid.id}/roles`, {roleId: roleId('security_auditor')}); + +// ---- Workspaces +const wantWs = [ + ['docs-sandbox', 'Docs sandbox'], + ['finance-analytics', 'Finance analytics'], + ['sales-operations', 'Sales operations'], + ['data-science-lab', 'Data science lab'], + ['legacy-reporting', 'Legacy reporting'], +]; +let wss = await list('/api/admin/workspaces'); +for (const [slug, name] of wantWs) { + if (!wss.find((w) => w.slug === slug)) { + must(await a.post('/api/admin/workspaces', {slug, name}), `create workspace ${slug}`); + console.log('workspace', slug); + } +} +wss = await list('/api/admin/workspaces'); +const wsId = (slug) => wss.find((w) => w.slug === slug)?.id; + +// ---- Workspace members +const members = { + 'docs-sandbox': [ + ['group', 'data-team', 'ws_editor'], + ['user', 'member-sso@egret.local', 'ws_viewer'], + ['user', 'alice.nguyen@egret.local', 'ws_admin'], + ['user', 'david.kim@egret.local', 'marketing_analyst'], + ['user', 'hiro.tanaka@egret.local', 'ws_viewer'], + ['group', 'finance-analysts', 'finance_reporter'], + ], + 'finance-analytics': [ + ['group', 'finance-analysts', 'ws_editor'], + ['user', 'alice.nguyen@egret.local', 'ws_admin'], + ['user', 'hiro.tanaka@egret.local', 'ws_viewer'], + ['user', 'farid.haddad@egret.local', 'ws_viewer'], + ], + 'sales-operations': [ + ['group', 'marketing', 'ws_editor'], + ['user', 'carla.mendes@egret.local', 'ws_viewer'], + ], + 'data-science-lab': [ + ['group', 'data-team', 'ws_editor'], + ['user', 'emma.schmidt@egret.local', 'ws_admin'], + ], +}; +for (const [ws, rows] of Object.entries(members)) { + const current = must(await a.get(`/api/ws/${ws}/members`), `members ${ws}`); + const have = JSON.stringify(current); + for (const [ptype, who, role] of rows) { + const pid = ptype === 'group' ? gid(who) : uid(who); + if (!pid) { console.log(` skip ${who}: not known yet`); continue; } + if (have.includes(pid)) continue; + const r = await a.post(`/api/ws/${ws}/members`, {principal_type: ptype, principal_id: pid, role}); + if (r.status >= 300) console.log(` member ${who} in ${ws}: ${r.status} ${JSON.stringify(r.body).slice(0, 200)}`); + } +} + +// ---- Docs sandbox content: a Trino connection, a dataset, two charts and the dashboard "Finance KPIs" with shares +const dbs = await list('/api/ws/docs-sandbox/databases'); +let db = dbs.find((d) => d.name === 'trino-sandbox'); +if (!db) { + db = must(await a.post('/api/ws/docs-sandbox/databases', {name: 'trino-sandbox', engine: 'trino', host: 'trino-demo', port: 8080, options: {user: 'egret', http_scheme: 'http', catalog: 'tpch'}}), 'database trino-sandbox'); + console.log('database trino-sandbox'); +} +const datasets = await list('/api/ws/docs-sandbox/datasets'); +let ds = datasets.find((d) => d.name === 'orders'); +if (!ds) { + ds = must(await a.post('/api/ws/docs-sandbox/datasets', {name: 'orders', database_id: db.id, description: 'TPC-H sample orders', source: {kind: 'physical', catalog: 'tpch', schema: 'tiny', table: 'orders'}}), 'dataset orders'); + await a.post(`/api/ws/docs-sandbox/datasets/${ds.id}/metrics`, {metric_name: 'revenue', verbose_name: 'Revenue', expression: 'sum(totalprice)'}); + await a.post(`/api/ws/docs-sandbox/datasets/${ds.id}/metrics`, {metric_name: 'order_count', verbose_name: 'Orders', expression: 'count(*)'}); + console.log('dataset orders'); +} +const spec = (metrics, dims, extra = {}) => ({metrics, dimensions: dims, columns: [], filters: [], orderby: [], row_limit: 1000, rollup_groups: [], ...extra}); +const charts = await list('/api/ws/docs-sandbox/charts'); +const chartByName = async (name, viz, s) => { + const c = charts.find((x) => x.name === name); + if (c) return c; + console.log('chart', name); + return must(await a.post('/api/ws/docs-sandbox/charts', {dataset_id: ds.id, name, viz_type: viz, params: {version: 2, settings: {}, spec: s}}), `chart ${name}`); +}; +const c1 = await chartByName('Revenue by order status', 'bar', spec(['revenue'], ['orderstatus'])); +const c2 = await chartByName('Orders by priority', 'table', spec(['order_count'], ['orderpriority'])); +const dashes = await list('/api/ws/docs-sandbox/dashboards'); +let dash = dashes.find((d) => d.name === 'Finance KPIs'); +if (!dash) { + dash = must(await a.post('/api/ws/docs-sandbox/dashboards', {name: 'Finance KPIs', description: 'Revenue and order volume'}), 'dashboard'); + await a.put(`/api/ws/docs-sandbox/dashboards/${dash.id}`, {name: 'Finance KPIs', description: 'Revenue and order volume', parameters: [], width: 'full', tabs: [], cards: [ + {chart_id: c1.id, card_type: 'chart', grid_col: 0, grid_row: 0, size_x: 12, size_y: 5, options: {}}, + {chart_id: c2.id, card_type: 'chart', grid_col: 12, grid_row: 0, size_x: 12, size_y: 5, options: {}}, + ]}); + console.log('dashboard Finance KPIs'); +} +const shares = [ + ['dashboard', dash.id, 'group', 'executives', 'ws_viewer', 'allow'], + ['dashboard', dash.id, 'user', 'farid.haddad@egret.local', 'ws_viewer', 'allow'], + ['dashboard', dash.id, 'user', 'carla.mendes@egret.local', 'ws_editor', 'allow'], + ['chart', c1.id, 'user', 'bao.tran@egret.local', 'ws_editor', 'allow'], + ['dataset', ds.id, 'user', 'hiro.tanaka@egret.local', 'ws_editor', 'deny'], +]; +for (const [kind, id, ptype, who, role, effect] of shares) { + const have = JSON.stringify(must(await a.get(`/api/ws/docs-sandbox/objects/${kind}/${id}/grants`), 'grants')); + if (have.includes(who)) continue; + const r = await a.put(`/api/ws/docs-sandbox/objects/${kind}/${id}/grants`, {principal_type: ptype, principal_id: who, role, effect}); + if (r.status >= 300) console.log(` share ${kind} ${who}: ${r.status} ${JSON.stringify(r.body).slice(0, 200)}`); +} + +// ---- Secrets (demo values only) +const wantSecrets = [ + {name: 'docs-warehouse-password', description: 'Password of the reporting warehouse user', type: 'kubernetes.io/basic-auth', data: {username: 'reporting', password: 'demo-only-not-secret'}}, + {name: 'finance-postgres', description: 'Finance Postgres read replica', type: 'kubernetes.io/basic-auth', data: {username: 'finance_ro', password: 'demo-only-not-secret'}}, + {name: 'marketing-api-token', description: 'Token for the marketing events API', type: 'Opaque', data: {token: 'demo-only-not-secret'}}, + {name: 'sandbox-lake-keys', description: 'Object-store keys for the sandbox lake', type: 'Opaque', data: {secret_key: 'demo-only-not-secret'}}, + {name: 'partner-sftp', description: 'SFTP drop for partner exports', type: 'kubernetes.io/basic-auth', data: {username: 'partner', password: 'demo-only-not-secret'}}, +]; +const secrets = await list('/api/control-plane/secrets'); +for (const s of wantSecrets) { + if (!secrets.find((x) => x.name === s.name)) { + must(await a.post('/api/control-plane/secrets', s), `secret ${s.name}`); + console.log('secret', s.name); + } +} + +// ---- Storage connection sandbox-lake (same demo object store, its own prefix; reuses the seeded demo-rustfs secret) +const storage = await list('/api/control-plane/storage'); +const rustfs = storage.find((s) => s.name === 'demo-rustfs'); +if (!storage.find((s) => s.name === 'sandbox-lake') && rustfs) { + must(await a.post('/api/control-plane/storage', {name: 'sandbox-lake', purpose: 'Scratch space for the docs sandbox', provider: 's3', endpoint: rustfs.endpoint, region: rustfs.region, bucket: rustfs.bucket, key_prefix: 'sandbox/', access_key: rustfs.access_key, secret_ref: rustfs.secret_ref, path_style: true}), 'storage sandbox-lake'); + console.log('storage sandbox-lake'); +} + +// ---- Catalogs: TPC-H for benchmarks (tpch_bench) +let catalogs = await list('/api/control-plane/catalogs'); +if (!catalogs.find((c) => c.name === 'tpch_bench')) { + must(await a.post('/api/control-plane/catalogs', {name: 'tpch_bench', kind: 'tpch', config: {}, description: 'Generated TPC-H data for benchmarks'}), 'catalog tpch_bench'); + console.log('catalog tpch_bench'); +} +// Governance uses the seeded warehouse catalog demo_warehouse: a second egret warehouse on the same bucket is refused +// ("Storage profile overlaps with existing warehouse demo-warehouse"). + +// ---- Classifications +const wantTags = [ + {key: 'personal-data-pii', name: 'Personal data (PII)', description: 'Columns that identify a person', color: '#e5484d'}, + {key: 'financial', name: 'Financial', description: 'Revenue, prices and payment data', color: '#f5a524'}, + {key: 'confidential', name: 'Confidential', description: 'Internal only, never shared outside the company', color: '#8e4ec6'}, + {key: 'public', name: 'Public', description: 'Safe to publish', color: '#30a46c'}, +]; +const tags = await list('/api/control-plane/tags'); +for (const t of wantTags) { + if (!tags.find((x) => x.key === t.key)) { + const r = await a.post('/api/control-plane/tags', t); + if (r.status >= 300) console.log(` tag ${t.key}: ${r.status} ${JSON.stringify(r.body).slice(0, 200)}`); + else console.log('tag', t.key); + } +} + +// ---- Access requests: filed by the requesters themselves (local users sign in once through the API) +const requestAs = async (email, ws, role, reason) => { + const pending = must(await a.get(`/api/ws/${ws}/access-requests`), 'requests'); + const rows = Array.isArray(pending) ? pending : pending.items ?? []; + if (rows.find((r) => r.requester_id === uid(email))) return; + const ctx = await request.newContext({baseURL: BASE}); + const login = await ctx.post('/api/auth/login', {data: {email, password: DEMO_PASSWORD}}); + if (!login.ok()) { console.log(` login ${email}: ${login.status()}`); return; } + const csrf = (await ctx.storageState()).cookies.find((c) => c.name === 'egret_csrf')?.value ?? ''; + const r = await ctx.post('/api/access-requests', {headers: {'X-Egret-CSRF': csrf}, data: {workspace_id: wsId(ws), role, reason}}); + console.log('request', email, ws, r.status()); + await ctx.dispose(); +}; +await requestAs('bao.tran@egret.local', 'sales-operations', 'ws_viewer', 'Quarterly pipeline review with the sales leads.'); +await requestAs('david.kim@egret.local', 'finance-analytics', 'ws_viewer', 'I need the revenue dashboards for the campaign ROI report.'); +await requestAs('emma.schmidt@egret.local', 'finance-analytics', 'ws_editor', 'Building the marketing spend vs. revenue charts for Q4.'); +await requestAs('carla.mendes@egret.local', 'data-science-lab', 'ws_viewer', 'Churn model notebooks.'); +if (process.env.MEMBER_STATE) { + const pending = JSON.stringify(must(await a.get('/api/ws/finance-analytics/access-requests'), 'requests')); + if (!pending.includes(uid('member-sso@egret.local') ?? '-')) { + const m = await api(process.env.MEMBER_STATE, BASE); + const r = await m.post('/api/access-requests', {workspace_id: wsId('finance-analytics'), role: 'ws_viewer', reason: 'I need to build dashboards on the orders dataset for the sales review.'}); + console.log('request member-sso finance-analytics', r.status); + } +} + +// ---- Legacy reporting is archived +const legacy = wss.find((w) => w.slug === 'legacy-reporting'); +if (legacy && legacy.status !== 'archived') { + const r = await a.del(`/api/admin/workspaces/${legacy.id}`); + console.log('archive legacy-reporting', r.status); +} +console.log('done'); diff --git a/scripts/seed-governance-ui.mjs b/scripts/seed-governance-ui.mjs index 40ef03c..1c49e99 100644 --- a/scripts/seed-governance-ui.mjs +++ b/scripts/seed-governance-ui.mjs @@ -1,19 +1,20 @@ // Register the demo Trino as an external engine (a record only: egret does not provision or monitor it). -// Usage: BASE=http://localhost:5180 STATE=/tmp/egret-state.json node scripts/seed-governance-ui.mjs +// Usage: BASE=http://egret.localtest.me:18080 STATE= node scripts/seed-governance-ui.mjs import {chromium} from 'playwright'; -const BASE = process.env.BASE ?? 'http://localhost:5180'; +const BASE = process.env.BASE ?? 'http://egret.localtest.me:18080'; const browser = await chromium.launch(); const ctx = await browser.newContext({viewport: {width: 1440, height: 900}, storageState: process.env.STATE}); const page = await ctx.newPage(); await page.goto(`${BASE}/control-plane/engines/register`, {waitUntil: 'networkidle'}); -await page.getByLabel('Name').fill('docs-trino'); -await page.getByLabel('Host').fill('localhost'); -await page.getByLabel('Port').fill('8091'); +await page.getByLabel('Name').fill('trino-demo-ext'); +await page.getByLabel('Host').fill('trino-demo'); +await page.getByLabel('Port').fill('8080'); await page.getByRole('combobox').filter({hasText: /choose a version/i}).click(); console.log('versions:', await page.getByRole('option').allTextContents()); -await page.getByRole('option').first().click(); +await page.getByRole('option', {name: '2.8.0'}).click(); +await page.getByText('tpch_bench', {exact: true}).click(); await page.getByRole('button', {name: 'Register engine'}).click(); await page.waitForTimeout(3000); console.log('after register:', page.url()); diff --git a/scripts/shots/admin-demo-data.md b/scripts/shots/admin-demo-data.md index 68a42e9..6cb2dc4 100644 --- a/scripts/shots/admin-demo-data.md +++ b/scripts/shots/admin-demo-data.md @@ -1,9 +1,38 @@ # Demo data assumed by admin.mjs +Captured on the Docker demo stack (`BASE=http://egret.localtest.me:18080`). Run in this order; every step is idempotent or +checks first. -- Engine `docs-trino` declares `demo_catalog` and `tpch_bench`, bound to profile `docs-trino-policy` (a `trino` profile whose `config.tagService` names the `tag` profile `docs-trino-tags`; both created through `POST /api/control-plane/policies/services`). -- On `demo_catalog.analytics.sessions`: a grant for user `member-sso`, a row filter, a mask by path, a mask by tag, tag `Personal data (PII)` on `user_id`, clearance `data_steward` on the tag, an engine permission (system session property `query_max_run_time` for `member-sso`). -- Workspaces `Docs sandbox` (members, connection `trino-sandbox`, dashboard `Finance KPIs` with two shares) and `Finance analytics` (access request from `member-sso`, approved at the end), archived workspace `Legacy reporting`. -- Role `finance_reporter` (inherits `ws_viewer`, plus `chart:manage`, `dashboard:manage`); catalog `tpch_bench`; secret `docs-warehouse-password`; storage connection `sandbox-lake`. -- `scripts/capture-member-request-dialog.mjs` captures the request dialog as `member-sso` (needs a saved SSO state). +1. Sign in once through SSO as `member-sso` and `admin-sso` (Keycloak users of the demo realm) and keep their storage + states outside the repository. `member-sso` is used for the request dialog (`scripts/capture-member-request-dialog.mjs`). +2. `node scripts/seed-admin.mjs` with `STATE` (admin) and `MEMBER_STATE` (member-sso): local users (`alice.nguyen`, + `bao.tran`, `carla.mendes`, `david.kim`, `emma.schmidt`, `farid.haddad`, `grace.lee` disabled, `hiro.tanaka`, all + `@egret.local`), groups `finance-analysts`, `marketing`, `data-stewards` (role `data_steward`), `executives`, roles + `finance_reporter`, `marketing_analyst`, `security_auditor`, workspaces `Docs sandbox`, `Finance analytics`, + `Sales operations`, `Data science lab` and the archived `Legacy reporting`, members, the connection `trino-sandbox` + (`trino-demo:8080`), dataset `orders`, two charts and the dashboard `Finance KPIs` with five shares, five secrets, the + storage connection `sandbox-lake`, the TPC-H catalog `tpch_bench`, four classifications and five access requests. +3. Through the UI (as documented on the pages): + - **Add engine** `docs-trino`: `demo_warehouse` primary, `tpch_bench` attached, worker replicas `0`; assign it to + `Docs sandbox`. Settings: `query.max-run-time` = `2h`. + - SQL on `docs-trino` in `Docs sandbox` (as admin): schemas `demo_warehouse.analytics` and `demo_warehouse.finance`, + tables `analytics.customers`, `analytics.sessions`, `analytics.orders`, `finance.invoices` (CTAS from + `tpch_bench.tiny`), view `analytics.revenue_by_country`. + - Governance: grant read on `analytics.sessions` to user `member-sso`; grants on `finance` (groups `finance-analysts`, + `data-team`), on the catalog (role `data_steward`, read and write), on `analytics.ord*` (groups `marketing`, + `data-team`), on `analytics.customers` (group `marketing`) and a deny on `analytics.customers.phone` (group + `marketing`); rules `EU sessions only` (row filter), `Hide revenue outside finance` (Nullify on `revenue`), + `Hash personal data` (Hash by tag `Personal data (PII)`); tags on `sessions.user_id`, `sessions.revenue`, + `customers.email/phone/name`, `orders.total_price` and the table `finance.invoices` (`Confidential`); clearance + role `data_steward` on `Confidential`; engine permission `query_max_run_time` for `member-sso`; role + `data_steward` given directly to Farid Haddad. + - **Register external engine** `trino-demo-ext` (`trino-demo:8080`, plugin 2.8.0) with `tpch_bench` and `finance_pg` + declared; profile pair `docs-legacy-tags` / `docs-legacy-trino` (API, `config.tagService`) bound to it, with the + hand-written policy `legacy: analysts read finance_pg` (migratable). Profiles `docs-external-policy` (Trino, seeded + defaults), `docs-external-trino` and `starrocks-prod` unbound. + - JDBC catalog `finance_pg` (secret `finance-postgres`), SMTP and Slack settings of `Docs sandbox` (Mailpit), a local + platform admin `ops.admin@egret.local` used for impersonation. +4. Capture: `node scripts/screenshots.mjs admin/ …` with the admin `STATE`. +The engines are removed at the end of a docs pass (**Delete engine**), so the engine shots can only be re-captured after +repeating step 3. diff --git a/scripts/shots/admin.mjs b/scripts/shots/admin.mjs index 5696404..53cbe95 100644 --- a/scripts/shots/admin.mjs +++ b/scripts/shots/admin.mjs @@ -1,32 +1,38 @@ -// Screenshots for the admin areas: Governance, Control plane, Workspace settings. -// Ids are prefixed `admin/`. The shots assume the objects created by the steps listed in scripts/shots/admin-demo-data.md -// (see scripts/shots/admin-demo-data.md): profile docs-trino-policy bound to engine docs-trino, grant/rules/tag on -// demo_catalog.analytics.sessions, workspaces "Docs sandbox" and "Finance analytics", role finance_reporter, -// a pending access request in Finance analytics, secret docs-warehouse-password. +// Screenshots for the admin areas: Governance, Control plane, Workspace settings. Ids are prefixed `admin/`. +// The shots assume the objects listed in scripts/shots/admin-demo-data.md: the seed (scripts/seed-admin.mjs), the provisioned +// engine docs-trino with its tables in demo_warehouse, the external engine trino-demo-ext, the grants, rules, tags and +// clearance written through the Governance UI. Run on the Docker demo stack: +// BASE=http://egret.localtest.me:18080 STATE= node scripts/screenshots.mjs admin/cp-users ... const gov = '/governance'; const cp = '/control-plane'; -const T = `${gov}/data/demo_catalog/analytics/sessions`; +const T = `${gov}/data/demo_warehouse/analytics/sessions`; const click = (text, opts = {}) => async (page) => { await page.getByText(text, {exact: true, ...opts}).first().click(); await page.waitForLoadState('networkidle'); }; +const link = (name) => async (page) => { + await page.locator('main').getByRole('link', {name, exact: true}).first().click(); + await page.waitForLoadState('networkidle'); +}; const tab = (name) => async (page) => { - await page.getByRole('tab', {name}).first().click(); + await page.getByRole('tab', {name, exact: true}).first().click(); await page.waitForTimeout(1200); }; const seq = (...fns) => async (page) => { for (const f of fns) await f(page); }; +// Open an engine of the Control Plane list by name (the list links each name). +const engine = (name) => seq(link(name), async (page) => page.waitForTimeout(1500)); export const shots = [ // ---- Governance {id: 'admin/gov-overview', path: gov, settle: 2500}, - {id: 'admin/gov-data-schema', path: `${gov}/data/demo_catalog/analytics`, settle: 2500}, + {id: 'admin/gov-data-schema', path: `${gov}/data/demo_warehouse/analytics`, settle: 2500}, {id: 'admin/gov-table-overview', path: T, settle: 2500}, {id: 'admin/gov-table-permissions', path: `${T}?tab=permissions`, settle: 2500}, {id: 'admin/gov-table-masks', path: `${T}?tab=masks`, settle: 2500}, - {id: 'admin/gov-table-lineage', path: `${T}?tab=lineage`, settle: 2000}, + {id: 'admin/gov-table-lineage', path: `${gov}/data/demo_warehouse/analytics/sessions?tab=lineage`, settle: 2500}, {id: 'admin/gov-table-details', path: `${T}?tab=details`, settle: 2000}, {id: 'admin/gov-column-panel', path: `${T}?column=user_id`, settle: 2500}, { @@ -34,18 +40,18 @@ export const shots = [ path: T, settle: 800, prepare: async (page) => { - await page.setViewportSize({width: 1440, height: 1250}); await page.getByRole('button', {name: /Grant access/}).click(); const d = page.getByRole('dialog'); + await d.getByRole('checkbox', {name: 'Write'}).check(); const inp = d.getByLabel('Include user search'); await inp.click(); - await inp.fill('member'); + await inp.pressSequentially('alice', {delay: 80}); await page.waitForTimeout(1500); await page.getByRole('option').first().click(); - await page.waitForTimeout(600); + await d.getByText('Principals', {exact: true}).click(); await d.getByRole('button', {name: 'Preview', exact: true}).click(); - await page.waitForTimeout(2500); - await page.waitForTimeout(300); + await page.waitForTimeout(3000); + await d.getByText(/Would recompile/).scrollIntoViewIfNeeded(); }, }, { @@ -53,18 +59,16 @@ export const shots = [ path: `${T}?tab=masks`, settle: 800, prepare: async (page) => { - await page.setViewportSize({width: 1440, height: 1250}); await page.getByRole('button', {name: 'Add rule'}).click(); const d = page.getByRole('dialog'); await d.getByRole('combobox', {name: 'Kind', exact: true}).click(); await page.getByRole('option', {name: 'Mask a column'}).click(); - await d.getByLabel('Name').fill('Hash visitor ids'); + await d.getByLabel('Name').fill('Hash session ids'); await d.getByRole('combobox', {name: 'Mask type'}).click(); - await page.getByRole('option', {name: 'Hash'}).click(); - await d.getByPlaceholder('e.g. ssn').fill('user_id'); + await page.getByRole('option', {name: 'Hash', exact: true}).click(); + await d.getByPlaceholder('e.g. ssn').fill('session_id'); await d.getByRole('button', {name: 'Preview', exact: true}).click(); - await page.waitForTimeout(2500); - await page.waitForTimeout(300); + await page.waitForTimeout(3000); }, }, { @@ -86,23 +90,25 @@ export const shots = [ {id: 'admin/gov-people-groups', path: `${gov}/people?kind=group`, settle: 2000}, {id: 'admin/gov-people-roles', path: `${gov}/people?kind=role`, settle: 2000}, {id: 'admin/gov-people-unmatched', path: `${gov}/people?kind=unmatched`, settle: 1500}, - { - id: 'admin/gov-person-overview', - path: `${gov}/people`, - settle: 1500, - prepare: click('Member SSO'), - }, - { - id: 'admin/gov-person-permissions', - path: `${gov}/people`, - settle: 1500, - prepare: seq(click('Member SSO'), tab('Permissions')), - }, + {id: 'admin/gov-person-overview', path: `${gov}/people`, settle: 2000, prepare: click('Member SSO')}, + {id: 'admin/gov-person-permissions', path: `${gov}/people`, settle: 2000, prepare: seq(click('Member SSO'), tab('Permissions'))}, + {id: 'admin/gov-person-clearances', path: `${gov}/people`, settle: 2000, prepare: seq(click('Farid Haddad'), tab('Clearances'))}, { id: 'admin/gov-person-check', path: `${gov}/people`, - settle: 1500, - prepare: seq(click('Member SSO'), tab('Check access')), + settle: 1000, + prepare: async (page) => { + await click('Member SSO')(page); + await tab('Check access')(page); + const main = page.locator('main'); + await main.getByRole('combobox').filter({hasText: /Select a catalog/}).click(); + await page.getByRole('option', {name: 'demo_warehouse'}).click(); + await main.getByRole('textbox').nth(0).fill('analytics'); + await main.getByRole('textbox').nth(1).fill('customers'); + await main.getByRole('button', {name: 'Explain'}).click(); + await page.waitForTimeout(4000); + await main.getByText('No policy matched').first().scrollIntoViewIfNeeded(); + }, }, {id: 'admin/gov-engines-index', path: `${gov}/engines`, settle: 2500}, {id: 'admin/gov-engine-permissions', path: `${gov}/engines`, settle: 2000, prepare: click('docs-trino')}, @@ -116,30 +122,75 @@ export const shots = [ const d = page.getByRole('dialog'); await d.getByRole('combobox').first().click(); await page.getByRole('option', {name: /Set a catalog session property/}).click(); - await page.waitForTimeout(600); + await page.waitForTimeout(800); }, }, {id: 'admin/gov-engine-policies', path: `${gov}/engines`, settle: 2000, prepare: seq(click('docs-trino'), tab('Policies'))}, {id: 'admin/gov-engine-plugin', path: `${gov}/engines`, settle: 2000, prepare: seq(click('docs-trino'), tab('Plugin'))}, {id: 'admin/gov-engine-advanced', path: `${gov}/engines`, settle: 2000, prepare: seq(click('docs-trino'), tab('Advanced'))}, - {id: 'admin/gov-profile', path: `${gov}/engines/profiles/docs-trino-profile`, settle: 2000}, - {id: 'admin/gov-policy-editor', path: `${gov}/engines/profiles/docs-trino-profile/advanced/policies/access/new`, settle: 2000}, + { + id: 'admin/gov-engine-migrate', + path: `${gov}/engines`, + settle: 1000, + prepare: async (page) => { + await click('trino-demo-ext')(page); + await tab('Migrate')(page); + await page.locator('main').getByRole('checkbox').last().check(); + await page.getByRole('button', {name: 'Preview migration'}).click(); + await page.waitForTimeout(5000); + }, + }, + {id: 'admin/gov-profile', path: `${gov}/engines`, settle: 2000, prepare: link('docs-external-policy')}, + { + id: 'admin/gov-policy-editor', + path: `${gov}/engines`, + settle: 2000, + prepare: async (page) => { + await click('trino-demo-ext')(page); + await tab('Advanced')(page); + await click('legacy: analysts read finance_pg')(page); + }, + }, + { + id: 'admin/gov-policy-editor-permissions', + path: `${gov}/engines`, + settle: 2000, + prepare: async (page) => { + await click('trino-demo-ext')(page); + await tab('Advanced')(page); + await click('legacy: analysts read finance_pg')(page); + await page.getByRole('button', {name: /^Permissions/}).click(); + }, + }, {id: 'admin/gov-classifications', path: `${gov}/classifications`, settle: 2000}, { id: 'admin/gov-tag-detail', path: `${gov}/classifications`, settle: 2500, prepare: click('Personal data (PII)'), - fullPage: true, }, - {id: 'admin/gov-tag-clearance', path: `${gov}/classifications`, settle: 2500, prepare: click('Personal data (PII)')}, + { + id: 'admin/gov-tag-rules', + path: `${gov}/classifications`, + settle: 2500, + prepare: seq(click('Personal data (PII)'), tab('Mask & filter')), + }, + { + id: 'admin/gov-tag-exposure', + path: `${gov}/classifications`, + settle: 3000, + prepare: seq(click('Personal data (PII)'), tab('View exposure')), + }, + {id: 'admin/gov-tag-clearance', path: `${gov}/classifications`, settle: 2500, prepare: seq(click('Confidential'), tab('Clearance'))}, { id: 'admin/gov-add-tag', path: `${T}?column=started_at`, settle: 800, prepare: async (page) => { await page.getByRole('button', {name: 'Add tag'}).last().click(); - await page.waitForTimeout(800); + await page.waitForTimeout(600); + await page.getByRole('dialog').getByRole('combobox').first().click(); + await page.waitForTimeout(600); }, }, @@ -155,12 +206,13 @@ export const shots = [ }, }, {id: 'admin/cp-workspace-members', path: `${cp}/workspaces`, settle: 1500, prepare: seq(click('Docs sandbox'), tab('Members'))}, - {id: 'admin/cp-workspace-resources', path: `${cp}/workspaces`, settle: 1500, prepare: seq(click('Default'), tab('Resources'))}, - {id: 'admin/cp-workspace-features', path: `${cp}/workspaces`, settle: 1500, prepare: seq(click('Default'), tab('Features')), fullPage: true}, + {id: 'admin/cp-workspace-resources', path: `${cp}/workspaces`, settle: 1500, prepare: seq(click('Docs sandbox'), tab('Resources'))}, + {id: 'admin/cp-workspace-features', path: `${cp}/workspaces`, settle: 1500, prepare: seq(click('Docs sandbox'), tab('Features'))}, + {id: 'admin/cp-workspace-requests', path: `${cp}/workspaces`, settle: 1500, prepare: seq(click('Finance analytics'), tab('Access requests'))}, {id: 'admin/cp-users', path: `${cp}/users`, settle: 1500}, - {id: 'admin/cp-user-detail', path: `${cp}/users`, settle: 2000, prepare: click('Member SSO'), fullPage: true}, + {id: 'admin/cp-user-detail', path: `${cp}/users`, settle: 2000, prepare: seq(click('Farid Haddad'))}, {id: 'admin/cp-groups', path: `${cp}/groups`, settle: 1500}, - {id: 'admin/cp-group-detail', path: `${cp}/groups`, settle: 2000, prepare: click('data-team'), fullPage: true}, + {id: 'admin/cp-group-detail', path: `${cp}/groups`, settle: 2000, prepare: seq(click('finance-analysts'))}, {id: 'admin/cp-roles', path: `${cp}/roles`, settle: 1500}, { id: 'admin/cp-role-new', @@ -171,11 +223,11 @@ export const shots = [ const d = page.getByRole('dialog'); await d.getByRole('combobox').click(); await page.getByRole('option', {name: 'Workspace'}).click(); - await d.locator('#new-role-name').fill('marketing_analyst'); - await d.locator('#new-role-description').fill('Reads datasets and builds charts'); + await d.locator('#new-role-name').fill('report_viewer'); + await d.locator('#new-role-description').fill('Views dashboards and receives reports'); }, }, - {id: 'admin/cp-role-detail', path: `${cp}/roles`, settle: 2000, prepare: click('finance_reporter'), fullPage: true}, + {id: 'admin/cp-role-detail', path: `${cp}/roles`, settle: 2000, prepare: seq(click('finance_reporter'))}, {id: 'admin/cp-permissions', path: `${cp}/permissions`, settle: 1500}, { id: 'admin/cp-explorer-user', @@ -183,9 +235,10 @@ export const shots = [ settle: 2500, prepare: async (page) => { await page.getByText('Select a user…').click(); - await page.getByText('Member SSO', {exact: false}).last().click(); + await page.keyboard.type('alice'); + await page.waitForTimeout(800); + await page.getByText('Alice Nguyen', {exact: false}).last().click(); }, - fullPage: true, }, { id: 'admin/cp-explorer-object', @@ -199,7 +252,6 @@ export const shots = [ await page.waitForTimeout(800); await page.getByText('Finance KPIs').last().click(); }, - fullPage: true, }, { id: 'admin/cp-roles-tree', @@ -215,34 +267,79 @@ export const shots = [ settle: 800, prepare: async (page) => { await page.getByRole('button', {name: 'Bulk select'}).click(); + await page.getByRole('checkbox').nth(2).check(); + await page.getByRole('checkbox').nth(3).check(); + }, + }, + {id: 'admin/cp-engines', path: `${cp}/engines`, settle: 2000}, + {id: 'admin/cp-engine-register', path: `${cp}/engines/register`, settle: 1500}, + { + id: 'admin/cp-engine-new', + path: `${cp}/engines/new`, + settle: 1000, + prepare: async (page) => { + await page.locator('#ec-name').fill('analytics'); + await page.getByRole('radio', {name: 'Make demo_warehouse primary'}).click(); await page.getByRole('checkbox').nth(1).check(); + await page.getByRole('button', {name: 'Worker', exact: true}).click(); + await page.locator('#ced-worker-replicas').fill('0'); + }, + }, + { + id: 'admin/cp-engine-security', + path: `${cp}/engines/new`, + settle: 1000, + prepare: async (page) => { + await page.getByRole('button', {name: 'Security & Exposure', exact: true}).click(); + await page.waitForTimeout(500); + await page.getByText('Ingress exposure', {exact: true}).scrollIntoViewIfNeeded(); + }, + }, + {id: 'admin/cp-engine-overview', path: `${cp}/engines`, settle: 2000, prepare: engine('docs-trino')}, + {id: 'admin/cp-engine-catalogs', path: `${cp}/engines`, settle: 2000, prepare: seq(engine('docs-trino'), tab('Catalogs'))}, + {id: 'admin/cp-engine-workspaces', path: `${cp}/engines`, settle: 2000, prepare: seq(engine('docs-trino'), tab('Workspaces'))}, + {id: 'admin/cp-engine-settings', path: `${cp}/engines`, settle: 2000, prepare: seq(engine('docs-trino'), tab('Settings'))}, + {id: 'admin/cp-engine-monitor', path: `${cp}/engines`, settle: 4000, prepare: seq(engine('docs-trino'), tab('Monitor'))}, + { + id: 'admin/cp-engine-logs', + path: `${cp}/engines`, + settle: 2500, + prepare: async (page) => { + await engine('docs-trino')(page); + await page.getByRole('button', {name: 'Logs'}).first().click(); + }, + }, + { + id: 'admin/cp-engine-delete', + path: `${cp}/engines`, + settle: 800, + prepare: async (page) => { + await engine('docs-trino')(page); + await page.getByRole('button', {name: 'More engine actions'}).click(); + await page.getByRole('menuitem', {name: 'Delete engine'}).click(); }, }, - {id: 'admin/cp-engines', path: `${cp}/engines`, settle: 1500}, - {id: 'admin/cp-engine-register', path: `${cp}/engines/register`, settle: 1500, fullPage: true}, - {id: 'admin/cp-engine-new', path: `${cp}/engines/new`, settle: 1500}, - {id: 'admin/cp-engine-overview', path: `${cp}/engines`, settle: 1500, prepare: click('docs-trino')}, - {id: 'admin/cp-engine-catalogs', path: `${cp}/engines`, settle: 1500, prepare: seq(click('docs-trino'), tab('Catalogs'))}, - {id: 'admin/cp-engine-settings', path: `${cp}/engines`, settle: 1500, prepare: seq(click('docs-trino'), tab('Settings'))}, - {id: 'admin/cp-engine-plugin', path: `${cp}/engines`, settle: 1500, prepare: seq(click('docs-trino'), tab('Plugin config')), fullPage: true}, + {id: 'admin/cp-engine-ext-overview', path: `${cp}/engines`, settle: 2000, prepare: engine('trino-demo-ext')}, + {id: 'admin/cp-engine-plugin', path: `${cp}/engines`, settle: 2000, prepare: seq(engine('trino-demo-ext'), tab('Plugin config'))}, + {id: 'admin/cp-engine-ext-settings', path: `${cp}/engines`, settle: 2000, prepare: seq(engine('trino-demo-ext'), tab('Settings'))}, {id: 'admin/cp-catalogs', path: `${cp}/catalogs`, settle: 1500}, { id: 'admin/cp-catalog-new', path: `${cp}/catalogs/new`, settle: 800, prepare: async (page) => { - await page.getByRole('combobox').click(); - await page.getByRole('option', {name: 'Iceberg', exact: true}).click(); + await page.getByRole('combobox').first().click(); + await page.getByRole('option', {name: 'TPC-H', exact: true}).click(); + await page.getByRole('textbox', {name: 'Name', exact: true}).fill('tpch_large'); }, - fullPage: true, }, - {id: 'admin/cp-catalog-config', path: `${cp}/catalogs`, settle: 1500, prepare: click('tpch_bench')}, - {id: 'admin/cp-catalog-engines', path: `${cp}/catalogs`, settle: 1500, prepare: seq(click('tpch_bench'), tab('Engines'))}, - {id: 'admin/cp-catalog-workspaces', path: `${cp}/catalogs`, settle: 1500, prepare: seq(click('tpch_bench'), tab('Workspaces')), fullPage: true}, - {id: 'admin/cp-catalog-roles', path: `${cp}/catalogs`, settle: 1500, prepare: seq(click('demo_catalog'), tab('Roles & grants'))}, - {id: 'admin/cp-catalog-browse', path: `${cp}/catalogs`, settle: 1500, prepare: seq(click('demo_catalog'), tab('Browse'), click('analytics'), click('sessions'))}, - {id: 'admin/cp-catalog-connect', path: `${cp}/catalogs`, settle: 1500, prepare: seq(click('demo_catalog'), tab('Connect')), fullPage: true}, - {id: 'admin/cp-catalog-settings', path: `${cp}/catalogs`, settle: 1500, prepare: seq(click('demo_catalog'), tab('Settings'))}, + {id: 'admin/cp-catalog-config', path: `${cp}/catalogs`, settle: 1500, prepare: link('tpch_bench')}, + {id: 'admin/cp-catalog-engines', path: `${cp}/catalogs`, settle: 1500, prepare: seq(link('tpch_bench'), tab('Engines'))}, + {id: 'admin/cp-catalog-workspaces', path: `${cp}/catalogs`, settle: 1500, prepare: seq(link('demo_warehouse'), tab('Workspaces'))}, + {id: 'admin/cp-catalog-roles', path: `${cp}/catalogs`, settle: 1500, prepare: seq(link('demo_warehouse'), tab('Roles & grants'))}, + {id: 'admin/cp-catalog-browse', path: `${cp}/catalogs`, settle: 2000, prepare: seq(link('demo_warehouse'), tab('Browse'), click('analytics'), click('sessions'))}, + {id: 'admin/cp-catalog-connect', path: `${cp}/catalogs`, settle: 1500, prepare: seq(link('demo_warehouse'), tab('Connect'))}, + {id: 'admin/cp-catalog-settings', path: `${cp}/catalogs`, settle: 1500, prepare: seq(link('demo_warehouse'), tab('Settings'))}, {id: 'admin/cp-storage', path: `${cp}/storage`, settle: 1500}, { id: 'admin/cp-storage-add', @@ -253,7 +350,7 @@ export const shots = [ }, }, {id: 'admin/cp-storage-detail', path: `${cp}/storage`, settle: 1500, prepare: click('demo-rustfs')}, - {id: 'admin/cp-storage-browse', path: `${cp}/storage`, settle: 2500, prepare: seq(click('sandbox-lake'), tab('Browse'))}, + {id: 'admin/cp-storage-browse', path: `${cp}/storage`, settle: 2500, prepare: seq(click('demo-rustfs'), tab('Browse'))}, {id: 'admin/cp-secrets', path: `${cp}/secrets`, settle: 1500}, { id: 'admin/cp-secret-add', @@ -273,13 +370,13 @@ export const shots = [ path: `${cp}/secrets`, settle: 600, prepare: async (page) => { - await click('docs-warehouse-password')(page); + await click('marketing-api-token')(page); await page.getByRole('button', {name: 'Rotate value'}).click(); }, }, {id: 'admin/cp-audit-log', path: `${cp}/audit-log`, settle: 2000}, - // ---- Workspace settings (Docs sandbox / Finance analytics are created for these pages) + // ---- Workspace settings (Docs sandbox / Finance analytics are created by scripts/seed-admin.mjs) {id: 'admin/ws-general', path: '/explore/docs-sandbox/settings/general', settle: 1500}, {id: 'admin/ws-members', path: '/explore/docs-sandbox/settings/members', settle: 1500}, { @@ -288,7 +385,7 @@ export const shots = [ settle: 600, prepare: async (page) => { await page.getByRole('button', {name: 'Add member'}).click(); - await page.getByRole('dialog').getByLabel('User email').fill('tester@egret.local'); + await page.getByRole('dialog').getByLabel('User email').fill('emma.schmidt@egret.local'); }, }, { @@ -297,6 +394,8 @@ export const shots = [ settle: 800, prepare: async (page) => { await page.getByRole('button', {name: 'Bulk select'}).click(); + await page.getByRole('checkbox').nth(3).check(); + await page.getByRole('checkbox').nth(4).check(); }, }, {id: 'admin/ws-roles', path: '/explore/docs-sandbox/settings/roles', settle: 1500}, @@ -304,8 +403,7 @@ export const shots = [ id: 'admin/ws-roles-expanded', path: '/explore/docs-sandbox/settings/roles', settle: 1000, - prepare: click('finance_reporter'), - fullPage: true, + prepare: seq(click('finance_reporter')), }, {id: 'admin/ws-requests', path: '/explore/finance-analytics/settings/requests', settle: 1500}, { @@ -318,7 +416,7 @@ export const shots = [ }, {id: 'admin/ws-members-finance', path: '/explore/finance-analytics/settings/members', settle: 1500}, {id: 'admin/ws-shares', path: '/explore/docs-sandbox/settings/shares', settle: 1500}, - {id: 'admin/ws-notifications', path: '/explore/docs-sandbox/settings/notifications', settle: 1500, fullPage: true}, + {id: 'admin/ws-notifications', path: '/explore/docs-sandbox/settings/notifications', settle: 1500}, {id: 'admin/ws-engines', path: '/explore/docs-sandbox/settings/engines', settle: 1500}, { id: 'admin/ws-add-connection', diff --git a/static/img/platform/admin/cp-audit-log.png b/static/img/platform/admin/cp-audit-log.png index 33eb656..bb87822 100644 Binary files a/static/img/platform/admin/cp-audit-log.png and b/static/img/platform/admin/cp-audit-log.png differ diff --git a/static/img/platform/admin/cp-catalog-browse.png b/static/img/platform/admin/cp-catalog-browse.png index 544273b..542f3cd 100644 Binary files a/static/img/platform/admin/cp-catalog-browse.png and b/static/img/platform/admin/cp-catalog-browse.png differ diff --git a/static/img/platform/admin/cp-catalog-config.png b/static/img/platform/admin/cp-catalog-config.png index 11b486a..719e56b 100644 Binary files a/static/img/platform/admin/cp-catalog-config.png and b/static/img/platform/admin/cp-catalog-config.png differ diff --git a/static/img/platform/admin/cp-catalog-connect.png b/static/img/platform/admin/cp-catalog-connect.png index 7bf8edd..48d81f7 100644 Binary files a/static/img/platform/admin/cp-catalog-connect.png and b/static/img/platform/admin/cp-catalog-connect.png differ diff --git a/static/img/platform/admin/cp-catalog-engines.png b/static/img/platform/admin/cp-catalog-engines.png index 1cbf99f..1f1bb8b 100644 Binary files a/static/img/platform/admin/cp-catalog-engines.png and b/static/img/platform/admin/cp-catalog-engines.png differ diff --git a/static/img/platform/admin/cp-catalog-new.png b/static/img/platform/admin/cp-catalog-new.png index d49d12c..228b0a0 100644 Binary files a/static/img/platform/admin/cp-catalog-new.png and b/static/img/platform/admin/cp-catalog-new.png differ diff --git a/static/img/platform/admin/cp-catalog-roles.png b/static/img/platform/admin/cp-catalog-roles.png index 666a497..7deb81c 100644 Binary files a/static/img/platform/admin/cp-catalog-roles.png and b/static/img/platform/admin/cp-catalog-roles.png differ diff --git a/static/img/platform/admin/cp-catalog-settings.png b/static/img/platform/admin/cp-catalog-settings.png index f36ed68..c89dd32 100644 Binary files a/static/img/platform/admin/cp-catalog-settings.png and b/static/img/platform/admin/cp-catalog-settings.png differ diff --git a/static/img/platform/admin/cp-catalog-workspaces.png b/static/img/platform/admin/cp-catalog-workspaces.png index 942823a..64b2595 100644 Binary files a/static/img/platform/admin/cp-catalog-workspaces.png and b/static/img/platform/admin/cp-catalog-workspaces.png differ diff --git a/static/img/platform/admin/cp-catalogs.png b/static/img/platform/admin/cp-catalogs.png index 9014309..2b83084 100644 Binary files a/static/img/platform/admin/cp-catalogs.png and b/static/img/platform/admin/cp-catalogs.png differ diff --git a/static/img/platform/admin/cp-engine-catalogs.png b/static/img/platform/admin/cp-engine-catalogs.png index ec99067..1328a9d 100644 Binary files a/static/img/platform/admin/cp-engine-catalogs.png and b/static/img/platform/admin/cp-engine-catalogs.png differ diff --git a/static/img/platform/admin/cp-engine-delete.png b/static/img/platform/admin/cp-engine-delete.png new file mode 100644 index 0000000..ffdf9c4 Binary files /dev/null and b/static/img/platform/admin/cp-engine-delete.png differ diff --git a/static/img/platform/admin/cp-engine-ext-overview.png b/static/img/platform/admin/cp-engine-ext-overview.png new file mode 100644 index 0000000..8ce8daa Binary files /dev/null and b/static/img/platform/admin/cp-engine-ext-overview.png differ diff --git a/static/img/platform/admin/cp-engine-ext-settings.png b/static/img/platform/admin/cp-engine-ext-settings.png new file mode 100644 index 0000000..80d5a51 Binary files /dev/null and b/static/img/platform/admin/cp-engine-ext-settings.png differ diff --git a/static/img/platform/admin/cp-engine-logs.png b/static/img/platform/admin/cp-engine-logs.png new file mode 100644 index 0000000..6c3b476 Binary files /dev/null and b/static/img/platform/admin/cp-engine-logs.png differ diff --git a/static/img/platform/admin/cp-engine-monitor.png b/static/img/platform/admin/cp-engine-monitor.png new file mode 100644 index 0000000..d135c8f Binary files /dev/null and b/static/img/platform/admin/cp-engine-monitor.png differ diff --git a/static/img/platform/admin/cp-engine-new.png b/static/img/platform/admin/cp-engine-new.png index 202a9a4..2601bee 100644 Binary files a/static/img/platform/admin/cp-engine-new.png and b/static/img/platform/admin/cp-engine-new.png differ diff --git a/static/img/platform/admin/cp-engine-overview.png b/static/img/platform/admin/cp-engine-overview.png index 61bf5e2..e2fed8a 100644 Binary files a/static/img/platform/admin/cp-engine-overview.png and b/static/img/platform/admin/cp-engine-overview.png differ diff --git a/static/img/platform/admin/cp-engine-plugin.png b/static/img/platform/admin/cp-engine-plugin.png index ead19ee..620ee8a 100644 Binary files a/static/img/platform/admin/cp-engine-plugin.png and b/static/img/platform/admin/cp-engine-plugin.png differ diff --git a/static/img/platform/admin/cp-engine-register.png b/static/img/platform/admin/cp-engine-register.png index 5f135c0..b97db22 100644 Binary files a/static/img/platform/admin/cp-engine-register.png and b/static/img/platform/admin/cp-engine-register.png differ diff --git a/static/img/platform/admin/cp-engine-security.png b/static/img/platform/admin/cp-engine-security.png new file mode 100644 index 0000000..38c2abd Binary files /dev/null and b/static/img/platform/admin/cp-engine-security.png differ diff --git a/static/img/platform/admin/cp-engine-settings.png b/static/img/platform/admin/cp-engine-settings.png index 1216aa1..e035ebf 100644 Binary files a/static/img/platform/admin/cp-engine-settings.png and b/static/img/platform/admin/cp-engine-settings.png differ diff --git a/static/img/platform/admin/cp-engine-workspaces.png b/static/img/platform/admin/cp-engine-workspaces.png new file mode 100644 index 0000000..bfee709 Binary files /dev/null and b/static/img/platform/admin/cp-engine-workspaces.png differ diff --git a/static/img/platform/admin/cp-engines.png b/static/img/platform/admin/cp-engines.png index 8aaf7cd..52ac658 100644 Binary files a/static/img/platform/admin/cp-engines.png and b/static/img/platform/admin/cp-engines.png differ diff --git a/static/img/platform/admin/cp-explorer-object.png b/static/img/platform/admin/cp-explorer-object.png index bfb8b2d..12e69a0 100644 Binary files a/static/img/platform/admin/cp-explorer-object.png and b/static/img/platform/admin/cp-explorer-object.png differ diff --git a/static/img/platform/admin/cp-explorer-user.png b/static/img/platform/admin/cp-explorer-user.png index cdd84db..6bb1345 100644 Binary files a/static/img/platform/admin/cp-explorer-user.png and b/static/img/platform/admin/cp-explorer-user.png differ diff --git a/static/img/platform/admin/cp-group-detail.png b/static/img/platform/admin/cp-group-detail.png index 88a9aca..1ed02ab 100644 Binary files a/static/img/platform/admin/cp-group-detail.png and b/static/img/platform/admin/cp-group-detail.png differ diff --git a/static/img/platform/admin/cp-groups.png b/static/img/platform/admin/cp-groups.png index 650ff25..437687f 100644 Binary files a/static/img/platform/admin/cp-groups.png and b/static/img/platform/admin/cp-groups.png differ diff --git a/static/img/platform/admin/cp-permissions.png b/static/img/platform/admin/cp-permissions.png index 4db59e4..39b629f 100644 Binary files a/static/img/platform/admin/cp-permissions.png and b/static/img/platform/admin/cp-permissions.png differ diff --git a/static/img/platform/admin/cp-role-detail.png b/static/img/platform/admin/cp-role-detail.png index 00c7e1a..e6d7840 100644 Binary files a/static/img/platform/admin/cp-role-detail.png and b/static/img/platform/admin/cp-role-detail.png differ diff --git a/static/img/platform/admin/cp-role-new.png b/static/img/platform/admin/cp-role-new.png index 74ec393..d0ca3e0 100644 Binary files a/static/img/platform/admin/cp-role-new.png and b/static/img/platform/admin/cp-role-new.png differ diff --git a/static/img/platform/admin/cp-roles-tree.png b/static/img/platform/admin/cp-roles-tree.png index 1ef60df..226f23a 100644 Binary files a/static/img/platform/admin/cp-roles-tree.png and b/static/img/platform/admin/cp-roles-tree.png differ diff --git a/static/img/platform/admin/cp-roles.png b/static/img/platform/admin/cp-roles.png index 995cc78..ed31fb3 100644 Binary files a/static/img/platform/admin/cp-roles.png and b/static/img/platform/admin/cp-roles.png differ diff --git a/static/img/platform/admin/cp-secret-add.png b/static/img/platform/admin/cp-secret-add.png index 7bd3745..058f98e 100644 Binary files a/static/img/platform/admin/cp-secret-add.png and b/static/img/platform/admin/cp-secret-add.png differ diff --git a/static/img/platform/admin/cp-secret-detail.png b/static/img/platform/admin/cp-secret-detail.png index 83e8825..58d5fa1 100644 Binary files a/static/img/platform/admin/cp-secret-detail.png and b/static/img/platform/admin/cp-secret-detail.png differ diff --git a/static/img/platform/admin/cp-secret-rotate.png b/static/img/platform/admin/cp-secret-rotate.png index d224072..bf77cfb 100644 Binary files a/static/img/platform/admin/cp-secret-rotate.png and b/static/img/platform/admin/cp-secret-rotate.png differ diff --git a/static/img/platform/admin/cp-secrets.png b/static/img/platform/admin/cp-secrets.png index b5ad25f..29f0815 100644 Binary files a/static/img/platform/admin/cp-secrets.png and b/static/img/platform/admin/cp-secrets.png differ diff --git a/static/img/platform/admin/cp-storage-add.png b/static/img/platform/admin/cp-storage-add.png index d25aaa0..5d2dcd3 100644 Binary files a/static/img/platform/admin/cp-storage-add.png and b/static/img/platform/admin/cp-storage-add.png differ diff --git a/static/img/platform/admin/cp-storage-browse.png b/static/img/platform/admin/cp-storage-browse.png index 59b85a0..c4f4314 100644 Binary files a/static/img/platform/admin/cp-storage-browse.png and b/static/img/platform/admin/cp-storage-browse.png differ diff --git a/static/img/platform/admin/cp-storage-detail.png b/static/img/platform/admin/cp-storage-detail.png index 634e8be..19eb11e 100644 Binary files a/static/img/platform/admin/cp-storage-detail.png and b/static/img/platform/admin/cp-storage-detail.png differ diff --git a/static/img/platform/admin/cp-storage.png b/static/img/platform/admin/cp-storage.png index 884d4c9..ae75c06 100644 Binary files a/static/img/platform/admin/cp-storage.png and b/static/img/platform/admin/cp-storage.png differ diff --git a/static/img/platform/admin/cp-user-detail.png b/static/img/platform/admin/cp-user-detail.png index d75b717..469b5a0 100644 Binary files a/static/img/platform/admin/cp-user-detail.png and b/static/img/platform/admin/cp-user-detail.png differ diff --git a/static/img/platform/admin/cp-users-bulk.png b/static/img/platform/admin/cp-users-bulk.png index 73ca5a2..ec38f1f 100644 Binary files a/static/img/platform/admin/cp-users-bulk.png and b/static/img/platform/admin/cp-users-bulk.png differ diff --git a/static/img/platform/admin/cp-users.png b/static/img/platform/admin/cp-users.png index 39625c8..0510353 100644 Binary files a/static/img/platform/admin/cp-users.png and b/static/img/platform/admin/cp-users.png differ diff --git a/static/img/platform/admin/cp-workspace-features.png b/static/img/platform/admin/cp-workspace-features.png index 7963bbf..93feca2 100644 Binary files a/static/img/platform/admin/cp-workspace-features.png and b/static/img/platform/admin/cp-workspace-features.png differ diff --git a/static/img/platform/admin/cp-workspace-members.png b/static/img/platform/admin/cp-workspace-members.png index 532b661..cd5fc16 100644 Binary files a/static/img/platform/admin/cp-workspace-members.png and b/static/img/platform/admin/cp-workspace-members.png differ diff --git a/static/img/platform/admin/cp-workspace-new.png b/static/img/platform/admin/cp-workspace-new.png index ea841eb..c90c86b 100644 Binary files a/static/img/platform/admin/cp-workspace-new.png and b/static/img/platform/admin/cp-workspace-new.png differ diff --git a/static/img/platform/admin/cp-workspace-requests.png b/static/img/platform/admin/cp-workspace-requests.png new file mode 100644 index 0000000..ed55eee Binary files /dev/null and b/static/img/platform/admin/cp-workspace-requests.png differ diff --git a/static/img/platform/admin/cp-workspace-resources.png b/static/img/platform/admin/cp-workspace-resources.png index 5d6e2ac..89f71c3 100644 Binary files a/static/img/platform/admin/cp-workspace-resources.png and b/static/img/platform/admin/cp-workspace-resources.png differ diff --git a/static/img/platform/admin/cp-workspaces.png b/static/img/platform/admin/cp-workspaces.png index 77fab35..f7d7260 100644 Binary files a/static/img/platform/admin/cp-workspaces.png and b/static/img/platform/admin/cp-workspaces.png differ diff --git a/static/img/platform/admin/gov-add-tag.png b/static/img/platform/admin/gov-add-tag.png index e3c69e6..6345df6 100644 Binary files a/static/img/platform/admin/gov-add-tag.png and b/static/img/platform/admin/gov-add-tag.png differ diff --git a/static/img/platform/admin/gov-classifications.png b/static/img/platform/admin/gov-classifications.png index c5a2254..3df6e91 100644 Binary files a/static/img/platform/admin/gov-classifications.png and b/static/img/platform/admin/gov-classifications.png differ diff --git a/static/img/platform/admin/gov-column-panel.png b/static/img/platform/admin/gov-column-panel.png index f7013ce..ab9c352 100644 Binary files a/static/img/platform/admin/gov-column-panel.png and b/static/img/platform/admin/gov-column-panel.png differ diff --git a/static/img/platform/admin/gov-data-schema.png b/static/img/platform/admin/gov-data-schema.png index 88a7d54..9c66e42 100644 Binary files a/static/img/platform/admin/gov-data-schema.png and b/static/img/platform/admin/gov-data-schema.png differ diff --git a/static/img/platform/admin/gov-engine-advanced.png b/static/img/platform/admin/gov-engine-advanced.png index 148a1e2..2e9b138 100644 Binary files a/static/img/platform/admin/gov-engine-advanced.png and b/static/img/platform/admin/gov-engine-advanced.png differ diff --git a/static/img/platform/admin/gov-engine-grant-dialog.png b/static/img/platform/admin/gov-engine-grant-dialog.png index 4ee59ef..19b6cde 100644 Binary files a/static/img/platform/admin/gov-engine-grant-dialog.png and b/static/img/platform/admin/gov-engine-grant-dialog.png differ diff --git a/static/img/platform/admin/gov-engine-migrate.png b/static/img/platform/admin/gov-engine-migrate.png new file mode 100644 index 0000000..8a485c9 Binary files /dev/null and b/static/img/platform/admin/gov-engine-migrate.png differ diff --git a/static/img/platform/admin/gov-engine-permissions.png b/static/img/platform/admin/gov-engine-permissions.png index 31b7e3e..da52a0f 100644 Binary files a/static/img/platform/admin/gov-engine-permissions.png and b/static/img/platform/admin/gov-engine-permissions.png differ diff --git a/static/img/platform/admin/gov-engine-plugin.png b/static/img/platform/admin/gov-engine-plugin.png index 056a339..e58e22b 100644 Binary files a/static/img/platform/admin/gov-engine-plugin.png and b/static/img/platform/admin/gov-engine-plugin.png differ diff --git a/static/img/platform/admin/gov-engine-policies.png b/static/img/platform/admin/gov-engine-policies.png index 21a29f9..2420823 100644 Binary files a/static/img/platform/admin/gov-engine-policies.png and b/static/img/platform/admin/gov-engine-policies.png differ diff --git a/static/img/platform/admin/gov-engines-index.png b/static/img/platform/admin/gov-engines-index.png index a942827..560bf50 100644 Binary files a/static/img/platform/admin/gov-engines-index.png and b/static/img/platform/admin/gov-engines-index.png differ diff --git a/static/img/platform/admin/gov-grant-dialog.png b/static/img/platform/admin/gov-grant-dialog.png index e781c5a..7748a6e 100644 Binary files a/static/img/platform/admin/gov-grant-dialog.png and b/static/img/platform/admin/gov-grant-dialog.png differ diff --git a/static/img/platform/admin/gov-overview.png b/static/img/platform/admin/gov-overview.png index e26bff8..a437e56 100644 Binary files a/static/img/platform/admin/gov-overview.png and b/static/img/platform/admin/gov-overview.png differ diff --git a/static/img/platform/admin/gov-people-groups.png b/static/img/platform/admin/gov-people-groups.png index 148f40a..7546baa 100644 Binary files a/static/img/platform/admin/gov-people-groups.png and b/static/img/platform/admin/gov-people-groups.png differ diff --git a/static/img/platform/admin/gov-people-roles.png b/static/img/platform/admin/gov-people-roles.png index 11a64ae..c9164e6 100644 Binary files a/static/img/platform/admin/gov-people-roles.png and b/static/img/platform/admin/gov-people-roles.png differ diff --git a/static/img/platform/admin/gov-people-unmatched.png b/static/img/platform/admin/gov-people-unmatched.png index 7fd3f4b..1014983 100644 Binary files a/static/img/platform/admin/gov-people-unmatched.png and b/static/img/platform/admin/gov-people-unmatched.png differ diff --git a/static/img/platform/admin/gov-people-users.png b/static/img/platform/admin/gov-people-users.png index 7ef8874..ef9c0b7 100644 Binary files a/static/img/platform/admin/gov-people-users.png and b/static/img/platform/admin/gov-people-users.png differ diff --git a/static/img/platform/admin/gov-person-check.png b/static/img/platform/admin/gov-person-check.png index c966d5c..269f894 100644 Binary files a/static/img/platform/admin/gov-person-check.png and b/static/img/platform/admin/gov-person-check.png differ diff --git a/static/img/platform/admin/gov-person-clearances.png b/static/img/platform/admin/gov-person-clearances.png new file mode 100644 index 0000000..3aa06dd Binary files /dev/null and b/static/img/platform/admin/gov-person-clearances.png differ diff --git a/static/img/platform/admin/gov-person-overview.png b/static/img/platform/admin/gov-person-overview.png index f6f4395..1713f82 100644 Binary files a/static/img/platform/admin/gov-person-overview.png and b/static/img/platform/admin/gov-person-overview.png differ diff --git a/static/img/platform/admin/gov-person-permissions.png b/static/img/platform/admin/gov-person-permissions.png index bdbcf29..41f7dfc 100644 Binary files a/static/img/platform/admin/gov-person-permissions.png and b/static/img/platform/admin/gov-person-permissions.png differ diff --git a/static/img/platform/admin/gov-policy-editor-permissions.png b/static/img/platform/admin/gov-policy-editor-permissions.png new file mode 100644 index 0000000..c61f94e Binary files /dev/null and b/static/img/platform/admin/gov-policy-editor-permissions.png differ diff --git a/static/img/platform/admin/gov-policy-editor.png b/static/img/platform/admin/gov-policy-editor.png index c3f88a0..226ae82 100644 Binary files a/static/img/platform/admin/gov-policy-editor.png and b/static/img/platform/admin/gov-policy-editor.png differ diff --git a/static/img/platform/admin/gov-profile.png b/static/img/platform/admin/gov-profile.png index 51f76de..865525b 100644 Binary files a/static/img/platform/admin/gov-profile.png and b/static/img/platform/admin/gov-profile.png differ diff --git a/static/img/platform/admin/gov-rule-dialog.png b/static/img/platform/admin/gov-rule-dialog.png index 309da0d..af02bce 100644 Binary files a/static/img/platform/admin/gov-rule-dialog.png and b/static/img/platform/admin/gov-rule-dialog.png differ diff --git a/static/img/platform/admin/gov-table-details.png b/static/img/platform/admin/gov-table-details.png index 8fa4d9f..344bac5 100644 Binary files a/static/img/platform/admin/gov-table-details.png and b/static/img/platform/admin/gov-table-details.png differ diff --git a/static/img/platform/admin/gov-table-lineage.png b/static/img/platform/admin/gov-table-lineage.png index 5ca8742..653b05a 100644 Binary files a/static/img/platform/admin/gov-table-lineage.png and b/static/img/platform/admin/gov-table-lineage.png differ diff --git a/static/img/platform/admin/gov-table-masks.png b/static/img/platform/admin/gov-table-masks.png index 3b7333c..67ba4e6 100644 Binary files a/static/img/platform/admin/gov-table-masks.png and b/static/img/platform/admin/gov-table-masks.png differ diff --git a/static/img/platform/admin/gov-table-overview.png b/static/img/platform/admin/gov-table-overview.png index 1866cd3..4aea2a7 100644 Binary files a/static/img/platform/admin/gov-table-overview.png and b/static/img/platform/admin/gov-table-overview.png differ diff --git a/static/img/platform/admin/gov-table-permissions.png b/static/img/platform/admin/gov-table-permissions.png index 19281ae..1a9a860 100644 Binary files a/static/img/platform/admin/gov-table-permissions.png and b/static/img/platform/admin/gov-table-permissions.png differ diff --git a/static/img/platform/admin/gov-tag-clearance.png b/static/img/platform/admin/gov-tag-clearance.png index f209621..3b079b8 100644 Binary files a/static/img/platform/admin/gov-tag-clearance.png and b/static/img/platform/admin/gov-tag-clearance.png differ diff --git a/static/img/platform/admin/gov-tag-detail.png b/static/img/platform/admin/gov-tag-detail.png index f209621..edfe456 100644 Binary files a/static/img/platform/admin/gov-tag-detail.png and b/static/img/platform/admin/gov-tag-detail.png differ diff --git a/static/img/platform/admin/gov-tag-exposure.png b/static/img/platform/admin/gov-tag-exposure.png new file mode 100644 index 0000000..602253c Binary files /dev/null and b/static/img/platform/admin/gov-tag-exposure.png differ diff --git a/static/img/platform/admin/gov-tag-rules.png b/static/img/platform/admin/gov-tag-rules.png new file mode 100644 index 0000000..06d9630 Binary files /dev/null and b/static/img/platform/admin/gov-tag-rules.png differ diff --git a/static/img/platform/admin/gov-who-can-read.png b/static/img/platform/admin/gov-who-can-read.png index e7ee59d..418b5a9 100644 Binary files a/static/img/platform/admin/gov-who-can-read.png and b/static/img/platform/admin/gov-who-can-read.png differ diff --git a/static/img/platform/admin/ws-add-connection.png b/static/img/platform/admin/ws-add-connection.png index c56da22..b454155 100644 Binary files a/static/img/platform/admin/ws-add-connection.png and b/static/img/platform/admin/ws-add-connection.png differ diff --git a/static/img/platform/admin/ws-add-member.png b/static/img/platform/admin/ws-add-member.png index 5deb2d0..15af4c7 100644 Binary files a/static/img/platform/admin/ws-add-member.png and b/static/img/platform/admin/ws-add-member.png differ diff --git a/static/img/platform/admin/ws-engines.png b/static/img/platform/admin/ws-engines.png index a3f3ac8..00a0d21 100644 Binary files a/static/img/platform/admin/ws-engines.png and b/static/img/platform/admin/ws-engines.png differ diff --git a/static/img/platform/admin/ws-general.png b/static/img/platform/admin/ws-general.png index bfe1e4b..63e1223 100644 Binary files a/static/img/platform/admin/ws-general.png and b/static/img/platform/admin/ws-general.png differ diff --git a/static/img/platform/admin/ws-members-bulk.png b/static/img/platform/admin/ws-members-bulk.png index 963939e..fa3b08d 100644 Binary files a/static/img/platform/admin/ws-members-bulk.png and b/static/img/platform/admin/ws-members-bulk.png differ diff --git a/static/img/platform/admin/ws-members-finance.png b/static/img/platform/admin/ws-members-finance.png index 71222ec..4f7cdc6 100644 Binary files a/static/img/platform/admin/ws-members-finance.png and b/static/img/platform/admin/ws-members-finance.png differ diff --git a/static/img/platform/admin/ws-members.png b/static/img/platform/admin/ws-members.png index 6020f93..c1e4dd3 100644 Binary files a/static/img/platform/admin/ws-members.png and b/static/img/platform/admin/ws-members.png differ diff --git a/static/img/platform/admin/ws-notifications.png b/static/img/platform/admin/ws-notifications.png index a6255db..75880a2 100644 Binary files a/static/img/platform/admin/ws-notifications.png and b/static/img/platform/admin/ws-notifications.png differ diff --git a/static/img/platform/admin/ws-request-approve.png b/static/img/platform/admin/ws-request-approve.png index 956f7ba..6359dd9 100644 Binary files a/static/img/platform/admin/ws-request-approve.png and b/static/img/platform/admin/ws-request-approve.png differ diff --git a/static/img/platform/admin/ws-request-dialog.png b/static/img/platform/admin/ws-request-dialog.png index 8a301e2..b6feb5f 100644 Binary files a/static/img/platform/admin/ws-request-dialog.png and b/static/img/platform/admin/ws-request-dialog.png differ diff --git a/static/img/platform/admin/ws-requests.png b/static/img/platform/admin/ws-requests.png index 280d825..edde621 100644 Binary files a/static/img/platform/admin/ws-requests.png and b/static/img/platform/admin/ws-requests.png differ diff --git a/static/img/platform/admin/ws-roles-expanded.png b/static/img/platform/admin/ws-roles-expanded.png index f110da0..6820b38 100644 Binary files a/static/img/platform/admin/ws-roles-expanded.png and b/static/img/platform/admin/ws-roles-expanded.png differ diff --git a/static/img/platform/admin/ws-roles.png b/static/img/platform/admin/ws-roles.png index 5931834..fc78830 100644 Binary files a/static/img/platform/admin/ws-roles.png and b/static/img/platform/admin/ws-roles.png differ diff --git a/static/img/platform/admin/ws-shares.png b/static/img/platform/admin/ws-shares.png index 6778297..41a1661 100644 Binary files a/static/img/platform/admin/ws-shares.png and b/static/img/platform/admin/ws-shares.png differ diff --git a/static/img/platform/control-plane-access-overview.png b/static/img/platform/control-plane-access-overview.png index 7b176ed..e43e0c0 100644 Binary files a/static/img/platform/control-plane-access-overview.png and b/static/img/platform/control-plane-access-overview.png differ diff --git a/static/img/platform/control-plane-permissions.png b/static/img/platform/control-plane-permissions.png index 0051a8b..39b629f 100644 Binary files a/static/img/platform/control-plane-permissions.png and b/static/img/platform/control-plane-permissions.png differ