Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion platform/data-api/activity.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 7

**Activity** is the access log of the Data API: every call to one of the workspace's endpoints, with its status, duration and whether the cache answered. Use it to see who is calling what, find the call behind an error report, and spot an endpoint that has started to fail.

![The Activity page with summary tiles, a per-endpoint table and the calls list](/img/platform/data-api/activity.png)
![The Activity page for 24h: 282 calls, a 20.2% error rate shown as Failing, latency and cache-hit tiles, the per-endpoint table with No matching endpoint last, and the newest calls including 429 rate_limited](/img/platform/data-api/activity.png)

Open **Explore → Data API → Activity**.

Expand Down
8 changes: 4 additions & 4 deletions platform/data-api/calling-endpoints.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ You need:
- a **published** endpoint (here, `GET /orders/by-status`);
- a [service account with an OAuth2 client](./clients), and its `client_id` and `client_secret`;
- an invoke [grant](./endpoints/access) for that service account on the endpoint;
- the base URL of your deployment. On the demo instance it is `http://localhost:8080`.
- the base URL of your deployment. On the Docker demo stack it is `http://egret.localtest.me:8080` (or the port set in `DEMO_PORT`).

Set them as shell variables:

```bash
export BASE=http://localhost:8080
export BASE=http://egret.localtest.me:8080
export WS=default
export CLIENT_ID=egret-dataapi-bf232700-aeb0-461f-9b13-f1ce18ba805e
export CLIENT_ID=egret-dataapi-cf9e7fe8-f970-45af-901d-66a9a24b9ab7
export CLIENT_SECRET='…the secret shown once when the client was issued…'
```

Expand Down Expand Up @@ -180,7 +180,7 @@ content-type: application/json
retry-after: 1
```

On the walk-through endpoint (**Per caller (qps)** 5, **Burst** 10), forty simultaneous requests produced twelve `200` answers and twenty-eight `429`. A client should wait the `Retry-After` interval and retry, with a little jitter:
On the walk-through endpoint (**Per caller (qps)** 5, **Burst** 10), thirty simultaneous requests from one client produced 10 to 12 `200` answers and the rest `429`. A client should wait the `Retry-After` interval and retry, with a little jitter:

```bash
for i in 1 2 3; do
Expand Down
10 changes: 5 additions & 5 deletions platform/data-api/clients.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 3

A consumer that calls your endpoints needs a credential. In the Data API that is an **OAuth2 client** owned by a **service account**. A service account is a machine identity: it is the "who" that endpoints grant access to and that the query engine sees. A client is one `client_id` and `client_secret` pair that lets a program prove it acts for that account.

![The Clients page with two service accounts and their clients](/img/platform/data-api/clients-list.png)
![The Clients page with three service accounts and their clients: Active, Revoked and Expired](/img/platform/data-api/clients-list.png)

Open **Explore → Data API → Clients**. Each service account is a header row with its clients listed beneath it.

Expand All @@ -26,7 +26,7 @@ One service account can have several clients, for example one per consuming syst
|---|---|
| **Client** | The service account's name and email, with the number of clients and when it was created; under it, each client's name. |
| **ID** | A shortened `client_id` (`egret-da…805e`) with a copy button. |
| **Status** | **Active**, **Revoked** (with *revoked 3 minutes ago*) or **Expired** (with the expiry date). |
| **Status** | **Active**, **Revoked** (with *revoked an hour ago*) or **Expired** (with *expires 9 months ago*, which means it expired then; KI-025). |
| **Last used** | When the client last minted a token, or *Never*. |

**Search service accounts…** filters by name. The **⋯** menu on a service account offers **Issue client…** and **Delete**; on an active client it offers **Revoke…**. A client that is already revoked has no menu.
Expand All @@ -48,15 +48,15 @@ Names are unique in a workspace. Reusing one is refused: `service account alread
2. Enter a **Name** for the credential and, optionally, **Expires**. Name it after the consumer that will use it, so revoking the right one later is obvious. Leave **Expires** blank for a client that lives until revoked.
3. Press **Create client**.

![The New client page](/img/platform/data-api/client-new.png)
![The New client page for orders-reporting-bot with the name weekly-export](/img/platform/data-api/client-new.png)

The next page shows three values:

| Field | Meaning |
|---|---|
| **Client ID** | Public identifier, `egret-dataapi-` followed by a UUID. |
| **Client secret** | The secret. **Shown only once.** Press **Copy** and store it in the consumer's secret manager now. |
| **Token endpoint** | Where the client exchanges its credentials, for example `http://localhost:8080/oauth/token`. |
| **Token endpoint** | Where the client exchanges its credentials, for example `http://egret.localtest.me:8080/oauth/token`. |

Reloading the page or returning to it later shows *Client secret no longer available*: the secret cannot be read back, by design. If it is lost, revoke the client and issue another.

Expand Down Expand Up @@ -86,7 +86,7 @@ The status becomes **Revoked**. A request for a token with a revoked secret is a
A token is a signed, stateless credential. Revoking a client stops it minting new tokens, but a token already issued keeps working until it expires, up to 10 minutes later. In the walk-through, a token minted shortly before revocation still returned `200` afterwards. To cut a consumer off at once, remove the **grant** on the endpoint: grants are read on every request and take effect on the next call.
:::

An **Expired** client (its **Expires** date has passed) behaves like a revoked one for new tokens.
An **Expired** client (its **Expires** date has passed) behaves like a revoked one for new tokens: on the demo stack the expired client `partner-pilot` got `{"error":"invalid_client"}`.

## Delete a service account

Expand Down
4 changes: 2 additions & 2 deletions platform/data-api/endpoints/access.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 7

Nobody can call an endpoint until you say so. The **Access** tab lists the workspace's service accounts and lets you grant each one permission to **invoke** this endpoint. Grants are per endpoint, so a consumer you give the sales endpoint cannot call the finance one.

![The Access tab with one granted and one ungranted service account](/img/platform/data-api/endpoint-access.png)
![The Access tab of Orders by status: finance-dashboard and orders-reporting-bot granted on this endpoint, partner-sandbox not granted](/img/platform/data-api/endpoint-access.png)

## Grant a service account access to an endpoint

Expand Down Expand Up @@ -37,7 +37,7 @@ The status under **Invoke** is one of:

Two notes appear on a row when they matter:

- **Revoking will not remove access**, shown on a granted account that also reaches invoke through the workspace: revoking removes the grant but not the access.
- **Revoking will not remove access**, meant for a granted account that also reaches invoke through the workspace: revoking removes the grant but not the access. On the demo stack this note is shown on **every** granted service account, including ones with no workspace role, and is wrong for them: revoking `finance-dashboard` on **Orders by status** made its next call answer `403 not_granted`. Trust the result of **Try it** after a revoke, not the note.
- **Revoking may not remove access**, shown when Egret cannot check the inherited access, for example while the endpoint is not live.

The check for inherited access runs the account's own OpenAPI document and looks for this endpoint in it, so it only works on a live endpoint. A draft or disabled endpoint says so above the table: *This endpoint is not live, so nothing is being served yet and egret cannot check workspace-inherited invoke rights against it. Grants recorded here are real and take effect when it is published.*
Expand Down
4 changes: 2 additions & 2 deletions platform/data-api/endpoints/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 4

The **Config** tab holds everything about an endpoint that is not its query or its response: its name and URL, whether answers are cached, how big and how slow an answer may be, and how often each caller may ask. Changes are saved with **Save** and reach consumers when you [publish](./publishing).

![The Config tab showing the Endpoint and Cache groups](/img/platform/data-api/endpoint-config.png)
![The Config tab of Orders by status: Endpoint, Cache (enabled, TTL 60, Per caller), Limits (100 rows, 30 seconds) and Rate limit (5 qps, burst 10) groups, and Validate saved draft](/img/platform/data-api/endpoint-config.png)

The tab is a table with **Setting**, **Details** and **Value** columns, grouped under **Endpoint**, **Cache**, **Limits** and **Rate limit**, followed by a **Validate** check.

Expand Down Expand Up @@ -90,7 +90,7 @@ How many requests may arrive at once, above the sustained rate, before throttlin

A ceiling across all callers of this endpoint together, whatever their individual limits. `0` means no endpoint-wide limit.

With **Per caller (qps)** `5` and **Burst** `10`, forty simultaneous requests from one client produced twelve answers and twenty-eight `429` responses with `Retry-After: 1`: the ten burst slots plus the tokens that refilled while the burst ran. Sequential calls at normal speed were not throttled.
With **Per caller (qps)** `5` and **Burst** `10` on **Orders by status**, thirty simultaneous requests from one client produced 10 to 12 answers and 18 to 20 `429` responses with `Retry-After: 1`, and sixteen simultaneous requests from a second client of the same service account produced eleven answers and five `429`: the ten burst slots plus the tokens that refilled while the burst ran. Each client credential has its own bucket. Sequential calls at normal speed were not throttled.

:::note
With one Egret replica, rate-limit counters and the cache live in the server's memory. Running more than one replica requires a shared store (Redis or Valkey); Egret refuses to start with several replicas and no store, so limits are never silently multiplied. If the shared store cannot be reached, a call to an endpoint that has a rate limit is refused with `503 rate_limiter_unavailable`, and a cache lookup that fails is treated as a miss.
Expand Down
18 changes: 9 additions & 9 deletions platform/data-api/endpoints/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 1

An endpoint is one query published at one URL. This page covers the list of endpoints, how to create an endpoint, and how the editor is organised. Each editor tab has its own page.

![The endpoints list showing six live endpoints](/img/platform/data-api/endpoints-list.png)
![The endpoints list with eight endpoints: six live, one draft and one disabled](/img/platform/data-api/endpoints-list.png)

## The endpoints list

Expand All @@ -16,14 +16,14 @@ Open **Explore → Data API → Endpoints**. The list shows every endpoint in th
|---|---|
| **Name** | The endpoint's name, with a star to bookmark it. |
| **Method & path** | The HTTP method and the path callers append to the workspace's base URL, for example `GET /orders/by-status`. |
| **State** | **Live**, **Draft**, or **Disabled** (see [lifecycle](./overview#endpoint-lifecycle)). |
| **State** | **Live**, **Draft**, or **Disabled · was live** (see [lifecycle](./overview#endpoint-lifecycle)). |
| **Engine** | The engine of the endpoint's database connection, for example Trino. |
| **Last edited** | When the draft last changed. |

Above the table:

- **Search endpoints…** filters by name.
- **State** and **Engine** narrow the list. The counters under the filters (`6 endpoints total · 6 live · 0 draft · 0 disabled`) are clickable shortcuts for the State filter.
- **State** and **Engine** narrow the list. The counters under the filters (`8 endpoints total · 6 live · 1 draft · 1 disabled`) are clickable shortcuts for the State filter.
- The toolbar buttons are **Bookmarks** (show only starred endpoints), **Bulk select** (select several rows and press **Move N to Trash**), **New folder**, and **Trash**.
- **Download OpenAPI** exports an OpenAPI document; see [OpenAPI export](./openapi).
- **+ Endpoint** opens the create page.
Expand All @@ -39,10 +39,10 @@ The **⋯** menu on a row offers:
| **Open** | Opens the editor. |
| **Disable** | Takes a live endpoint out of service. Calls get `404 not_found`. Its live version and grants are kept. |
| **Enable** | Puts a disabled endpoint back, serving the version it had. Not available for an endpoint that was never published. |
| **Move to folder** | Moves it into a folder or a new folder. |
| **Move to Trash** | Soft-deletes the endpoint. Disabled for a **Live** endpoint: the menu says *Disable it first*. |
| **Move to new folder…** | Creates a folder and moves the endpoint into it. With folders in the workspace the menu also offers **Move to "folder"** for each one, and **Remove from folder** for an endpoint in a folder. |
| **Move to Trash** | Soft-deletes the endpoint. Greyed out for a **Live** endpoint, with the hint *Disable it first*. |

![The row menu for an endpoint](/img/platform/data-api/endpoint-actions-menu.png)
![The row menu of the live endpoint Top customers: Open, Disable, Move to new folder and a greyed-out Move to Trash with Disable it first](/img/platform/data-api/endpoint-actions-menu.png)

In **Trash**, a row offers **Restore** (the endpoint returns **disabled**, never straight to serving) and **Delete permanently**, which cannot be undone.

Expand All @@ -52,7 +52,7 @@ In **Trash**, a row offers **Restore** (the endpoint returns **disabled**, never
2. Fill in **Name**, **Path**, an optional **Description**, **Method**, **Database** and **SQL**.
3. Press **Create endpoint**.

![The New endpoint page filled in](/img/platform/data-api/endpoint-new.png)
![The New endpoint page filled in for Customer spend on trino-demo](/img/platform/data-api/endpoint-new.png)

The create page collects only what an endpoint cannot exist without. Parameters, response shape, cache and limits are configured in the editor you land in next. Nothing is served yet: a new endpoint is a **Draft**.

Expand Down Expand Up @@ -104,10 +104,10 @@ Opening an endpoint shows its editor. The header carries the name, a state badge
| **Try it** | Make a real call as a service account. | [Try it](./endpoints/try-it) |
| **Versions** | Publish, review history, roll back. | [Publishing](./endpoints/publishing) |

The header badge reads **Draft · never published**, **Live · v2**, **Live · v2 · unpublished changes** (the saved draft differs from what is served) or **Disabled · was live**. Pressing the badge opens **Versions**.
The header badge reads **Draft · never published**, **Live · v2**, **Live · v1 · unpublished changes** (the saved draft differs from what is served, as on **Top customers** in the walk-through) or **Disabled · was live**. Pressing the badge opens **Versions**.

:::caution Changing the path moves the live URL
In the version of Egret used for these docs, saving a new **Path** in **Config** changes the URL that is served immediately, before you publish. A call to the old path returns `404 not_found` while the new path answers with the live version's data. Publishing then reports the change as breaking. Treat a path change as a breaking change to make deliberately. See [Publishing](./endpoints/publishing#breaking-changes).
Saving a new **Path** in **Config** changes the URL that is served immediately, before you publish. A call to the old path returns `404 not_found` while the new path answers with the live version's data. Publishing then reports the change as breaking. Treat a path change as a breaking change to make deliberately. See [Publishing](./endpoints/publishing#breaking-changes).
:::

## Limits and known issues
Expand Down
2 changes: 1 addition & 1 deletion platform/data-api/endpoints/parameters.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 2

Parameters let a caller change what an endpoint returns: filter by a status, pick a date, choose a sort column. You declare each parameter on the **Query** tab and refer to it in the SQL as `{{ name }}`. Egret checks every value before the query runs and sends it to the engine as a bound value, never as pasted text.

![The Query tab with the parameters table open](/img/platform/data-api/endpoint-params.png)
![The Query tab of Orders by status with Database and parameters open: priority (enum) and min_total (number, 0 to 1000000)](/img/platform/data-api/endpoint-params.png)

## Declare a parameter

Expand Down
6 changes: 3 additions & 3 deletions platform/data-api/endpoints/publishing.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 8

Consumers never see your draft. They see a **version**: a frozen snapshot of the endpoint's contract (SQL, parameters, response settings, cache, limits and rate limit) taken when you press **Publish**. You can edit the draft as much as you like without affecting anyone, and go back to an earlier version in one click.

![The Versions tab listing v2 as Live and v1](/img/platform/data-api/endpoint-versions.png)
![The Versions tab of Orders by status: v2 Live with a 1 breaking change badge over its note, and v1 with Roll back](/img/platform/data-api/endpoint-versions.png)

## Publish an endpoint

Expand Down Expand Up @@ -41,7 +41,7 @@ Version numbers are per endpoint and never reused.

A change is **breaking** when a consumer built against the live version could stop working. Before you publish, the dialog compares the draft with the live version.

![The Publish dialog listing a breaking change and requiring acknowledgement](/img/platform/data-api/publish-dialog-breaking.png)
![The Publish dialog for Top customers: column spend of statement main is gone, and Publish stays disabled until the acknowledgement is ticked](/img/platform/data-api/publish-dialog-breaking.png)

Examples Egret classifies as breaking, each with its own message:

Expand Down Expand Up @@ -71,7 +71,7 @@ Rolling back makes an earlier version the live one again. Consumers calling the
2. Press **Roll back** on the version you want back.
3. Type `rollback` to confirm and press **Roll back**. The dialog says *Every deployed client calling this endpoint's URL will immediately start receiving this version's frozen contract instead of the current live one. This does not delete anything — publish again to move forward once you're ready.*

![The rollback confirmation dialog](/img/platform/data-api/rollback-dialog.png)
![The Roll back to v1? dialog with rollback typed and the red Roll back button enabled](/img/platform/data-api/rollback-dialog.png)

A message confirms *Rolled back to vN*. Nothing is deleted. The draft is untouched, so the header may read **Live · v1 · unpublished changes** when the draft is ahead of the version being served.

Expand Down
4 changes: 2 additions & 2 deletions platform/data-api/endpoints/response.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 5

The **Response** tab shows what a consumer actually receives when it calls the endpoint: the frozen columns of the live version, and the settings that decide how rows become JSON. Use it to rename keys, return positional arrays, drop the envelope, or build a custom document with a script.

![The Response tab with frozen columns and the Output shape settings](/img/platform/data-api/endpoint-response.png)
![The Response tab of Orders by status: frozen columns status, revenue and order_count, and the Output shape settings](/img/platform/data-api/endpoint-response.png)

## The default response

Expand Down Expand Up @@ -72,7 +72,7 @@ Every other column is in every response.

Tick **Build the response with a script** to replace the standard `data` with a document you build. This is for a shape no SQL projection can express, such as a single object that joins a total with a list. Most endpoints do not need it: **Shape** and **Naming** already cover renaming keys and objects versus arrays.

![The Response tab of an endpoint with a transform script and declared return columns](/img/platform/data-api/endpoint-response-transform.png)
![The Response tab of Yearly revenue summary: two statements' frozen columns, the transform script (227 / 8,192 bytes) and three declared returns](/img/platform/data-api/endpoint-response-transform.png)

The script is Starlark, a deterministic Python subset with no input or output, no network, no clock and no randomness. It must define `transform(params, statements)`:

Expand Down
Loading