Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 30 additions & 5 deletions .github/workflows/app.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
name: app

# Tests the Impulse App in apps/adhoc-reporting/. Not a required check, and it runs only on PRs that
# Tests the Impulse App in apps/adhoc-reporting/. The test job runs only on PRs that
# change the app: Codecov (require_ci_to_pass) holds back its status while any other check fails, so
# an app check on framework PRs would affect the framework's CI. Framework changes (src/) run it
# once they are on main, where the contract tests build the app's generated notebooks against the
# framework source of that commit. To check a framework branch before merging, run it by hand.
on:
pull_request:
types: [ opened, synchronize, ready_for_review ]
paths:
- 'apps/adhoc-reporting/**'
- '.github/workflows/app.yml'
push:
branches:
- main
Expand All @@ -37,9 +34,29 @@ jobs:
steps:
- run: echo "Not a fork PR, proceeding"

changes:
needs: not-a-fork
runs-on:
group: databrickslabs-protected-runner-group
labels: linux-ubuntu-latest
outputs:
app: ${{ steps.diff.outputs.app }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 2

- name: Does the PR change the app?
id: diff
run: |
if [ "$GITHUB_EVENT_NAME" = pull_request ] && git diff --quiet HEAD^ HEAD -- apps/adhoc-reporting .github/workflows/app.yml; then
echo "app=false" >> "$GITHUB_OUTPUT"
fi

test:
name: Test the app
needs: not-a-fork
needs: [ not-a-fork, changes ]
if: ${{ !cancelled() && needs.not-a-fork.result != 'skipped' && needs.changes.outputs.app != 'false' }}
runs-on:
group: databrickslabs-protected-runner-group
labels: linux-ubuntu-latest
Expand Down Expand Up @@ -79,3 +96,11 @@ jobs:

- name: Build the frontend
run: make frontend

- name: The committed frontend matches its source
run: |
if [ -n "$(git status --porcelain -- frontend/dist)" ]; then
git status --short -- frontend/dist
echo "::error::frontend/dist doesn't match frontend/src. Run 'make frontend' in apps/adhoc-reporting and commit frontend/dist."
exit 1
fi
2 changes: 1 addition & 1 deletion apps/adhoc-reporting/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ node_modules/
CLAUDE.md
.databricks/
.vscode/
frontend/dist/
!frontend/dist/
*.tsbuildinfo
.isaac/
.env
3 changes: 3 additions & 0 deletions apps/adhoc-reporting/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,13 @@ Run the commands from this folder. `make test` needs no workspace. The other tar
| `make smoke` | Runs the fixture report on serverless without writing gold tables |
| `make smoke-persist` | Same, but writes the fixture report's gold tables, which `make live` reads |
| `make eval` | Runs the assistant scenarios against FMAPI |
| `make frontend` | Builds the frontend into `frontend/dist/` |
| `make deploy` | Builds the frontend and deploys the app (commit the app first) |
| `make logs` | Shows the app logs |
| `make lock` | Updates `uv.lock` and `requirements.txt`, keeping both lockfiles on the public registries |

The built frontend in `frontend/dist/` is committed, so deployments need neither Node.js nor access to the npm registry. After changing `frontend/`, run `make frontend` and commit the rebuilt `frontend/dist/` with the change. CI fails when the two differ, and `make deploy` refuses to deploy until the rebuild is committed.

## Updating Impulse

```
Expand Down
15 changes: 14 additions & 1 deletion apps/adhoc-reporting/databricks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@ resources:
impulse_app:
name: ${var.app_name}
description: Impulse report builder
source_code_path: .
user_api_scopes:
- sql
- workspace.workspace
Expand All @@ -52,3 +51,17 @@ targets:
default: true
workspace:
root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}
resources:
apps:
impulse_app:
source_code_path: .
github:
resources:
apps:
impulse_app:
git_repository:
url: https://github.com/databrickslabs/impulse
provider: gitHub
git_source:
commit: ${bundle.git.commit}
source_code_path: ${bundle.git.bundle_root_path}
19 changes: 19 additions & 0 deletions apps/adhoc-reporting/frontend/dist/assets/index-Bnk2zU0V.js

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

15 changes: 15 additions & 0 deletions apps/adhoc-reporting/frontend/dist/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="color-scheme" content="dark" />
<link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'%3E%3Crect width='32' height='32' rx='7' fill='%23FF3621'/%3E%3Cpath d='M5 17h5l3-7 5 13 3-6h6' fill='none' stroke='white' stroke-width='2.6' stroke-linecap='round' stroke-linejoin='round'/%3E%3C/svg%3E" />
<title>Impulse</title>
<script type="module" crossorigin src="/assets/index-Bnk2zU0V.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-CbcUiC_u.css">
</head>
<body>
<div id="root"></div>
</body>
</html>
1 change: 1 addition & 0 deletions apps/adhoc-reporting/frontend/src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ export const api = {
channels: (catalog: string, schema: string, uuts: string[], window: Window) =>
request<{ channels: Channel[]; issues: string[] }>("POST", "/api/data/channels", { catalog, schema, uuts, window }),
preflight: (name: string) => request<Preflight>("POST", `${reportPath(name)}/preflight`),
grant: (name: string) => request<{ granted: string }>("POST", `${reportPath(name)}/grants`),
run: (name: string) => request<{ preflight: Preflight; run: { job_id: number; run_id: number } | null }>("POST", `${reportPath(name)}/runs`),
runs: (name: string) => request<{ runs: RunSummary[] }>("GET", `${reportPath(name)}/runs`).then((r) => r.runs),
runDetail: (name: string, runId: number) => request<RunSummary>("GET", `${reportPath(name)}/runs/${runId}`),
Expand Down
48 changes: 37 additions & 11 deletions apps/adhoc-reporting/frontend/src/components/steps/ReviewStep.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { CheckCircle2, CircleX, Copy, ExternalLink, Play, Square } from "lucide-react";
import { CheckCircle2, CircleX, Copy, ExternalLink, Play, ShieldCheck, Square } from "lucide-react";
import { useCallback, useEffect, useState } from "react";
import { api } from "../../api";
import { reportHref, useAsync } from "../../hooks";
Expand Down Expand Up @@ -57,7 +57,7 @@ function badge(run: RunSummary): string {
return run.result === "SUCCESS" ? "badge ok" : "badge bad";
}

function PreflightProblems({ preflight }: { preflight: Preflight }) {
function PreflightProblems({ preflight, busy, onGrant }: { preflight: Preflight; busy?: string; onGrant: () => void }) {
return (
<div className="card-body stack preflight">
{preflight.user.length > 0 && (
Expand All @@ -80,9 +80,7 @@ function PreflightProblems({ preflight }: { preflight: Preflight }) {
{preflight.app.length > 0 && (
<div>
<h3>The app's service principal</h3>
<p className="sub">
Report jobs run as {preflight.app_principal.name}. A workspace or catalog admin grants it access once per catalog and schema:
</p>
<p className="sub">Report jobs run as {preflight.app_principal.name}, which needs access once per catalog and schema:</p>
<ul className="todo">
{preflight.app.map((p) => (
<li key={`${p.privilege}:${p.object}`}>
Expand All @@ -92,10 +90,24 @@ function PreflightProblems({ preflight }: { preflight: Preflight }) {
))}
</ul>
{preflight.grants && (
<div className="grants">
<pre className="code">{preflight.grants}</pre>
<CopyButton text={preflight.grants} label="Copy SQL" />
</div>
<>
<div className="grants">
<pre className="code">{preflight.grants}</pre>
<CopyButton text={preflight.grants} label="Copy SQL" />
</div>
{preflight.user.length === 0 && (
<div className="row grant-row">
<button className="btn primary" onClick={onGrant} disabled={!!busy}>
{busy === "grant" ? <span className="spinner" /> : <ShieldCheck size={15} />}
Grant and run
</button>
<span className="hint">
Runs these statements as you, so it works only if you may grant access to these catalogs and schemas. Otherwise copy the
SQL for someone who can.
</span>
</div>
)}
</>
)}
</div>
)}
Expand All @@ -108,7 +120,7 @@ function RunPanel({ onSuccess }: { onSuccess: () => void }) {
const [history, setHistory] = useState<RunSummary[]>();
const [current, setCurrent] = useState<RunSummary>();
const [preflight, setPreflight] = useState<Preflight>();
const [busy, setBusy] = useState<"run" | "cancel">();
const [busy, setBusy] = useState<"run" | "cancel" | "grant">();
const [error, setError] = useState<string>();
const now = useNow(!!current?.active);

Expand Down Expand Up @@ -174,6 +186,20 @@ function RunPanel({ onSuccess }: { onSuccess: () => void }) {
}
};

const grantAndRun = async () => {
setBusy("grant");
setError(undefined);
try {
if (!(await flushNow())) throw new Error("The report could not be saved. Fix the save error first.");
await api.grant(spec.name);
} catch (e) {
setError((e as Error).message);
setBusy(undefined);
return;
}
await start();
};

const cancel = async () => {
if (!current) return;
setBusy("cancel");
Expand Down Expand Up @@ -213,7 +239,7 @@ function RunPanel({ onSuccess }: { onSuccess: () => void }) {
<Banner>{error}</Banner>
</div>
)}
{preflight && !preflight.ok && <PreflightProblems preflight={preflight} />}
{preflight && !preflight.ok && <PreflightProblems preflight={preflight} busy={busy} onGrant={() => void grantAndRun()} />}
{current && (
<div className="card-body stack">
<div className="row">
Expand Down
8 changes: 8 additions & 0 deletions apps/adhoc-reporting/frontend/src/styles/app.css
Original file line number Diff line number Diff line change
Expand Up @@ -1210,6 +1210,14 @@ kbd {
right: 8px;
}

.grant-row {
margin-top: 10px;
}

.grant-row .btn {
flex-shrink: 0;
}

.run-error {
padding: 12px 14px;
border: 1px solid rgba(248, 113, 113, 0.35);
Expand Down
7 changes: 7 additions & 0 deletions apps/adhoc-reporting/server/routes/runs.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,13 @@ def preflight(name: ReportName, user: User = Depends(current_user)) -> dict:
return runs.preflight(user, generated.config)


@router.post("/grants")
def grant(name: ReportName, user: User = Depends(current_user)) -> dict:
_, generated = _generated(user, name)
runs.grant(user, generated.config)
return {"granted": name}


@router.post("/runs")
def start(name: ReportName, user: User = Depends(current_user)) -> dict:
_, generated = _generated(user, name)
Expand Down
14 changes: 14 additions & 0 deletions apps/adhoc-reporting/server/runs.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,20 @@ def preflight(user: User, config: dict) -> dict:
}


def grant(user: User, config: dict) -> None:
tables = list(config["source"].values())
sink = config["unity_sink"]
app_id, _ = runner_identity()
failures = []
for statement in grant_script(app_id, tables, sink).splitlines():
try:
sql.query(user, statement)
except ApiError as e:
failures.append(e)
if failures and _checks(User(app_id, runner()), tables, sink):
raise failures[0]


def _job_name(email: str, report: str) -> str:
return f"Impulse {report} ({email})"

Expand Down
3 changes: 2 additions & 1 deletion apps/adhoc-reporting/skills/app/impulse-app-run/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ Call `run_report` only when the user asks. It first checks access:
- The user needs SELECT on every source table and CREATE TABLE in the output schema. Missing privileges are
listed under `preflight.user`; the user asks a data owner for them.
- Report jobs run as the app's service principal. Missing privileges are listed under `preflight.app`, and
`preflight.grants` holds the GRANT statements an admin runs once. Show them unchanged.
`preflight.grants` holds the GRANT statements an admin runs once. Show them unchanged, and tell the user
that someone who may grant them can also click **Grant and run** on the Review & run step.

When it starts, the run takes one to three minutes on serverless compute. Tell the user the app shows
progress on the Review & run step; call `get_run_status` when the user asks how it is going.
Expand Down
1 change: 1 addition & 0 deletions apps/adhoc-reporting/tests/unit/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
("DELETE", ""),
("POST", "/check"),
("POST", "/preflight"),
("POST", "/grants"),
("POST", "/runs"),
("GET", "/runs"),
("GET", "/runs/1"),
Expand Down
57 changes: 57 additions & 0 deletions apps/adhoc-reporting/tests/unit/test_runs.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,31 @@
import pytest

from server import runs
from server.errors import ApiError
from server.runs import grant_script

CONFIG = {"source": {"containers": "c.silver.container_metrics"}, "unity_sink": {"catalog": "c", "schema": "gold"}}
GRANTS = [
"GRANT USE CATALOG ON CATALOG `c` TO `app-sp`;",
"GRANT USE SCHEMA, SELECT ON SCHEMA `c`.`silver` TO `app-sp`;",
"GRANT USE SCHEMA, SELECT, MODIFY, CREATE TABLE ON SCHEMA `c`.`gold` TO `app-sp`;",
]


@pytest.fixture
def app_principal(monkeypatch):
monkeypatch.setattr(runs, "runner_identity", lambda: ("app-sp", "Impulse App"))
monkeypatch.setattr(runs, "runner", lambda: None)


def deny_catalog_grant(monkeypatch, executed):
def query(user, statement):
executed.append(statement)
if statement == GRANTS[0]:
raise ApiError(403, "missing_privilege", "User does not have MANAGE on Catalog 'c'")

monkeypatch.setattr(runs.sql, "query", query)


def test_grant_script_covers_source_and_sink():
tables = ["c1.silver.container_metrics", "c1.silver.channels", "c2.raw.channel_tags"]
Expand All @@ -19,3 +45,34 @@ def test_grant_script_merges_shared_schema():
"GRANT USE CATALOG ON CATALOG `c` TO `sp`;",
"GRANT USE SCHEMA, SELECT, MODIFY, CREATE TABLE ON SCHEMA `c`.`s` TO `sp`;",
]


def test_grant_runs_the_generated_statements_as_the_user(monkeypatch, app_principal):
executed = []
user = object()
monkeypatch.setattr(runs.sql, "query", lambda principal, statement: executed.append((principal, statement)))

runs.grant(user, CONFIG)

assert executed == [(user, statement) for statement in GRANTS]


def test_grant_continues_when_the_app_already_has_the_denied_privilege(monkeypatch, app_principal):
executed = []
deny_catalog_grant(monkeypatch, executed)
monkeypatch.setattr(runs, "_checks", lambda principal, tables, sink: [])

runs.grant(object(), CONFIG)

assert executed == GRANTS


def test_grant_reports_the_denial_when_the_app_still_lacks_access(monkeypatch, app_principal):
executed = []
deny_catalog_grant(monkeypatch, executed)
missing = [{"object": "c.gold", "privilege": "CREATE TABLE", "message": "PERMISSION_DENIED"}]
monkeypatch.setattr(runs, "_checks", lambda principal, tables, sink: missing)

with pytest.raises(ApiError, match="MANAGE on Catalog"):
runs.grant(object(), CONFIG)
assert executed == GRANTS
5 changes: 3 additions & 2 deletions docs/impulse/app/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ flowchart LR
read.
- **The app's service principal.** Report jobs run as the service principal, which needs access to
the silver and gold schemas. Before each run the app checks that both the user and the service
principal can read the source tables and create tables in the output schema, and it shows the
missing `GRANT` statements otherwise. The user gets `CAN_MANAGE_RUN` on their report's job.
principal can read the source tables and create tables in the output schema. Otherwise it shows
the missing `GRANT` statements, which **Grant and run** executes as the signed-in user if that
user may grant them. The user gets `CAN_MANAGE_RUN` on their report's job.

## How a report runs

Expand Down
2 changes: 1 addition & 1 deletion docs/impulse/app/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,5 @@ the [query engine](/docs/references/query_engine) isn't part of the app.
## Next steps

- [Requirements](requirements.md): workspace features, data and permissions.
- [Install](install.md): build, deploy and grant access.
- [Install](install.md): deploy and grant access.
- [Architecture](architecture.md): components, identities and where data is stored.
Loading
Loading