Skip to content

Add the UCX Databricks App and the deploy-app command - #5000

Draft
FastLee wants to merge 1 commit into
mainfrom
feat/db_app_2026
Draft

FastLee wants to merge 1 commit into
mainfrom
feat/db_app_2026

Conversation

@FastLee

@FastLee FastLee commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds the UCX App, a Databricks App that runs the main table migration steps from a web UI, and a new databricks labs ucx deploy-app command to deploy it.

Page What it does CLI equivalent
Overview UCX installation, latest run of each workflow, next step workflows
Assessment Runs the assessment workflow, embeds the assessment dashboard ensure-assessment-run
Table Mapping Creates the mapping from the assessment; review and edit it in a grid with validation create-table-mapping + editing mapping.csv
Migration Runs any of migrate-tables, hiveserde-in-place and CTAS, sequentially migrate-tables

An Active runs window on every page loads the UCX runs in progress when the app opens (including runs started from the CLI or the Jobs UI), polls every 10 s, and keeps finished runs with their final state for 5 minutes. Workflows always run asynchronously.

deploy-app downloads the released UCX wheel of the installed version, verifies its sha256 against the package index, deploys it as an asset bundle (databricks bundle deploy + bundle run), and grants the app's service principal CAN_MANAGE_RUN on the UCX jobs.

Docs: docs/ucx/docs/reference/app.mdx, the deploy-app command reference, and a developer section in contributing.mdx.

Design decisions

  • Stack: FastAPI backend (src/databricks/labs/ucx/app/), served by gunicorn with uvicorn workers, and a Next.js static export (app-ui/), the same stack as other internal Databricks Apps. The backend reuses WorkspaceContext and the CLI's code paths (TableMapping, DeployedWorkflows, VerifyProgressTracking).
  • Auth is a hybrid. By default, workspace files, SQL, Unity Catalog checks and identity use the logged-in user's forwarded token. Databricks Apps user authorization has no scope for the Jobs API, so starting and watching workflows always uses the app's service principal. Chained run names record (requested by <user>), and the app logs who started what. --run-as-sp true uses the SP for everything; deploy-app then also grants it CAN_EDIT on the install folder.
  • Sequential migration: the selected workflows run as one one-time job run (jobs.submit), with a run_job_task per workflow, each depends_on the previous one. The chain survives app restarts, and a failure stops the workflows after it.
  • Artifacts: databricks labs install ucx installs from the source archive, which doesn't contain the built frontend. deploy-app therefore deploys the released wheel from PyPI (or PIP_INDEX_URL), or a local wheel with --wheel. The wheel ships app/bundle/requirements.txt, generated from uv.lock with hashes, so the app runs exactly the locked dependency set in pip hash-checking mode.
  • pyproject.toml gets a new optional extra app (fastapi, uvicorn, gunicorn), so CLI and job installs don't grow. The CLI does not import FastAPI.

Supply chain

  • Python deps use compatible ranges, locked in uv.lock. The registry URLs in uv.lock are rewritten to pypi.org the same way as make lock-dependencies. httpx is added to the pytest group for FastAPI's TestClient.
  • Frontend deps are pinned exactly to the newest versions released at least 7 days ago, with pnpm-lock.yaml and a frozen install. This cooldown is applied by hand to direct dependencies only; pnpm 9.15.9 can't enforce minimumReleaseAge, so transitive dependencies are not held back.
  • TypeScript is 7.1.0-beta. The internal npm proxy currently serves no stable TypeScript release (5.x and 6.x return 404; only 7.1 pre-releases are listed). This should move to a stable release once it's available.
  • New actions in push.yml are pinned by SHA: actions/setup-node v7.1.0 and pnpm/action-setup v6.1.1, both verified commits at least 7 days old. JFrog auth configures npm with ignore-scripts=true.

Release requirement

The release workflow was removed in #4739, so releases are built outside this repo. Release builds must run make build (it now runs make app-ui and make app-requirements first, and needs Node 26 + pnpm 9.15.9). Otherwise the released wheel has no app and deploy-app refuses it. The new app CI job runs make build and fails if the wheel is missing the app.

Testing

  • Unit tests: 2077 pass, including 52 new ones in tests/unit/app/ (API, auth, deployer, CLI). make lint checks are clean: black, ruff, mypy, pylint 10.00/10.
  • Frontend: tsc --noEmit and next build pass. Clicked through all pages in a browser against the real FastAPI app with canned data: mapping edits and duplicate-destination validation, chained run tracking, no console errors.
  • Packaging: make build produces a wheel with the frontend and hashed requirements. Staged the app from that wheel with AppDeployer, installed it in a clean venv with --require-hashes, and started it with the exact app.yaml gunicorn command: pages return 200, and the API returns 403 without a user token.
  • Docs site builds (yarn build, with onBrokenLinks: throw).
  • Integration tests (tests/integration/app/) have not passed yet. The test workspace's IP access list blocked the machine I ran them from. One partial run in another workspace confirmed that chained run_job_tasks are accepted in a submitted run and that a failed step stops the next one (UPSTREAM_FAILED). Its first no-op child job failed for a reason I couldn't see; the test now reports child run errors. This PR's acceptance run should cover test_mapping_roundtrip, test_status_dashboard_and_runs and test_chained_migration_run.
  • test_deploy_app_end_to_end is skipped unless UCX_APP_WHEEL points to a wheel from make build. It needs a manual run (or a follow-up to build the wheel in acceptance.yml).

Not yet verified in a workspace

  • That the workspace.workspace user scope covers reading and writing the install folder (config.yml, state.json, mapping.csv).
  • The CAN_EDIT grant on the install folder in --run-as-sp mode.
  • Embedding the Lakeview dashboard in an iframe (needs embedding enabled and the app domain allow-listed; the page falls back to Open in new tab).
  • The PEP 691 JSON download path in deploy-app against PyPI. It's unit-tested with mocked responses only, and some package proxies may not support the JSON API.

This pull request and its description were written by Isaac.

A Databricks App to run the assessment, review and edit the table mapping, and
migrate tables from a web UI, deployed with `databricks labs ucx deploy-app`.

- Backend (FastAPI) in databricks.labs.ucx.app, reusing the CLI's code paths
- Frontend (Next.js static export) in app-ui/, built into the wheel
- On-behalf-of-user access, with the app's service principal for the Jobs API
- Selected migration workflows run sequentially as one chained job run
- deploy-app deploys the released wheel as an asset bundle
- CI job that builds the frontend and checks the wheel contains the app

Co-authored-by: Isaac <no-reply@databricks.com>
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.72802% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.02%. Comparing base (06f771e) to head (9536054).

Files with missing lines Patch % Lines
src/databricks/labs/ucx/app/deploy.py 96.89% 2 Missing and 2 partials ⚠️
src/databricks/labs/ucx/app/service.py 97.14% 1 Missing and 3 partials ⚠️
src/databricks/labs/ucx/app/routes/dependencies.py 70.00% 3 Missing ⚠️
src/databricks/labs/ucx/app/context.py 91.30% 1 Missing and 1 partial ⚠️
src/databricks/labs/ucx/app/main.py 93.75% 1 Missing and 1 partial ⚠️
src/databricks/labs/ucx/cli.py 88.88% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5000      +/-   ##
==========================================
+ Coverage   87.79%   88.02%   +0.23%     
==========================================
  Files         123      135      +12     
  Lines       17595    18084     +489     
  Branches     3717     3790      +73     
==========================================
+ Hits        15448    15919     +471     
- Misses       1458     1466       +8     
- Partials      689      699      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant