Skip to content

fix: normalize DBFS paths before file downloads - #1635

Open
Swarneil1905 wants to merge 1 commit into
databricks:mainfrom
Swarneil1905:fix/1630-download-fallback-path
Open

Swarneil1905 wants to merge 1 commit into
databricks:mainfrom
Swarneil1905:fix/1630-download-fallback-path

Conversation

@Swarneil1905

Copy link
Copy Markdown

Summary

Normalize dbfs:/Volumes/... paths in files.download() and files.download_to() so downloads can fall back to the Files API using a valid URL.

Fixes #1630.

Why

When an offloaded download fails, the SDK falls back to the Files API. Paths containing the dbfs: prefix are currently appended directly to the API endpoint, producing an invalid path such as /api/2.0/fs/filesdbfs%3A/Volumes/... and causing a 404.

Removing the leading scheme before download processing allows the fallback to request the correct file.

What changed

Interface changes

No signature changes. Updated both download method docstrings to document dbfs:/Volumes/... paths.

Behavioral changes

  • Normalize the leading dbfs: prefix before download and metadata requests.
  • Preserve existing /Volumes/... paths and dbfs: text within filenames.
  • Apply normalization to the original download implementation when experimental features are disabled.

Internal changes

  • Added mocked regression tests for a cloud download returning 403 followed by a successful Files API fallback.
  • Covered streaming and file downloads, Files API versions 2.0 and 3.0, special-character escaping, and overwriting an existing destination.
  • Extended parallel download coverage to include dbfs: paths.

How is this tested?

Validated locally on Windows:

  • uv run --locked pytest tests/ -q -o log_cli=false --tb=short: 4036 passed, 192 skipped, 13 warnings.
  • uv run --locked ruff check databricks tests: passed.
  • uv run --locked ruff format --check databricks tests: passed.
  • git diff --check: passed.

The new tests use mocked HTTP responses. Parallel download tests remain skipped on Windows. No live Databricks workspace testing was performed.

Signed-off-by: Swarneil Pradhan <swarneil1905@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Databricks SDK exception downloading file on Linux x64

1 participant