Skip to content

Fixed BucketPath using backslash separators on Windows - #325

Open
Ramnath0521 wants to merge 2 commits into
exasol:mainfrom
Ramnath0521:fix/issue-322
Open

Ramnath0521 wants to merge 2 commits into
exasol:mainfrom
Ramnath0521:fix/issue-322

Conversation

@Ramnath0521

Copy link
Copy Markdown

Closes #322

Cause

BucketPath stored its path as PurePath(path), which is a PureWindowsPath on Windows. Only _get_relative_posix() converted it back to forward slashes. read(), write(), rm(), __str__, iterdir() and as_udf_path() used str(self._path) directly and so sent backslash paths to BucketFS, as described in the issue.

Change

Following the suggestion in the issue, BucketPath now stores a PurePosixPath:

  • __init__ builds a PurePosixPath. A PurePath argument (for example a PureWindowsPath or WindowsPath) is converted with as_posix() first, so joinpath() with Windows path objects also works.
  • _get_relative_posix() no longer needs its PureWindowsPath special case.
  • as_udf_path() builds the result with PurePosixPath, so the UDF path uses / on every platform.

Behaviour on Linux and macOS is unchanged, because PurePath is already PurePosixPath there.

Two things on Windows are different now and worth a look:

  • A plain string containing a backslash is no longer split into path parts; paths in a bucket are POSIX paths. PureWindowsPath/WindowsPath objects are still accepted and converted.
  • relative_to() returns a PurePosixPath instead of a PureWindowsPath. I updated the one existing assertion that compared against PurePath(...) accordingly.

Tests

New test/unit/test_bucket_path_separators.py (7 tests) uses a recording fake bucket API and PureWindowsPath inputs, so it runs the same on every OS and does not need to patch PurePath the way the issue's Linux emulation does. It covers str(), write()/read()/rm(), joinpath(), iterdir() and as_udf_path().

On the unfixed code (Windows 11, Python 3.12) 5 of the new tests fail, with the backslash paths from the issue ('dir\sub\a.txt' == 'dir/sub/a.txt'); with the fix all 7 pass.

I also ran the whole test/unit folder on Windows. It does not pass on Windows even before this change (17 failures). With this change it is 14 failures, none new, and test_joinpath, test_truediv and test_archive_as_udf_path now pass. The remaining 14 come from the bucket_fake fixture: MountedBucket.files on a local directory returns backslash paths on Windows. I left that out because it is a separate code path (a locally mounted bucket) from the one in this issue, but I'm happy to look at it in a follow-up if you want it.

black --check, isort --check and mypy are clean on the changed files. I did not run the integration tests or the full nox sessions.

Checklist

  • Mentioned the issue this closes
  • Updated the changelog in doc/changes/unreleased.md

Written with AI assistance (Claude Code). I reproduced the failure on my own Windows machine, read the fix, and verified the added tests fail before it and pass after. Errors are mine.

🤖 Generated with Claude Code

BucketPath stored its path as PurePath, which is a PureWindowsPath on
Windows, so read(), write(), rm(), iterdir() and as_udf_path() sent
backslash paths to BucketFS. Store a PurePosixPath instead, converting
PurePath arguments with as_posix().

Closes exasol#322

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@ahsimb ahsimb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tackling this! I found a few issues; inline comments cover the changed lines, and these points are about code outside the diff:

  1. Root cause in MountedBucket is still there (exasol/bucketfs/_buckets.py, MountedBucket.files). On Windows it still returns backslash paths (dir\a.txt), while _navigate() now compares against a strictly POSIX path_str. So exists(), iterdir(), rm() and walk() still fail with a mounted bucket on Windows. Returning pth.relative_to(self.root).as_posix() there (and a POSIX udf_path) would fix it at the source.
  2. relative_to() return type changes on Windows (_path.py, relative_to). It now returns PurePosixPath instead of PureWindowsPath, so caller code like bp.relative_to(root) == PurePath('child') is always False on Windows. That is a public API change (the existing unit test had to be changed for it), so it should probably be mentioned in the changelog rather than only listed as a bug fix.
  3. Lookup and I/O use different strings (_path.py, read/write/rm). They send str(self._path), which includes the root, while _navigate() uses _get_relative_posix() without the anchor. For example, BucketPath('/dir/a.txt', api).exists() finds dir/a.txt, but rm() calls delete('/dir/a.txt').
  4. Outdated docstring (_path.py, __init__). It still says Windows forms like \\abc\...\ are accepted, which is no longer true for plain strings.

Comment thread exasol/bucketfs/_path.py Outdated
:param bucket_api: An object supporting the Bucket API protocol.
"""
self._path = PurePath(path)
self._path = PurePosixPath(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression for backslash strings on Windows. A plain str is now parsed by PurePosixPath, so os.path.join('dir', 'a.txt') or str(WindowsPath(...)) ('dir\a.txt') becomes a single component with a literal backslash. Previously _get_relative_posix replaced \ with /, so exists() worked; now exists() returns False, .name is 'dir\a.txt', .parent is '.', and write() uploads a file whose name contains a backslash. Consider normalizing strings too (e.g. PureWindowsPath(path).as_posix() on Windows, or always replacing \ with /).

Comment thread exasol/bucketfs/_path.py Outdated
"""
self._path = PurePath(path)
self._path = PurePosixPath(
path.as_posix() if isinstance(path, PurePath) else path

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as_posix() keeps the drive / UNC anchor. BucketPath(PureWindowsPath(r'C:\data\a.txt'), api) becomes C:/data/a.txt, so lookup uses C:/data/a.txt instead of data/a.txt (the old code stripped the anchor). A UNC path \\srv\share\x becomes //srv/share/x, which write()/rm() send to BucketFS with a leading //. Should the anchor be dropped (e.g. via path.relative_to(path.anchor)) before converting?

Also: joinpath() and __truediv__ don't go through this normalization. On Python 3.10/3.11 (both supported), BucketPath('a', api) / PureWindowsPath('C:/x/y') yields a/C:\/x/y, and .joinpath(PureWindowsPath(r'\x\y')) yields a/\/x/y. They should apply the same conversion.

Comment thread exasol/bucketfs/_path.py
def as_udf_path(self) -> str:
return str(
PurePath(self._bucket_api.udf_path) / _remove_archive_suffix(self._path)
PurePosixPath(self._bucket_api.udf_path)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MountedBucket.udf_path returns str(Path(...)), which uses backslashes on Windows, so wrapping it in PurePosixPath produces mixed separators, e.g. \buckets\bfsdefault\default/dir/a.txt. Either MountedBucket.udf_path should return a POSIX string, or this should convert it.

assert path.exists()


def test_joinpath_with_string_uses_posix_separators(bucket_api):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test uses only forward slashes, so it also passes on the unfixed code and doesn't exercise the bug. Missing coverage: backslash strings, absolute / drive-letter / UNC PureWindowsPath, and / / joinpath with Windows path objects. Those would catch the regressions noted on _path.py.

Convert every path that enters BucketPath through one helper, so a
Windows string, a Windows path object, joinpath() and "/" all end up as
the same POSIX path: separators become "/", a drive or UNC share is
dropped and a root is kept. Make MountedBucket.files and udf_path POSIX
as well, which is the root cause the reviewer pointed out and also makes
the existing unit tests pass on Windows.

Document the relative_to() return type change under Breaking Changes and
cover backslash strings, drive and UNC anchors, joinpath() and "/" with
Windows path objects and MountedBucket in the unit tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@Ramnath0521

Copy link
Copy Markdown
Author

Thanks for the careful review, all of this was right. I pushed a follow-up commit (764ec93) that addresses it:

  • Strings and Windows path objects now go through one helper (_to_posix_path), also used by joinpath() and /. On Windows a string is parsed as a Windows path, so os.path.join('dir', 'a.txt') is found again; a drive letter or UNC share is dropped, the separators become /, and a leading root is kept as /, like for a POSIX path. That also covers BucketPath('a') / PureWindowsPath('C:/x/y').
  • MountedBucket: files and udf_path now return POSIX strings, which was the root cause you pointed out. With that, the whole test/unit folder passes on Windows (103 passed; before my first commit it had 17 failures there).
  • Docstring no longer promises the Windows string forms; the relative_to() return type change is under "Breaking Changes" in doc/changes/unreleased.md.
  • Tests: the separator tests now use backslash-style input (native-joined strings, drive and UNC PureWindowsPath, joinpath and / with Windows objects) and MountedBucket; I removed the forward-slash-only test that passed on the unfixed code.

One point I left alone on purpose: read()/write()/rm() send str(self._path) while the lookup strips the anchor. That already differs for a rooted POSIX path such as /dir/a.txt on Linux, so it is not specific to Windows, and changing it would change what Linux callers send to BucketFS. I can do it in a separate PR if you want it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BucketPath uses backslash separators on Windows

2 participants