Skip to content

Invalidate stale caches after cloud mutations - #600

Open
terminalchai wants to merge 3 commits into
drivendataorg:masterfrom
terminalchai:fix/invalidate-cache-after-mutations
Open

terminalchai wants to merge 3 commits into
drivendataorg:masterfrom
terminalchai:fix/invalidate-cache-after-mutations

Conversation

@terminalchai

@terminalchai terminalchai commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Deleting or moving a cloud object can leave stale cached files behind. Recreating the source with exclusive mode can then raise FileExistsError, and append can reuse deleted content. External cloud deletions cause the same problem.

Clear obsolete caches after successful unlink, rename/replace, move/move_into, rmdir and rmtree operations, including ADLS Gen2 directory renames. Moves invalidate both source and destination caches. Failed mutations and self-moves preserve their caches. When a remote file is missing, refresh discards its clean cached file while retaining directory contents and same-path pending writes using the existing _dirty flag.

Closes #388.

The full mocked/local suite passes on Windows/Python 3.11: 2,369 passed and 127 skipped across all provider rigs. Black, flake8, mypy (31 source files), source/wheel builds, the regenerated MkDocs documentation and diff checks pass. Independent specification and code-quality reviews found no remaining actionable findings. Live cloud backends were not tested locally; fork live CI requires maintainer approval.

Contributor checklist:

Prepared with Codex assistance, including automated implementation and independent automated reviews. Local validation results are reported above.

@pjbull pjbull left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! A few other removals still leave the cache in place, so the same FileExistsError and stale append can still happen. Let's get those as part of this as well.

  • Clear both caches in same-client move / move_into. They call client._move_file from _copy (cloudpathlib/cloudpath.py:1385) and never enter replace, so the source cache stays after the object is gone.
  • Clear both caches on ADLS Gen2 directory rename. AzureBlobPath.replace (cloudpathlib/azure/azblobpath.py:90) catches CloudPathIsADirectoryError and returns client._move_file directly, so nested cached files survive the rename.
  • In _refresh_cache (cloudpathlib/cloudpath.py:1713), delete a non-dirty local file when stat() raises NoStatError. That is the backstop from the issue thread, and it covers deletes made outside these methods. We should clear the cache when we catch NoStatError
  • Add tests for same-client move / move_into, an ADLS Gen2 directory rename with a cached nested file, and client._remove followed by open("x") and open("a") with no cloudpathlib mutation in between.

Comment thread cloudpathlib/cloudpath.py
f"Path {self} is a directory; call rmdir instead of unlink."
)
self.client._remove(self, missing_ok)
self.clear_cache()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check if the cache actually has content that may need to still be uploaded with something like.

existed = self.exists()
self.client._remove(self, missing_ok)
if existed or not self._dirty:
    self.clear_cache()

An explicit unlink of an object that is there should still clear a dirty cache.

Comment thread HISTORY.md Outdated
- Added streaming I/O via `FileCacheMode.streaming` for S3, Azure Blob Storage, Google Cloud Storage, and HTTP/HTTPS. In this mode `CloudPath.open()` returns the standard `io` file objects (`BufferedReader`/`BufferedWriter`/`TextIOWrapper`) backed by ranged reads and multipart uploads instead of the local cache, so only the bytes you read are downloaded and written data is uploaded as you write it. Read streams are seekable; append and update modes fall back to the cache and remove the cache file on close. `buffering` has its builtin `open()` meaning and sets the ranged-request size (default 5 MiB). A new `streaming_max_concurrency` client option (default 4) bounds background read-ahead and part uploads per stream; the buffer size, concurrency, and multipart part size can also be set with the `CLOUDPATHLIB_STREAMING_BUFFER_SIZE`, `CLOUDPATHLIB_STREAMING_MAX_CONCURRENCY`, and `CLOUDPATHLIB_[<PROVIDER>_]STREAMING_PART_SIZE` environment variables. Opening a file for append or update in streaming mode warns that it falls back to the cache. `copy`/`rename`/`replace` stream between clients, `fspath` raises `CloudPathNotImplementedError`, and failures raise `cloudpathlib.exceptions` types including the new `CloudPathStreamingError`. See the [Streaming I/O](https://cloudpathlib.drivendata.org/stable/streaming_io/) docs. (PR [#535](https://github.com/drivendataorg/cloudpathlib/pull/535))
- Changed append mode (`a`, `ab`, `a+`, and `ab+`) to create missing cloud files, matching Python file behavior. Appending to existing files still preserves their content. (Issue [#585](https://github.com/drivendataorg/cloudpathlib/issues/585))

- Fixed stale local caches after `unlink`, `rename`, `replace`, `rmdir`, and `rmtree`. Deleted files can be recreated with exclusive or append mode, and moved files no longer reuse cached destination content. (Issue [#388](https://github.com/drivendataorg/cloudpathlib/issues/388))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add this PR number to the bullet.

@terminalchai terminalchai changed the title Clear stale caches after cloud file and directory mutations Invalidate stale cloud caches without discarding pending writes Oct 12, 2026

@pjbull pjbull left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_dirty_cache_files machinery is a separate feature that we could consider separately, but I don't want to implement as part of this fix. could you extract that and just keep the pieces that fix the current issues? thanks!

@terminalchai terminalchai changed the title Invalidate stale cloud caches without discarding pending writes Invalidate stale caches after cloud mutations Oct 18, 2026

This branch is waiting to be deployed

1 waiting deployment
reviewed-live-cloud — df37206c Waiting Oct 18, 2026 by terminalchai via Live Tests (azure_gen2) #47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Methods that move/remove files on the cloud don't remove the cached version

2 participants