Repository navigation
Invalidate stale caches after cloud mutations - #600
terminalchai wants to merge 3 commits into
Conversation
pjbull
left a comment
There was a problem hiding this comment.
Thanks! A few other removals still leave the cache in place, so the same FileExistsError and stale append can still happen. Let's get those as part of this as well.
- Clear both caches in same-client
move/move_into. They callclient._move_filefrom_copy(cloudpathlib/cloudpath.py:1385) and never enterreplace, so the source cache stays after the object is gone. - Clear both caches on ADLS Gen2 directory rename.
AzureBlobPath.replace(cloudpathlib/azure/azblobpath.py:90) catchesCloudPathIsADirectoryErrorand returnsclient._move_filedirectly, so nested cached files survive the rename. - In
_refresh_cache(cloudpathlib/cloudpath.py:1713), delete a non-dirty local file whenstat()raisesNoStatError. That is the backstop from the issue thread, and it covers deletes made outside these methods. We should clear the cache when we catch NoStatError - Add tests for same-client
move/move_into, an ADLS Gen2 directory rename with a cached nested file, andclient._removefollowed byopen("x")andopen("a")with no cloudpathlib mutation in between.
| f"Path {self} is a directory; call rmdir instead of unlink." | ||
| ) | ||
| self.client._remove(self, missing_ok) | ||
| self.clear_cache() |
There was a problem hiding this comment.
We should check if the cache actually has content that may need to still be uploaded with something like.
existed = self.exists()
self.client._remove(self, missing_ok)
if existed or not self._dirty:
self.clear_cache()An explicit unlink of an object that is there should still clear a dirty cache.
| - Added streaming I/O via `FileCacheMode.streaming` for S3, Azure Blob Storage, Google Cloud Storage, and HTTP/HTTPS. In this mode `CloudPath.open()` returns the standard `io` file objects (`BufferedReader`/`BufferedWriter`/`TextIOWrapper`) backed by ranged reads and multipart uploads instead of the local cache, so only the bytes you read are downloaded and written data is uploaded as you write it. Read streams are seekable; append and update modes fall back to the cache and remove the cache file on close. `buffering` has its builtin `open()` meaning and sets the ranged-request size (default 5 MiB). A new `streaming_max_concurrency` client option (default 4) bounds background read-ahead and part uploads per stream; the buffer size, concurrency, and multipart part size can also be set with the `CLOUDPATHLIB_STREAMING_BUFFER_SIZE`, `CLOUDPATHLIB_STREAMING_MAX_CONCURRENCY`, and `CLOUDPATHLIB_[<PROVIDER>_]STREAMING_PART_SIZE` environment variables. Opening a file for append or update in streaming mode warns that it falls back to the cache. `copy`/`rename`/`replace` stream between clients, `fspath` raises `CloudPathNotImplementedError`, and failures raise `cloudpathlib.exceptions` types including the new `CloudPathStreamingError`. See the [Streaming I/O](https://cloudpathlib.drivendata.org/stable/streaming_io/) docs. (PR [#535](https://github.com/drivendataorg/cloudpathlib/pull/535)) | ||
| - Changed append mode (`a`, `ab`, `a+`, and `ab+`) to create missing cloud files, matching Python file behavior. Appending to existing files still preserves their content. (Issue [#585](https://github.com/drivendataorg/cloudpathlib/issues/585)) | ||
|
|
||
| - Fixed stale local caches after `unlink`, `rename`, `replace`, `rmdir`, and `rmtree`. Deleted files can be recreated with exclusive or append mode, and moved files no longer reuse cached destination content. (Issue [#388](https://github.com/drivendataorg/cloudpathlib/issues/388)) |
There was a problem hiding this comment.
Add this PR number to the bullet.
pjbull
left a comment
There was a problem hiding this comment.
_dirty_cache_files machinery is a separate feature that we could consider separately, but I don't want to implement as part of this fix. could you extract that and just keep the pieces that fix the current issues? thanks!
Deleting or moving a cloud object can leave stale cached files behind. Recreating the source with exclusive mode can then raise
FileExistsError, and append can reuse deleted content. External cloud deletions cause the same problem.Clear obsolete caches after successful
unlink,rename/replace,move/move_into,rmdirandrmtreeoperations, including ADLS Gen2 directory renames. Moves invalidate both source and destination caches. Failed mutations and self-moves preserve their caches. When a remote file is missing, refresh discards its clean cached file while retaining directory contents and same-path pending writes using the existing_dirtyflag.Closes #388.
The full mocked/local suite passes on Windows/Python 3.11: 2,369 passed and 127 skipped across all provider rigs. Black, flake8, mypy (31 source files), source/wheel builds, the regenerated MkDocs documentation and diff checks pass. Independent specification and code-quality reviews found no remaining actionable findings. Live cloud backends were not tested locally; fork live CI requires maintainer approval.
Contributor checklist:
Closes #388in the summaryPrepared with Codex assistance, including automated implementation and independent automated reviews. Local validation results are reported above.