Remote arrays, hierarchical store discovery, and unified caching - #708

Merged

FrancescAlted merged 89 commits into

mainfrom

Sep 11, 2026

Commits on Sep 3, 2026

Commits on Sep 4, 2026

Clarify and extend remote array caching
FrancescAlted
committed
bca3cf4
Reconstruct remote sources from proxy caches
FrancescAlted
committed
eb214c1
Count metadata in remote traffic
FrancescAlted
committed
266e8e8
Report persistent cache status
FrancescAlted
committed
5200b64
Update C2Array test doubles for traffic
FrancescAlted
committed
408899d
Reduce default test suite runtime
FrancescAlted
committed
47fd165
Add persistable RemoteProxy
FrancescAlted
committed
1d0e2b9
Harden remote proxy source refresh
FrancescAlted
committed
38890c0

Commits on Sep 5, 2026

Commits on Sep 6, 2026

Commits on Sep 7, 2026

Commits on Sep 8, 2026

Support remote HDF5 arrays via kerchunk and unify dataset syntax
FrancescAlted
committed
4b1af61
Add native remote array reads from B2Z hierarchies
FrancescAlted
committed
3f9dc19
Optimize remote B2Z metadata reads
FrancescAlted
committed
8fc6fb5
Support meta and vlmeta access and caching in RemoteProxy
FrancescAlted
committed
480a413
Add RemoteProxy attrs alias and filter HDF5 dimension metadata
FrancescAlted
committed
00a1ab8
Read public Caterva2 attributes in remote arrays
FrancescAlted
committed
ee7d4fd
Add remote array support to b2view
FrancescAlted
committed
27df9e5
Add attrs as the recommended user metadata interface
FrancescAlted
committed
6248887
Hide hierarchy details for standalone b2view objects
FrancescAlted
committed
c26ece6
Fix Zarr decoder startup in b2view
FrancescAlted
committed
a7070c1

Commits on Sep 9, 2026

Add remote hierarchy browsing
FrancescAlted
committed
89548d9
Fix meta annotation
FrancescAlted
committed
57b693f
Update remote arrays guide and B2ZNDSource docs
FrancescAlted
committed
6fc8ed6
Fix remote table paging layout
FrancescAlted
committed
565cdd0
Exercise remote array backends in CI
FrancescAlted
committed
8583d46
Document HTTP/HTTPS and general fsspec support for remote arrays
FrancescAlted
committed
bec0f5e
Some more fixes for the test suite, and remote arrays docs review
FrancescAlted
committed
824d86a
Handle unavailable S3 test credentials
FrancescAlted
committed
b891fb6

Commits on Sep 10, 2026

Add RemoteArray and initial RemoteStore discovery
FrancescAlted
committed
b672180
Add shared RemoteStore memory caching
FrancescAlted
committed
0f82b2e
Add persistent RemoteStore disk caching
FrancescAlted
committed
169a04f
Migrate b2view to shared RemoteStore caching
FrancescAlted
committed
f75ed01
Step 6, and v13 plan is complete now
FrancescAlted
committed
a0c19cb
Add portable RemoteStore exports and immutable caches
FrancescAlted
committed
5ab63dd
Fix column capping when paging backward to start in b2view
FrancescAlted
committed
6b40b5a
Fix invalid Windows filename characters in Zarr proxy test
FrancescAlted
committed
e3de169
Updated docs for the new remote features
FrancescAlted
committed
5f175d5
Add --cache-dir option to b2view for persistent remote caching
FrancescAlted
committed
e23003b
C-Blosc2 bumped to latest 3.3.4
FrancescAlted
committed
d975eda
Fix Windows hang in remote store tests and cache lock cleanup
FrancescAlted
committed
af0ce37
Fix flaky timeout in remote listing retry and shutdown test
FrancescAlted
committed
8717439
Fix remote persistence and protect conversion inputs
FrancescAlted
committed
a9fa7cc
Wait for remote viewer results in asynchronous tests
FrancescAlted
committed
c5d0397
Disable in-process HTTP server tests on Windows
FrancescAlted
committed
c89e66f
Drop file-backed index handles between tests
FrancescAlted
committed
9d41755
Bound the two-writer barrier so a failed sibling cannot hang
FrancescAlted
committed
64677af
Name the test behind a CI hang instead of hanging
FrancescAlted
committed
e8b3d83
Give the b2view waits a wall-clock budget
FrancescAlted
committed
c35ea12
Keep the deadman's stacks where CI can show them
FrancescAlted
committed
f1f3a1f
Let the warm cache restart wait for the first browser to close
FrancescAlted
committed
33bdf00
Catch hangs outside tests with a session deadman
FrancescAlted
committed
381e2db
Bound the zarr sync bridge under kerchunk's HDF5 translation
FrancescAlted
committed
b740310
Keep zarr Buffer objects out of a cut-short HDF5 manifest
FrancescAlted
committed
dbae1e2

Commits on Sep 11, 2026

Abort an HDF5 translation whose zarr bridge timed out
FrancescAlted
committed
9c7aca9
Add shared RemoteStore sparse caches
FrancescAlted
committed
fd1324c
Update B2Z sparse attachment test
FrancescAlted
committed
4066959
Reset Zarr resources before HDF5 scans
FrancescAlted
committed
2eba2fc
Apply blacken-docs formatting
FrancescAlted
committed
2f2964d
Keep Ruff format off Markdown files
FrancescAlted
committed
0c6525e
Address review feedback on remote arrays, b2view, and zarr conversion
FrancescAlted
committed
8f0f132
Wait for the previous remote browser close before refresh reopens
FrancescAlted
committed
8d27fe3
Emit numcodecs compressors for b2nd-to-zarr v2 output
FrancescAlted
committed
677da11
Coordinate Zarr sync resets with in-flight reads
FrancescAlted
committed
13e85a2
Partition RemoteArray caches by fsspec storage options
FrancescAlted
committed
951ae13
Partition RemoteStore caches by fsspec storage options
FrancescAlted
committed
6196566
Keep format and dataset selections when localizing fsspec opens
FrancescAlted
committed
dc00d7d
Restore full-strength locking stress runs behind a heavy marker
FrancescAlted
committed
e0a69b3
Document the Zarr adapter's fixed-size dtype support
FrancescAlted
committed
5e2ff0b
Wait for End to take effect in the remote paging test
FrancescAlted
committed
edccebc
Run TUI tests once, outside the blocking CI matrix
FrancescAlted
committed
cbf4ab9
Fix remote cache lifecycle and compatibility regressions
FrancescAlted
committed
ab17e27