Remote arrays, hierarchical store discovery, and unified caching

[FrancescAlted]

This branch introduces comprehensive remote data access capabilities to Python-Blosc2. Users can now open, navigate, and slice remote arrays and multi-dataset containers (.b2nd, .b2z, Zarr, HDF5, and Caterva2) without pre-downloading entire datasets. Data chunks are fetched lazily on demand and managed through configurable, shared cache budgets.

Key Features

1. High-Level Remote APIs

• RemoteArray: Direct, slice-on-demand access to standalone remote files or individual container datasets.
• RemoteStore: Tree-like container discovery and navigation (store.keys(), store["group/dataset"]) across B2Z, Zarr, and HDF5 archives, sharing a single cache budget across all leaves.
• blosc2.open(..., lazy=True): Unified entry point for remote URLs (https://, s3://, gcs://, abfs://, etc.) with storage_options.

2. Remote Format Adapters (proxy_source.py)

• b2z_source.py: Native Blosc2 chunk/block range reads from external ZIP_STORED NDArray members in .b2z archives without unzipping.
• zarr_source.py: Zero-copy/converted reads from remote Zarr v2 and v3 arrays (including scalar, empty, and fixed dtypes).
• hdf5_source.py: Remote HDF5 slicing via kerchunk byte-range pre-indexing, with reference map reuse across container leaves and sessions (blosc2.available_datasets).
• proxy_source.py: Standard byte-range fetches over arbitrary fsspec protocols.

3. Unified Caching & Portable Snapshots

• init.py: Support for NONE, MEMORY, and persistent DISK caching with LRU eviction and memory bounds.
• TreeStore-backed disk layout: Generations are persisted as standard .b2d structures.
• Portable reference exports: RemoteStore.save("snapshot.b2z") serializes remote manifests with optional retained caches.
• In-place snapshot reads: Opening exported .b2z reference snapshots via blosc2.open() runs directly in-place with zero disk overhead for immutable caches, falling back to transient memory reads on misses.
• Caterva2 integration: Sparse-cache backend support via with_sparse_cache().

4. User Metadata & CLI / TUI Tools

• Standardized .attrs interface: Exposes .attrs as the recommended user metadata interface across arrays, containers, and proxy sources (aliasing .vlmeta while preserving Caterva2 server attributes).
• b2view TUI enhancement: Terminal browsing of remote containers and hierarchies with persistent disk caching (--cache-dir) and configurable cache budgets (--max-cache-bytes).
• b2nd-to-zarr CLI: New command-line tool in b2nd_to_zarr.py for converting .b2nd files to Zarr format.

5. Maintenance & Upgrades

• Bundled C-Blosc2 updated to 3.3.4.
• Comprehensive remote benchmarks (bench/remote_array_traffic.py), guide documentation (doc/guides/remote_arrays.md), and reference tests.

Quick Example

import blosc2

# Discover and navigate a remote container (B2Z, Zarr, or HDF5)
with blosc2.RemoteStore("https://datasets.example.org/data.h5") as store:
    print(store.keys())
    arr = store["experiment/temperature"]  # RemoteArray leaf
    slice_data = arr[:100]  # Only requested chunk ranges are fetched
    metadata = arr.attrs[:]

# Or open standalone remote arrays directly
a = blosc2.open("s3://my-bucket/dataset.b2nd", lazy=True)
values = a[0:50, 0:50]

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate findings remain in b2view lifecycle/concurrency, cache identity/resource coordination, and B2ND-to-Zarr conversion.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds lazy remote array/store access, shared caching, format adapters, b2view integration, and B2ND-to-Zarr conversion.

Changes:

• Adds RemoteArray/RemoteStore support for B2Z, Zarr, HDF5, fsspec, and Caterva2.
• Adds memory/disk caching, snapshots, metadata handling, and storage options.
• Adds CLI/TUI tooling, documentation, examples, benchmarks, and tests.

File summaries

File	Reviewed change
tests/test_zarr_source.py	Zarr adapter coverage
tests/test_vlmeta.py	Metadata behavior
tests/test_python_blosc.py	Core regression coverage
tests/test_objectarray.py	Object-array compatibility
tests/test_locking.py	Locking stress coverage
tests/test_fsspec_s3.py	fsspec and S3 access
tests/test_ctable_cframe.py	CTable frame coverage
tests/test_b2view_model.py	Remote browser model
tests/test_b2objects.py	Caterva2 object coverage
tests/test_b2nd_to_zarr.py	Conversion coverage
tests/test_attrs.py	Attribute interface
tests/ndarray/test_proxy.py	Proxy array behavior
tests/ndarray/test_indexing.py	Array indexing
tests/ndarray/test_c2array_writes.py	C2Array writes
tests/ndarray/test_c2array_blocks.py	C2Array blocks
tests/ndarray/test_c2array_async.py	Async C2Array behavior
tests/ctable/test_utf8.py	UTF-8 columns
tests/ctable/test_object_spec.py	Object column specifications
tests/ctable/test_dictionary_column.py	Dictionary columns
tests/ctable/test_ctable_indexing.py	CTable indexing
tests/conftest.py	Test fixtures
tests/b2view/tui_wait.py	TUI test helper
tests/b2view/tree_store_gen.py	Tree-store test generation
tests/b2view/test_sort.py	Browser sorting
tests/b2view/test_render.py	Browser rendering
tests/b2view/test_group.py	Browser grouping
tests/b2view/test_cli.py	b2view CLI coverage
tests/b2view/test_basics.py	b2view basics
src/blosc2/zarr_source.py	Zarr remote adapter
src/blosc2/tree_store.py	Tree-store persistence
src/blosc2/remote_store_cache.py	Persistent cache generations
src/blosc2/ref.py	Reference serialization
src/blosc2/proxy_source.py	Remote range-source infrastructure
src/blosc2/objectarray.py	Object-array implementation
src/blosc2/ndarray.py	NDArray behavior
src/blosc2/msgpack_utils.py	MessagePack utilities
src/blosc2/list_array.py	List-array behavior
src/blosc2/lazyexpr.py	Lazy expressions
src/blosc2/dict_store.py	Dictionary store
src/blosc2/ctable.py	CTable implementation
src/blosc2/core.py	Core URL and storage handling
src/blosc2/cli/__init__.py	CLI registration
src/blosc2/c2array.py	C2Array implementation
src/blosc2/batch_array.py	Batch-array behavior
src/blosc2/b2z_source.py	B2Z range adapter
src/blosc2/b2view/render.py	b2view rendering
src/blosc2/b2view/model.py	Remote browser model
src/blosc2/b2view/cli.py	b2view CLI
src/blosc2/b2objects.py	Caterva2 objects
src/blosc2/__init__.py	Public API exports
RELEASE_NOTES.md	Release documentation
pyproject.toml	Packaging and dependencies
plans/remote-proxy-v3.md	Remote proxy plan
plans/remote-proxy-v11.md	Remote proxy plan
examples/vlmeta.py	Metadata example
examples/remote/store-browse.py	Remote store browsing
examples/remote/s3-access.py	Remote S3 access
examples/remote/rw-fsspec.py	fsspec read/write example
examples/remote/proxy-carray.py	Proxy CArray example
examples/remote/fsspec-cat2-access.py	Caterva2 fsspec access
examples/remote/concurrent-fsspec.py	Concurrent fsspec access
examples/remote/c2array-traffic.py	C2Array traffic example
examples/remote/c2array-get-slice.py	C2Array slicing
examples/remote/c2array_expr.py	C2Array expressions
doc/tutorials/06.remote_proxy.ipynb	Remote proxy tutorial
doc/reference/zarrndsource.rst	Zarr source reference
doc/reference/tree_store.rst	Tree-store reference
doc/reference/schunk.rst	Schunk reference
doc/reference/remotestore.rst	RemoteStore reference
doc/reference/remotearray.rst	RemoteArray reference
doc/reference/ref.rst	Reference API
doc/reference/ndarray.rst	NDArray reference
doc/reference/msgpack_serialization.rst	Serialization reference
doc/reference/lazyarray.rst	Lazy-array reference
doc/reference/hdf5ndsource.rst	HDF5 source reference
doc/reference/fsspecndsource.rst	fsspec source reference
doc/reference/classes.rst	Class reference
doc/reference/c2array.rst	C2Array reference
doc/reference/b2zndsource.rst	B2Z source reference
doc/guides/b2view.rst	b2view guide
doc/getting_started/installation.rst	Installation documentation
CMakeLists.txt	Build configuration
bench/remote_array_traffic.py	Remote traffic benchmark
bench/remote_array_traffic.md	Benchmark documentation
.github/workflows/build.yml	CI workflow

Review details

Suppressed comments (9)

doc/reference/remotearray.rst:110

• The documentation says authorized B2Z sparse attachment is unsupported, but RemoteArray.with_sparse_cache is implemented for this source and the B2Z tests exercise it successfully. This prevents users from discovering the supported persistent sparse-cache path; update the capability statement to describe the eligible external NDArray leaves and retain only the Caterva2 limitation.
  src/blosc2/b2view/app.py:2257
• The remote browser is closed only in a daemon thread during unmount. RemoteStore.close() publishes the cache manifest, so app.run() can return and terminate the process before this thread finishes; fetched payloads may then be left without an active manifest and be missed on the next run. Make shutdown wait for the closer (or otherwise guarantee the close/publish completes before exit).
  src/blosc2/b2view/cli.py:62
• The PR description promises a configurable cache budget for b2view, but this adds only --cache-dir; there is no --max-cache-bytes argument or value forwarded to StoreBrowser, whose remote store still hardcodes 64 MiB. Users therefore cannot configure the advertised shared-cache limit. Add the option and thread it through the CLI, app, and browser/store construction.
  src/blosc2/b2view/model.py:305
• The PR description advertises a configurable b2view cache budget via --max-cache-bytes, but this path hard-codes every remote store to 64 << 20 and b2view/cli.py exposes no corresponding option. Users therefore cannot configure the advertised limit; add the CLI/app plumbing and pass the selected value here, or remove the claim.
  src/blosc2/cli/b2nd_to_zarr.py:397
• Valid Blosc2 empty arrays can have a zero-valued chunk extent (for example, shape (0, 100) with chunks (0, 9)). The resulting zero copy step makes range(0, s, step) raise, and passing the inherited zero chunk extent to Zarr is invalid, so this converter cannot handle an otherwise valid empty NDArray. Normalize output chunk extents to positive Zarr values and special-case empty batch generation.
  src/blosc2/cli/b2nd_to_zarr.py:302
• This slice verification also uses np.array_equal with its default equal_nan=False, so a matching slice containing NaNs is reported as corrupt. Pass equal_nan=True here, consistently with the sample and full-array checks.
  src/blosc2/remote_array.py:394
• When a Zarr dataset is opened from a URL containing a query or fragment (for example a signed HTTP URL), this appends the dataset after the query string, producing ...?token=.../dataset instead of adding it to the URL path. The resulting object cannot be opened and signed URLs/fsspec options are lost; append the dataset to the parsed path and rebuild the URL while preserving its query and fragment, with a regression test for such a URL.
  src/blosc2/remote_array.py:220
• The validation accepts B2ZNDSource below, but this error message omits it and tells callers that only Fsspec, Zarr, or HDF5 sources are valid. Include the B2Z source in the message so an invalid authorized-source error accurately describes the supported API.
  src/blosc2/remote_store.py:1112
• Each refresh publishes a new generation into the disk cache, but unlike initial construction it never calls discard_old_generations. Repeated refreshes therefore retain every prior .b2d generation and its leaf payloads indefinitely, allowing the configured cache directory to grow without bound. Discard old generations after publishing the replacement manifest.

• Files reviewed: 100/112 changed files
• Comments generated: 9
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.