feat(array): add ArrayIO and AsyncArrayIO, the explicit-IO layer under Array and AsyncArray

changes/419.feature.md

@@ -0,0 +1,6 @@
+Added `zarr.ArrayIO` and `zarr.AsyncArrayIO`, an explicit-IO layer over Zarr arrays. Constructing one performs no IO against storage: storing metadata (`store_metadata`), storing or reading chunks (`setitem` / `getitem`), and their erasing inverses (`erase_metadata`, `erase_chunk`, `erase_chunks`) are explicit method calls that run in whatever order the caller chooses, so chunks can be written before metadata. The declaring constructors, `build_v2` and `build_v3`, are synchronous — binding is pure — take a `Store` or `StorePath`, and take the fields of each format's metadata document, with nothing hiding the difference between the formats; the documents themselves are built by the module-level `build_array_metadata_v2` and `build_array_metadata_v3`. `open` auto-detects the Zarr format by default, as `zarrs` does, and `verify` reads the stored metadata and raises, naming each differing field, when it does not match the metadata the instance is bound to.
+`Array` and `AsyncArray` are built on this layer: every `AsyncArray` wraps an `AsyncArrayIO` and delegates its metadata and selection IO to it, the wrapped layer is reachable as `Array.io` / `AsyncArray.io`, and `to_array()` returns an array over a copy of the binding, so nothing done through the array (an attribute update, a resize) changes the binding it came from.
+The binding holds its metadata as a `zarr-metadata` model — `zarr-python` now depends on `zarr-metadata`, which raises its minimum `typing-extensions` to 4.16 — reachable as `ArrayIO.metadata`, with `ArrayIO.metadata_legacy` the same metadata resolved into the legacy classes: metadata operations (`store_metadata`, `erase_metadata`, `erase_chunk`, `erase_chunks`, `verify`) never resolve codec or data type names, so an array whose codecs this environment lacks can still be opened, verified and erased, and names resolve into classes only when chunks are read or written.
+`with_metadata`, `with_store` and `with_config` return new bindings with one part replaced.
+`ArrayIO` responds to the global configuration the way `Array` always has: the codec pipeline implementation is captured when the binding is made, the runtime configuration defaults from `zarr.config`, and `build_v2`'s `order` defaults from `array.order`.
+Whether a stored array document needs a repair that moves chunks (such as a stored chunk size of 0) is now recorded per store object, in a cache of the metadata documents zarr has read and stored, instead of on the metadata read from it. Every array bound to one store object shares what any of them learned, derived bindings carry nothing, and the first chunk write through any binding that knows the document, including a pickled one, stores the repair; nothing is read that was not read before.

docs/api/zarr/array.md

@@ -1,2 +1,4 @@
 ::: zarr.Array
 ::: zarr.AsyncArray
+::: zarr.ArrayIO
+::: zarr.AsyncArrayIO

docs/user-guide/installation.md

@@ -9,7 +9,7 @@ Required dependencies include:
 - [numpy](https://numpy.org) (2.0 or later)
 - [numcodecs](https://numcodecs.readthedocs.io/en/stable/) (0.14 or later)
 - [google-crc32c](https://github.com/googleapis/python-crc32c) (1.5 or later)
-- [typing_extensions](https://typing-extensions.readthedocs.io/en/latest/) (4.14 or later)
+- [typing_extensions](https://typing-extensions.readthedocs.io/en/latest/) (4.16 or later)
 - [donfig](https://donfig.readthedocs.io/en/latest/) (0.8 or later)
 
 ## pip

pyproject.toml

@@ -50,9 +50,10 @@ dependencies = [
     # 0.16.6 and 0.17.1 refuse to unpickle untrusted data (GHSA-jqc8-qjfv-5m6r); 0.17.0 does not.
     'numcodecs>=0.16.6,!=0.17.0',
     'google-crc32c>=1.5',
-    'typing_extensions>=4.14',
+    'typing_extensions>=4.16',
     'donfig>=0.8',
     'msgspec>=0.19',
+    'zarr-metadata>=0.6.0',
 ]
 
 dynamic = [
@@ -257,7 +258,7 @@ extra-dependencies = [
     'fsspec==2023.10.0',
     's3fs==2023.10.0',
     'universal_pathlib==0.2.0',
-    'typing_extensions==4.14.*',
+    'typing_extensions==4.16.*',
     'donfig==0.8.*',
     'obstore==0.5.*',
     'msgspec==0.19.*',

src/zarr/__init__.py

@@ -34,7 +34,7 @@
     zeros,
     zeros_like,
 )
-from zarr.core.array import Array, AsyncArray
+from zarr.core.array import Array, ArrayIO, AsyncArray, AsyncArrayIO
 from zarr.core.config import config
 from zarr.core.group import AsyncGroup, Group
 
@@ -147,7 +147,9 @@ def set_format(log_format: str) -> None:
 
 __all__ = [
     "Array",
+    "ArrayIO",
     "AsyncArray",
+    "AsyncArrayIO",
     "AsyncGroup",
     "Group",
     "__version__",

src/zarr/api/asynchronous.py

@@ -18,6 +18,7 @@
     create_array,
     from_array,
     get_array_metadata,
+    observe_read,
 )
 from zarr.core.array_spec import ArrayConfigLike, parse_array_config
 from zarr.core.buffer import NDArrayLike
@@ -425,7 +426,11 @@ async def open(
                 zarr_format = _metadata_dict["zarr_format"]
                 is_v3_array = zarr_format == 3 and _metadata_dict.get("node_type") == "array"
                 if is_v3_array or zarr_format == 2:
-                    return AsyncArray(store_path=store_path, metadata=_metadata_dict, config=config)
+                    array = AsyncArray(
+                        store_path=store_path, metadata=_metadata_dict, config=config
+                    )
+                    observe_read(store_path, metadata_dict)
+                    return array
             except (FileNotFoundError, NodeTypeValidationError):
                 pass
         _check_no_group_config(config, store_path)