fix: keep chunk-cached DataArray.data lazy (#2910) - #2912
RanaPriyansh wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
I don't think that this test file is necessary. FWICT its pretty much just testing that dataarray.data is a Dask array (and testing Dask functionality).
I don't think its worth including in our test suite
| def get_duck_array(self): | ||
| return self.array.compute() | ||
| return self.array |
There was a problem hiding this comment.
Though there is the problem that users writing interpolators should never really be using da.data when using ChunkedArrays (since it falls back to Dask, which is less performant, or (before this PR) uses Numpy, which causes eager computation of results.
Implementing get_duck_array at all could result in users writing interpolators that have really bad performance.
I'm thinking maybe the solution is just to do a raise NotImplementedError here. This would mean that users can't inspect the data using a da.data, but I think thats acceptable (users won't be inspecting this anyway).
Thoughts @erikvansebille ?
There was a problem hiding this comment.
Agreed that .data bypasses the chunk cache. I tested raising from get_duck_array at this head. It also makes .values and np.asarray(data_array) raise NotImplementedError; vectorized .isel(...).data still works. The API decision therefore includes whether explicit whole-array materialization should remain supported.
For test scope, the no-computation assertion fails on the original Parcels implementation because get_duck_array computes the chunks. The separate module can be reduced to a focused integration regression once the accessor contract is settled.
|
superceded by #2916 |
Summary
Accessing
DataArray.dataon aChunkCachedArrayfield currently computes every backing Dask chunk. Return the backing Dask array fromget_duck_array()so.datastays lazy. Vectorized selection continues to use the chunk cache.Validation
.dataaccess and explicit.valuesandnp.asarraymaterialization.Fixes #2910
AI disclosure
Codex used.