Stabilize zarr.core.sync.sync() for downstream async integration

[aladinor]

Context

We're adding async support to xarray's open_datatree (xarray PR #10742) to enable concurrent I/O when opening multi-group zarr stores. This achieves at least ~6x speedup on cloud data (8.5s → 1.4s) by opening groups concurrently using zarr's async APIs.

During review, @shoyer raised concerns about xarray relying on zarr internal APIs that could break without notice.

What xarray uses from zarr

API	Public?	Usage
zarr.AsyncGroup	Yes (in __all__)	Type checking, async group operations
AsyncGroup.members()	Yes (documented method)	Listing child groups asynchronously
AsyncGroup.open()	Yes (documented classmethod)	Opening groups asynchronously
zarr.core.sync.sync()	No (not re-exported, not in __all__)	Bridging async→sync in xarray's open_datatree()

Why we need zarr.core.sync.sync()

xarray's open_datatree() is synchronous, but we want to use zarr's AsyncGroup APIs internally to open multiple groups concurrently. We use zarr.core.sync.sync() to bridge async→sync — the same function zarr uses internally via SyncMixin._sync() to implement its sync Group class.

Without a supported bridge, xarray would need to manage its own event loop, which is error-prone and could conflict with zarr's internal loop management.

Example usage in xarray:

from zarr.core.sync import sync as zarr_sync

# In xarray's sync open_datatree path
results = zarr_sync(create_indexes_async())

The ask

1. Could zarr.core.sync.sync() be made public? It would just need to be re-exported in zarr/__init__.py and added to __all__. This seems low-cost since zarr already depends on it extensively internally.

2. Or is there an alternative supported mechanism for downstream libraries to run async zarr operations from sync contexts?

3. Can we also confirm that AsyncGroup.members() return type (AsyncGenerator[tuple[str, AsyncArray | AsyncGroup], None]) is considered stable?

Alternatives we've considered

• xarray manages its own event loop — risky, likely conflicts with zarr's loop
• Higher-level zarr API (e.g., Group.open_members() returning all child groups concurrently) — would eliminate the need for xarray to manage concurrency, but more work on zarr's side

References

• xarray PR: Implement async support for open_datatree pydata/xarray#10742
• xarray async strategy discussion: How should Xarray control asynchronous calls? pydata/xarray#10622

cc @TomNicholas @shoyer @keewis @dcherian @ilan-gold @d-v-b

We'd love to hear your thoughts, suggestions, or alternative approaches. Any feedback from the zarr core team on the best path forward would be greatly appreciated!

[aladinor]

Also worth noting: @BorisTheBrave has proposed the same approach in xarray PR #11171 — using zarr's sync() to share zarr's internal event loop so the sync API gets async benefits transparently. See his comment: pydata/xarray#10622 (comment)

This reinforces the case for making zarr.core.sync.sync() a stable, public API — multiple downstream efforts are converging on it as the right bridge between sync and async.

[shoyer]

Another (non-exclusive) idea worth considering is forking Zarr's Jupyter-safe event loop logic into a separate library, which could be used by other projects like Xarray. I think we would still need a way to to invoke Zarr's event loop from inside Xarray for async Zarr IO, though.

[d-v-b]

do you need to interface with existing async xarray routines, or do you just need to synchronously invoke something that should be done asynchronously under the hood?

If it's the second option, shouldn't we add an API to zarr-python that does this? Like, you say you need to open multiple groups concurrently. That means reading N metadata documents concurrently. that means using the async _get_many method for stores, then wrapping that in blocking code, and making it public. We already have the async method, but we don't have a public API for it.

More broadly, we don't have any public synchronous API for our store classes, which IMO is a major oversight, because this would evidently be useful to people. I tried to move in this direction with this pr: #3638, but the result was private methods. IMO we should consider a public, synchronous store API, so people can do useful concurrency-backed store operations without needing to touch our event loop.

[ilan-gold]

I think in this case, you're right @d-v-b since the use-case seems focused on just getting metadata (right @aladinor?).

When I think of the use-cases, though, the following pipeline comes to mind (Which I am not sure applies to xarray):

async def _one_array_work(my_async_zarr_array: zarr.Array, ...):
    res = await my_async_zarr_array.getitem(...)
    mutated_res = do_work(res)
    return mutated_res
zarr.sync(asyncio.gather(*(_one_array_work(args) for args in args_list)))

To me, zarr python could offer a collect method that might produce code that looks like

getitem_res_list = zarr.collect(arrays, subsets)
[do_work(res) for res in getitem_res_list]

But then you lose the interleaving, which I think is actually desirable, no? Maybe we just need a little callback for collect to allow for interleaving?

getitem_res_list = zarr.collect(arrays, subsets, callback= do_work)

[ilan-gold]

But I personally even have a use-case that looks like

async def _two_array_work(my_async_zarr_array: zarr.Array, my_other_async_zarr_array: zarr.Array,...):
    res1 = await my_async_zarr_array.getitem(...)
    res2 = await my_other_async_zarr_array.getitem(...)
    res = do_work(res1, res2)
    return res
zarr.sync(asyncio.gather(*(_two_array_work(args) for args in args_list)))

which now has 2 arrays. collect starts to look a little weirder now

[d-v-b]

IMO what we need to support this kind of operation would be something like lazy indexing, where you can lazily write to a subregion of an array and defer the actual IO. IIRC some of this is working in my lazy indexing prototype

[TomNicholas]

Xarray needs to fetch/set metadata of arbitrary groups/arrays concurrently, and also concurrently fetch/set all the data for some of those arrays.

I don't think the xarray (datatree) use case has much to do with lazy indexing, as most of what we want is either metadata operations or getting/setting entire arrays (the coordinate arrays).

Lets disambiguate the two questions here:

1. "What is the ideal async interface for zarr operations in the future?"
2. "Can xarray rely on the zarr.core.sync.sync() function, so that we can merge PRs that enormously speed up xarray today, without xarray breaking due to unexpected zarr API changes?"

We suggested that @aladinor raise this issue specifically to ask about (2).

[d-v-b]

these are related questions. I would rather add useful API to zarr-python (question number 1) instead of exposing our internals to downstream consumers (question number 2).