Implement adapter for `OpenWithKerchunk | OpenWithXarray`

Question

Implement adapter for `OpenWithKerchunk | OpenWithXarray`

cisaacstern opened this issue 10 months ago · comments

In theory, OpenWithKerchunk should be able to provide inputs for OpenWithXarray, which can help address issues such as #361. IIUC a version of this existed in 0.9.4 (or at least was in development there). Discussion in leap-stc/cmip6-leap-feedstock#16 (comment) reminded me that this would be a useful thing to implement (or re-implement, as the case may be).

Raphael Hagen · Answer 1 · Tue Oct 03 2023 23:12:17 GMT+0800 (China Standard Time)

This seems like a great idea!

As far as design. OpenWithKerchunk returns a PCollection of references in memory. It seems like there would need to be either:

An additional PTransform to convert the PCollection of references to a PCollection of fsspec mappers that could be read by OpenWithXarray?
An option within OpenWithKerchunk that returns fsspec mapppers.

Any thoughts here @cisaacstern?

Charles Stern · Answer 2 · Tue Oct 03 2023 23:20:38 GMT+0800 (China Standard Time)

Good questions, @norlandrhagen.

An option within OpenWithKerchunk that returns fsspec mapppers.

I think I'd lean towards this option. The downside this that it introduces multiple return types into OpenWithKerchunk, but the benefit is it keeps the user-facing API simpler.

Justus Magin · Answer 3 · Tue Oct 03 2023 23:25:35 GMT+0800 (China Standard Time)

another option would be to have OpenWithXarray use an engine (xr.open_dataset backend) that immediately knows what to do with the references (the "kerchunk" engine discussed in fsspec/kerchunk#360?)

Raphael Hagen · Answer 4 · Tue Oct 03 2023 23:32:00 GMT+0800 (China Standard Time)

That would be the best way!

@keewis are you working on that PR/issue or do you know if there is any development on it?

Justus Magin · Answer 5 · Tue Oct 03 2023 23:46:50 GMT+0800 (China Standard Time)

I'm not working on this nor am I planning to at the moment (and I'm not aware of anyone else doing so), but the development will most likely happen on the kerchunk repo.

Charles Stern · Answer 6 · Wed Oct 04 2023 00:04:42 GMT+0800 (China Standard Time)

Thanks for mentioning this @keewis.

Whichever solution we choose here, let's link fsspec/kerchunk#360 in a comment, and mention that the implementation here is a shim until that issue is resolved.