Query optimization
tomwhite opened this issue · comments
There are a couple of possibilities for doing query optimization that have come up recently.
Dask-expr will support arrays soon (dask/dask-expr#446). It would be interesting to see if the expression system can be used in Cubed, and if there are any changes we'd need to contribute back.
egglog "is a Python package that provides bindings to the Rust library egglog, allowing you to use e-graphs in Python for optimization". Interestingly, it has a prototype of the Array API, which might make it a good candidate for providing query optimization for Cubed. This tutorial has an example of using the Array API implementation to optimize a scikit-learn function. (@saulshanabrook told us about egglog at yesterday's Pangeo Distributed Computing Working Group.)
Ah wish I could've made the discussion. egglog looks cool!
I would be happy to do another call to get into more specifics of trying to implement the types of rewrites cubed needs with egglog. I did something similar a few weeks ago around the PyTensor project, using a concrete example to drive some exploration of how it could be implemented: https://egglog-python.readthedocs.io/latest/explanation/2023_11_17_pytensor.html
EDIT: Both the python bindings and the upstream rust library are in active development, so it's definitely useful to see if anything can be improved to support this kind of use case.
@saulshanabrook that would be awesome - thanks for the offer!