`magic_duckdb` based magic?
eitsupi opened this issue · comments
In combination with DuckDB, it seems that magic_duckdb may provide better performance than jupysql.
https://github.com/iqmo-org/magic_duckdb
Yes, looks interesting. I'm not sure whether it's sufficiently mature or not — I notice it's new and has unfinished sentences on the Readme :) (all tools start somewhere!).
The perf does look much better though, I'm surprised the existing approach has so much overhead
jupysql is really amazing. This performance issue should be solved...
ploomber/jupysql#470
https://twitter.com/ploomber/status/1661824499224915970
https://github.com/ploomber/jupysql/blob/master/CHANGELOG.md
0.7.5 (2023-05-24)
[Feature] Using native DuckDB .df() method when using autopandas
yes, we fixed the performance issue when %autopandas is turned on. And we just started working on ploomber/jupysql#470 to provide further improvements.
If you encounter any performance issues, let us know and we'll fix them!
Thank you @edublancas !