multiprocessio / dsq

Commandline tool for running SQL queries against JSON, CSV, Excel, Parquet, and more.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support for caching SQL-ingested data

eatonphil opened this issue · comments

This should be relatively easy. When a -c, --cache flag is on, instead of generating a random project name, generate one based on the hashed contents of the files being queried.

Don't use an in-memory :memory: SQLite table but put the SQLite file on disk with the same name as the project (based on the hash).

But in actually eval-ling the SQL/Program panel there might need to be a new flag added to DataStation and set in the evalContext maybe that says "don't reingest" and just run the SQL query with the DM_getPanel() calls substituted as usual for table names in SQL.

Finally, just don't delete the generated directory at the end of the script.