Shillelagh (ʃɪˈleɪlɪ) is a Python library and CLI that allows you to query many resources (APIs, files, in memory objects) using SQL. It's both user and developer friendly, making it trivial to access resources and easy to add support for new ones.
The library is an implementation of the Python DB API 2.0 based on SQLite (using the APSW library):
There is also a SQLAlchemy dialect:
And a command-line utility:
Sharks have been around for a long time. They're older than trees and the rings of Saturn, actually! The reason they haven't changed that much in hundreds of millions of years is because they're really good at what they do.
SQL has been around for some 50 years for the same reason: it's really good at what it does.
Picture a leprechaun hitting APIs with a big stick so that they accept SQL.
Shillelagh allows you to easily query non-SQL resources. For example, if you have a Google Spreadsheet you can query it directly as if it were a table in a database:
You can even run INSERT
/DELETE
/UPDATE
queries against the spreadsheet:
Queries like this are supported by adapters. Currently Shillelagh has the following adapters:
Name | Type | URI pattern | Example URI |
---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
There are also 3rd-party adapters:
A query can combine data from multiple adapters:
The query above reads timestamps from a Google sheet, uses them to filter weather data from WeatherAPI, and writes the chance of rain into a (pre-existing) CSV file.
New adapters are relatively easy to implement. There's a step-by-step tutorial that explains how to create a new adapter to an API or filetype.
Install Shillelagh with pip
:
You also need to install optional dependencies, depending on the adapter you want to use:
$ pip install 'shillelagh[console]' # to use the CLI
$ pip install 'shillelagh[datasetteapi]' # for Datasette
$ pip install 'shillelagh[genericjsonapi]' # for Generic JSON
$ pip install 'shillelagh[githubapi]' # for GitHub
$ pip install 'shillelagh[gsheetsapi]' # for GSheets
$ pip install 'shillelagh[htmltableapi]' # for HTML tables
$ pip install 'shillelagh[pandasmemory]' # for Pandas in memory
$ pip install 'shillelagh[s3selectapi]' # for S3 files
$ pip install 'shillelagh[socrataapi]' # for Socrata API
$ pip install 'shillelagh[systemapi]' # for CPU information
$ pip install 'shillelagh[weatherapi]' # for WeatherAPI
Alternatively, you can install everything with: