API REST source type

[Carlos Sansón]

Feature Description

A new source type that allows to use Latitude on API REST endpoints.

Proposal

Usage

An API source type can be configured with the the following schema:

type: api
config:
  method: GET
  headers:
    Authorization: <custom auth header>
    (... other custom headers)

Then, the sql query can be used as follows:

{@config method = 'POST'} -- Optional configuration overrides
SELECT *
FROM {api('https://api.example.com/v1/data')}
WHERE id = 1

Implementation

This api function is a exclusive method for this source type. In compile time, it will make a request based on the function parameters and source configuration, save the response as a temporary parquet file, and include it in the query. Then, the query will be executed as usual by a DuckDB connector.

To be able to parse the response, currently it must follow a given JSON schema previously defined by us.

Alternatively, we can require the user to define each endpoint in the source config file, or maybe in a {@config} tag within the query, following the OpenAPI specification. This would allow the user to define the endpoints and their parameters in a standard way, and then use it in the query by refering to the endpoint id and passing parameters as a second argument.

[Gerard]

fantastic 👌🏼