mjirv / demeter

Build a REST API on top of your data warehouse

Demeter

The dbt Metrics API

Demeter turns your dbt project into a metrics platform. You get a REST/GraphQL API so you can query, build interactive apps, and use your metrics from anywhere.

View live demo

• Demeter
  • The dbt Metrics API
  • About
  • Installation
    • Prerequisites
    • Quickstart (local)
  • Usage
    • Routes
      • GET /metrics
      • GET /metrics/:metric_name
      • POST /metrics/:metric_name
      • POST /graphql
  • Advanced
    • Running a production instance
    • Authentication

About

Demeter gives you a REST API (including a GraphQL endpoint) to query your dbt metrics

Installation

Prerequisites

1. A dbt project with metrics
2. Node
   • Run node --version
   • If there is no output or node is not found, follow the steps at https://heynode.com/tutorial/install-nodejs-locally-nvm/ to install Node.

Quickstart (local)

1. 📦 Install dbt-metrics-api
   • Add the following to your dbt project's packages.yml file:

   - git: "https://github.com/mjirv/demeter.git"
     revision: main

   • Run dbt deps
2. ✅ Run
   • Run cd dbt_packages/dbt_metrics_api/server && npm i && node dist/
   • Navigate to http://localhost:3001/graphql in your browser to start querying your API!

Usage

Routes

dbt-metrics-api has 4 routes:

GET /metrics

• returns a JSON array of your project's metrics

• supports "name", "type", "model", and "package_name" query strings for filtering

  $ curl -X GET "http://localhost:3001/metrics?type=count&package_name=jaffle_shop"
  
  [{"unique_id":"metric.jaffle_shop.orders","package_name":"jaffle_shop","model":"ref('orders')","name":"orders","description":"The number of orders","label":"Orders","type":"count","filters":[],"time_grains":["day","week","month","quarter","year"],"dimensions":["status","customer_id"]},{"unique_id":"metric.jaffle_shop.orders2","package_name":"jaffle_shop","model":"ref('orders')","name":"orders2","description":"The number of orders","label":"Orders","type":"count","filters":[],"time_grains":["day","week","month","quarter","year"],"dimensions":["status","customer_id"]}]
  

GET /metrics/:metric_name

• returns a JSON object with keys unique_id, package_name, model, name, description, label, type, filters, time_grains, dimensions

  $ curl -X GET "http://localhost:3001/metrics/orders"
  
  {"unique_id":"metric.jaffle_shop.orders","package_name":"jaffle_shop","model":"ref('orders')","name":"orders","description":"The number of orders","label":"Orders","type":"count","filters":[],"time_grains":["day","week","month","quarter","year"],"dimensions":["status","customer_id"]}
  

POST /metrics/:metric_name

• Queries a metric and returns the result

• Accepts a JSON object in the request body with the following properties: grain, dimensions, start_date, end_date (start_date and end_date are optional)

• Returns a JSON object or CSV depending on your Accept: header (application/json or text/csv)

  $ curl http://localhost:3001/metrics/orders -H "Content-Type: application/json" -H "Accept: application/json" -d '{"grain": "year", "start_date": "2017-01-01", "end_date": "2019-01-01"}'
  
  [{"period": "2017-01-01", "orders": 0.0}, {"period": "2018-01-01", "orders": 99.0}, {"period": "2019-01-01", "orders": 0.0}]
  

POST /graphql

• GraphQL API for your metrics
• Visit YOUR_SERVER_PATH/graphql (e.g. http://localhost:3001/graphql) to see the GraphiQL client and schema

Advanced

Running a production instance

On your favorite cloud hosting service (e.g. Heroku or GCP Cloud Run) install this repo via the GitHub URL.

Look in .example.env for the available environment variables and configure them as needed.

Authentication

dbt-metrics-api supports Kable for authentication.

To get started:

1. Sign up at https://kable.io (the free tier includes unlimited test requests and up to 10,000 live requests)

2. Note your client ID and secret and add them to your server/.env.local file:

   KABLE_CLIENT_ID=
   KABLE_CLIENT_SECRET=
   
   # you will need to set to LIVE in production
   KABLE_ENV=TEST

3. Requests will now require authentication with a customer ID and secret that you set up in your Kable dashboard using X-CLIENT-ID and X-API-KEY headers

   $ curl -X GET "http://localhost:3001/metrics/orders" -H "X-CLIENT-ID: test-customer-1" -H "X-API-KEY: sk_test.some.secret.key"