Kong Heroku Postgres app
Kong 0.8.0rc1 as a 12-factor app with external supervisor patches.
Uses the Kong Postgres buildpack and the Runit buildpack
This is pre-release software
Both Kong itself and this app are actively in development. MIT license is in effect.
Requirements
- Heroku CLI
- Postgres datastore
- Private network for clustering
- Heroku Common Runtime
- Only a single-dyno is fully supported,
heroku ps:scale web=1
- Kong's cluster will be bound to localhost,
127.0.0.1:7946
. - Multiple dynos will not be recognized in the cluster.
- Only a single-dyno is fully supported,
- Heroku Private Space
- Scale horizontally from one to hundreds of dynos,
heroku ps:scale web=10
- Kong's cluster connects via private subnet in the Space.
- Scale horizontally from one to hundreds of dynos,
- Heroku Common Runtime
Usage
Get started by cloning heroku-kong-pg and deploying it to a new Heroku app.
The serf
command must be installed locally to generate the cluster's shared secret. Download Serf
git clone https://github.com/chrisanderton/heroku-kong-pg.git
cd heroku-kong-pg
# Create app in Common Runtime:
heroku create my-proxy-app --buildpack https://github.com/heroku/heroku-buildpack-multi.git
# …or in a Private Space:
heroku create my-proxy-app --buildpack https://github.com/heroku/heroku-buildpack-multi.git --space my-private-space
heroku config:set KONG_CLUSTER_SECRET=`serf keygen`
# Add Heroku Postgres (this will give a warning if you're using a Private Space)
heroku addons:create heroku-postgresql:hobby-dev
git push heroku master
# …the first build will take approximately ten minutes; subsequent builds approx two-minutes.
The Procfile uses runit to supervise all of Kong's processes defined in Procfile.web.
Commands
To use Kong CLI in a console:
$ heroku run bash
# Run Kong in the background, so you can issue commands:
~ $ kong start -c $KONG_CONF
# …Kong will start & continue running in the background of this interactive console.
# Example commands:
~ $ kong --help
~ $ kong migrations list -c $KONG_CONF
~ $ curl http://localhost:8001/status
Configuration
The Heroku app must have several config vars, as defined in the buildpack.
Kong is automatically configured at runtime with the .profile.d/kong-12f.sh
script, which:
- renders the
config/kong.yml
file - exports environment variables (see:
.profile.d/kong-env
in a running dyno)
Revise config/kong.yml.etlua
to suite your application.
See: Kong 0.7 Configuration Reference
Postgres
You may connect to any Postgres datastore accessible to your Heroku app using the DATABASE_URL
config var as documented in the buildpack.
Once Postgres is attached to the app, Kong will automatically create the keyspace and run migrations.
Then, initialize DB schema using a console:
~ $ kong migrations reset -c $KONG_CONF
Kong plugins & additional Lua modules
See buildpack usage
Protecting the Admin API
Kong's Admin API has no built-in authentication. Its exposure must be limited to a restricted, private network.
For Kong on Heroku, the Admin API listens on the dyno's localhost port 8001.
That's the admin_api_port
set in config/kong.yml.etlua
.
console
Access viaMake API requests to localhost with curl.
$ heroku run bash
> kong start -c $KONG_CONFIG
> curl http://localhost:8001
Authenticated Admin API
Using Kong itself, you may expose the Admin API with authentication & rate limiting.
From the console:
# Create the authenticated `/kong-admin` API, targeting the localhost port:
curl -i -X POST --url http://localhost:8001/apis/ --data 'name=kong-admin' --data 'upstream_url=http://localhost:8001/' --data 'request_path=/kong-admin' --data 'strip_request_path=true'
curl -i -X POST --url http://localhost:8001/apis/kong-admin/plugins/ --data 'name=request-size-limiting' --data "config.allowed_payload_size=8"
curl -i -X POST --url http://localhost:8001/apis/kong-admin/plugins/ --data 'name=rate-limiting' --data "config.minute=12"
curl -i -X POST --url http://localhost:8001/apis/kong-admin/plugins/ --data 'name=key-auth' --data "config.hide_credentials=true"
curl -i -X POST --url http://localhost:8001/apis/kong-admin/plugins/ --data 'name=acl' --data "config.whitelist=kong-admin"
# Create a consumer with username and authentication credentials:
curl -i -X POST --url http://localhost:8001/consumers/ --data 'username=8th-wonder'
curl -i -X POST --url http://localhost:8001/consumers/8th-wonder/acls --data 'group=kong-admin'
curl -i -X POST --url http://localhost:8001/consumers/8th-wonder/key-auth
# …this response contains the `"key"`.
Now, access Kong's Admin API via the protected, public-facing proxy:
# Set the request header:
curl -H 'apikey: {kong-admin key}' https://kong-proxy-public.herokuapp.com/kong-admin/status
# or use query params:
curl https://kong-proxy-public.herokuapp.com/kong-admin/status?apikey={kong-admin key}
API Rate Limiting
Demo:Request this Bay Lights API more than five times in a minute, and you'll get HTTP Status 429: API rate limit exceeded, along with X-Ratelimit-Limit-Minute
& X-Ratelimit-Remaining-Minute
headers to help the API consumers regulate their usage.
Try it in your shell terminal:
curl -I https://kong-proxy-public.herokuapp.com/bay-lights/lights
# HTTP/1.1 200 OK
curl -I https://kong-proxy-public.herokuapp.com/bay-lights/lights
# HTTP/1.1 200 OK
curl -I https://kong-proxy-public.herokuapp.com/bay-lights/lights
# HTTP/1.1 200 OK
curl -I https://kong-proxy-public.herokuapp.com/bay-lights/lights
# HTTP/1.1 200 OK
curl -I https://kong-proxy-public.herokuapp.com/bay-lights/lights
# HTTP/1.1 200 OK
curl -I https://kong-proxy-public.herokuapp.com/bay-lights/lights
# HTTP/1.1 429
Here's the whole configuration for this API rate limiter:
curl -i -X POST --url http://localhost:8001/apis/ --data 'name=bay-lights' --data 'upstream_url=https://bay-lights-api-production.herokuapp.com/' --data 'request_path=/bay-lights' --data 'strip_request_path=true'
curl -i -X POST --url http://localhost:8001/apis/bay-lights/plugins/ --data 'name=request-size-limiting' --data "config.allowed_payload_size=8"
curl -i -X POST --url http://localhost:8001/apis/bay-lights/plugins/ --data 'name=rate-limiting' --data "config.minute=5"
# Demo loading app-specific Kong plugins & Lua modules.
curl -i -X POST --url http://localhost:8001/apis/bay-lights/plugins/ --data 'name=hello-world-header'
Demo: API translation, XML as JSON
JSON/REST has taken over as the internet API lingua franca, shedding the complexity of XML/SOAP. The National Digital Forecast Database [NDFD] is a legacy XML/SOAP service.
Here we demonstrate a custom plugin ndfd-xml-as-json to expose an JSON/REST API that fetches the maximum temperatures forecast for a location from the NDFD SOAP service. Using the single-resource concept of REST, the many variations of a SOAP interface may be broken out into elegant, individual JSON APIs.
Try it in your shell terminal:
curl --data '{"latitude":37.733795,"longitude":-122.446747}' https://kong-proxy-public.herokuapp.com/ndfd-max-temps
# Response contains max temperatures forecast for San Francisco, CA
curl --data '{"latitude":27.964157,"longitude":-82.452606}' https://kong-proxy-public.herokuapp.com/ndfd-max-temps
# Response contains max temperatures forecast for Tampa, FL
curl --data '{"latitude":41.696629,"longitude":-71.149994}' https://kong-proxy-public.herokuapp.com/ndfd-max-temps
# Response contains max temperatures forecast for Fall River, MA
Much more elegant than the legacy API. See the sample request body:
curl --data @spec/data/ndfd-request.xml -H 'Content-Type:text/xml' -X POST http://graphical.weather.gov/xml/SOAP_server/ndfdXMLserver.php
# Response contains wrapped XML data. Enjoy decoding that.
This technique may be used to create a suite of cohesive JSON APIs out of various legacy APIs.
Here's the configuration for this API translator:
curl -X POST -v http://localhost:8001/apis --data 'name=ndfd-max-temps' --data 'upstream_url=http://graphical.weather.gov/xml/SOAP_server/ndfdXMLserver.php' --data 'request_path=/ndfd-max-temps' --data 'strip_request_path=true'
curl -X POST -v http://localhost:8001/apis/ndfd-max-temps/plugins/ --data 'name=request-size-limiting' --data "config.allowed_payload_size=8"
curl -X POST -v http://localhost:8001/apis/ndfd-max-temps/plugins/ --data 'name=rate-limiting' --data "config.minute=5"
curl -X POST -v http://localhost:8001/apis/ndfd-max-temps/plugins/ --data 'name=ndfd-xml-as-json'
Librato
Demo: API analytics,Collect per-API metrics, explore, and set alerts on them with Librato. This librato-analytics
plugin demonstrates near-realtime (~1-minute delay), batch-oriented (up 300 metrics/post), asynchronous (non-blocking to proxy traffic) pushes of Kong/Nginx metrics to Librato's Metrics API.
The per-API metrics are sent by source, named "kong-{API-NAME}":
- request size (bytes)
- response size (bytes)
- kong latency (milliseconds)
- upstream latency (milliseconds)
- response latency (milliseconds)
This demo requires your own Heroku Kong instance with the Librato add-on. Kong sends custom metrics, so a paid plan of any level is required.
Here's the plugin configuration. Example based on the Bay Lights API example above:
curl -X POST -v http://localhost:8001/apis/bay-lights/plugins/ --data 'name=librato-analytics' --data "config.verify_ssl=false"
The LIBRATO_*
config vars set-up by the add-on will be used for authorization, but can be overridden by explicitly setting config.username
& config.token
for specific instances of the Kong plugin.
Dev Notes
Learning the Language of Kong
- Definitely an openresty guide
- An Introduction To OpenResty - Part 1, 2, & 3
- Nginx API for Lua,
ngx
reference, for use in Kong plugins- Nginx variables, accessible through
ngx.var
- Nginx variables, accessible through
Programming with Lua
- Lua 5.1, Note: Kong is not compatible with the newest Lua version
- Classic Objects, the basis of Kong's plugins
- Moses, functional programming
- Lubyk, realtime programming (performance- & game-oriented)
- resty-http, Nginx-Lua co-routine based HTTP client
- Serpent, inspect values
- Busted, testing framework
Using Environment Variables in Plugins
As a 12-factor app, Heroku Kong already uses environment variables for configuration. Here's how to use those vars within your own code.
- Whitelist the variable name for use within Nginx
- In the kong.yml config files
nginx:
property addenv MY_VARIABLE;
- Access the variable in Lua plugins
- Use
os.getenv('MY_VARIABLE')
to retrieve the value
Local Development
To work with Kong locally on Mac OS X.
Setup
- Install Kong using the .pkg
- Install Postgres
- Execute
./bin/setup
Running
- Postgres needs to be running
- Execute
./bin/start
- Logs in
/usr/local/var/kong/logs/
Testing
Any test-specific Lua rocks should be specified in .luarocks_test
file, so that they are not installed when the app is deployed.
- Add tests in
spec/
- Uses the Busted testing framework
- See also Kong integration testing
- Execute
source .profile.local
- Execute
busted
to run the tests