SQL Runner
[ ![Build Status] travis-image ] travis [ ![Release] release-image ] releases [ license-image ] license
Overview
Run playbooks of SQL scripts in series and parallel on Amazon Redshift and PostgreSQL.
Used with Snowplow snowplow for scheduled SQL-based transformations of event stream data.
User Quickstart
Assuming you are running on 64bit Linux:
> wget http://dl.bintray.com/snowplow/snowplow-generic/sql_runner_0.1.0_linux_amd64.zip
> unzip sql_runner_0.1.0_linux_amd64.zip
> ./sql-runner -usage
See the User Guide section below for more.
Developer Quickstart
Building
Assuming git, [Vagrant] vagrant-install and [VirtualBox] virtualbox-install installed:
host> git clone https://github.com/snowplow/sql-runner
host> cd sql-runner
host> vagrant up && vagrant ssh
guest> cd /opt/gopath/src/github.com/snowplow/sql-runner
guest> godep go build
Testing
Assuming Building complete:
guest> psql -c 'create database sql_runner_tests_1' -U postgres
guest> psql -c 'create database sql_runner_tests_2' -U postgres
guest> ./sql-runner -playbook ./integration-tests/postgres.yml
Publishing
Assuming Travis travis is green and versions updated:
guest> godep go build
host> vagrant push
User guide
Playbooks
A playbook consists of one of more steps, each of which consists of one or more queries. Steps are run in series, queries are run in parallel within the step.
Each query contains the path to a query file. See Query files for details
All steps are applied against all targets. All targets are processed in parallel.
For the playbook template see: [config/config.yml.sample] example-config
Query files
A query file contains one or more SQL statements. These are executed "raw" (i.e. not in a transaction) in series by SQL Runner.
If the query file is flagged as a template in the playbook, then the file is pre-processed as a template before being executed. See Templates for details
Templates
Templates are run through Golang's [text template processor] go-text-template. The template processor can access all variables defined in the playbook.
The following custom functions are also supported:
nowWithFormat [timeFormat]
- wheretimeFormat
is a valid Golang [time format] go-time-format
For an example query file using templating see: [integration-tests/postgres-sql/good/3.sql] example-query
Failure modes
If a statement fails in a query file, the query will terminate and report failure.
If a query fails, its sibling queries will continue running, but no further steps will run.
Failures in one target do not affect other targets in any way.
Copyright and license
SQL Runner is copyright 2015 Snowplow Analytics Ltd.
Licensed under the [Apache License, Version 2.0] license (the "License"); you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.