Deploy, Manage and Query Cloud Infrastructure using SQL

[Documentation] [Developer Guide] [BYO Providers]

Cloud infrastructure coding using SQL

StackQL allows you to create, modify and query the state of services and resources across all three major public cloud providers (Google, AWS and Azure) using a common, widely known DSL...SQL.

Its as easy as...

SELECT * FROM google.compute.instances WHERE zone = 'australia-southeast1-b' AND project = 'my-project' ;

select d1.name, d1.id, d2.name as d2_name, d2.status, d2.label, d2.id as d2_id from google.compute.disks d1 inner join okta.application.apps d2 on d1.name = d2.label where d1.project = 'lab-kr-network-01' and d1.zone = 'australia-southeast1-a' and d2.subdomain = 'dev-79923018-admin';

Provider development

Keen to expose some new functionality though stackql? We are very keen on this!

Please see registry_contribution.md.

Design

HLDD

Providers

Please see the stackql-provider-registry repository

Providers include:

Google.
Okta.
...

Build

Native Build

In shell

env CGO_ENABLED=1 go build \
  --tags "json1 sqleanall" \
  -ldflags "-X github.com/stackql/stackql/internal/stackql/cmd.BuildMajorVersion=${BUILDMAJORVERSION:-1} \
  -X github.com/stackql/stackql/internal/stackql/cmd.BuildMinorVersion=${BUILDMINORVERSION:-1} \
  -X github.com/stackql/stackql/internal/stackql/cmd.BuildPatchVersion=${BUILDPATCHVERSION:-1} \
  -X github.com/stackql/stackql/internal/stackql/cmd.BuildCommitSHA=$BUILDCOMMITSHA \
  -X github.com/stackql/stackql/internal/stackql/cmd.BuildShortCommitSHA=$BUILDSHORTCOMMITSHA \
  -X \"github.com/stackql/stackql/internal/stackql/cmd.BuildDate=$BUILDDATE\" \
  -X \"stackql/internal/stackql/planbuilder.PlanCacheEnabled=$PLANCACHEENABLED\" \
  -X github.com/stackql/stackql/internal/stackql/cmd.BuildPlatform=$BUILDPLATFORM" -o ./build ./stackql

System requirements

These are the system requirements for local development, build and test

golang>=1.18
openssl>=1.1.1
python>=3.10
- python packages as per the requirements file
docker

Docker Build

docker build -t stackql:${STACKQL_TAG} -t stackql:latest .

Run

Native Run

Help message

./build/stackql --help

Shell

# Amend STACKQL_AUTH as required, angle bracketed strings must be replaced.
export STACKQL_AUTH='{ "google": { "credentialsfilepath": "</path/to/google/sa-key.json>", "type": "service_account" }, "okta": { "credentialsenvvar": "<OKTA_SECRET_KEY>", "type": "api_key" }, "github": { "type": "basic", "credentialsenvvar": "<GITHUB_CREDS>" }, "aws": { "type": "aws_signing_v4", "credentialsfilepath": "</path/to/aws/secret-key.txt>", "keyID": "<YOUR_AWS_KEY_NOT_A_SECRET>" }, "k8s": { "credentialsenvvar": "<K8S_TOKEN>", "type": "api_key", "valuePrefix": "Bearer " } }'

./build/stackql --auth="${STACKQL_AUTH}" shell

Docker Run

NOTE: on some docker versions, the argument --security-opt seccomp=unconfined is required as a hack for a known issue in docker.

Docker single query

docker compose run --rm stackqlsrv "bash" "-c" "stackql exec 'show providers;'"

Docker interactive shell

export AWS_KEY_ID='<YOUR_AWS_KEY_ID_NOT_A_SECRET>'

export DOCKER_AUTH_STR='{ "google": { "credentialsfilepath": "/opt/stackql/keys/sa-key.json", "type": "service_account" }, "okta": { "credentialsenvvar": "OKTA_SECRET_KEY", "type": "api_key" }, "github": { "type": "basic", "credentialsenvvar": "GITHUB_CREDS" }, "aws": { "type": "aws_signing_v4", "credentialsfilepath": "/opt/stackql/keys/integration/aws-secret-key.txt", "keyID": "'${AWS_KEY_ID}'" }, "k8s": { "credentialsenvvar": "K8S_TOKEN", "type": "api_key", "valuePrefix": "Bearer " } }'

export DOCKER_REG_CFG='{ "url": "https://registry.stackql.app/providers" }'

docker compose -p shellrun run --rm -e OKTA_SECRET_KEY=some-dummy-api-key -e GITHUB_SECRET_KEY=some-dummy-github-key -e K8S_SECRET_KEY=some-k8s-token -e REGISTRY_SRC=test/registry-mocked stackqlsrv bash -c "stackql shell --registry='${DOCKER_REG_CFG}' --auth='${DOCKER_AUTH_STR}'"

Docker PG Server

mTLS Server Stock as a rock

From the root directory of this repository...

docker compose -f docker-compose-credentials.yml run --rm credentialsgen 

docker compose up stackqlsrv

Then...

psql -d "host=127.0.0.1 port=5576 user=myuser sslmode=verify-full sslcert=./vol/srv/credentials/pg_client_cert.pem sslkey=./vol/srv/credentials/pg_client_key.pem sslrootcert=./vol/srv/credentials/pg_server_cert.pem dbname=mydatabase"

When finished, clean up with:

docker compose down

Examples

./stackql exec "show extended services from google where title = 'Service Directory API';"

More examples in examples/examples.md.

Developers

Testing

Server mode

Please see the server mode section of the developer docs.

Alpha Features

GC, cacheing and concurrent users

Acknowledgements

Forks of the following support our work:

We gratefully acknowledge these pieces of work.

Licensing

Please see the stackql LICENSE.

Licenses for third party software we are using are included in the /licenses directory.

kieranrimmer / stackql