Houston is the API for the Astronomer Platform. It is primarily a GraphQL server built with Apollo Server 2 and Prisma. It also supports several RESTful endpoints using the same Express server that Apollo is using. These endpoints are primarily used for integrating with external authentication, Alertmanager, Docker Registry and other external dependencies. This is the second iteration of this API. This version was built to support the same external API, and is just an internal re-implementation.
Houston leverages Prisma as a GraphQL backend on top of Postgres. The Prisma service sits between Houston and Postgres and exposes a secure CRUD API on top of the datamodel defined in this project.
Houston can be configured via YAML files under ./config, and can be overridden via environment variables, which are defined in /config/custom-environment-variables.yaml. To add or override any variables locally, create a .env file in the project directory.
All source code is nested under src. This directory contains:
libcontains all shared, library code. This directory is listed as a root for the babel module loader. All modules defined under this directory can be imported directly, rather than specifying a relative path.resolverscontains all the Queries, Mutations and Types for the GraphQL API.routescontains all RESTful route definitions.index.jsis the entrypoint for the application.schema.graphlcontains the entire application schema.
In general, Queries and Mutations should be granular enough, such that they can be protected using the @auth directive. There may be cases where a user can list a group of resources, but maybe not read or update the details of that resource. For instance maybe a user can list deployments, but they can not get or update those deployments. In this case, we would want to support a deployments query, as well as a singular deployment query that returns a single instance of a Deployment. Each query could then require a different permission.
- Queries and Subscriptions should have the form
scopeObject, likeworkspaceUsersorworkspaceDeployments. Sometimes we'll have a query and subscription that are used together, like we do for the deployment logs. In these cases, the query and subscription should have the same name. - Mutations should have the form
scopeObjectVerb, likeworkspaceUserInviteorworkspaceDeploymentCreate.
Houston uses the concept of scopes as part of its RBAC system, which is tied to the application schema. There are currently three scopes - System, Workspace and Deployment. These are the bondaries that a RoleBinding will apply to. System is used to denote operations that apply at the Astronomer Platform layer and generally have global implications. The Workspace scope groups users and deployments together. In cloud mode, this is usually a company or organization. In enterprise mode, this is usually a team within a company or organization. The Deployment scope denotes operations that affect a single Deployment, which is always nested under a Workspace.
Houston supports multiple authentication methods. Currently it supports the built-in username/password method, as well as Google OAuth, as well as Auth0 OAuth. By default, the local authentication method is enabled, as well as a default Auth0 account. This can be overridden with a custom configuration to override the default values under the auth section.
Houston currently defines several default role values in it's datamodel here. These values are backed by a configurable permission mapping via a config file. The default permission mappings are defined here. These permissions are currently enforced using a GraphQL Directive.
Houston is written in ES6 and beyond, and it's currently built with Babel. The easiest way to run the project locally is to first run docker-compose up, or docker-compose up -d to detach and run the containers in the background, freeing up your terminal. This will start the postgres database, as well as the prisma service. From here you can run npm install, then npm start to start the project locally, connecting to the previously started data stack. This uses Nodemon and will restart when any source files change. The only exception is ./database/datamodel.graphql. Changes to this file require you to restart, triggering a prisma deploy and prisma generate. We could probably automate that process on change as well. Houston could also be added to the docker-compose services, but it's easier to interpret the logs when it's on its own. Houston will start the Playground by default, but it's also possible to start a playground that will expose the application API, as well as the prisma API. To start this on port 3000, just run graphql playground, or send it to the background with graphql playground &.
Houston is currently using Jest for running tests. Typically test files will live near the unit being tested and be named similarly with .unit.test.js as its extension. Jest can be ran as a one-off or can be run in watch mode. Both modes allow you to specify a regex path to limit what tests are running. npm run test will run all tests once and report back. npm run test -- src/resolvers/create-user --watch will run in watch mode, and only for the tests for the create-user resolver.
Sometimes it's useful to manually take over the role of commander, in order to test other components, like the airflow chart. Since commander requires the chart to live in a remote repository, it's difficult to quickly iterate on changes to the chart locally. One way to test this type of scenario is to disable commander and enable logging of the helm values instead. This can be done by setting configuration commander.enabled: false and deployments.logHelmValues: true. These can be overridden via environment variables.
Create a new deployment via the UI or CLI, then grab the values that Houston logs out. You can then use these values to pass to a helm install manually. Ex: helm install -f values.yaml -n <release-name> --namespace <namespace>.
- Fetch credentials for the running cluster (i.e.
gcloud container clusters get-credentials ${CLUSTER_NAME} --zone ${ZONE} --project ${PROJECT}) - Switch to namespace (i.e.
kubens ${NAMESPACE}), You may need tobrew install kubectx. - Port forward to a production Prometheus (i.e.
kubectl port-forward svc/${SERVICE_NAME} 9090:9090) - Hard code to read from a production Airflow deployment's metrics stream, uncomment/change 25 of
/src/resolvers/subscription/metrics/index.js
npm startstarts the develpment server. Restarts automatically with nodemon.npm run buildbuilds with babel and place output into./dist.npm run servestarts the production server from built files under./dist.npm run testruns tests usingjest.npm run test -- --watchruns tests in watch mode.npm run test -- src/resolvers/create-user --watchruns a subset of tests in watch mode.npm run coverageruns tests and outputs a coverage report to./coverage.npm run lintruns eslint.npm run playgroundstarts a single playground for application as well as prisma CRUD.docker-compose upstarts a postgres container and the prisma service container.