Automatic reloading of PostgreSQL credentials with Ecto

This repository is an example of how to automatically reload the credentials with Ecto of a PostgreSQL instance rotated automatically by Vault.

Note that Vault is not strictly required to rotate credentials, as long as an automatic process update these credentials. However, being the de-facto server to manage secrets, it's useful to see that it's easy to make Ecto and Vault work together.

Tested with

Elixir 1.12.3
Ecto 3.7.1
Vault 1.8.4
Postgres 14.1

How it works

Vault agent renews credentials automatically and renders them in a file;
We use secrets_watcher to detect changes to this file;
When a change is detected, we use disconnect_all/3 from db_connection to force connections to the database to disconnect (they will automatically reconnect after a backoff);
Upon restart, these connections will reconfigure themselves using a MFA given when configuring the repo.

⚠️ It requires db_connection >= 2.4.1, make sure your dependencies are up to date.

Steps

Launch server instances

In a dedicated terminal (will set the root token to root), run vault dev server:
```
vault server -dev -dev-root-token-id root
```
⚠️ in memory only, when the server is shutdown, everything is lost.
Have a postgres instance running.
- On macOS (default user/pass: postgres/postgres): sh brew install postgres brew services start postgresql
- With docker:
```
docker run --name postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:14.1
```

Create and configure database

Connect to instance

$ PGPASSWORD=postgres psql -U postgres -p 5432 -h 127.0.0.1

Create database
```
CREATE DATABASE my_database;
```

Forbid connection to database by default

REVOKE ALL ON DATABASE my_databse FROM public;

Create role that will be inherited by the accounts generated by Vault
```
CREATE ROLE base_role NOLOGIN;
```

Grant permission to connect to database

GRANT CONNECT ON DATABASE my_database TO base_role;

Grant permission to create schemas (not strictly necessary here):
```
GRANT CREATE ON DATABASE my_database TO base_role;
```

Configure Vault database engine

In another terminal, login to vault as root:
```
export VAULT_ADDR=http://127.0.0.1:8200
vault login root
```
👉 You can connect to the Vault GUI at http://127.0.0.1:8200/ui
Enable vault database secret engine:
```
vault secrets enable database
```

Configure database plugin for my_database database:

vault write database/config/my_database \
  plugin_name=postgresql-database-plugin \
  allowed_roles="vault_ecto" \
  connection_url="postgresql://{{username}}:{{password}}@localhost:5432/my_database?sslmode=disable" \
  username="postgres" \
  password="postgres"

ℹ️ username and password are the admin postgres instance credentials ℹ️ note the ?sslmode=disable to connect to the dev instance (which is obviously a bad idea in production!)

Create role vault_ecto:

vault write database/roles/vault_ecto \
  db_name=my_database \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE base_role;"\
  revocation_statements="REASSIGN OWNED BY \"{{name}}\" TO base_role; DROP USER \"{{name}}\";"\
  default_ttl="60" \
  max_ttl="120"

Create a policy to authorize reading vault_ecto database credentials:

vault policy write read_vault_ecto_creds ./vault/read_vault_ecto_creds_policy.hcl

Configure AppRole for vault agent

ℹ️ Vault agent provides many ways to authenticate to vault. However, using an approle is the fastest way for a local setup.

Enable approle backend:
```
vault auth enable approle
```

Create approle:

vault write auth/approle/role/vault-agent\
    secret_id_ttl=43200m\
    token_num_uses=9999\
    token_ttl=43200m\
    token_max_ttl=43200m\
    secret_id_num_uses=99999\
    policies=read_vault_ecto_creds\
    token_policies=read_vault_ecto_creds

Launch vault agent

Get agent's approle role-id:

vault read -format=json auth/approle/role/vault-agent/role-id | jq -r '.data.role_id' > ./vault/role_id

Get agent's approle secret-id:

vault write -format=json -f auth/approle/role/vault-agent/secret-id | jq -r '.data.secret_id' > ./vault/secret_id

Launch vault agent:

vault agent -config ./vault/vault_agent_config.hcl

Launch vault_ecto and initialize database

Launch vault_ecto:
```
iex -S mix
```

Create table:

iex> Ecto.Migrator.with_repo(VaultEcto.Repo, &Ecto.Migrator.run(&1, :up, all: true))

ℹ️ This code applies all migrations in priv/repo/migrations.

Seed table with some data:

iex> Code.eval_file("priv/repo/seed.exs")

Cheatsheet

`vault_ecto`

Select query:

VaultEcto.Person |> Ecto.Query.first() |> VaultEcto.Repo.one()

Insert query:

 %VaultEcto.Person{first_name: "foo", last_name: "bar", age: 42} |> VaultEcto.Repo.insert()

VaultEcto.insert()

Long transaction:

VaultEcto.Repo.transaction(
  fn ->
    %VaultEcto.Person{first_name: "foo", last_name: "bar", age: 42}
    |> VaultEcto.Repo.insert!()

    :timer.sleep(100_000)

    %VaultEcto.Person{first_name: "foo", last_name: "bar", age: 42}
    |> VaultEcto.Repo.insert!()
  end,
  timeout: :infinity)

VaultEcto.long_transaction(100_000)

👉 Running this long running transaction proves that even if the credentials are rotated during its execution, it will not be interrupted.

Vault

To get database credentials:
```
vault read database/creds/vault_ecto
```

TOKEN=$(vault write auth/approle/login role_id=@./vault/role_id secret_id=@./vault/secret_id -format=json | jq -r '.auth.client_token')
vault login ${TOKEN}

Postgres

Connect to database:

psql -U postgres -p 5432 -h 127.0.0.1 -d vault_ecto

List roles
```
\du
```
List tables
```
\dt
```

List permissions

select * from information_schema.role_table_grants where grantee='YOUR_USER';

ahamez / vault_ecto