Env variables: how to?
ndrean opened this issue Β· comments
Desperate to understand how to load and use them. Seems like a crazy problem..... What should be put in "runtime" and in "config" and in "prod"?
@ndrean great question as ever.
Anything you don't mind "leaking" i.e. being read by OpenAI
(which you better believe has access to all private
GitHub repos because of their MSFT
deal...) can be in /config/prod.exs
. The sensitive things like API/AWS keys should be an environment variable.
A second rule of thumb is: a variable that you don't want to require a code update for should be an environment variable too. e.g. a feature flag that you want to toggle in the environment and then just restart the app without re-deploying.
e.g. the Debug Level
if you need to debug something quickly without re-deploying an App you can simply toggle the DEBUG="info"
environment variable, power-cycle the App and then all the debugging will be enabled. Then once you're done debugging set it back to DEBUG="error"
and cycle the app again.
At least this is how we've done it in the past with bigger teams in companies with well-defined change-control processes.
i.e. All PRs even "debugging" ones have to be approved by 2 people and QA-tested before they can go to production
so having the ability to toggle debugging as env var is super useful.
Nice, thks! Yes, I pass most as env variables.
Nice tip for DEBUG
Second step is passing them in github secrets, later.
But my problem is more simple. I can't pass the env vars to fly.io, so I looked in 2 repo:
They put the github credentials in "runtime.exs", populate with System.fetch_env!
and calls with Application.fetch_env!
:
# runtime.exs
config :live_beats, :github,
client_id: System.fetch_env!("LIVE_BEATS_GITHUB_CLIENT_ID"),
client_secret: System.fetch_env!("LIVE_BEATS_GITHUB_CLIENT_SECRET")
When they want to use them, they do:
# github.ex
defp client_id, do: LiveBeats.config([:github, :client_id])
defp secret, do: LiveBeats.config([:github, :client_secret])
where LiveBeats.config
is basically does for example:
Application.fetch_env!(:live_beats, :github) |> Keyword.fetch(:client_id)
- I looked into https://github.com/dwyl/imgup/blob/main/config/runtime.exs, same thing:
AWS config is in "runtime.exs" but with a System.get_env
this time, and the Env var is called via a Application.get_env
.
So, simple? Humm... here comes the famous this does not work for me π yes yes!!. I don't know why, of course.
Yes, indeed https://github.com/dwyl/imgup/blob/a7d18ce6a4f0d3ceb512a5cdc494b2d7b3683050/config/runtime.exs#L67-L73 is a good example of how we use environment variables for AWS
keys which we definitely don't want to leak anywhere.
What part is not working for you? π
Are you sure the environment variables are available?
We have a simple checker in our MVP to confirm the variables are available: https://mvp.fly.dev/init
Ref: https://github.com/dwyl/mvp/blob/main/lib/app_web/controllers/init_controller.ex
Yes, I did something similar, not that nice, but just print them on the landing page. Nada.
Ah yes, I have another test: I need an env var to set up a module. If I simply use System.get_env
, fly.io won't even compile. I only manage to compile if I hard code the env var.... means nothing passes, although they are set in fly.io.
I can put this in "prod.exs" or "runtime.exs", it is not loaded:
config :ex_aws,
access_key_id: System.get_env("AWS_ACCESS_KEY_ID"),
secret_access_key: System.get_env("AWS_SECRET_ACCESS_KEY"),
region: System.get_env("AWS_REGION"),
bucket: System.get_env("AWS_S3_BUCKET"),
request_config_override: %{}
config :up_img, :github,
github_client_id: System.get_env("GITHUB_CLIENT_ID"),
github_client_secret: System.get_env("GITHUB_CLIENT_SECRET")
config :up_img, :google,
google_client_id: System.get_env("GOOGLE_CLIENT_ID"),
google_client_secret: System.get_env("GOOGLE_CLIENT_SECRET")
config :up_img, :vault_key, System.get_env("CLOAK_KEY")
The "reader" module does nothing more (or less) than previously shown:
# reader.ex
def fetch_key(main, key),
do:
Application.fetch_env!(:up_img, main) |> Keyword.get(key)
def gh_id, do: fetch_key(:github, :github_client_id)
def gh_secret, do: fetch_key(:github, :github_client_secret)
def google_id, do: fetch_key(:google, :google_client_id)
def google_secret, do: fetch_key(:google, :google_client_secret)
def vault_key, do: Application.get_env(:up_img, :vault_key)
def bucket, do: Application.get_env(:ex_aws, :bucket)
And I checked:
> printenv GITHUB_CLIENT_ID
1dd13991a3....
> fly secrets set GITHUB_CLIENT_ID=1dd13991a3.....
but nothing is there when I print the following in the landing page (in fly):
<p><%= UpImg.gh_id()%></p>
<p><%= UpImg.gh_secret()%></p>
<p><%= UpImg.google_id()%></p>
<p><%= UpImg.google_secret()%></p>
<p><%= UpImg.bucket()%></p>
<p><%= UpImg.vault_key()%></p>
Not convinced? I ssh
into the app:
This is very strange. And feels like a support topic for the Fly/Elixir forum. π
Guaranteed your code is not "rubbish". π
Did you use fly launch
when setting up your app? π
Without access to your fly.toml
or Dockerfile
can't see if it's loading the env
correctly. π€·ββοΈ
My advice would be to re-run fly launch
and re-create the deployment files.
Yes I used fly launch
, threw everything 2 times already. The Dockerfile is the standard one generated by fly (I only added inotify-tools
<=> chokidar
but I will remove this).
Well, I decided to run a DockerCompose since this is what fly does, and good news (or bad depends π), it breaks too! digging...
I ran it under MIX_ENV=dev and moved the confi from "dev.exs" to "runtime.exs", and bad (?) news, it works now on docker: the env vars are loaded and read. So they need to be located in "runtime.exs". ok. So a release accepts to run a config set in "dev.exs" but an image (which passes through a release stage) needs it in "runtime.exs", both with MIX_ENV=dev....seriously?!
I didn't change a single line to the code, config in "runtime.exs", as already shown .... and now it is deployed π₯΅, at least the env vars are loaded.βοΈ. It even compiles with the env var that sets fields for the email encryption/hasing. I don't understand: 6 tries....π½π».
One step further, even the db is working, the one-tap that reaches Google public keys, the Cloak encryption
the preview with I/O on the server,
and even the upload to S3! It works! halleluja! π
Fun part: I used :httpc
not to be depend of the HTTP client library used and thought it was part of Erlang, but:
Request: POST /google/callback
2023-09-19T17:34:22Z app[9185369a17d538] cdg [info]** (exit) an exception was raised:
2023-09-19T17:34:22Z app[9185369a17d538] cdg [info]
** (UndefinedFunctionError) function :httpc.request/4 is undefined (module :httpc is not available)
2023-09-19T17:34:22Z app[9185369a17d538] cdg [info]
:httpc.request(:get, {~c"https://www.googleapis.com/oauth2/v1/certs", []}, [], [])
2023-09-19T17:34:22Z app[9185369a17d538] cdg [info]
(up_img 0.1.0) lib/libraries/google_certs.ex:67: ElixirGoogleCerts.fetch/1
A good example (I think) of using "compiled" config is when you use a mock: with Application.compile_env
as a module attribute. Declare it in "text.exs" and "dev/prod.exs" and it will change upon the context. Nice in fact.
You use this in ElixirGithubAuth.
I was wondering if reading the env variable could be slow ?? Is it worth loading them all in an ETS table when the app starts and retrieve them from ETS rather than reading? I implemented a Task to do so, it centralises the calls to get these env vars - could be from ETS or reading.
For what its worth, I have the following results which will make we adopt the ETS version, which is only a few liens ofr code more:
start = System.monotonic_time()
Application.fetch_env!(:up_img, :cleaning_timer)
Logger.info(%{dur: System.monotonic_time() - start})
4875 / 4167 / 4625
start = System.monotonic_time()
:ets.lookup(:env_vars, :cleaning_timer)
Logger.info(%{dur: System.monotonic_time() - start})
3625 / 3762 / 2708
When I run this on Fly.io, I get more clear results (even if it is microseconds):
- from_ETS : 1600-2000
- Application.fetch_env: 6000-7000
The module is just a supervied Task:
defmodule UpImg.EnvReader do
@moduledoc """
Task to load in an ETS table the env variables started in the Application module.
"""
use Task
def start_link(arg) do
:envs = :ets.new(:envs, [:set, :public, :named_table])
Task.start_link(__MODULE__, :run, [arg])
end
def run(_arg) do
# setters in ETS
:ets.insert(:envs, {:google_id, read_google_id()})
:ets.insert(:envs, {:gh_id, read_gh_id()})
:ets.insert(:envs, {:gh_secret, read_gh_secret()})
:ets.insert(:envs, {:google_secret, read_google_secret()})
:ets.insert(:envs, {:bucket, read_bucket()})
:ets.insert(:envs, {:cleaning_timer, read_cleaning_timer()})
end
defp fetch_key!(main, key),
do:
Application.get_application(__MODULE__)
|> Application.fetch_env!(main)
|> Keyword.get(key)
defp lookup(key), do: :ets.lookup(:envs, key) |> Keyword.get(key)
# readers from "runtime.exs"
defp read_gh_id, do: fetch_key!(:github, :github_client_id)
defp read_gh_secret, do: fetch_key!(:github, :github_client_secret)
defp read_google_id, do: fetch_key!(:google, :google_client_id)
defp read_google_secret, do: fetch_key!(:google, :google_client_secret)
defp read_vault_key, do: Application.fetch_env!(:up_img, :vault_key)
defp read_bucket, do: Application.fetch_env!(:ex_aws, :bucket)
defp read_cleaning_timer, do: Application.fetch_env!(:up_img, :cleaning_timer)
# public lookups in ETS
def bucket, do: lookup(:bucket)
def google_id, do: lookup(:google_id)
def google_secret, do: lookup(:google_secret)
def gh_id, do: lookup(:gh_id)
def gh_secret, do: lookup(:gh_secret)
def cleaning_timer, do: lookup(:cleaning_timer)
end