Non-configurable req_keepvars_extra
NPotvin opened this issue · comments
Bug description
While configuring a JupyterHub server on a cluster, I encountered several issues related to environment variables not being properly exported on the compute nodes by the batchspawner.
Expected behaviour
In the default batch script, a line is responsible for exporting the variables (e.g. #SBATCH --export={{keepvars}}
for slurm). The keepvars
variable is basically a concatenation of two coma separated lists of variables req_keepvars
and req_keepvars_extra
.
The former is not configurable and already contains some required variables.
The latter should be configurable and is expected to provide the user with a convenient way to pass additional environment variables to the batch script.
Actual behaviour
The attribute BatchSpawnerBase.req_keepvars_extra
does not seem to be used when producing the submitted batch script. The extra variables are not present in the list of exported environment variables.
Workaround
Overriding the default batch script in the config by hardcoding the variables to export does the trick but I am not comfortable with that approach. In the future, the idea would be to load the shell script from a file that should be agnostic of those variables.
(In other words, I want to define a list of additional environment variables in the config, and keep {keepvars}
at the top of a .sh file read in the jupyterhub config)
Proposed fix
I managed to get the expected behaviour by changing this segment of code:
#batchspawner.py: line 169
req_keepvars_extra = Unicode(
help="Extra environment variables which should be configured, "
"added to the defaults in keepvars, "
"comma separated list.",
)
and adding .tag(config=True)
:
req_keepvars_extra = Unicode(
help="Extra environment variables which should be configured, "
"added to the defaults in keepvars, "
"comma separated list.",
).tag(config=True)
How to reproduce
- Install a jupyterhub server on a cluster
- Configure it to use a batch spawner that inherits from BatchSpawnerBase
- Try to export additional environment variables using
c.BatchSpawnerBase.req_keepvars_extra = "YOUR,VARS,HERE"
- Start the server (in my case using systemctl) and see it fail when attempting to spawn a new batch job (checking the produced script reveals that the environment variables do not contain the extra variables)
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋
Welcome to the Jupyter community! 🎉
It seems I can't tag the PR in development section, but here it is: #295