clicumu / doepipeline

A python package for optimizing processing pipelines using statistical design of experiments (DoE).

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Prepending scripts with env-variable setting and setup-scripts

RickardSjogren opened this issue · comments

I was asked about the possibility to set environment variables when running remotely. This does not currently work at all since paramiko executes each command in an isolated session.

I suggest this is solved by letting environment variable setting and setup script execution be prepended at each execution command in a similar manner as directory change is now:

    if execution_dir:
        cd = [path for path in execution_dir
              if not 'cd {path}'.format(path=path) in command]
        prefix = '. ./.bash_profile; cd {path};'.format(path=posixpath.join(*cd)) 
    else:
        prefix = '. ./.bash_profile;'

    full_command = prefix + command

Reading of .bash_profile could also be then specified as a "setup-script". This would be a generalization of current functionality, since now the package assumes that BASH is the terminal used at the server.

BaseSSHExecutor can simply override BasePipelineExecutor._set_env_variables to build a prefix that can be fetched in execute_command.


Sidenote:
I also think that the remote-executor could override BasePipelineExecutor._cd in a similar manner to instead keep check of current location which can be fetched as a script-prefix. This would tidy up the now butt-ugly conditional checking in BaseSSHExecutor.execute_command.

Remote functionality removed, closes issue.