Docker step not working
GuillaumeDesforges opened this issue · comments
Describe the bug
Making a SimpleFlow that runs docker, but it fails:
[2020-05-19 10:16:20][FFExecutorD.executeLoop][Info][poseideep][PID 21434][ThreadId 22][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:183:9] Initialising connection to coordinator.
[2020-05-19 10:16:21][FFExecutorD.executeLoop.process][Info][poseideep][PID 21434][ThreadId 22][processId:CreateProcess {cmdspec = RawCommand "docker" ["run","--user=1000","--workdir=/output","--volume=/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/:/output","ubuntu","pwd"], cwd = Just "/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/", env = Nothing, std_in = Inherit, std_out = UseHandle {handle: /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout}, std_err = UseHandle {handle: /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stderr}, close_fds = True, create_group = False, delegate_ctlc = False, detach_console = False, create_new_console = False, new_session = False, child_group = Nothing, child_user = Nothing, use_process_jobs = False}][task:52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:127:9] Executing
[2020-05-19 10:16:21][FFExecutorD.executeLoop][Error][poseideep][PID 21434][ThreadId 22][task:52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:214:19] Executor failed: docker: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
/!\ Something went wrong:ExternalTaskFailed (TaskDescription {_tdOutput = ContentHash "52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff", _tdTask = ExternalTask {_etCommand = "docker", _etParams = [Param [ParamText "run"],Param [ParamText "--user=",ParamUid],Param [ParamText "--workdir=/output"],Param [ParamText "--volume=",ParamOut,ParamText ":/output"],Param [ParamText "ubuntu"],Param [ParamText "pwd"]], _etEnv = EnvInherit, _etWriteToStdOut = NoOutputCapture}}) (KnownTask (Failed (ExecutionInfo {_eiExecutor = Executor "poseideep", _eiElapsed = TimeSpec {sec = 0, nsec = 0}}) 2)) (Just "/tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout") (Just "/tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stderr")
To Reproduce
git clone https://github.com/GuillaumeDesforges/funflow-sandbox.git
git checkout example-docker
nix-shell
stack run
Expected behavior
Should output
/output
in /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout
Environment
- OS name + version: NixOS 20.03
- Version of the code: master 7d8f088
Additional context
I can run the docker command fine in terminal
$ docker run --user=1000 --workdir=/output --volume=/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/:/output ubuntu pwd
/output
I have just reproduced this error with:
=== Running examples of flows ===
# Running the example 'a flow running a task in Docker'
[2020-05-19 14:27:22][FFExecutorD.executeLoop][Info][matto-lap][PID 16912][ThreadId 14][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:183:9] Initialising connection to coordinator.
[2020-05-19 14:27:23][FFExecutorD.executeLoop.process][Info][matto-lap][PID 16912][ThreadId 14][processId:CreateProcess {cmdspec = RawCommand "docker" ["run","--user=1000","--workdir=/output","--volume=/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/:/output","ubuntu","pwd"], cwd = Just "/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/", env = Nothing, std_in = Inherit, std_out = UseHandle {handle: /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout}, std_err = UseHandle {handle: /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stderr}, close_fds = True, create_group = False, delegate_ctlc = False, detach_console = False, create_new_console = False, new_session = False, child_group = Nothing, child_user = Nothing, use_process_jobs = False}][task:52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:127:9] Executing
[2020-05-19 14:27:23][FFExecutorD.executeLoop][Error][matto-lap][PID 16912][ThreadId 14][task:52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:214:19] Executor failed: docker: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
/!\ Something went wrong:ExternalTaskFailed (TaskDescription {_tdOutput = ContentHash "52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff", _tdTask = ExternalTask {_etCommand = "docker", _etParams = [Param [ParamText "run"],Param [ParamText "--user=",ParamUid],Param [ParamText "--workdir=/output"],Param [ParamText "--volume=",ParamOut,ParamText ":/output"],Param [ParamText "ubuntu"],Param [ParamText "pwd"]], _etEnv = EnvInherit, _etWriteToStdOut = NoOutputCapture}}) (KnownTask (Failed (ExecutionInfo {_eiExecutor = Executor "matto-lap", _eiElapsed = TimeSpec {sec = 0, nsec = 0}}) 2)) (Just "/tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout") (Just "/tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stderr")
I used my own stack
with the above repo and ran stack install && stack run
@GuillaumeDesforges @YPares , can we add this example to the test suite of funflow?
It seems that @Divesh-Otwani also had a similar issue a while ago:
here is the solution: we need to add docker
to the stack.yaml
because stack uses a pure nix shell.
actually it seems that any dependent package just needs to add this to its stack.yaml:
nix:
packages:
- docker
I don't fully understand how stack handles this.
actually it seems that any dependent package just needs to add this to its stack.yaml:
nix: packages: - docker
I don't fully understand how stack handles this.
What if the user does not use nix ? Does it work without it ?
Anyway, this should be documented.
In order for Stack to work with Nix, you need either to have nix: enable: true
, or to pass --nix
to the executable, as described here: https://docs.haskellstack.org/en/stable/nix_integration/
In the stack.yaml
, enable
is omitted, which means that Nix support should be off. Maybe Matthias was enabling it by hand? Maybe the solution is simpler: do you have Docker installed in your machine? Maybe a solution is just to add it here: https://github.com/GuillaumeDesforges/funflow-sandbox/blob/master/shell.nix
In order for Stack to work with Nix, you need either to have
nix: enable: true
, or to pass--nix
to the executable, as described here: https://docs.haskellstack.org/en/stable/nix_integration/
The Stack-Nix integration is automatically enabled on NixOS: commercialhaskell/stack#3938
This would explain why that worked for Matthias. Nice to know.
So, maybe running with
nix-shell -p stack ghc --command "stack build && stack --no-nix --system-ghc exec external-c-computation"
or with
nix-shell -p stack ghc docker zlib --command "stack build && stack --no-nix --system-ghc exec external-c-computation"
if docker
and zlib
are not available system wide should suffice?
In that case we only need to add some entry to the README.rst
.
I could make it work with the following in package.yml
:
nix:
enable: true
pure: false
$ cat /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout
/output
It is important to be sure that it works without Nix too. @GuillaumeDesforges, since you are a non-NixOS user, can you check:
- Whether you have Docker installed in your machine.
- If the error persists, that Docker is accessible to Funflow.
Otherwise, this can be a real problem, and some comments should be added to the README.
I am a NixOS user
Ah, too bad!