Docker step not working

Question

Docker step not working

GuillaumeDesforges opened this issue 4 years ago · comments

Guillaume Desforges commented 4 years ago

Describe the bug
Making a SimpleFlow that runs docker, but it fails:

[2020-05-19 10:16:20][FFExecutorD.executeLoop][Info][poseideep][PID 21434][ThreadId 22][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:183:9] Initialising connection to coordinator.
[2020-05-19 10:16:21][FFExecutorD.executeLoop.process][Info][poseideep][PID 21434][ThreadId 22][processId:CreateProcess {cmdspec = RawCommand "docker" ["run","--user=1000","--workdir=/output","--volume=/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/:/output","ubuntu","pwd"], cwd = Just "/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/", env = Nothing, std_in = Inherit, std_out = UseHandle {handle: /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout}, std_err = UseHandle {handle: /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stderr}, close_fds = True, create_group = False, delegate_ctlc = False, detach_console = False, create_new_console = False, new_session = False, child_group = Nothing, child_user = Nothing, use_process_jobs = False}][task:52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:127:9] Executing
[2020-05-19 10:16:21][FFExecutorD.executeLoop][Error][poseideep][PID 21434][ThreadId 22][task:52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:214:19] Executor failed: docker: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
/!\ Something went wrong:ExternalTaskFailed (TaskDescription {_tdOutput = ContentHash "52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff", _tdTask = ExternalTask {_etCommand = "docker", _etParams = [Param [ParamText "run"],Param [ParamText "--user=",ParamUid],Param [ParamText "--workdir=/output"],Param [ParamText "--volume=",ParamOut,ParamText ":/output"],Param [ParamText "ubuntu"],Param [ParamText "pwd"]], _etEnv = EnvInherit, _etWriteToStdOut = NoOutputCapture}}) (KnownTask (Failed (ExecutionInfo {_eiExecutor = Executor "poseideep", _eiElapsed = TimeSpec {sec = 0, nsec = 0}}) 2)) (Just "/tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout") (Just "/tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stderr")

To Reproduce

git clone https://github.com/GuillaumeDesforges/funflow-sandbox.git
git checkout example-docker

nix-shell

stack run

Expected behavior
Should output

/output

in /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout

Environment

OS name + version: NixOS 20.03
Version of the code: master 7d8f088

Additional context
I can run the docker command fine in terminal

$ docker run --user=1000 --workdir=/output --volume=/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/:/output ubuntu pwd
/output

Matthias Meschede · Answer 1 · Tue May 19 2020 22:29:17 GMT+0800 (China Standard Time)

I have just reproduced this error with:

=== Running examples of flows ===

# Running the example 'a flow running a task in Docker'
[2020-05-19 14:27:22][FFExecutorD.executeLoop][Info][matto-lap][PID 16912][ThreadId 14][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:183:9] Initialising connection to coordinator.
[2020-05-19 14:27:23][FFExecutorD.executeLoop.process][Info][matto-lap][PID 16912][ThreadId 14][processId:CreateProcess {cmdspec = RawCommand "docker" ["run","--user=1000","--workdir=/output","--volume=/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/:/output","ubuntu","pwd"], cwd = Just "/tmp/funflow/pending-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/", env = Nothing, std_in = Inherit, std_out = UseHandle {handle: /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout}, std_err = UseHandle {handle: /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stderr}, close_fds = True, create_group = False, delegate_ctlc = False, detach_console = False, create_new_console = False, new_session = False, child_group = Nothing, child_user = Nothing, use_process_jobs = False}][task:52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:127:9] Executing
[2020-05-19 14:27:23][FFExecutorD.executeLoop][Error][matto-lap][PID 16912][ThreadId 14][task:52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff][funflow-1.6.0-Ls51xdfLQhPYK80R5oo4y:Control.Funflow.External.Executor src/Control/Funflow/External/Executor.hs:214:19] Executor failed: docker: createProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
/!\ Something went wrong:ExternalTaskFailed (TaskDescription {_tdOutput = ContentHash "52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff", _tdTask = ExternalTask {_etCommand = "docker", _etParams = [Param [ParamText "run"],Param [ParamText "--user=",ParamUid],Param [ParamText "--workdir=/output"],Param [ParamText "--volume=",ParamOut,ParamText ":/output"],Param [ParamText "ubuntu"],Param [ParamText "pwd"]], _etEnv = EnvInherit, _etWriteToStdOut = NoOutputCapture}}) (KnownTask (Failed (ExecutionInfo {_eiExecutor = Executor "matto-lap", _eiElapsed = TimeSpec {sec = 0, nsec = 0}}) 2)) (Just "/tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout") (Just "/tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stderr")

I used my own stack with the above repo and ran stack install && stack run

Matthias Meschede · Answer 2 · Tue May 19 2020 22:30:27 GMT+0800 (China Standard Time)

@GuillaumeDesforges @YPares , can we add this example to the test suite of funflow?

Matthias Meschede · Answer 3 · Tue May 19 2020 22:50:11 GMT+0800 (China Standard Time)

It seems that @Divesh-Otwani also had a similar issue a while ago:

https://github.com/tweag/funflow/blob/d1d5b8f4cae370fe178d82cf0ec45005604d5a78/funflow-cwl/test/guide/output

Matthias Meschede · Answer 4 · Wed May 20 2020 01:37:44 GMT+0800 (China Standard Time)

here is the solution: we need to add docker to the stack.yaml because stack uses a pure nix shell.

Matthias Meschede · Answer 5 · Wed May 20 2020 02:21:34 GMT+0800 (China Standard Time)

actually it seems that any dependent package just needs to add this to its stack.yaml:

nix:
    packages:
        - docker

I don't fully understand how stack handles this.

Guillaume Desforges · Answer 6 · Wed May 20 2020 04:41:12 GMT+0800 (China Standard Time)

actually it seems that any dependent package just needs to add this to its stack.yaml:
nix:
    packages:
        - docker
I don't fully understand how stack handles this.

What if the user does not use nix ? Does it work without it ?

Anyway, this should be documented.

Juan Raphael Diaz Simões · Answer 7 · Wed May 20 2020 11:20:23 GMT+0800 (China Standard Time)

In order for Stack to work with Nix, you need either to have nix: enable: true, or to pass --nix to the executable, as described here: https://docs.haskellstack.org/en/stable/nix_integration/

In the stack.yaml, enable is omitted, which means that Nix support should be off. Maybe Matthias was enabling it by hand? Maybe the solution is simpler: do you have Docker installed in your machine? Maybe a solution is just to add it here: https://github.com/GuillaumeDesforges/funflow-sandbox/blob/master/shell.nix

Leonhard Markert · Answer 8 · Wed May 20 2020 14:11:23 GMT+0800 (China Standard Time)

In order for Stack to work with Nix, you need either to have nix: enable: true, or to pass --nix to the executable, as described here: https://docs.haskellstack.org/en/stable/nix_integration/

The Stack-Nix integration is automatically enabled on NixOS: commercialhaskell/stack#3938

Juan Raphael Diaz Simões · Answer 9 · Wed May 20 2020 14:25:21 GMT+0800 (China Standard Time)

This would explain why that worked for Matthias. Nice to know.

Matthias Meschede · Answer 10 · Wed May 20 2020 15:43:54 GMT+0800 (China Standard Time)

So, maybe running with

nix-shell -p stack ghc --command "stack build && stack --no-nix --system-ghc exec external-c-computation"

or with

nix-shell -p stack ghc docker zlib --command "stack build && stack --no-nix --system-ghc exec external-c-computation"

if docker and zlib are not available system wide should suffice?

In that case we only need to add some entry to the README.rst.

Guillaume Desforges · Answer 11 · Wed May 20 2020 16:45:44 GMT+0800 (China Standard Time)

I could make it work with the following in package.yml:

nix:
  enable: true
  pure: false

$ cat /tmp/funflow/metadata/hash-52f84e4e97ac7cfaeed95b2b83af28f1d2d582b142361febe436b43599bcb3ff/stdout 
/output

Juan Raphael Diaz Simões · Answer 12 · Wed May 20 2020 16:56:12 GMT+0800 (China Standard Time)

It is important to be sure that it works without Nix too. @GuillaumeDesforges, since you are a non-NixOS user, can you check:

Whether you have Docker installed in your machine.
If the error persists, that Docker is accessible to Funflow.

Otherwise, this can be a real problem, and some comments should be added to the README.

Guillaume Desforges · Answer 13 · Wed May 20 2020 17:13:07 GMT+0800 (China Standard Time)

I am a NixOS user

Juan Raphael Diaz Simões · Answer 14 · Wed May 20 2020 17:47:50 GMT+0800 (China Standard Time)

Ah, too bad!