Multiple bash commands
rosskevin opened this issue · comments
I'm having problems issuing multiple commands:
command: bundle install --jobs 4 --retry 3 && bundle exec spring binstub --all && bin/rake log:clear && bundle exec docker-rails-db-check && bin/rake db:rebuild_testUnknown switches '--all'
I'm sure it's simple but I'm pulling my hair out. What's the best way to do this (I don't want it in an external script).
If I remove the second statement, I get ERROR: "bundle install" was called with arguments ["&&", "bin/rake", "log:clear", "&&", "bundle", "exec", "docker-rails-db-check", "&&", "bin/rake", "db:rebuild_test"], so this seems to be a simple bash usage failure on my part.
Running this via cmd line works, so perhaps the command needs prefixed with something?
Found it! finally. http://stackoverflow.com/a/30064175/2363935
bash -c "<long string of commands>"
If you're running that many commands, wouldn't it be better to use a Dockerfile?
@thaJeztah what makes the Dockerfile a better choice? I plan to use the same Dockerfile, but run some slightly different commands and configurations per environment i.e. development | test | parallel_tests | staging | production, in this case, it is easier for me to use a single Dockerfile but provide manipulation above yaml and generate/use a docker-file.yml based on the desired env. See alienfast/docker-rails#4
I'm sorry, but I found a way for the multiline :)
https://github.com/indiehosters/piwik/blob/master/docker-compose.yml#L29-L36
Feedback is welcomed :)
Dockerfile just is not suitable for some commands. Let's say we want to run database migrations before app is started. It can't be done from Dockerfile because database server is not launched at the moment and event it's hostname (from docker-compose.yml) can't be resolved.
Using bash -c is a workaround but it's not really nice and may require extra dependencies for minimalistic images without bash.
Dockerfile just is not suitable for some commands. Let's say we want to run database migrations before app is started.
Correct; the Dockerfile is meant to build the image (e.g. install the software that needs to be in the image); doing so allows you to test the actual image you're going to run before deploying it. Installing the software at runtime could make it a hit and miss; for example, packages you're installing may have been updated since you last tried, and there's no way to verify what went into the image you are running.
For actions that are needed at runtime , consider using an entrypoint script (see here for an example) that checks if migration is needed, performs the migration, and after migration has completed, switches to the container's main process, using exec. The entrypoint script can be parameterized using (e.g.) environment variables, which would allow you to pass certain settings.
@thaJeztah Thank you for useful hints. I'm actually not an expert in Docker and I like that most things works as expected. But this particular case seems very confusing to me. It would be good to have an option to specify list for commands: in docker-compose.yml.
Special script for entry point can solve problems, but it also can be the source of problems by itself :) More code = more chances for errors.
And with bash -c we can write like that:
command: >
bash -c "./manage.py migrate &&
./manage.py collectstatic --noinput &&
./manage.py runserver 0.0.0.0:8000"Yes we could add an extra script which chains these commands but its already a bunch of devops scripts in modern projects :) I think that explicit call of standard script. i.e. manage.py for Django, is cleaner.
And we don't always want to call all commands, for example there could be:
command: ./manage.py runworkersMore code = more chances for errors.
Putting the code in the docker-compose file doesn't change that of course; keeping it in the image at least makes sure that the code works well together with the image (and can be verified in your CI/CD).
Having said that, I fully understand that sometimes a longer "command" may be needed; docker-compose is a bit tied to what the yaml format supports here.
It would be good to have an option to specify list for
commands:indocker-compose.yml.
This may not be easy; we need to take into account that docker-compose does not have knowledge about the image itself; how should those commands be combined? Or would it simply be; "concatenate the commands with a space"? e.g.
command:
- "bash -c"
- "./manage.py migrate &&"
- "./manage.py collectstatic --noinput &&"
- "./manage.py runserver 0.0.0.0:8000"Which would result in;
bash -c ./manage.py migrate && ./manage.py collectstatic --noinput && ./manage.py runserver 0.0.0.0:8000Notice the missing quotes in that case; working around that would quickly lead to
command:
- "bash -c \""
- "./manage.py migrate &&"
- "./manage.py collectstatic --noinput &&"
- "./manage.py runserver 0.0.0.0:8000\""Not sure if that would be more readable than the current situation 😞
@thaJeztah I would expect commands to run separately one after another:
command:
- ./manage.py migrate
- ./manage.py collectstatic --noinput
- ./manage.py runserver 0.0.0.0:8000Just like if I'd run:
docker run app_site ./manage.py migrate
docker run app_site ./manage.py collectstatic --noinput
docker run app_site ./manage.py runserver 0.0.0.0:8000
but with all extra options provided by docker-compose.yml like environment variables etc.
Actually I think current approach is fine and disadvantages are minor. Just probably it could be better, I don't know about possible pitfalls.
I would expect commands to run separately one after another:
That won't be possible; a container runs as long as it's primary process is running; the container will stop the moment the first command completes.
It's a bit different when running interactively; in that case bash is the primary process, and the process keeps running because an interactive shell is opened; your proposal would mean that docker-compose runs a command, detects when it completes, then triggers the next command. That's not something that can be implemented (or at least, not without a major change in the way compose and/or docker work).
@thaJeztah I see, sounds like a major change. Just an idea, to keep command: behaviour as is but introduce optional prepare: like that:
prepare:
- ./manage.py migrate
- ./manage.py collectstatic --noinput
command: manage.py runserver 0.0.0.0:8000
Any progress on this?
@rudyryk prepare was a suggestion, it's not implemented at all
you can do it like this
command:
- /bin/sh
- -c
- |
./manage.py migrate
./manage.py collectstatic --noinput
./manage.py runserver 0.0.0.0:8000
@rosskevin in my case I didn't have bash, so I used sh -c "command && other command", just for anyone that is having this issue as well
What about having another entrypoint defined in the Dockerfile to handle multi commands?
ENTRYPOINT ["/bin/sh"]
ENTRYPOINT_MULTI ["/bin/sh", "-c"]
@thaJeztah I see, sounds like a major change. Just an idea, to keep
command:behaviour as is but introduce optionalprepare:like that:prepare: - ./manage.py migrate - ./manage.py collectstatic --noinput command: manage.py runserver 0.0.0.0:8000
within a command command is it possible to use variables?
Something like :
command: / bin / bash -c "envsubst </etc/nginx/conf.d/template.conf> /etc/nginx/conf.d/${NGINX_HOST}.conf" ...
Unsupported config option for services.python: 'prepare'
This is quite lame that command: first_command && second_command doesn't work.
I haven't understood the design reason for this, nor the technical restriction if any.
But at least, shouldn't there be a warning somewhere?
I lost much time on this and I guess I'm not the only one!
This is quite lame that command: first_command && second_command doesn't work.
Have you tried;
command: "sh -c 'first_command && second_command'"Thanks @thaJeztah for getting back. sh -c 'commands' does work but
- this is much less elegant
- this is very unpredictable from anyone starting with
docker-composehence a dead end for many.
Just a workaround then.
This is quite lame that command: first_command && second_command doesn't work.
Have you tried;
command: "sh -c 'first_command && second_command'"
I have tried all approaches mentioned above, but nothing works as of now. Any ideas? What else I could try?