Running container commands from `commands_pre`

[Jonathan Green]

I'm trying to migrate a project to use tox 4 and tox-docker 4, and one problem I am having is running commands in the container, before running my tests.

Previously, since the container name was known, I was able to do something like this:

[testenv]
commands_pre =
    docker: docker exec es elasticsearch-plugin -s install analysis-icu
    docker: docker restart es

I realize I could build and push my own docker images then use those, but I really don't want to maintain docker images just for CI testing. I would rather use the images maintained by elasticsearch.

Would you be willing to look at a PR that addresses this by either:

1. Adding an optional tox-docker configuration that disables the unique name
2. Adds a substitution so that the container name could be templated
3. Merge something like #139 so a command can be specified

[Matt Baker]

Option 2 seems very easy to implement. Something like the existing <container-name>_HOST.

diff --git a/tox_docker/plugin.py b/tox_docker/plugin.py
index eec57fd..c20d3ab 100644
--- a/tox_docker/plugin.py
+++ b/tox_docker/plugin.py
@@ -161,7 +161,9 @@ def stop_containers(
 def get_env_vars(
     container_config: ContainerConfig, container: Container
 ) -> Mapping[str, str]:
-    env = {}
+    env = {
+        escape_env_var(f"{container_config.name}_NAME"): container.name
+    }
     gateway_ip = get_gateway_ip(container)
     for containerport, hostports in container.attrs["NetworkSettings"]["Ports"].items():
         if hostports is None:

[Dan Crosta]

Presently, it's not possible for plugins to contribute to the substitutions that tox performs, so the commands_pre approach won't work.

I think passing a custom command also won't do quite what you want here -- or rather, it will, but it will couple your tox not only to the setup you need to do within the container, but also the correct command for the service itself (elastic, it looks like, in this case). However, there is prior art with the healthcheck implementation, which is farily similar.

I could also see adding the ability to set the container name via a directive in the [docker:foo] sections -- so it would be predictable, but in user control. The downside here is that tox runs will fail if a container already exists by that name. (We make efforts to always clean up, but there are ways that can fail, and also --docker-dont-stop=...)

I haven't investigated the feasibility, but what about a fourth option, of having something like

[docker:foo]
build = Dockerfile.withMyCustomizations
image = imageName

This would effectively docker build -f Dockerfile.withMyCustomizations -t imageName . and then docker run imageName. This would actually be a "graceful" solution to any kind of customization -- the entrypoint/command, additional steps, exported ports, etc.

Thoughts?

[Jonathan Green]

@dcrosta I hadn't considered that option but it sounds like a winner to me. It would certainly fit my use case and provides a lot of flexibility for customization.

[Bernát Gábor]

Presently, it's not possible for plugins to contribute to the substitutions that tox performs, so the commands_pre approach won't work.

While you can't extend tox substition, you can redefine commands_pre or alter it yourself so I think you could achieve the same effect.

[Jonathan Green]

While you can't extend tox substition, you can redefine commands_pre or alter it yourself so I think you could achieve the same effect.

@gaborbernat do you happen to know of an existing plugin that alters commands_pre like this that you could point me to? I'd like to try to understand how this could be accomplished.

[Bernát Gábor]

I don't have that, but you can either insert a memory loader that overrides it, as described here tox-dev/tox#2882 (comment) or modify it inplace

@impl
def tox_add_env_config(env_conf: EnvConfigSet, state: State) -> None:  # noqa: U100
	 commands_pre: list[Command] = env_conf["commands_pre"]
     commands_pre[0].args[1] = apply_my_substitution(commands_pre[0].args[1])
     commands_pre.insert(Command())

https://github.com/tox-dev/tox/blob/main/src/tox/config/types.py#LL9C7-L9C14

[Dan Crosta]

@gaborbernat I'm guessing since you suggest it that this is/will be a supported operation, not something we come to regret if I make this change that way?

We'd need to invent some new substitution-like syntax (I think we could not re-use the {...} syntax, that would cause errors due to unknown substitutions from the main code? or will it preserve subs it doesn't recognize?), so this feels like a tricky road to go down...

[Bernát Gábor]

It's supported. The substitution engine will ignore unknown ones.

[Dan Crosta]

Awesome! Thank you!

[Dan Crosta]

@jonathangreen want to try to version on #156 and let me know if it works for your needs? It seems to work OK with simple builds, but there are a bunch of options we could pass to build(), and I'd love more real-world feedback before launching this...