USER and Layers Discussion
duglin opened this issue · comments
While reviewing #28499 in today's maintainer meeting we decided to take a step back and talk about whether we should consider this at a higher level.
It seems there are two issues at play here:
1 - under what id should a certain Dockerfile command be executed under? root vs USER
2 - is the USER --chown issue really a work-around because of the layering issue?
Meaning, are people ok with COPY+RUN chmod... if an extra layer wasn't created? Which is related to @stevvooe's comment: https://github.com/docker/docker/issues/29853#issuecomment-271469066
So, we decided to open up this issue to have a discussion around how to move forward.
Let's start with some questions:
1 - If we were starting from scratch (and didn't have to worry about backwards compat) would people prefer for USER to change the id under which subsequent COPY/ADD/RUN/... commands run?
2 - is it true that while being able to COPY --user john ... is perhaps easier to write than COPY ... + RUN chmod ..., are people ok with two commands if an extra layer were not created?
Personal opinion:
- I think
USERimpacting subsequent commands is more natural, so I would be in favor ofUSER --persist johntype of flag to enable this. This won't break backwards compat. - I liked @stevvooe's suggestion of using an
@prefix on a Dockerfile command to indicate that we should skip the commit on this command. I have no idea what kind of impact this would have on caching yet, but if we can make it work I like that better than previous suggestions of BEGIN/END type of commands. It seems so non-intrusive. And yes I know we have a squash command on build now, but there are time when people still want layers, just not as many as they get today.
Please post your comments/opinions here instead of in #28499 and moby/buildkit#4242
ping @docker/maintainers
I think the whole USER Dockerfile command is quite odd anyway, and a docker build -u uid option which just ran everything as that user would make more sense (yes I know it doesnt work for some use cases).
I also think that if you could mount the build context as a filesystem life would be easier as you could just do
MOUNTCONTEXT /mnt
USER foo
RUN cp /mnt/* /usr/
which would work as expected.
Yes, mounting the context works against any potential lazy loading of context, but it does have the advantage that you can use files without having to ADD or COPY then first eg
MOUNTCONTEXT /go/src/project
RUN go build -o /usr/bin/program /go/src/project
which saves committing useless source code to your output.
I still have some code around, someplace, that would automatically mount the build context into each build container under a well-defined spot in the FS to do exactly what you mentioned. At the time I wrote this, ages ago it seems like, there wasn't much interest. Perhaps we should reconsider. But that's a slight tangent :-)
re: "build -u" I wouldn't see an issue with that as an option, but I do suspect there are lots of people who will still need to switch to/from root in the middle of a build. Plus, I think the original purpose of USER was to just set the user id under which the runtime container was executed w/o them having to specify it on each "docker run" cmd, so I don't think we can ever really get rid of that variant.
2 - is the USER --chown issue really a work-around because of the layering issue? Meaning, are people ok with COPY+RUN chmod... if an extra layer wasn't created? Which is related to @stevvooe's comment: moby/buildkit#4242 (comment)
I'm here because this was NOT acceptable to me. We have a fairly unremarkable NodeJS application, yet running the chown -R on the node_modules directory was taking upwards of 5 minutes.
I also would advocate for implementing @stevvooe's suggestion of using an @ prefix on a Dockerfile command to indicate that we should skip the commit on this command.
As detailed on #14298 in this comment: we have been waiting to fix this "user:group when doing a COPY" problem for more then 2 years now. Many people have invested their time proposing similar fixes to this problem (@thaJeztah @duglin @icecrime @kostickm and many others). Many times the attempts are blocked for more discussion.
I may be missing some issue and sound ungrateful for other peoples intentions or anything (if so, please correct me), but it seems to me that the positive intention of carefully moving forward had resulted into a painful inability to fix this issue for our users.
Personally I would much appreciate a COPY --chown because: 1) it fixes this problem, 2) I find it's meaning clear and 3) I do not have to use it if I do not want to. Adding a @ prefix can in my opinion also improve the Dockerfile format for this and many other reasons. But even with a @ prefix I would much enjoy the COPY --chown over having to add an additional @USER root, @RUN chown <same path again> and @USER <restore previous USER>. If I understand correctly this is also what @robhaswell advocates a few comments ago.
So I think the @ prefix is a valuable suggestion, but I do not think the discussion around that should derail fixing this COPY problem. Is there anybody that heartedly opposed to introduction of COPY --chown (after more than 2 years of waiting)? E.g. by using #28499 (closed by @dnephin 2 days ago awaiting this thread).
Confirmed, one of our services is running as root for now because running chown -R on the build directory causes an unacceptable delay.
I am disappointed at Docker Inc's approach to this issue. @pjvleeuwen I believe Docker's official line is that they will not make any changes to the Dockerfile builder because they intend to throw away the code and start again. I wonder how their enterprise customers feel about being denied easy fixes for real customer pain.
With each PR that addresses the issue some hope is build up, so I recognize the sentiment of disappointment (without wishing to sound frustrated or judgemental). I much appreciate the effort of @thaJeztah @duglin @icecrime @kostickm @dnephin and others, that is why I think it is such a waste that it does not come to a close for the COPY --chown problem. It has been fixed now many times it seems.
Last thing I want is to turn this thread into just chickens (speaking for myself) complaining to the pigs. Just a plea to consider all those you help by pushing e.g. #28499 to a merge. My thanks to anybody helping on that front. Cheers!
I personally think both of the solutions have merit. I am concerned that some scope creep may be making its way into the discussion -- @ is about preserving layer efficiency, not applying user:group. If people disagree on @ then it should not block USER/COPY/ADD proposals. Maybe these two should be split from each other?
Count me as another customer that suffers from the COPY + RUN pattern causing severe build times. I would place my vote on adding --chown to COPY/ADD since it's already backwards compatible, opens up options for --chmod, and can co-exist with any --persist changes to the USER command if that decision is also made.
That said, props for making sure a thorough discussion is in place to back up any decision. I know how painful it is to remove something after it's added to the spec!
We recently encountered this behavior in an enterprise setting, where it's particularly painful to deal with. To aid in moving this discussion towards closure I'll describe our basic use case and then examine the proposed alternatives in this context.
To allow dev teams to build standard container images we provide a baseline Tomcat image, derived off an enterprise-approved OS base image and with some additional utilities and best practices wired in. A user only needs to COPY their expanded WAR into /apps/tomcat/webapps and they're ready to roll. Following Docker best practices we include a USER directive to configure Tomcat run as a custom "tomcat" user - we create this user and ensure all files under /apps end up owned by this user. A developer's Dockerfile would ideally be as simple as:
FROM <internal-registry>/tomcat:8
COPY myapp /apps/tomcat/webapps/
Indeed in our old days of letting Tomcat run as root in the container it was actually this simple. Alas, once we added in a USER directive in the Tomcat base image we ended up in distress, because now the /apps/tomcat/webapps/myapp directory ends up owned by root, and token-replacement scripts run from within our CMD script before Tomcat is launched now fail due to lack of permissions for the tomcat user to modify these root-owned files.
Currently our workaround for this is to have developers use the following Dockerfile pattern:
FROM <internal-registry>/tomcat:8
COPY myapp /apps/tomcat/webapps/
USER root
chown -R tomcat:tomcat /apps/tomcat/webapps/myapp
USER tomcat
This is an unsatisfying solution - we end up with an entire second copy of the WAR file contents in the image in a new redundant layer. We could use docker build --squash to eventually fix that - but it's still marked experimental (https://docs.docker.com/engine/reference/commandline/build/#squash-an-images-layers---squash-experimental-only) so we've shied away from that for the time being. Worse, the entire Docker base image abstraction is compromised since the developer now needs to know implementation details of the base image - most notably what user Tomcat runs as, and every developer needs to wire this workaround into their Dockerfile.
The ADD/COPY --chown flags proposed in #28499 would let us trim the developer's workload down to:
FROM <internal-registry>/tomcat:8
COPY --chown tomcat:tomcat myapp /apps/tomcat/webapps/
This is an improvement - we've removed the redundant layer from the final image without needing to resort to squashing, and we've trimmed four commands down to one. However, we're still forcing the developer to break the base image abstraction by knowing which user the Tomcat base image is using. If we ever want to change which user Tomcat runs as (perhaps one day we'll define a cross-base-image standard user) we'd be faced with the ugly task of asking dozens of dev teams to change their Dockerfile COPY directives as they take a new base image version, and in tandem for all of them using our tomcat:8 latest-Tomcat-8-image tracker tag.
The DEFATTR directive proposed in moby/buildkit#4242 is one potential solution to this problem, as we'd then be able to add a DEFATTR chown=tomcat:tomcat directive to our base image, and developers would be back to their original two-line Dockerfile without needing to know which user Tomcat is running as. In our use case I don't see a need for the chmod aspect of DEFATTR, which looks to be the more controversial aspect of that proposal from the comments.
Alternatively, if the USER directive triggered all subsequent COPY/ADD directives to run as that user then our problems would be solved immediately, and we'd return to the original two-liner with no need for developers to know base image internals details.
Therefore, for our use case either of the following approaches solves our problems:
- Implementing the changes defined in #28499 and moby/buildkit#4242 (at least the chown side of moby/buildkit#4242).
- Changing the semantics of USER to control which user COPY/ADD operations run as in addition to which user RUN and CMD operations run as.
Which option is preferred by the maintainers?
Thanks for your input @brodmerkle!
Changing the semantics of
USERto control which userCOPY/ADDoperations run as in addition to which userRUNandCMDoperations run as.
This would have been my preference, and preference of most maintainers, however the behavior of not following USER was already in place and therefore would result in a backward breaking change; as a result of which we're having this issue (and a couple of other issues/PR's before this)
Thanks for the feedback @thaJeztah - I can appreciate how changing the semantics of an unadorned USER directive at this point would probably be a disruptive mess.
Given that, it sounds like the first option of implementing #28499 and moby/buildkit#4242 wins by default. Currently #28499 is closed with comments that it be reopened once the discussion in this issue wraps up. Is there anything left to hash out here, or are we ready to reopen #28499, and for that matter to get some traction on moby/buildkit#4242 to fully close out this scenario?
If people would prefer to change the semantics of USER, one option is to add a flag, e.g.:
USER --chown joe
and its fully backwards compatible.
Thanks @duglin - this sounds like an extension of your proposed code in #28499 to add --chown to ADD/COPY, in this case allowing the same optional argument on USER to set a default for subsequent ADD/COPY operations (basically piggybacking the DEFATTR approach proposed in moby/buildkit#4242 onto the USER command).
How would this syntax look for specifying users and groups? I'm thinking it will by necessity need to be a bit different than what's shown above to handle groups, something more like:
USER joe- Configures RUN/CMD operations to run as joe, while ADD/COPY operations continue to create root-owned files.USER --chown joe:users joe- Configures RUN/CMD operations to run as joe, and configures ADD/COPY operations to create files owned by joe:users.
A subsequent ADD/COPY directive can then always override the chown defaults from a prior USER directive by specify its own chown flag. For example:
USER alice
COPY alice.txt /apps/
USER --chown joe:users joe
COPY joe.txt /apps/
COPY --chown frank:user frank.txt /apps/
Here we'd end up with /apps/alice.txt owned by root:root, /apps/joe.txt owned by joe:users, and /apps/frank.txt owned by frank:users.
My knowledge of Go is rather limited, but I'd be happy to at least look over a proposed commit to this effect, and propose some wording for the documentation side if it helps.
@thaJeztah - Given that this thread has now reached what seems to be a consensus conclusion (with no further feedback in the past week), I believe that the condition @dnephin laid down in #28499 for @duglin to reopen that issue has now been met. Are the next steps now for #28499 to be reopened, for @duglin to rebase the earlier PR and add --chown defaulting behavior on USER as well, and then for this PR to get merged?
Bringing this to conclusion will be a huge win for a large swath of Docker users...
Now that multi-stage builds have been merged we should bring back the discussion about mounting build context(s) for builder's RUN commands but with the same caching rules as COPY --from=. In addition to fixing cases where context files are not needed inside image at all and potentially fixing the need for build -v by providing reusable cache mounts that would also provide a fix for chown issue. Maybe COPY --chown is important enough on its own to justify the extra UI but implementation wise they should be equal with RUN --mount.
@tianon @duglin
I saw the discussion of what the actual flags should be in the pull request. If it is helpful I thought I could point out that the install command is used for copying and setting file owner/mode.
Its flags look like this:
-g, --group=GROUP set group ownership, instead of process' current group
-m, --mode=MODE set permission mode (as in chmod), instead of rwxr-xr-x
-o, --owner=OWNER set ownership (super-user only)
Maybe these should be used? It might be nice to be able to use those short options (maybe in the future).
What does everyone else think?
The rsync syntax we are currently talking about looks ok too.
现在对于这个问题，发现居然已经讨论了两年了，怪不得这么人不指定用户，我只能呵呵~
只好对于所有的镜像，我选择一切基于 root 用户和组，为什么？
- 宿主机挂载的目录文件权限全部是 root （不论你是否之前创建以及授权，第一次挂载时 docker 好像都会把它改变为 root）
- 在子的dockerfile 如果之前设定了用户，那么下次使用 dockerfile 时无法安装，因为权限不足，为啥？docker会把之前的用户来运行下一个shell，好比如 USER docker ，yum install ？？呵呵！（不要天真，你以为这样就完事了？先指定用户只 run/copy 指令，这些都是 root 的所有者）
- 还有一些别的，就上面两条我已经放弃使用 docker 的用户（安全？只要严谨一些就好了，不要顾虑这么多了）
现在我有这些建议选择：
- 提供一个构建参数，可以指定全局的，针对变更的层设置用户和组/权限，同时也可以指定某些文件或者目录（e.g : docker build -ug docker:docker -p -wrx-wrx-wrx，这个是全局的，只针对这个变更的层，docker build -ug docker:docker -d /opt 这个针对指定目录的，可以多个，docker build -ug docker:docker -f /opt/why.sh 这个指定变更的文件，可以多个），这样的话则兼容之前的，不要在意参数的命名，我随意写的
- 就是在构建之前把相关的用户和组/权限同时复制进镜像中 #32940
这只是我的一些想法，或许有更多没有考虑到。
Google：
Now for this problem, found that has been discussed for two years, no wonder so people do not specify the user, I can only Oh ~
Almost for all mirroring, I chose everything based on root users and groups, why?
- host directory mount the directory file permissions are all root (whether you have previously created and authorized, the first time docking docker seems to change it to root)
- in the child's dockerfile If you set the user before, then the next use dockerfile can not be installed, because the lack of authority, why? Docker will be the previous user to run the next shell, such as USER docker, yum install? The Ha ha! (Do not be naive, do you think this is done? First specify the user only run / copy instructions, these are the root of the owner)
- there are some other, on the above two I have to give up the use of docker users (security? As long as some of the strict enough, do not worry so much)
Now I have these suggestions to choose:
- Provide a build parameter, you can specify the global, for the changes to set the user and group / permissions, but also can specify some files or directories (eg: docker build -ug docker: docker-p-wrx-wrx-wrx , This is the overall, only for this change in the layer, docker build -ug docker: docker-d / opt for the specified directory, can be multiple, docker build -ug docker: docker-f /opt/why.sh this Specify the change of the file, you can more than one), so it is compatible with the previous, do not care about the naming of parameters, I am free to write
- is in the construction before the relevant user and group / permissions at the same time copied into the mirror # 32940
This is just some of my ideas, maybe there is more not taken into account.
@lanmingle would this proposal #32507 solve your use case?
In my opinion, the ideal would be to break compatibility at some point. This seems like a design flaw of sorts. RUN respects USER? Then COPY/ADD should also.
As suggested:
COPY --chown frank:user
Could be a good way around the problem. One thing is for sure: a solution, even if imperfect, is better than what we have now.
A couple of months have elapsed here, so it's worth a review of where the different options for solving this particular problem stand, in the interest of getting something implemented. I'll reuse my original Tomcat base image example since the backstory for that scenario is already covered above in #30110 (comment). Our metric of success here is both remove the duplicate layer and to simplify the end user's Dockerfile as much as possible from the current setup of:
FROM <internal-registry>/tomcat:8
COPY myapp /apps/tomcat/webapps/
USER root
chown -R tomcat:tomcat /apps/tomcat/webapps/myapp
USER tomcat
The RUN --mount support in #32507 would allow us to solve the dual layer problem, by converting the end user Dockerfile to:
FROM <internal-registry>/tomcat:8
RUN --mount=src=/myapps,dst=/tmp/myapp,ro=true \
cp -R /tmp/myapp /apps/tomcat/webapps && \
chown -R tomcat:tomcat /apps/tomcat/webapps/myapp
While we've solved the duplicate layer problem, the new Dockerfile content isn't really any simpler than before - and it gets even uglier if the user has multiple WAR directories to copy in (something that was reasonably easy to do in the original Dockerfile syntax). I also still need to know what user/group the parent container runs as - ick.
Would I actually use this in real life? I might just to get past the duplicate image layers for the WAR contents, but at that point I may just say keep using the current syntax and use docker build --squash to tackle the layer overhead. Does anyone know when the "Experimental Only" designation is due to be removed from https://docs.docker.com/engine/reference/commandline/build/#squash-an-images-layers-squash-experimental-only? The term "experimental" is pretty much kryptonite for using the feature in an enterprise.
To be clear - the RUN --mount functionality looks like it will be useful in quite a number of scenarios and should move forward; I just find that trying to use it to work around the ADD/COPY ownership model is kludgy. I'll probably only use it as a solution if #32507 is due to end up in a Docker release significantly sooner than the squash flag will be deemed non-experimental.
That leaves #28499 to add a --chown flag to ADD/COPY as the best alternative, since getting that merged would allow me to eliminate the duplicate layers and simplify the end user Dockerfile down to:
FROM <internal-registry>/tomcat:8
COPY --chown tomcat:tomcat myapp /apps/tomcat/webapps/
At this point I get 80% of what I'm looking for (as does the rest of the community that keeps referencing this issue) - the only remaining wart is that I'm still having to specify the tomcat user/group inside the end user Dockerfile. That scenario is covered in moby/buildkit#4242, but that issue drifted into deadlock regarding different syntax options and usage of squash versus new flags.
@thaJeztah @duglin @dnephin - in the interest of getting to a solution here, can we proceed with re-opening #28499 and rebasing the commit? That issue was put on hold pending discussion in this issue, but in the end we've come back to needing the exact semantics that the commit in #28499 already implements. Once #28499 is completed and merged this issue can then probably be closed as well, with the last followups for defaulting the ADD/COPY user/group ownership covered under moby/buildkit#4242.
I can add some thoughts to moby/buildkit#4242 around reasons that as a user I can see value in setting ADD/COPY user/group ownership defaults, but that's not in the critical path.
We came across another issue - if you are doing multi stage builds where you are doing a FROM scratch bit and you do not have chown in the image, there is no way you can add a file owned by a non root user at all from another layer, as the uid and gid are always reset to zero. So regardless of anything else we really need COPY --chown ... or COPY --keep-ownership ... for --from type use cases (arguably with --from it could always do that as it is totally under control of the user at that point, although that would be a bit unexpected.
cc @tonistiigi
@justincormack yes as said in #32816 (comment) --chown would be a very great thing to start to implement at this point.
@justincormack Yes, multi-stage itself doesn't solve this issue. copy --chown or run --mount would. For example, buildkit doesn't have this issue because it supports (equivalent of) run --mount already.
@tonistiigi @justincormack so, is the conclusion that we should add the --chown (and -a / --archive) options to the Dockerfile COPY instruction?
Also, COPY --chown 1000:1000 src ./src could be a shorthand of this more powerful yet verbose RUN --mount=src=/some/local/host/path,dst=/some/temp/mount/path,uid=1000,gid=1000 cp -a /some/temp/mount/path/src ./src no?
@shouze Bind mounts don't support uid/gid so we probably can't support them in --mount as well. The files in the source path will have their original owners though, not rewritten to 0.
@tonistiigi ok I see so uid/gid remap should be managed aside in another PR maybe and then will fix #32816. Even ignored from the build hash algorithm or make it possible to remap uid/gid as I was 1st thinking in #32507 (comment), don't know actual implementations enough to say what's the best thing to do at this point.
Discussing in the maintainers meeting with @tonistiigi and @cpuguy83, and after considering the options, we think adding the --chown flag is ok to add. The --mount option can be added in future, and should not conflict with this functionality.
Also discussing; when copying between stages in multi-stage builds, we can preserve the permissions from the previous build-stage, and require people to set --chown 0:0 to reset to root.
This is a change in behaviour, but given that multi-stage builds are still young, and most builds will have 0:0 already, probably defendable (if documented)
I would like to work on it but I'm not sure I have the time right now. If someone else wants to carry it for me in the meantime that would be great.
I may be way out of date, but just came across this thread. We often add a large code base into a container and then have to chown all the files to get file permissions right for security reasons, doubling the image size. So would love the ability to add files with the right file ownership. (Using ADD with a .tar.gz file was a big help to add a large directory tree of files with file permissions set correctly per file - it was only the ownership of the files that was painful to then fix afterwards - root was unacceptable for these files for us.)
Personally I don't care as much about the performance of the chmod - its just the image size that was annoying just to change file ownership.
I personally dislike solutions that remember state from previous commands (like the USER affecting subsequent COPY/ADD commands). Its too easy for someone to then break a long Dockerfile by reordering commands and not noticing the USER command. I would prefer a new COPY-AS-USER capability so the command is atomic. However I would not lose sleep over it.
Looks like this can be closed with #34263 merged yes 👍