rename and repurpose this repository
ktdreyer opened this issue · comments
The "ceph-container" GitHub project started as an experiment to containerize Ceph and make it integrate with Kubernetes. It has met this goal. I'm opening this ticket to propose how to containerize Ceph in the future.
Years ago, ceph-container provided a lot of extra experimental code (the different "flavors" like "ceph demo", the "sree" app, confd stuff, etc) and supported a wide variety of operating systems (Ubuntu, CentOS, Alpine, etc). Neither of these things are true today. Guillaume has simplified the code now so that it is almost a simple Dockerfile and entrypoint script. We also only publish Ceph images with a single OS (CentOS 8 Stream), greatly simplifying QE effort.
The templating system introduced in 2018 does not provide value today.
I suggest that we have a single Dockerfile at the root of https://github.com/ceph/ceph that will be "the" Ceph container image. (see ceph/ceph#43292). Users who deploy Ceph with cephadm or Rook will use those stable sets of images. This ceph-container GitHub repository will no longer host the Dockerfile for that.
There are a few people in the Ceph community who have discussed maintaining Ubuntu- or Debian-based container images. To provide a common place for these, I suggest renaming this GitHub repo from ceph-container to ceph-containers or alt-containers, and rewrite the README to explain that this is now the location for the non-official or "alternative" container images. You could store simple, untemplated Dockerfiles within subdirectories like ubuntu, debian, etc.
Great to hear about the simplification. What about the builds that currently come out of the ceph-container repo? For example, in Rook we rely on these tags to test against the latest merges to the various branches:
quay.io/ceph/daemon-base:latest-quincy-develquay.io/ceph/daemon-base:latest-reef-devel
When the images are build from the ceph/ceph repo, can we get these tags also built? Even better if we could just get these tags in the quay.io/ceph/ceph repo, it has always been confusing having the daemon-base repo.
@travisn good question.
Personally, I think it makes sense to go one of two ways:
- Use
quay.io/ceph/cephfor all images including dev builds, OR - Use a new
quay.io/ceph/devellibrary for the dev builds
Probably the latter is best.
I think all the different builds can be handled by the configuration of GitHub actions, combined with any special changes needed in release branches.
In the main branch, the Dockerfile will use the latest main build of Ceph, and the GitHub action will publish to quay.io/ceph/devel:latest-main as well as the commithash version.
In release branches, Dockerfiles will use the latest corresponding release build of Ceph, and the GitHub action will still publish to quay.io/ceph/devel:latest-<release> as well as the commithash version.
When a particular Ceph version on GitHub is tagged as a release, it should be given a tag like v18.2.1 using GitHub's "Releases" feature. When the GitHub action sees a tag like this, it should publish an image with that tag to quay.io/ceph/ceph:<tag>, plus quay.io/ceph/ceph:<tag>-<date> (in addition to the regular commithash and latest-<release> publish to quay.io/ceph/devel). This process can also be used for release candidates with tags like v18.2.1-RC0 if desired.
One of the other things that GitHub actions should probably do is to regularly re-build tagged, released images with the latest base image. The base image (stream8, stream9-minimal, etc.) will have regular, ongoing security fixes that some users may want. A GH action cron that runs daily on v* tags can do this, and there is a github action that should make it easy to only publish new image versions when there are base image changes: https://www.flypenguin.de/2021/07/30/auto-rebuild-docker-images-if-base-image-changes-using-github-actions/
I think renaming this repo to alt-containers might be good to signify that its purpose is changed dramatically. I suspect that it is not widely used, and that we won't see many changes or much interest in it after the move Ken is proposing.
Great to hear about the simplification. What about the builds that currently come out of the ceph-container repo? For example, in Rook we rely on these tags to test against the latest merges to the various branches:
quay.io/ceph/daemon-base:latest-quincy-develquay.io/ceph/daemon-base:latest-reef-develWhen the images are build from the
ceph/cephrepo, can we get these tags also built? Even better if we could just get these tags in thequay.io/ceph/cephrepo, it has always been confusing having thedaemon-baserepo.
I assume however we move forward we'll continue building regular images of the stable branches. We already do daily builds of the pacific onward stable branches to make sure they can be nuilt and used in baseline runs. Those are currently pushed to quay.ceph.io/ceph-ci/ceph:<release> e.g. quay.ceph.io/ceph-ci/ceph:reef and it would be detrimental to lose those. I'm not sure how the naming would be moving forward but I don't see a world where people are happy with us not making regular builds of the stable branches.
I really like this idea. I think it has the potential to simplify processes involved in testing and building ceph containers downstream as well as upstream.
hear, hear! I'm all for simplifying how the container image(s) are built. I've felt for a while that this code is too complex for the result it produces. I might have some quibbles with the exact structure or location for this Containerfile but that can be handled elsewhere.
As for repos and tagging, I feel obligated to mention that ceph produces release containers at quay.io/ceph/ceph as well as ci/testing containers at quay.ceph.io/ceph-ci/ceph. The 'go-ceph' project uses both in it's ci - the former for required tests against major releases and the latter for integration tests against unreleased branches. If the latter is suitable for the rook use case, perhaps no new repos need to be created?
I think quay.ceph.io/ceph-ci/ceph is a good option for what I proposed above as quay.io/ceph/devel. I didn't know about those images.
As you note, this has been proposed before. All the issues above are valid; I'm paranoid about finding all the users of ceph-container, but I think the time is right to simplify (having just spent another few hours trying to understand make macros in ceph-container)
Great idea! I propose to consider to use https://github.com/GoogleContainerTools/distroless images as base images.
FYI: Starting to set up the skeleton of the new Containerfile and github actions builds here: ceph/ceph#54575