Giters

qgis / QGIS-Enhancement-Proposals

QEP's (QGIS Enhancement Proposals) are used in the process of creating and discussing new enhancements for QGIS

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Docker images strategy

3nids opened this issue · comments

commented

QGIS Enhancement: Docker images strategy

Date 2021/03/22

Author Denis Rouzaud @3nids

Contact denis@opengis.ch

maintainer @3nids

Version all new releases

Current situation

  • Current builds of Docker images are failing due to a limit of space.
  • The image produced is super heavy ~7gb.
  • There is a confusion between QGIS CI purpose with the base image (qgis/qgis3-build-deps) and the need of a public image for other purposes (testing or production with server or PyQGIS apps/plugins).

Due to the failing jobs, we need to propose something soon.

Proposed solution

The idea is to produce production/testing ready images (one for PyQGIS, one for server).

PyQGIS (qgis/qgis)

This new workflow would simply use the existing packages from qgis-debian apt repo.
The main image qgis/qgis would be for PyQGIS.
The docker file would simply be: https://github.com/opengisch/qgis-docker/blob/master/Dockerfile
The workflow is regularly testing what needs to be build: https://github.com/opengisch/qgis-docker/blob/master/.github/workflows/publish-docker.yml

Steps to be done:

  • Transfer opengisch/qgis-docker to qgis/qgis (we put the workflow in the main repo)
  • add Alessandro's work with QGIS runners.
  • Add the nightlies build in addition to existing LTR and LR (regular release)

Server (qgis/qgis-server)

First find a consensus on an image to reuse (with credits to original authors).

Proposed images:
https://github.com/gem/oq-qgis-server/blob/master/Dockerfile

Potential issues

2 issues are identified:

  • qgis/qgis will not have server anymore. But this image is in reality not usable in production in its current state.
  • we will rely on the debian packages for the nightlies and not on the build of the source code anymore (but I don't see how this is an issue).

Advantages of the approach

  • official images
  • new LTR and LR tags than can be used in CIs
  • usable in production (mainly for the server)
  • much lighter images
  • less maintenance work for the existing workflows

Votes

(required)

commented

Sounds good!

commented

This new workflow would simply use the existing packages from qgis-debian apt repo.

For me, this didn't sound good because it looks like simple but when we should do a new release it's not so simple (because we depend on an external resource), and this is also valid when we should debug the image..., we should have a way to directly have an image from the sources, for example by using multistage build at Camptocamp we have this for example:

https://github.com/camptocamp/docker-qgis-server/blob/master/Dockerfile

The first stage
https://github.com/camptocamp/docker-qgis-server/blob/master/Dockerfile#L1-L61
we build the server

Then we install the server:
https://github.com/camptocamp/docker-qgis-server/blob/master/Dockerfile#L63-L65

Build the desktop:
https://github.com/camptocamp/docker-qgis-server/blob/master/Dockerfile#L67-L87

Create a base runner:
https://github.com/camptocamp/docker-qgis-server/blob/master/Dockerfile#L89-L108

Create the final server image:
https://github.com/camptocamp/docker-qgis-server/blob/master/Dockerfile#L110-L176

Create the final client image:
https://github.com/camptocamp/docker-qgis-server/blob/master/Dockerfile#L178-L195

I don't say that it should be like it, I just want what we can do with multistage build:

  • Multiple FROM using different base images
  • COPY --from=... to copy a file from an image to another

It probably shouldn't redo a full build but get the build artefacts from the CI build.

commented

For me, this didn't sound good because it looks like simple but when we should do a new release it's not so simple (because we depend on an external resource), and this is also valid when we should debug the image..., we should have a way to directly have an image from the sources

I guess it all depends what we are talking about. Is this for production server? etc.
I am no expert for the server part.
To move forward, I would propose to leave this question apart for now and move on for the PyQGIS image.
This could be moved to a second QEP.

Does this sound acceptable?

commented

@sbrunner @3nids server is a particular case because almost everyone has different needs.

I think we should provide an official server image that perhaps will not fit for all foreseeable advanced use cases but still provides a working image for people that wants to try it out.

I would recommend to keep is simple and load it with a test project and the landing page that points to the folder where the test project is stored: something that works out of the box and that can be used as a starting point for more complex setups.

@sbrunner I think multistage is overkill here, we can just use debian official packages and install them with apt, no need to overengineer or build from sources.

commented

@elpaso should we move .docker/qgis_resources/test_runner to QGIS resources so they are installed in share?
It would make sense to have this part of the installation, no?

commented

@elpaso should we move .docker/qgis_resources/test_runner to QGIS resources so they are installed in share?
It would make sense to have this part of the installation, no?

Yes, that sounds a good idea to me.

commented

Would this affect the doctest test in qgis-documentation?

commented

Great idea, thanks @3nids !

I agree with @sbrunner to build the Docker directly from sources to have a up-to-date, for each version (LTR, LR, possibly master) QGIS server docker.

We could use multi-stage to remove build related dependencies (*-dev, clang, ninja) and lower the size of docker images produced(example here .

We could also reuse what has been done for the OGC worflow (docker file and build script).

I think we should provide an official server image that perhaps will not fit for all foreseeable advanced use cases but still provides a working image for people that wants to try it out.

I agree.

commented

to build the Docker directly from sources to have a up-to-date for each version (LTR, LR, possibly master)

the solution with the packages would satisfy this, with a delay of about 1 day for master and maybe a week or so for the releases.

As the PyQGIS API docs building process is now broken, I will first introduce the PyQGIS image as I think there is no objection for this one.

For the server, looking at the multi-stage solution, I have mixed feelings. While it looks nice to be able to quickly produce images from an update in the code base, I don't think it's worth the maintenance effort.
It seems also to introduce some very specific things in there.
Discussing with some people a bit more involved with server, it looks like the openquake image is quite widely used.
Of course, complex use-cases will require fine tuning, and these will always use their own images.

More generally, if we talk about production images, I believe that we should not bring the complexity of the building steps into the image itself. This complexity should be kept on the CI/testing side.
For reference, there is another QEP about publishing official images where there are similar opinions: #142

As one of the main maintainer of the CI and Docker images, I'd rather ship something simple to maintain that works for a large panel of users (and not targeting the most advanced ones).

I think we can always bring a second parallel solution later on for the server if there is a need for it.

commented

the solution with the packages would satisfy this, with a delay of about 1 day for master and maybe a week or so for the releases.

OK, I was not aware that packages was up-to-date without (not too much) delay. So I agree it seems better to use them to build the docker images

commented

Hi, speaking here as a user and GIS administrator.

Definitly we (I) need official images dedicated to documented use cases with a landing page explaining this somewhere (in dockerhub?), especially for QGIS server.

For the desktop use case, it is not that easy. Since 2 years, I play with dockerised QGIS desktop, but it remains limited and needs tweaking to mount volumes, point to profiles where you need it to be, and have correct interaction with the host (open the help pages for instance) etc..

I would also recommend the lighter images possible, and if I get it well, multi stage is a good idea. It seems to be a good practice in the industry, so I would be in favor of making the effort going this way.

@elemoine did something similar at https://gitlab.com/Oslandia/qgis/docker-qgis where one image builds QGIS and generates .deb that are installed to a light production image.

commented

Here's a plan:

  1. transfer opengisch/qgis-docker to qgis.org
  2. replace official images by these (meaning that we will have a few days of delay as explained)
  3. if anyone feels the need to build on master to gain a couple of days or have more flexibility on the build, we could reuse Oslandia's work https://gitlab.com/Oslandia/qgis/docker-qgis
  4. turn down the docker build workflow on the main CI (jobs are taking 6 hours, and sometimes failing)
  5. I will update PyQGIS API docs build process (taking the same approach with automatically getting which is the last version, so no need to update anymore the source at each release)
commented

@pka the repo has been transferred. Contributions to the Dockerfile are very welcome!!!