Make it possible to run backend components locally

[Henrik Plate]

Is your feature request related to a problem? Please describe.

The necessity to setup and run a number of backend components makes it difficult to quickly setup and test Eclipse Steady. Moreover, smaller development organizations or single developers may not want or cannot afford to operate always-on server components.

Describe the solution you'd like

It should be possible to run all components of Steady locally, e.g., on a developer machine or build system, without the need to operate a server. Ideally, the local setup does not require any explicit action or input from the user but happens transparently.

At least the following two topics must be covered:

• The footprint of the existing Docker Compose environment must be reduced to limit the consumption of client resources. The frontend-apps and frontend-bugs can be removed altogether. The services rest-backend and rest-lib-utils only require one container each. Other improvements in regards to allocated memory may be possible.
• The setup of the local environment must be done in an automated or semi-automated fashion, and there are several possibilities:
  • Create a new goal setup (subclass of org.eclipse.steady.goals.AbstractGoal), which has to be called explicitly before running analysis goals
  • When running analysis goals, check whether vulas.shared.backend.serviceUrl is set to localhost and, if yes, check and setup as needed
  • Use an existing Maven plugin to check and setup the (reduced) Docker Compose environment. This approach would exclude CLI, Gradle and Python users to run Steady locally.

In any case, it is clear that the sync with Project KB will impact the initial installation time, similar to the NVD download in case of OWASP Dependency Check. Subsequent goal executions, however, should not be impacted.

Describe alternatives you've considered

None

Additional context

None

[sumeet patil]

My investigation is as follows -

1. There is no standard maven plugin to use our existing docker-compose files to run the containers.
2. The existing plugins use docker socket APIs to docker run, docker pull, docker volume, and more. Nothing related to docker-compose.
3. We need to build our own maven plugin which does the following -
   1. Parse the docker-compose. Of course, we need to split our existing docker-compose files. Like an Inheritance structure. One compose file with just clients which are used by this new maven plugin and other overrides and adds extra features for server.
   2. Use Spotify java docker client or call docker APIs on our own to run the containers from the parsed docker-compose.

I suppose docker-compose is been written in the same way ;)

[Henrik Plate]

Rather than developing a new Maven plugin from scratch, I would subclass AbstractGoal, and create thin wrappers (goals) for the existing Maven and Gradle plugins.

If I understand correctly, such subclass would download and build all images it finds in docker-compose-local.yml (a slim version of the existing docker-compose.yml) before running docker-compose up, correct?

[sumeet patil]

We will not build any images here. Rather just pull the images from the docker hub.
The process would be to pull images in docker-compose-local.yml, run the containers with different parameters, and then once everything is up it will trigger the analysis.
And yes I will make use of AbstractGoal.

[sumeet patil]

• I will be falling back to running an exec "docker-compose up" as parsing the yml and running the containers manually is a big task and also the java docker clients are not yet evolved
• I would also need a small rest-endpoint for kb-importer which gives us the progress. Example - {"initUpdate":"Completed", "cronUpdate":"InProgess"}. It will help me check the progress of kb-importer from the maven plugin

[Henrik Plate]

With release 3.2.1, the local setup and execution has been facilitated with help of two bash scripts (#502). Different usage profiles (core, ui, vdb, all) allow running different sets of containers to reduce resource consumption.