Refactor github build/test model

[spbolton]

Changes in #27858 exposed limitations in how we running or dev/test flow with regards to how we go from a PR to deploying tested code in master_latest.

The original flow

1. PR builds and tests only based upon the actual changes made to the code
2. When PR build/test passes the PR goes into the merge queue which could combine with other PRs if configured on the Queue. The changes in this temporary branch define what is built/tested
3. Merge queue on success pushes the commit onto master branch, the commit is Fast Forward pushed so the exact same commit that was tested on the merge queue is now on master including test results on the commit.
4. The push to master triggered a full build test. Adds its test step results to the commit.

Issues
As the merge queue does not always run all steps it can merge quickly into master. As we are running full build and test on each commit merged to master we can create a bottleneck if we run these in serial. If we run these in parallel we may exhaust the number of active runners that is limited to 20. When we use concurrency control to run in parallel Github only allows 1 pending workflow if another request to build comes in while one is pending, the pending one is cancelled.

We have some duplication of testing. The build and test on master runs on the same exact commit that the build queue workflow ran on. Even though it may run some extra steps, unless there is a flakey test or an environmental issue these should not fail, other tests and the build should produce exactly the same results. Finding an issue with the build and test in the master branch should be an exception case and not based upon a direct issue with the code, it is much better to find these issues in the build queue and prevent merge into master rather than find these after they have already been merged. If we are concerned about making sure that possible flakey integration or postman tests are run it is better for us to make sure the full test suite is included into the merge queue workflow and then accept that whatever has been merged has been appropriately tested to be ready to deploy to master qa server. Flakey tests that did manage to get through to master can be found by running periodic full build cycle e.g. nightly build/deploy or scheduled periodic builds that do not need to run on every commit.

Running workflow steps on the completion of others has its downsides. Currently we do this to call build-report.yml workflow after the main ci workflow. The reason for this is due to additional security of secrets and permissions required to update the PR with check run information and to post to slack. This requires this separate build-report.yml to only be effective after updates have been pushed to master. It can also be difficult in the Github UI to see the relation between the triggered and triggering workflow as well as control properly the concurrency of these asynchronous workflows. When we are running a workflow that runs off the code that is in the master branch like our deployment to qa master, or in building a release it is harder to inject code that could expose secrets as it should have already been approved. We should therefore be able to include the functionality in build-report.yml into the main workflow we are creating. The way we can enable the best of both is to make use of githubs reusable workflow functionality to include the same code to be triggered in both cases either in build-report.yml from a PR or directly embedded into a single workflow for build/test/deploy.

Todo
Break down the .github/workflows/maven-cicd-pipeline.yml using reusable workflows, allow configuration of the build / test / deploy allowing for the constraints of PR, merge-queue, master and nightly e.g.

• full vs incremental build/test
• build of artifacts from current build vs using those from another previous build (e.g. from merge queue for same sha )
• Allow for SonarQube to only run for PR and master branches