obotezatu / awesome-jenkins

This ansible playbook installs Jenkins on specified host

Prerequisites

• Run the ansible playbook on Debian or Ubuntu. Used was VM with Jammy Ubuntu. Use the script to create VM on Proxmox.

• Use $HOME/awesome-jenkins/inventory/localhost/hosts.yaml if you are installing the Jenkins on the same host where Ansible is running. Use $HOME/awesome-jenkins/inventory/example/hosts.yaml if you are installing the Jenkins on the remote host.

  In our examples, we use $HOME/awesome-jenkins/inventory/localhost/hosts.yaml file.

• Install Ansible: Follow the second step

• Install molecule on Ubuntu Linux. Molecule project is designed to aid in the development and testing of Ansible roles.

  apt update
  apt install pip
  python3 -m pip install molecule ansible-core
  pip3 install 'molecule-plugins[docker]'
  

Playbook variables used in Jenkins server installation:

1. The HTTP port for Jenkins web interface:

   jenkins_http_port: 8085
   

2. Admin account credentials which will be created the first time Jenkins is installed:

   jenkins_admin_username: admin
   jenkins_admin_password: admin
   

3. Java version:

   java_packages: 
     - openjdk-17-jdk
   

4. Install global tools. Maven versions:

   jenkins_maven_installations:
     - 3.8.4
     - 3.9.0
   

5. List of plugins that will be installed

6. Multibranch pipeline job's repository url. Please change this parameter to the url of your fork:

   multibranch_repository_url: "https://github.com/Alliedium-demo-test/springboot-api-rest-example.git"
   

Instructions to install Jenkins with ansible playbook

1. Clone repo:

git clone https://github.com/Alliedium/awesome-jenkins.git $HOME/awesome-jenkins

2. Installing Jenkins on remote host

• Copy $HOME/awesome-jenkins/inventory/example to $HOME/awesome-jenkins/inventory/my-jenkins folder.

  cp -r $HOME/awesome-jenkins/inventory/example $HOME/awesome-jenkins/inventory/my-jenkins
  

• Change the variables in the files $HOME/awesome-jenkins/inventory/my-jenkins/hosts.yml as you need

• Installing Jenkins on localhost does not require any changes to $HOME/awesome-jenkins/inventory/localhost/hosts.yml file.

3. Install ansible roles for Java, Git, and Jenkins using commands:

ansible-galaxy install -r $HOME/awesome-jenkins/requirements.yml

4. Run ansible playbook

This playbook contains multiple tasks that install git, java, Jenkins, as well as plugins, tools and pipelines in Jenkins. Using Ansible tags you can run a part of tasks. In our playbook we use 7 tags: always, step1, step2, step3, step4, step5 and step6. Use -t <tag_name> flag to specify desired tag. They form a hierarchy of tags from always to step6. In this hierarchy, each subsequent tag includes both the tasks marked by this tag as well as tasks relating to all preceding tags, e.g. if you run playbook with step3 tag, tasks tagged with always, step1, step2 and step3 will be run.

1. Before running tasks, check the list of tasks that will be executed using --list-tasks flag

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost --list-tasks

You will receive a list of all tasks. Using -t step2 when getting a list of tasks.

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost -t step2 --list-tasks

You will receive a list of tasks, tagged always, step1 and step2.

2. Run all the available tasks from playbook.yml playbook.

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost

3. Run without installing any plugins in Jenkins:

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost -t step1

4. Run with installing plugins in Jenkins:

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost -t step2

5. Use step3 tag - install python-jenkins

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost -t step3

6. step4 - Add maven tool

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost -t step4

7. step5 - Create and launch Jenkins pipeline job

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost -t step5

8. step6 - Create and launch Jenkins multibranch pipeline job

ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost -t step6

5. Checkup Jenkins

1. Go to the host specified in the $HOME/awesome-jenkins/inventory/localhost/hosts.yml file, open browser and check that Jenkins is available at http://localhost:8085/.
2. Login to Jenkins using the credentials.
3. You will see Jenkins dashboard. Open job.
4. The main branch will be run for the single pipeline job
5. Pull requests will be run for the multibranch pipeline job.

5. Ansible playbook local testing with molecule

The molecule configuration files are located in the $HOME/awesome-jenkins/molecule/default folder.

molecule.yml - this is the core file for Molecule. Used to define your testing steps, scenarios, dependencies, and other configuration options.

converge.yml - this is the playbook that Molecule will run to provision the targets for testing.

verify.yml - this is the playbook that is used to validate that the already converged instance state matches the desired state.

Before running the molecule command, go to awesome-jenkins project

cd $HOME/awesome-jenkins

• Run Ansible playbook test after which all previously created resources are deleted.

molecule test

The test command will run the entire scenario; creating, converging, verifying.

• Ansible playbook execution or role in target infrastructure, without testing. In this case, molecule will run the Ansible playbook in docker

molecule converge

• Run Ansible playbook test after the infrastructure has been converged using the "molecule converge" command. All previously created resources are not deleted

molecule verify

• Navigate to the target infrastructure - the docker container with the debug or check target

molecule login

• Reset molecule temporary folders.

molecule reset

• Finally, to clean up, we can run

molecule destroy

This removes the containers that we deployed and provisioned with create or converge. Putting us into a great place to start again.

6. Ansible playbook remote testing with GitHub Actions

The $HOME/awesome-jenkins/.github/workflows/ci.yml file describes the steps for GitHub Actions testing.

After creating or updating a pull request, tests are launched on the GitHub server and the results can be viewed here

Jenkins and GitHub integration

1. Set Resource Root URL

2. Creating your organization in GitHub

3. Creating GitHub apps

4. Generate and download SSH key

5. Install your app for repositories

6. Convert your generated key

openssl pkcs8 -topk8 -inform PEM -outform PEM -in key-in-your-downloads-folder.pem -out converted-github-app.pem -nocrypt

key-in-your-downloads-folder.pem - your generated SSH key

converted-github-app.pem - converted key

7. Fork your repo for testing purposes on GitHub

8. Create multibranch pipeline in Jenkins

9. On GitHub create new branch and pull request

After creating new pull request on Jenkins scan repository

10. Run your build

11. See build result on GitHub

Project:

As the example we used the following project

Job configuration:

Job configuration is set in the templates/job-config.xml.j2 - pipeline config and templates/multibranch-pipeline-config.xml.j2

GitHub Actions

Get familiar with GitHub workflows

1. Get familiar with GitHub actions functionality by following the examples from GitHub Actions examples
2. Fork the repositories to run examples with GitHub actions workflows

Run GitHub Actions

1. Fork repository on GitHub.
2. The pipeline workflow is described in the ci.yaml file in .github/workflows/ repository.
3. Navigate to Actions and enable them if needed.
4. The existing workflows can be run manually by following steps marked with the numbers 1-4 from the Figure below or triggered by pull request, see marks 5-7. -

Create Jenkins node on VM

Prerequisite:

Use VM with Rocky9.2. Use the script to create VM on Proxmox. Connect to your VM via ssh and enter password

ssh <username>@<vm_ip_address>

Next steps should be executed on your VM machine

1. Install git

   sudo dnf install git
   

2. Install java 17 and make it default

   sudo dnf install java-17-openjdk java-17-openjdk-devel
   java -version
   alternatives --list
   sudo alternatives --config java
   java -version
   

3. Create directory <agent_jenkins_dir> for Jenkins on your VM. In this directory the Jenkins associated files (settings, jobs) will be stored.

   mkdir <agent_jenkins_dir>
   

Do on your Jenkins controller machine

1. Navigate to

    cd /var/lib/jenkins
   

2. Create directory

   mkdir ./ssh
   

3. Change its owner

   sudo chown -R jenkins:jenkins /var/lib/jenkins/.ssh
   

4. Change user to jenkins

   sudo su jenkins
   

5. Create file known_hosts

   touch ./.ssh/known_hosts
   

6. Add VM to the known_hosts

   ssh-keyscan host <your_vm_ip> >> /var/lib/jenkins/.ssh/known_hosts
   

7. Go to your Jenkins. Open Manage Jenkins => Nodes

8. Configure your Jenkins agent-node:

Write Name (1), indicate Number of runners (2), Remote root directory should be the same as in the p.5 (3), add Labels that will trigger your agent (4); select type of Usage (5); choose launch method via SSH

Create credentials. Choose kind SSH Username with private key. Use username from your VM machine.

Enter directly private key from your VM. Use key for ssh access generated on VM creation.

Indicate IP address of your VM machine and add credentials for it; set Host Key Verification Strategy to Known hosts file verification strategy

Go to Advanced settings and set path to your JVM:

9. Disable agent on your Jenkins controller

Set number of runners to 0

Create Jenkins input job

1. On your host machine go to the directory with awesome-jenkins project

   cd $HOME/awesome-jenkins
   

2. Run step7 from ansible playbook - Create and launch Jenkins pipeline input job.

   ansible-playbook $HOME/awesome-jenkins/playbooks/create-job.yml -i $HOME/awesome-jenkins/inventory/localhost -t step7
   

3. Open Jenkins in your browser: 127.0.0.1:8085

4. Go to the pipeline-input-job and run the build. It will stop after some seconds.

5. Connect to your VM machine with Jenkins node

6. Go to the repository <agent_jenkins_dir> set for Jenkins

   cd <agent_jenkins_dir>
   

7. Explore it. Your may found installed tools in the tools directory

8. Your job workspaces is in the workspaces/pipeline-input-job directory

9. After exploring go back to Jenkins on your VM machine and input any name to continue the build.

Nektos Act

Install Nektos Act on Ubuntu Jammy

sudo apt install act

To install Nektos Act on other OS follow the instructions from section

1. View all jobs that are triggered by pull_request event act -l
2. View all jobs triggered by events, e.g. by pull_request

   act pull_request -l
   

or in the certain workflow file

act main.yaml -l

3. Run job with a specific name:

   act -j <job_name> 
   

4. Your may also explicitly indicate the workflow and job to run using flags --workflowand --job, respectively

 act --workflows .github/workflows/main.yml --verbose --job my-job

5. Use alternative environment to run your workflows

   act -P ubuntu-18.04=nektos/act-environments-ubuntu:18.04
   

References

Ansible roles used in playbook

1. Ansible galaxy Java role
2. Ansible galaxy Git role
3. Ansible galaxy Jenkins role

GitOps workflow

4. DevOps guide: pipeline challenges latest trends
5. Gitflow workflow
6. GitOps principles

Maven profiles

7. Maven profiles documentation
8. Maven profiles guide
9. Maven profiles. Examples

CI/CD

10. CI/CD basics
11. CI/CD basics
12. CI/CD fundamentals

CI/CD Tools

13. CI/CD tools review
14. Jenkins vs Jenkins X
15. Jenkins user documentation
16. Jenkins X
17. GitHub actions docs
18. Argo Workflows - The workflow engine for Kubernetes
19. Tekton - Cloud Native CI/CD
20. GitLab CI/CD docs

Jenkins pipelines

21. Jenkins pipelines
22. Jenkinsfile
23. Jenkins pipeline syntax
24. Pipeline stage view Jenkins plugin
25. Rendering User Content

Debugging Jenkins pipeline

26. Jenkins script console
27. Debugging Jenkins pipeline
28. Debugging tips
29. Testing framework for Jenkins pipelines

Static code analysis

30. Spotbugs maven plugin
31. PMD/CPD static analysis plugin
32. Checkstyle plugin
33. Code coverage tools
34. Comparison of findbugs, pmd and checkstyle
35. Spotbugs docs
36. Spotbugs maven plugin docs
37. Code style analysis reports publisher on Jenkins
38. Code coverage tools
39. Jacoco documentationJacoco examples
40. Jacoco examples
41. Code coverage reports publisher on Jenkins

Jenkins and GitHub integration

42. GitHub checks Jenkins plugin
43. Disable GitHub multibranch status Jenkins plugin
44. How to integrate Jenkins with GitHub
45. How to create GitHub App
46. Manage protected branches on GitHub
47. Setup GitHub checks in Jenkinsfile

Jenkins nodes

48. How to configure Jenkins master and slave nodes
49. Jenkins ssh-slaves plugin
50. Managing Jenkins nodes
51. Video: Jenkins node setup

Manage Jenkins users

52. Matrix authorization strategy Jenkins plugin
53. Role strategy Jenkins plugin
54. Login to Jenkins using your own, self-hosted or public openid connect server
55. How to reset Jenkins admin password

GitHub Actions

56. GitHub Actions workflows
57. GitHub Actions workflows basics, examples and a quick tutorial

Act

58. Act
59. GitHub Actions on your local machine
60. Debug GitHub Actions locally with act