moosh3 / jenkinsfile-runner

A command line tool to run Jenkinsfile

Jenkinsfile Runner

Introduction

Jenkinsfile Runner is an experiment to package Jenkins pipeline execution as a command line tool. The intended use cases include:

• Assist editing and testing Jenkinsfile locally
• Use Jenkins in Function-as-a-Service context
• Integration test shared libraries

Please report issues on Jira

Usage

Jenkinsfile Runner is a command line tool you can invoke from any project directory containing a Jenkinsfile. It's a standalone executable but requires java in your PATH so it can run a Jenkins headless master.

Jenkinsfile Runner will:

• download latest Jenkins LTS
• setup a temporary JENKINS_HOME directory to run a headless jenkins for your Pipeline
• install plugins as defined by a plugins.txt file in project directory. If none are set, it will install the latest workflow-aggregator
• optionally create a .secret directory with secrets from a secrets.gpg GPG-encrypted file. Read more in sensitive data
• run Jenkins master headlessly with a custom plugin installed to immediately run a single job based on local Jenkinsfile, then shutdown on completion.

Docker image

For your convenience you can use the jenkins/jenkinsfile-runner docker image:

$ docker run -it -v $(pwd):/workspace jenkins/jenkinsfile-runner

To avoid repeated downloads we suggest you define volumes for the download caches:

$ docker run -it -v $(pwd):/workspace                         \
    -v jenkinsfile-runner-cache:/var/jenkinsfile-runner-cache \
    jenkins/jenkinsfile-runner

Advanced docker image use and recommendations

We recommend that you make the transient JENKINS_HOME a temporary volume so that you don't consume all disk space with repeated builds:

$ docker run -it -v $(pwd):/workspace                         \
    --tmpfs /var/jenkinsfile-runner                           \
    -v jenkinsfile-runner-cache:/var/jenkinsfile-runner-cache \
    jenkins/jenkinsfile-runner

Alternatively you might want to reuse this folder between runs. This can be useful to collect build output and diagnose jenkinsfile-runner issues:

$ docker run -it -v $(pwd):/workspace                         \
    -v $(pwd)/jenkinsfile-runner:/var/jenkinsfile-runner      \
    -v jenkinsfile-runner-cache:/var/jenkinsfile-runner-cache \
    jenkins/jenkinsfile-runner

Use with knative

see sample here

Build

Currently there's no released distribution, so you must first build the code by yourself. You need Maven and Go SDK installed. We also use dep for golang dependencies and vendor folder management.

You can either build with Dockerfile:

docker build -t jenkins/jenkinsfile-runner . 

or if you have adequate tools installed:

mvn install
dep ensure
go install 

Assuming you have $GOBIN well set and declared in your PATH, you now have command line jenkinsfile-runner available to run from any directory containing a Jenkinsfile.

Jenkins core version

You can choose the version of jenkins to run by passing a -version argument. Default value latest is an alias for "latest LTS release" which is checked once a day. The requested jenkins.war is downloaded to download cache before jenkins is started from local .jenkinsfile-runner JENKINS_HOME

Plugins

You can include a plugins.txt file with plugins required to run your pipeline. Jenkinsfile-runner will download those plugins and dependencies into download cache and setup .jenkinsfile-runner JENKINS_HOME accordingly.

plugins.txt file is a plain text format with a plugin per line, as <shortname>:<version>

If you use a custom update site to host your own plugins, you can suffix plugins with optional @updatesiteId and pass -site argument using id=url format to Jenkinsfile-runner.

Note: once JENKINS-34002 is implemented we will also pick required dependencies from Jenkinsfile.

Download Cache

As Jenkinsfile-runner downloads jenkins.war and plugins on-demand, it relies on a download cache. Default location is $HOME/.jenkinsfile-runner but you can override using the -cache option.

Master Configuration

For non-trivial pipelines you'll need some way to configure Jenkins master, for example to declare some credentials referred by ID in your Jenkinsfile. Jenkinsfile-runner relies on Configuration-as-Code for this purpose. You only need to provide a jenkins.yaml file alongside your Jenkinsfile to have the transient jenkins master configured accordingly.

Sensitive data

Your pipeline might require some sensitive data that you don't want to store in plain text in your SCM. For this purpose you can define them in a secrets file in java properties format, and encrypt it with GPG allowing your teammates to decrypt this file:

gpg --encrypt --recipient [e-mail] --recipient [e-mail] (...) secrets

This will produce an encrypted secrets.gpg file you can safely commit to SCM. Never commit the initial secrets file (once encrypted, you can delete it). We highly recommend you add it to your .gitignore to avoid mistakes.

We suggest you manage teammates' gpg public keys in your SCM as a keyring to ensure everybody in the team knows each other GPG identity.

When such a secrets.gpg file exists alongside your Jenkinsfile (or is set by -secrets option), Jenkinsfile-runner will create a .secrets directory in .jenkinsfile-runner JENKINS_HOME following the Docker secrets layout, fully compatible with Configuration-as-Code so you can define your configuration with secret replacements:

jenkins:
    somethingSecret: ${NEVER_STORED_IN_PLAIN_TEXT}

To enforce security, this folder is deleted when Jenkinsfile-runner completes.

Implementation

Jenkinsfile Runner is not a re-implementation of the Pipeline execution library used by Jenkins. This would be a huge effort, and anyway all the power of Pipeline comes from various Jenkins plugins to provide DSL keywords. Jenkinsfile-runner is actually booting a (headless) Jenkins master, create a one-shot job to run once the Jenkinsfile script in your local directory, sending build log to stdout.

Jenkinsfile Runner is a go program to setup a transient, headless Jenkins master. It automatically install the set of plugins defined in a plugins.txt file stored aside your Jenkinsfile. It also will apply a jenkins.yaml Configuration-as-Code file if present, which you can rely on to define the expected Jenkins configuration required for your pipeline script.

Within this transient jenkins master, a dedicated plugin is injected and will automatically create a job to run your Jenkinsfile from local filesystem. This plugin also hacks Jenkins to redirect the build log to stdout, and shut down master when job is completed.