Giters

ataliba / gitlab-ci-pipelines-exporter

Prometheus / OpenMetrics exporter for GitLab CI pipelines insights

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

🦊 gitlab-ci-pipelines-exporter

GoDoc Go Report Card Docker Pulls Build Status Coverage Status

gitlab-ci-pipelines-exporter allows you to monitor your GitLab CI pipelines with Prometheus or any monitoring solution supporting the OpenMetrics format.

TL:DR

Here is a Grafana dashboard I have been able to craft, using those metrics:

grafana_dashboard

If you are interested into trying it out, have a look into the example/ folder which contains documentation to provision test version of the exporter, prometheus and also grafana in ~5min using docker-compose

Install

Go

~$ go get -u github.com/mvisonneau/gitlab-ci-pipelines-exporter

Homebrew

~$ brew install mvisonneau/tap/gitlab-ci-pipelines-exporter

Docker

~$ docker run -it --rm mvisonneau/gitlab-ci-pipelines-exporter

Scoop

~$ scoop bucket add https://github.com/mvisonneau/scoops
~$ scoop install gitlab-ci-pipelines-exporter

Binaries, DEB and RPM packages

Have a look onto the latest release page to pick your flavor and version. Here is an helper to fetch the most recent one:

~$ export GCPE_VERSION=$(curl -s "https://api.github.com/repos/mvisonneau/gitlab-ci-pipelines-exporter/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
# Binary (eg: linux/amd64)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_amd64.tar.gz
~$ tar zxvf gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_amd64.tar.gz -C /usr/local/bin

# DEB package (eg: linux/386)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_386.deb
~$ dpkg -i gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_386.deb

# RPM package (eg: linux/arm64)
~$ wget https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/releases/download/${GCPE_VERSION}/gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_arm64.rpm
~$ rpm -ivh gitlab-ci-pipelines-exporter_${GCPE_VERSION}_linux_arm64.rpm

HELM

If you want to make it run on kubernetes, there is a helm chart available for this purpose.

You can check chart/values.yml for configuration options.

# Clone the repository locally
~$ git clone git@github.com:mvisonneau/gitlab-ci-pipelines-exporter.git

# Configure a minimal configuration for the exporter
~$ cat <<EOF > values.yml
config:
  gitlab:
    url: https://gitlab.example.com
    # You can also configure the token using --gitlab-token
    # or the $GCPE_GITLAB_TOKEN environment variable
    token: xrN14n9-ywvAFxxxxxx
  projects:
    - name: foo/project
EOF

# Release the chart on your Kubernetes cluster
~$ helm upgrade -i gitlab-ci-pipelines-exporter ./chart -f values.yml

Configuration syntax

The complete configuration syntax is maintained here.

Quickstart

# Write a minimal config file somewhere on disk
~$ cat <<EOF > $(pwd)/config.yml
config:
  gitlab:
    url: https://gitlab.example.com
    # You can also configure the token using --gitlab-token
    # or the $GCPE_GITLAB_TOKEN environment variable
    token: <your_token>
  projects:
    - name: foo/project
    - name: bar/project
  wildcards:
    - owner:
        name: foo
        kind: group
EOF

# If you have installed the binary
~$ gitlab-ci-pipelines-exporter --config /etc/config.yml

# Otherwise if you have docker available, it is as easy as :
~$ docker run -it --rm \
   --name gitlab-ci-pipelines-exporter \
   -v $(pwd)/config.yml:/etc/config.yml \
   -p 8080:8080 \
   mvisonneau/gitlab-ci-pipelines-exporter:latest \
   --config /etc/config.yml

You should then be able to see the following logs

INFO[0000] starting exporter                             gitlab-endpoint="https://gitlab.com" on-init-fetch-refs-from-pipelines=true polling-pipelines-every=60s polling-projects-every=15s polling-refs-every=10s polling-workers=2 rate-limit=10rps
INFO[0000] configured wildcards                          count=1
INFO[0000] found new project                             project-name=foo/project wildcard-archived=false wildcard-owner-include-subgroups=false wildcard-owner-kind=group wildcard-owner-name=foo wildcard-search=
INFO[0000] found new project                             project-name=foo/bar wildcard-archived=false wildcard-owner-include-subgroups=false wildcard-owner-kind=group wildcard-owner-name=foo wildcard-search=
INFO[0000] configured projects                           count=3
INFO[0000] started, now serving requests                 listen-address=":8080"
INFO[0000] found project refs                            project-path-with-namespace=foo/project project-ref=master
INFO[0000] found project refs                            project-path-with-namespace=bar/project project-ref=master
INFO[0000] found project refs                            project-path-with-namespace=foo/bar project-ref=master

And this is an example of the metrics you should expect to retrieve

~$ curl -s localhost:8080/metrics | grep gitlab_ci
# HELP gitlab_ci_pipeline_last_run_duration_seconds Duration of last pipeline run
# TYPE gitlab_ci_pipeline_last_run_duration_seconds gauge
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/project",ref="dev",topics="",variables=""} 81
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/project",ref="master",topics="",variables=""} 420
gitlab_ci_pipeline_last_run_duration_seconds{project="bar/project",ref="master",topics="",variables=""} 334
gitlab_ci_pipeline_last_run_duration_seconds{project="foo/bar",ref="master",topics="",variables="FOO:BAR"} 55
# HELP gitlab_ci_pipeline_last_run_id ID of the most recent pipeline
# TYPE gitlab_ci_pipeline_last_run_id gauge
gitlab_ci_pipeline_last_run_id{project="foo/project",ref="dev",topics="",variables=""} 4.0059611e+07
gitlab_ci_pipeline_last_run_id{project="foo/project",ref="master",topics="",variables=""} 1.25351545e+08
gitlab_ci_pipeline_last_run_id{project="bar/project",ref="master",topics="",variables=""} 1.33308085e+08
gitlab_ci_pipeline_last_run_id{project="foo/bar",ref="master",topics="",variables="FOO:BAR"} 1.40420947e+08
# HELP gitlab_ci_pipeline_last_run_status Status of the most recent pipeline
# TYPE gitlab_ci_pipeline_last_run_status gauge
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="failed",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="dev",status="success",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="master",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="master",status="failed",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="master",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="master",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="master",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="master",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="foo/project",ref="master",status="success",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="master",status="canceled",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="master",status="failed",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="master",status="manual",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="master",status="pending",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="master",status="running",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="master",status="skipped",topics="",variables=""} 0
gitlab_ci_pipeline_last_run_status{project="bar/project",ref="master",status="success",topics="",variables=""} 1
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="master",status="canceled",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="master",status="failed",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="master",status="manual",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="master",status="pending",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="master",status="running",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="master",status="skipped",topics="",variables="FOO:BAR"} 0
gitlab_ci_pipeline_last_run_status{project="foo/bar",ref="master",status="success",topics="",variables="FOO:BAR"} 1
# HELP gitlab_ci_pipeline_run_count GitLab CI pipeline run count
# TYPE gitlab_ci_pipeline_run_count counter
gitlab_ci_pipeline_run_count{project="foo/project",ref="dev",topics="",variables=""} 1
gitlab_ci_pipeline_run_count{project="foo/project",ref="master",topics="",variables=""} 2
gitlab_ci_pipeline_run_count{project="bar/project",ref="master",topics="",variables=""} 1
gitlab_ci_pipeline_run_count{project="foo/bar",ref="master",topics="",variables="FOO:BAR"} 2
# HELP gitlab_ci_pipeline_time_since_last_run_seconds Elapsed time since most recent GitLab CI pipeline run.
# TYPE gitlab_ci_pipeline_time_since_last_run_seconds gauge
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/project",ref="dev",topics="",variables=""} 4.3368877e+07
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/project",ref="master",topics="",variables=""} 4.151883e+06
gitlab_ci_pipeline_time_since_last_run_seconds{project="bar/project",ref="master",topics="",variables=""} 1.907042e+06
gitlab_ci_pipeline_time_since_last_run_seconds{project="foo/bar",ref="master",topics="",variables="FOO:BAR"} 65456

If fetch_pipeline_job_metrics is enabled, expect additional metrics:

~$ curl -s http://localhost:8080/metrics | grep job
# HELP gitlab_ci_pipeline_job_run_count GitLab CI pipeline job run count
# TYPE gitlab_ci_pipeline_job_run_count counter
gitlab_ci_pipeline_job_run_count{job="build",project="bar/project",ref="master",stage="build",topics=""} 1
gitlab_ci_pipeline_job_run_count{job="test",project="bar/project",ref="master",stage="build",topics=""} 1
# HELP gitlab_ci_pipeline_last_job_run_artifact_size Filesize of the most recent job artifacts
# TYPE gitlab_ci_pipeline_last_job_run_artifact_size gauge
gitlab_ci_pipeline_last_job_run_artifact_size{job="build",project="bar/project",ref="master",stage="build",topics=""} 1.3793677e+07
gitlab_ci_pipeline_last_job_run_artifact_size{job="test",project="bar/project",ref="master",stage="build",topics=""} 257737
# HELP gitlab_ci_pipeline_last_job_run_duration_seconds Duration of last job run
# TYPE gitlab_ci_pipeline_last_job_run_duration_seconds gauge
gitlab_ci_pipeline_last_job_run_duration_seconds{job="build",project="bar/project",ref="master",stage="build",topics=""} 826.064469
gitlab_ci_pipeline_last_job_run_duration_seconds{job="test",project="bar/project",ref="master",stage="test",topics=""} 519.873374
# HELP gitlab_ci_pipeline_last_job_run_status Status of the most recent job
# TYPE gitlab_ci_pipeline_last_job_run_status gauge
gitlab_ci_pipeline_last_job_run_status{job="build",project="bar/project",ref="master",stage="build",status="failed",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="build",project="bar/project",ref="master",stage="build",status="running",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="build",project="bar/project",ref="master",stage="build",status="success",topics=""} 1
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="failed",topics=""} 1
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="running",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="success",topics=""} 0 
# HELP gitlab_ci_pipeline_time_since_last_job_run_seconds Elapsed time since most recent GitLab CI job run.
# TYPE gitlab_ci_pipeline_time_since_last_job_run_seconds gauge
gitlab_ci_pipeline_time_since_last_job_run_seconds{job="build",project="bar/project",ref="master",stage="build",topics=""} 52422
gitlab_ci_pipeline_time_since_last_job_run_seconds{job="test",project="bar/project",ref="master",stage="test",topics=""} 1.0260727e+07

If the number of metrics generated by fetching jobs becomes a problem, you can enable output_sparse_status_metrics on a global, per-project or per-wildcard basis. When enabled, only labels matching the previous pipeline or job status will be submitted (with value 1) rather than all label combinations submitted but with 0 value where the status does not match the previous run, for example:

# output_sparse_status_metrics: false
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="canceled",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="failed",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="manual",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="pending",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="running",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="skipped",topics=""} 0
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="success",topics=""} 1

# output_sparse_status_metrics: true
gitlab_ci_pipeline_last_job_run_status{job="test",project="bar/project",ref="master",stage="test",status="success",topics=""} 1

Usage

~$ gitlab-ci-pipelines-exporter --help
NAME:
   gitlab-ci-pipelines-exporter - Export metrics about GitLab CI pipelines statuses

USAGE:
   gitlab-ci-pipelines-exporter [global options] command [command options] [arguments...]

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --config file, -c file                          config file (default: "~/.gitlab-ci-pipelines-exporter.yml") [$GCPE_CONFIG]
   --enable-pprof                                  Enable profiling endpoints at /debug/pprof [$GCPE_ENABLE_PPROF]
   --gitlab-token token                            GitLab access token. Can be use to override the gitlab token in config file [$GCPE_GITLAB_TOKEN]
   --listen-address address:port, -l address:port  listen-address address:port (default: ":8080") [$GCPE_LISTEN_ADDRESS]
   --log-level level                               log level (debug,info,warn,fatal,panic) (default: "info") [$GCPE_LOG_LEVEL]
   --log-format format                             log format (json,text) (default: "text") [$GCPE_LOG_FORMAT]
   --help, -h                                      show help

Develop / Test

If you use docker, you can easily get started using :

~$ make dev-env
# You should then be able to use go commands to work onto the project, eg:
~docker$ make fmt
~docker$ gitlab-ci-pipelines-exporter

Contribute

Contributions are more than welcome! Feel free to submit a PR.

About

Prometheus / OpenMetrics exporter for GitLab CI pipelines insights

License:Apache License 2.0

Languages

Language:Go 93.1%Language:Makefile 4.8%Language:Smarty 1.4%Language:Dockerfile 0.7%