bosh-prometheus / bosh_exporter

BOSH Prometheus Exporter

BOSH Prometheus Exporter

A Prometheus exporter for BOSH metrics. Please refer to the FAQ for general questions about this exporter.

Architecture overview

Installation

Binaries

Download the already existing binaries for your platform:

$ ./bosh_exporter <flags>

From source

Using the standard go install (you must have Go already installed in your local machine):

$ go install github.com/bosh-prometheus/bosh_exporter
$ bosh_exporter <flags>

Docker

To run the bosh exporter as a Docker container, run:

$ docker run -p 9190:9190 boshprometheus/bosh-exporter <flags>

Cloud Foundry

The exporter can be deployed to an already existing Cloud Foundry environment:

$ git clone https://github.com/bosh-prometheus/bosh_exporter.git
$ cd bosh_exporter

Modify the included application manifest file to include your BOSH properties. Then you can push the exporter to your Cloud Foundry environment:

$ cf push

BOSH

This exporter can be deployed using the Prometheus BOSH Release.

Usage

Flags

Flag / Environment Variable	Required	Default	Description
bosh.url BOSH_EXPORTER_BOSH_URL	Yes		BOSH URL
bosh.username BOSH_EXPORTER_BOSH_USERNAME	[1]		BOSH Username
bosh.password BOSH_EXPORTER_BOSH_PASSWORD	[1]		BOSH Password
bosh.uaa.client-id BOSH_EXPORTER_BOSH_UAA_CLIENT_ID	[1]		BOSH UAA Client ID
bosh.uaa.client-secret BOSH_EXPORTER_BOSH_UAA_CLIENT_SECRET	[1]		BOSH UAA Client Secret
bosh.log-level BOSH_EXPORTER_BOSH_LOG_LEVEL	No	ERROR	BOSH Log Level (DEBUG, INFO, WARN, ERROR, NONE)
bosh.ca-cert-file BOSH_EXPORTER_BOSH_CA_CERT_FILE	Yes		BOSH CA Certificate file
filter.deployments BOSH_EXPORTER_FILTER_DEPLOYMENTS	No		Comma separated deployments to filter
filter.azs BOSH_EXPORTER_FILTER_AZS	No		Comma separated AZs to filter
filter.collectors BOSH_EXPORTER_FILTER_COLLECTORS	No		Comma separated collectors to filter. If not set, all collectors will be enabled (Deployments, Jobs, ServiceDiscovery)
filter.cidrs BOSH_EXPORTER_FILTER_CIDRS	No	0.0.0.0/0	Comma separated CIDR to filter instance IPs
metrics.namespace BOSH_EXPORTER_METRICS_NAMESPACE	No	bosh	Metrics Namespace
metrics.environment BOSH_EXPORTER_METRICS_ENVIRONMENT	Yes		Environment label to be attached to metrics
sd.filename BOSH_EXPORTER_SD_FILENAME	No	bosh_target_groups.json	Full path to the Service Discovery output file
sd.processes_regexp BOSH_EXPORTER_SD_PROCESSES_REGEXP	No		Regexp to filter Service Discovery processes names
web.listen-address BOSH_EXPORTER_WEB_LISTEN_ADDRESS	No	:9190	Address to listen on for web interface and telemetry
web.telemetry-path BOSH_EXPORTER_WEB_TELEMETRY_PATH	No	/metrics	Path under which to expose Prometheus metrics
web.auth.username BOSH_EXPORTER_WEB_AUTH_USERNAME	No		Username for web interface basic auth
web.auth.password BOSH_EXPORTER_WEB_AUTH_PASSWORD	No		Password for web interface basic auth
web.tls.cert_file BOSH_EXPORTER_WEB_TLS_CERTFILE	No		Path to a file that contains the TLS certificate (PEM format). If the certificate is signed by a certificate authority, the file should be the concatenation of the server's certificate, any intermediates, and the CA's certificate
web.tls.key_file BOSH_EXPORTER_WEB_TLS_KEYFILE	No		Path to a file that contains the TLS private key (PEM format)

[1] When BOSH delegates user managament to UAA, either bosh.username and bosh.password or bosh.uaa.client-id and bosh.uaa.client-secret flags may be used; otherwise bosh.username and bosh.password will be required. When using UAA and the bosh.username and bosh.password authentication method, tokens are not refreshed, so after a period of time the exporter will be unable to communicate with the BOSH API, so use this method only when testing the exporter. For production, it is recommended to use the bosh.uaa.client-id and bosh.uaa.client-secret authentication method.

Metrics

The exporter returns the following metrics:

Metric	Description	Labels
metrics.namespace_scrapes_total	Total number of times BOSH was scraped for metrics	environment, bosh_name, bosh_uuid
metrics.namespace_scrape_errors_total	Total number of times an error occured scraping BOSH	environment, bosh_name, bosh_uuid
metrics.namespace_last_scrape_error	Whether the last scrape of metrics from BOSH resulted in an error (1 for error, 0 for success)	environment, bosh_name, bosh_uuid
metrics.namespace_last_scrape_timestamp	Number of seconds since 1970 since last scrape from BOSH	environment, bosh_name, bosh_uuid
metrics.namespace_last_scrape_duration_seconds	Duration of the last scrape from BOSH	environment, bosh_name, bosh_uuid

The exporter returns the following Deployments metrics:

Metric	Description	Labels
metrics.namespace_deployment_release_info	Labeled BOSH Deployment Release Info with a constant 1 value	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_release_name, bosh_release_version
metrics.namespace_deployment_release_job_info	Labeled BOSH Deployment Release Job Info with a constant 1 value	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_release_name, bosh_release_version, bosh_release_job_name
metrics.namespace_deployment_release_package_info	Labeled BOSH Deployment Release Package Info with a constant 1 value	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_release_name, bosh_release_version, bosh_release_package_name
metrics.namespace_deployment_stemcell_info	Labeled BOSH Deployment Stemcell Info with a constant 1 value	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_stemcell_name, bosh_stemcell_version, bosh_stemcell_os_name
metrics.namespace_deployment_instances	Number of instances in the deployment	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_vm_type
metrics.namespace_last_deployments_scrape_timestamp	Number of seconds since 1970 since last scrape of Deployments metrics from BOSH	environment, bosh_name, bosh_uuid
metrics.namespace_last_deployments_scrape_duration_seconds	Duration of the last scrape of Deployments metrics from BOSH	environment, bosh_name, bosh_uuid

The exporter returns the following Jobs metrics:

Metric	Description	Labels
metrics.namespace_job_healthy	BOSH Job Healthy (1 for healthy, 0 for unhealthy)	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_load_avg01	BOSH Job Load avg01	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_load_avg05	BOSH Job Load avg05	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_load_avg15	BOSH Job Load avg15	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_cpu_sys	BOSH Job CPU System	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_cpu_user	BOSH Job CPU User	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_cpu_wait	BOSH Job CPU Wait	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_mem_kb	BOSH Job Memory KB	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_mem_percent	BOSH Job Memory Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_swap_kb	BOSH Job Swap KB	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_swap_percent	BOSH Job Swap Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_system_disk_inode_percent	BOSH Job System Disk Inode Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_system_disk_percent	BOSH Job System Disk Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_ephemeral_disk_inode_percent	BOSH Job Ephemeral Disk Inode Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_ephemeral_disk_percent	BOSH Job Ephemeral Disk Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_persistent_disk_inode_percent	BOSH Job Persistent Disk Inode Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_persistent_disk_percent	BOSH Job Persistent Disk Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip
metrics.namespace_job_process_info	BOSH Job Process Info with a constant '1' value. Release can be found only if process name is the same as release job name.	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip, bosh_job_process_name, bosh_job_process_release_name, bosh_job_process_release_version
metrics.namespace_job_process_healthy	BOSH Job Process Healthy (1 for healthy, 0 for unhealthy)	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip, bosh_job_process_name
metrics.namespace_job_process_uptime_seconds	BOSH Job Process Uptime in seconds	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip, bosh_job_process_name
metrics.namespace_job_process_cpu_total	BOSH Job Process CPU Total	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip, bosh_job_process_name
metrics.namespace_job_process_mem_kb	BOSH Job Process Memory KB	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip, bosh_job_process_name
metrics.namespace_job_process_mem_percent	BOSH Job Process Memory Percent	environment, bosh_name, bosh_uuid, bosh_deployment, bosh_job_name, bosh_job_id, bosh_job_index, bosh_job_az, bosh_job_ip, bosh_job_process_name
metrics.namespace_last_jobs_scrape_timestamp	Number of seconds since 1970 since last scrape of Job metrics from BOSH	environment, bosh_name, bosh_uuid
metrics.namespace_last_jobs_scrape_duration_seconds	Duration of the last scrape of Job metrics from BOSH	environment, bosh_name, bosh_uuid

The exporter returns the following ServiceDiscovery metrics:

Metric	Description	Labels
metrics.namespace_last_service_discovery_scrape_timestamp	Number of seconds since 1970 since last scrape of Service Discovery from BOSH	environment, bosh_name, bosh_uuid
metrics.namespace_last_service_discovery_scrape_duration_seconds	Duration of the last scrape of Service Discovery from BOSH	environment, bosh_name, bosh_uuid

Service Discovery

If the ServiceDiscovery collector is enabled, the exporter will write a json file at the sd.filename location containing a list of static configs that can be used with the Prometheus file-based service discovery mechanism:

[
  {
    "targets": [
      "10.244.0.12"
    ],
    "labels": {
      "__meta_bosh_job_process_name": "bosh_exporter"
    }
  },
  {
    "targets": [
      "10.244.0.11",
      "10.244.0.12",
      "10.244.0.13",
      "10.244.0.14"
    ],
    "labels": {
      "__meta_bosh_deployment": "deployment1",
      "__meta_bosh_deployment_releases": "exporters_release:1.0,other_release:0.2",
      "__meta_bosh_job_process_name": "node_exporter",
      "__meta_bosh_job_process_release":"exporters_release:1.0"
    }
  }
]

[!NOTE] __meta_bosh_job_process_release has the same value as the labels: bosh_job_process_release_name:bosh_job_process_release_version. The process release can be found only if the process name (label bosh_job_process_name) is the same as BOSH release job name. BOSH release job name is not the same as the label bosh_job_name which is the instance group name (BOSH deployment manifest).

The list of targets can be filtered using the sd.processes_regexp flag.

Prometheus file-based service discovery example:

- job_name: node_exporter
  metrics_path: /metrics
  scheme: http
  file_sd_configs:
    - files:
        - /var/vcap/store/bosh_exporter/bosh_target_groups.json
  relabel_configs:
    - source_labels: [ __meta_bosh_job_process_name ]
      regex: 'node_exporter'
      action: keep
    - source_labels: [ __meta_bosh_deployment_releases ]
      regex: '.*exporters_release:1\..*'
      action: keep

Filtering IPs

Available instance IPs can be filtered using the filter.cidrs flag.

The first IP that matches a CIDR is used as target. CIDRs are tested in the order specified by the comma-seperated list. The instance is dropped if no IP is included in any of the CIDRs.

Contributing

Refer to the contributing guidelines.

License

Apache License 2.0, see LICENSE.