prom-ngxlog-exporter

Export request metrics from Nginx access log

Build & Run

clone the project then build:

go build -o prom-ngxlog-exporter

write a config file then run:

./ prom-ngxlog-exporter -config-file /path/to/config.yml

Configuration

Example

listen:
  port: 4040
  address: 0.0.0.0

sentry:
  dsn: ""
  debug: true

namespaces:
  - name: api
    labels:
      - method
      - status
      - path

    format: "$remote_addr - $remote_user [$time_local] \"$request\" $status r:$request_length s:$bytes_sent($gzip_ratio) \"$http_referer\" \"$http_user_agent\" ($upstream_addr) $request_time $upstream_response_time $pipe"

    default_relabels:
      - name: method
        source: request
        split: 1
      - name: status
        source: status
      - name: path
        source: request
        split: 2
        preprocesses:
          - regexp: "^(.*)\\?.*$"
            replacement: "$1"
        exact_matches:
          - match: "/api/happen"
            replacement: "/api/happen"
        regex_matches:
          - regexp: "^/static/(.*)$"
            replacement: "/static/<filename>"
          - regexp: "^.*$"
            replacement: ""

    histogram_buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]

    apps:
      - name: app1
        source_files:
          - "./dev_data/access.log"

Concepts

Namespace: a group of metric, take name as metric name prefix
Label: label of metrics, in a namespace, the set of labels is fixed.
Nginx Log Format: log_format defined in Nginx config file, see nginx docs for detail
Relabel: convert value from nginx log to value that set to Label, It may need several step to extract that value.
HistogramBuckets: see prometheus docs for detail
App: Application tha share a namespace of metrics, it tells exporter where to find the accesslog files and it can have its specific Relabel config.

Relabel Procedure

+-------------------------+
|      source value       | -+
+-------------------------+  |
  |                          |
  | split > 0                |
  v                          |
+-------------------------+  |
|     split by space      |  |
|    get the Nth part     |  | split = 0
+-------------------------+  |
  |                          |
  |                          |
  v                          |
+-------------------------+  |
|       preprocess        | <+
+-------------------------+
  |
  | result
  v
+-------------------------+
| exact match and replace | -+
+-------------------------+  |
  |                          |
  | not matched              |
  v                          |
+-------------------------+  |
| regex match and replace |  | matched
+-------------------------+  |
  |                          |
  |                          |
  v                          |
+-------------------------+  |
|      got the value      | <+
+-------------------------+

Please read config schema from config.go for unmetioned config params, it's pretty straight forward.

euclidr / prom-ngxlog-exporter