criblio / elastic-cribl-monitoring

Cribl Monitoring in ElasticSearch

Introduction

If you want to look into the health of your Cribl environments, you are usually pretty set with the default Monitoring functionality that Cribl offers out of the box.

But Cribl also lets you send Internal Metrics/Logs to external monitoring tools, so you can take advantage of the advanced searching, visualization and alerting capabilities of Elasticsearch.

How It Works

We prepare Elasticsearch before sending the data with the appropriate mappings.

Metrics are saved in Time Series Data Stream (TSDS).

Logs are saved in Data Streams.

We connect the Cribl Internal Metrics and Logs sources to an Elasticsearch destination using QuickConnect.

We select a custom pipeline as a pre-processing pipeline for logs before sending to Elasticsearch.

We have created some dashboards and alerts for you, from the perspective of a Cribl Support Engineer that we can import into Kibana.

Prerequisites

• Cribl Stream or Edge Version 3.3+
• Elasticsearch and Kibana Version 8.8+

Getting started

We provide 2 options to set you up quickly.

Option 1: You can prepare Elasticsearch and Cribl automatically using a bootstrap.sh script. This is ideal for larger environments, to prevent having to configure many worker groups at the same time.

Option 2: You can quickly configure Elasticsearch and Cribl manually as well.

For both options you will then have to manually commit and deploy Cribl's changes and import the Dashboards into Kibana.

You can then also choose to collect logs from the Leader separately as an optional step.

Option 1 - Bootstrap Script

This script will run the manual steps for every Stream Worker Group or Edge Fleet, which could save some time.

Prerequisites for the script are: Bash, jq, curl

Files to run the script are in the Github repository.

ℹ️ The following variables will need to be updated in the .env file:

Variable(s)	Description
ES_ELASTIC_URL	The endpoint to your Elasticsearch deployment (e.g. https://elasticsearch.domain.com:9200)
ES_KIBANA_URL	The endpoint to your Kibana instance (e.g. https://kibana.domain.com:5601)
ES_CRIBL_URL	The endpoint to your Cribl Leader node or your Cribl.Cloud URL . (e.g. https://cribl.domain.com:9000 or https://main-happy-margulis-topxery.cribl.cloud)
CRIBL_USER, CRIBL_PASSWORD	On-Premise specific options. Your typical Username and Password combination for Authenticating through the Cribl UI.
CRIBL_CLIENT_ID , CRIBL_CLIENT_SECRET	Cribl.Cloud specific options. The Client ID and secret created by following instructions here.
ES_ELASTIC_USER, ES_ELASTIC_PASSWORD	Elasticsearch username and password to authenticate Elasticsearch API calls with.
ES_ELASTIC_BEARER_TOKEN	(optional) If you want to use token-based authentication service to authenticate Elasticsearch API calls (limited to service-accounts and token-service)
ES_ELASTIC_PASSWORD	The password for your Cribl Writer user
ES_CRIBL_WORKERGROUP_NAME	The Stream groups or Edge Fleet names for which to apply the bootstrap script to (e.g. ("default" "defaultHybrid") for multiple or ("defaultHybrid") for a single group/fleet)
ES_CRIBL_CUSTOM_IDENTIFIER	(optional) Any value for a custom identifier that you want to add, such as a data centre name. Will be added as a field to the events.
ES_CRIBL_ELASTIC_OUTPUT_ID (optional)	The ID of the Elasticsearch output that will be created/updated. This id will be the same across all your worker groups.

ℹ️ The bootstrap script has some debugging options which can be enabled by uncommenting lines at the top of the file. This will create a log file with very verbose information in the same working directory as the script.

Run the bootstrap script:

1. Make bootstrap.sh executable (chmod +x bootstrap.sh)
2. Run: ./bootstrap.sh
3. Commit and deploy from Cribl to make sure the changes take effect
4. Continue to the Importing Saved Objects section.

Common issues while running the script:
- Be sure to commit and deploy, for the changes in Stream to take effect!
- If the ES_*_URL endpoints require a TLS connection while the Certificate Authorities certificates are not in your local trust stores, you will have to adjust the curl commands to use the -k flag

Option 2 - Manual Configuration

Here are the manual steps to configure Elasticsearch first and then Cribl Stream. The instructions here are for Stream, but will work for Edge fleets too.

1. Elasticsearch Configuration

Please copy & paste and then execute the appropriate commands into your Developer Console one by one in the order that they are displayed:

ℹ️ Please update the password for the user and/or the permissions of the corresponding user role.

1. Roles

PUT _security/role/cribl_writer
{
  "cluster": ["manage_index_templates", "monitor", "manage_ilm"],
  "indices": [
    {
      "names": [ "metrics-cribl-internal", "logs-cribl-internal" ],
      "privileges": ["write","create","create_index","manage","manage_ilm"]
    }
  ]
}

2. Users

PUT _security/user/cribl_internal
{
  "password" : "cribl-test-password",
  "roles" : [ "cribl_writer"],
  "full_name" : "Internal Cribl User"
}

3. Component Templates

Metrics:

Endpoint: PUT _component_template/metrics-cribl-internal

Payload:

elastic-cribl-monitoring/ecm_metrics_component_template.json

Lines 1 to 88 in b6f4ead

	{
	"template": {
	"settings": {
	"index": {
	"look_ahead_time": "10m",
	"codec": "best_compression"
	}
	},
	"mappings": {
	"_data_stream_timestamp": {
	"enabled": true
	},
	"dynamic": "runtime",
	"properties": {
	"custom_id": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"@timestamp": {
	"type": "date"
	},
	"_metric_type": {
	"type": "keyword"
	},
	"host": {
	"properties": {
	"name": {
	"type": "keyword",
	"time_series_dimension": true
	}
	}
	},
	"_metric": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"_value": {
	"type": "double",
	"time_series_metric": "gauge"
	},
	"cribl_wp": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"group": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"input": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"output": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"name": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"pipeline": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"project": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"route": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"source": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"subscription": {
	"type": "keyword",
	"time_series_dimension": true
	},
	"sourcetype": {
	"type": "keyword",
	"time_series_dimension": true
	}
	}
	}
	}
	}

Logs:

Endpoint: PUT _component_template/logs-cribl-internal

Payload:

elastic-cribl-monitoring/ecm_logs_component_template.json

Lines 1 to 4054 in b6f4ead

	{
	"template": {
	"mappings": {
	"dynamic": false,
	"properties": {
	"@timestamp": {
	"type": "date"
	},
	"BRANCH": {
	"type": "keyword"
	},
	"TIMESTAMP": {
	"type": "date"
	},
	"VERSION": {
	"type": "keyword"
	},
	"_raw": {
	"type": "text"
	},
	"abortCxn": {
	"type": "long"
	},
	"ack": {
	"type": "long"
	},
	"action": {
	"type": "keyword"
	},
	"activeCxn": {
	"type": "long"
	},
	"activeEP": {
	"type": "long"
	},
	"activeId": {
	"type": "keyword"
	},
	"activelyClosing": {
	"type": "long"
	},
	"activityLogSampleRate": {
	"type": "long"
	},
	"actualReloadPeriodMs": {
	"type": "long"
	},
	"addIdToStagePath": {
	"type": "boolean"
	},
	"alive": {
	"type": "keyword"
	},
	"all": {
	"properties": {
	"values": {
	"type": "long"
	}
	}
	},
	"allCollectors": {
	"type": "text"
	},
	"apiKey": {
	"type": "keyword"
	},
	"apiName": {
	"type": "keyword"
	},
	"apiVersion": {
	"type": "long"
	},
	"appId": {
	"type": "keyword"
	},
	"arch": {
	"type": "keyword"
	},
	"args": {
	"type": "text"
	},
	"assumeRoleArn": {
	"type": "keyword"
	},
	"audience": {
	"type": "keyword"
	},
	"authMethod": {
	"type": "keyword"
	},
	"authToken": {
	"type": "keyword"
	},
	"authTokens": {
	"type": "keyword"
	},
	"authType": {
	"type": "keyword"
	},
	"authUrl": {
	"type": "keyword"
	},
	"authenticationTimeout": {
	"type": "long"
	},
	"avoidDuplicates": {
	"type": "boolean"
	},
	"awsAccountId": {
	"type": "keyword"
	},
	"awsApiKey": {
	"type": "keyword"
	},
	"b": {
	"type": "long"
	},
	"baseFileName": {
	"type": "keyword"
	},
	"batchSize": {
	"type": "long"
	},
	"bearerTokenLen": {
	"type": "long"
	},
	"blockedEP": {
	"type": "long"
	},
	"blockedSince": {
	"type": "long"
	},
	"breakerRulesets": {
	"type": "keyword"
	},
	"broker": {
	"type": "keyword"
	},
	"brokers": {
	"type": "text"
	},
	"bucket": {
	"type": "keyword"
	},
	"bufSize": {
	"type": "long"
	},
	"buffered": {
	"type": "long"
	},
	"build": {
	"type": "keyword"
	},
	"cacheCleanupNumEvents": {
	"type": "long"
	},
	"cacheExpirationMs": {
	"type": "long"
	},
	"caller": {
	"properties": {
	"groupId": {
	"type": "keyword"
	},
	"guid": {
	"type": "keyword"
	},
	"inputId": {
	"type": "keyword"
	},
	"workerId": {
	"type": "keyword"
	}
	}
	},
	"capabilities": {
	"properties": {
	"ack": {
	"type": "keyword"
	},
	"compression": {
	"type": "keyword"
	}
	}
	},
	"captureHeaders": {
	"type": "boolean"
	},
	"cfgWorkerCount": {
	"type": "long"
	},
	"channel": {
	"type": "keyword"
	},
	"checkFileModTime": {
	"type": "boolean"
	},
	"checksum": {
	"type": "keyword"
	},
	"cid": {
	"type": "keyword"
	},
	"cleanFields": {
	"type": "boolean"
	},
	"cleanupInterval": {
	"type": "long"
	},
	"client": {
	"type": "keyword"
	},
	"clientId": {
	"type": "keyword"
	},
	"closeCxn": {
	"type": "long"
	},
	"closingFiles": {
	"type": "long"
	},
	"collected": {
	"type": "boolean"
	},
	"collectible": {
	"properties": {
	"__pageNum": {
	"type": "long"
	},
	"collectedEvents": {
	"type": "long"
	},
	"collectorId": {
	"type": "keyword"
	},
	"contentCreated": {
	"type": "date"
	},
	"contentExpiration": {
	"type": "date"
	},
	"contentId": {
	"type": "keyword"
	},
	"contentTime": {
	"type": "float"
	},
	"contentType": {
	"type": "keyword"
	},
	"contentUri": {
	"type": "keyword"
	},
	"discoveredEvents": {
	"type": "long"
	},
	"earliest": {
	"type": "long"
	},
	"fakeDiscover": {
	"type": "boolean"
	},
	"filteredEvents": {
	"type": "long"
	},
	"flushedBuffers": {
	"type": "long"
	},
	"guid": {
	"type": "keyword"
	},
	"host": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"latest": {
	"type": "long"
	},
	"source": {
	"type": "keyword"
	},
	"taskId": {
	"type": "keyword"
	}
	}
	},
	"collectorId": {
	"type": "keyword"
	},
	"collectorIds": {
	"type": "text"
	},
	"collectors": {
	"properties": {
	"cronSchedule": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	}
	}
	},
	"commitFrequency": {
	"type": "long"
	},
	"compareExpression": {
	"type": "keyword"
	},
	"compress": {
	"type": "keyword"
	},
	"compression": {
	"type": "keyword"
	},
	"concurrency": {
	"type": "long"
	},
	"concurrentJobLimit": {
	"type": "long"
	},
	"concurrentScheduledJobLimit": {
	"type": "long"
	},
	"concurrentSystemJobLimit": {
	"type": "long"
	},
	"confReloadPeriodSec": {
	"type": "long"
	},
	"configHelper": {
	"type": "boolean"
	},
	"configHelperSocketDir": {
	"type": "keyword"
	},
	"conflictingFields": {
	"properties": {
	"time": {
	"type": "long"
	}
	}
	},
	"connected": {
	"type": "boolean"
	},
	"connectionTimeout": {
	"type": "long"
	},
	"connections": {
	"properties": {
	"output": {
	"type": "keyword"
	}
	}
	},
	"consecutiveSkips": {
	"type": "long"
	},
	"consumerOpts": {
	"properties": {
	"groupId": {
	"type": "keyword"
	},
	"heartbeatInterval": {
	"type": "long"
	},
	"maxBytes": {
	"type": "long"
	},
	"maxBytesPerPartition": {
	"type": "long"
	},
	"rebalanceTimeout": {
	"type": "long"
	},
	"sessionTimeout": {
	"type": "long"
	}
	}
	},
	"consumerRunOpts": {
	"properties": {
	"autoCommit": {
	"type": "boolean"
	}
	}
	},
	"consumerRunning": {
	"type": "boolean"
	},
	"containerName": {
	"type": "keyword"
	},
	"content": {
	"type": "text"
	},
	"contentCreated": {
	"type": "date"
	},
	"contentExpiration": {
	"type": "date"
	},
	"contentId": {
	"type": "keyword"
	},
	"contentTime": {
	"type": "float"
	},
	"contentType": {
	"type": "keyword"
	},
	"contentUri": {
	"type": "keyword"
	},
	"context": {
	"type": "keyword"
	},
	"copyStatsKey": {
	"type": "keyword"
	},
	"correlationId": {
	"type": "long"
	},
	"count": {
	"type": "long"
	},
	"cpuPerc": {
	"type": "float"
	},
	"cpuProfile": {
	"type": "boolean"
	},
	"cpus": {
	"type": "long"
	},
	"createContainer": {
	"type": "boolean"
	},
	"createQueue": {
	"type": "boolean"
	},
	"createdAt": {
	"type": "long"
	},
	"criblAPI": {
	"type": "keyword"
	},
	"cribl_pipe": {
	"type": "keyword"
	},
	"current": {
	"type": "long"
	},
	"currentReadables": {
	"type": "long"
	},
	"currentState": {
	"type": "keyword"
	},
	"customContentType": {
	"type": "keyword"
	},
	"customDropWhenNull": {
	"type": "boolean"
	},
	"customEventDelimiter": {
	"type": "keyword"
	},
	"custom_id": {
	"type": "keyword"
	},
	"customSourceExpression": {
	"type": "keyword"
	},
	"data": {
	"properties": {
	"__cloneCount": {
	"type": "long"
	},
	"__criblEventType": {
	"type": "keyword"
	},
	"__final": {
	"type": "boolean"
	},
	"__jsonFail": {
	"type": "boolean"
	},
	"__raw": {
	"type": "text"
	},
	"__socketAddr": {
	"type": "keyword"
	},
	"__srcIpPort": {
	"type": "keyword"
	},
	"_raw": {
	"type": "text"
	},
	"_time": {
	"type": "float"
	},
	"body": {
	"type": "text"
	},
	"opts": {
	"properties": {
	"_dst": {
	"type": "keyword"
	},
	"_src": {
	"type": "keyword"
	},
	"streamOptions": {
	"properties": {
	"highWaterMark": {
	"type": "long"
	}
	}
	},
	"timeout": {
	"type": "long"
	}
	}
	},
	"req": {
	"type": "text"
	},
	"reqId": {
	"type": "long"
	},
	"type": {
	"type": "keyword"
	},
	"workerId": {
	"type": "keyword"
	}
	}
	},
	"dead": {
	"type": "keyword"
	},
	"delimiterRegex": {
	"type": "keyword"
	},
	"destPath": {
	"type": "keyword"
	},
	"destinationDir": {
	"type": "keyword"
	},
	"diagfile": {
	"type": "keyword"
	},
	"diaghost": {
	"type": "keyword"
	},
	"dir": {
	"type": "keyword"
	},
	"directorties": {
	"type": "text"
	},
	"directory": {
	"type": "text"
	},
	"disabled": {
	"type": "boolean"
	},
	"diskUsage": {
	"properties": {
	"bytesAvailable": {
	"type": "long"
	},
	"bytesUsed": {
	"type": "long"
	},
	"diskPath": {
	"type": "text"
	},
	"totalDiskSize": {
	"type": "long"
	}
	}
	},
	"dnsResolvePeriodSec": {
	"type": "long"
	},
	"downloadUrl": {
	"type": "text"
	},
	"dropEventsMode": {
	"type": "boolean"
	},
	"dropped": {
	"type": "long"
	},
	"droppedEvents": {
	"type": "long"
	},
	"duration": {
	"type": "long"
	},
	"durationSeconds": {
	"type": "long"
	},
	"earliest": {
	"type": "long"
	},
	"elapsed": {
	"type": "long"
	},
	"elasticAPI": {
	"type": "keyword"
	},
	"eluPerc": {
	"type": "float"
	},
	"enableACK": {
	"type": "boolean"
	},
	"enableAssumeRole": {
	"type": "boolean"
	},
	"enableHeader": {
	"type": "boolean"
	},
	"enableMultiMetrics": {
	"type": "boolean"
	},
	"enableProxyHeader": {
	"type": "boolean"
	},
	"enableSQSAssumeRole": {
	"type": "boolean"
	},
	"enableUnixPath": {
	"type": "boolean"
	},
	"enabled": {
	"type": "long"
	},
	"endpoint": {
	"properties": {
	"host": {
	"type": "keyword"
	},
	"maxVersion": {
	"type": "keyword"
	},
	"minVersion": {
	"type": "keyword"
	},
	"port": {
	"type": "long"
	},
	"rejectUnauthorized": {
	"type": "boolean"
	},
	"servername": {
	"type": "keyword"
	},
	"tls": {
	"type": "boolean"
	}
	}
	},
	"endpoints": {
	"properties": {
	"family": {
	"type": "long"
	},
	"host": {
	"type": "keyword"
	},
	"key": {
	"type": "keyword"
	},
	"resolvedHost": {
	"type": "keyword"
	},
	"stats": {
	"properties": {
	"bytes": {
	"type": "long"
	},
	"errors": {
	"type": "long"
	},
	"events": {
	"type": "long"
	},
	"health": {
	"type": "long"
	},
	"lastFlushBytes": {
	"type": "long"
	},
	"lastFlushTime": {
	"type": "long"
	},
	"requests": {
	"type": "long"
	},
	"totalBytes": {
	"type": "long"
	},
	"totalErrors": {
	"type": "long"
	},
	"totalEvents": {
	"type": "long"
	},
	"totalRequests": {
	"type": "long"
	}
	}
	},
	"url": {
	"type": "keyword"
	},
	"weight": {
	"type": "long"
	}
	}
	},
	"endtime": {
	"type": "date"
	},
	"env": {
	"type": "keyword"
	},
	"err": {
	"properties": {
	"code": {
	"type": "keyword"
	},
	"errno": {
	"type": "long"
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"name": {
	"type": "keyword"
	},
	"originalError": {
	"properties": {
	"code": {
	"type": "keyword"
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"stack": {
	"type": "text"
	}
	}
	},
	"path": {
	"type": "keyword"
	},
	"req": {
	"properties": {
	"body": {
	"type": "text"
	},
	"opts": {
	"properties": {
	"_dst": {
	"type": "keyword"
	},
	"_src": {
	"type": "keyword"
	},
	"service": {
	"type": "keyword"
	},
	"workerProcessFilter": {
	"type": "keyword"
	}
	}
	},
	"req": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"stack": {
	"type": "text"
	},
	"syscall": {
	"type": "keyword"
	}
	}
	},
	"error": {
	"properties": {
	"__cloneCount": {
	"type": "long"
	},
	"__criblEventType": {
	"type": "keyword"
	},
	"__final": {
	"type": "boolean"
	},
	"__raw": {
	"type": "text"
	},
	"__socketAddr": {
	"type": "keyword"
	},
	"__srcIpPort": {
	"type": "keyword"
	},
	"details": {
	"properties": {
	"host": {
	"type": "keyword"
	},
	"method": {
	"type": "keyword"
	},
	"path": {
	"type": "keyword"
	},
	"port": {
	"type": "keyword"
	}
	}
	},
	"error": {
	"properties": {
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"stack": {
	"type": "text"
	}
	}
	},
	"host": {
	"type": "keyword"
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"method": {
	"type": "keyword"
	},
	"name": {
	"type": "keyword"
	},
	"path": {
	"type": "keyword"
	},
	"port": {
	"type": "keyword"
	},
	"reason": {
	"properties": {
	"caused_by": {
	"properties": {
	"reason": {
	"type": "text"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"details": {
	"properties": {
	"host": {
	"type": "keyword"
	},
	"method": {
	"type": "keyword"
	},
	"path": {
	"type": "keyword"
	},
	"port": {
	"type": "keyword"
	}
	}
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"name": {
	"type": "keyword"
	},
	"stack": {
	"type": "text"
	},
	"status": {
	"type": "long"
	},
	"type": {
	"type": "keyword"
	},
	"statusCode": {
	"type": "long"
	}
	}
	},
	"req": {
	"type": "text"
	},
	"reqId": {
	"type": "long"
	},
	"rpc": {
	"type": "boolean"
	},
	"stack": {
	"type": "text"
	},
	"status": {
	"type": "long"
	},
	"statusCode": {
	"type": "long"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"errors": {
	"type": "long"
	},
	"eventBreakerRegex": {
	"type": "keyword"
	},
	"excludeFields": {
	"properties": {
	"fieldname": {
	"type": "keyword"
	},
	"regexList": {
	"properties": {
	"negated": {
	"type": "boolean"
	}
	}
	}
	}
	},
	"excludeSelf": {
	"type": "boolean"
	},
	"exclusive": {
	"type": "boolean"
	},
	"executor": {
	"properties": {
	"collectStep": {
	"type": "keyword"
	},
	"collectibles": {
	"properties": {
	"collectorId": {
	"type": "keyword"
	},
	"contentCreated": {
	"type": "date"
	},
	"contentExpiration": {
	"type": "date"
	},
	"contentId": {
	"type": "keyword"
	},
	"contentTime": {
	"type": "float"
	},
	"contentType": {
	"type": "keyword"
	},
	"contentUri": {
	"type": "keyword"
	},
	"earliest": {
	"type": "long"
	},
	"fakeDiscover": {
	"type": "boolean"
	},
	"guid": {
	"type": "keyword"
	},
	"host": {
	"type": "keyword"
	},
	"latest": {
	"type": "long"
	},
	"source": {
	"type": "keyword"
	},
	"taskId": {
	"type": "keyword"
	}
	}
	},
	"heartbeatPeriod": {
	"type": "long"
	},
	"input": {
	"properties": {
	"breakerRulesets": {
	"type": "keyword"
	},
	"filter": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"metadata": {
	"properties": {
	"name": {
	"type": "keyword"
	},
	"value": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	}
	}
	},
	"preprocess": {
	"properties": {
	"disabled": {
	"type": "boolean"
	}
	}
	},
	"sendToRoutes": {
	"type": "boolean"
	},
	"staleChannelFlushMs": {
	"type": "long"
	},
	"throttleRatePerSec": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"executorId": {
	"type": "keyword"
	},
	"existing": {
	"properties": {
	"genId": {
	"type": "long"
	},
	"guid": {
	"type": "keyword"
	},
	"missedHBLimit": {
	"type": "long"
	},
	"period": {
	"type": "keyword"
	},
	"url": {
	"type": "keyword"
	}
	}
	},
	"existingOrNew": {
	"type": "keyword"
	},
	"existingRule": {
	"type": "keyword"
	},
	"exit": {
	"type": "long"
	},
	"exitCode": {
	"type": "long"
	},
	"expired": {
	"properties": {
	"guid": {
	"type": "keyword"
	},
	"lastHB": {
	"type": "long"
	},
	"task": {
	"properties": {
	"dir": {
	"type": "keyword"
	},
	"executor": {
	"properties": {
	"collectStep": {
	"type": "keyword"
	},
	"collectibles": {
	"type": "keyword"
	},
	"heartbeatPeriod": {
	"type": "long"
	},
	"input": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"jobId": {
	"type": "keyword"
	},
	"logLevel": {
	"type": "keyword"
	},
	"task": {
	"properties": {
	"conf": {
	"type": "keyword"
	},
	"destructive": {
	"type": "boolean"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"taskId": {
	"type": "keyword"
	}
	}
	},
	"workerProcess": {
	"type": "keyword"
	}
	}
	},
	"expiryTime": {
	"type": "long"
	},
	"expression": {
	"type": "keyword"
	},
	"extractDir": {
	"type": "keyword"
	},
	"extractedSource": {
	"type": "keyword"
	},
	"failed": {
	"type": "long"
	},
	"failedRequestLoggingMode": {
	"type": "keyword"
	},
	"failover": {
	"properties": {
	"missedHBLimit": {
	"type": "long"
	},
	"period": {
	"type": "keyword"
	},
	"volume": {
	"type": "keyword"
	}
	}
	},
	"fakeDiscover": {
	"type": "boolean"
	},
	"fieldsLineRegex": {
	"type": "keyword"
	},
	"file": {
	"type": "keyword"
	},
	"fileFilter": {
	"type": "keyword"
	},
	"fileName": {
	"type": "keyword"
	},
	"fileNameSuffix": {
	"type": "keyword"
	},
	"filename": {
	"type": "keyword"
	},
	"filenames": {
	"type": "text"
	},
	"files": {
	"properties": {
	"created": {
	"type": "keyword"
	},
	"deleted": {
	"type": "keyword"
	},
	"modified": {
	"type": "keyword"
	}
	}
	},
	"first": {
	"properties": {
	"__offset": {
	"type": "long"
	}
	}
	},
	"firstOffset": {
	"type": "keyword"
	},
	"flushEventCount": {
	"type": "long"
	},
	"flushPeriodSec": {
	"type": "long"
	},
	"force": {
	"type": "boolean"
	},
	"format": {
	"type": "keyword"
	},
	"fromBeginning": {
	"type": "boolean"
	},
	"fullFidelity": {
	"type": "boolean"
	},
	"function": {
	"type": "keyword"
	},
	"functions": {
	"type": "keyword"
	},
	"fwd": {
	"type": "keyword"
	},
	"fwdType": {
	"type": "keyword"
	},
	"garbageCollectionPeriodSecs": {
	"type": "long"
	},
	"genId": {
	"type": "long"
	},
	"getRecordsLimit": {
	"type": "long"
	},
	"getRecordsLimitTotal": {
	"type": "long"
	},
	"group": {
	"type": "keyword"
	},
	"groupId": {
	"type": "keyword"
	},
	"groupProtocol": {
	"type": "keyword"
	},
	"guid": {
	"type": "keyword"
	},
	"guids": {
	"type": "text"
	},
	"hashFile": {
	"type": "keyword"
	},
	"hashUrl": {
	"type": "keyword"
	},
	"headerLineRegex": {
	"type": "keyword"
	},
	"heartbeatInterval": {
	"type": "long"
	},
	"highWatermark": {
	"type": "keyword"
	},
	"hint": {
	"type": "keyword"
	},
	"history": {
	"properties": {
	"args": {
	"properties": {
	"collector": {
	"properties": {
	"conf": {
	"type": "text"
	},
	"destructive": {
	"type": "boolean"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"groupId": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"input": {
	"properties": {
	"breakerRulesets": {
	"type": "text"
	},
	"filter": {
	"type": "keyword"
	},
	"metadata": {
	"type": "keyword"
	},
	"output": {
	"type": "keyword"
	},
	"pipeline": {
	"type": "keyword"
	},
	"preprocess": {
	"type": "keyword"
	},
	"sendToRoutes": {
	"type": "boolean"
	},
	"staleChannelFlushMs": {
	"type": "long"
	},
	"throttleRatePerSec": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"resumeOnBoot": {
	"type": "boolean"
	},
	"run": {
	"properties": {
	"earliest": {
	"type": "keyword"
	},
	"expression": {
	"type": "keyword"
	},
	"jobTimeout": {
	"type": "keyword"
	},
	"latest": {
	"type": "keyword"
	},
	"logLevel": {
	"type": "keyword"
	},
	"maxTaskReschedule": {
	"type": "long"
	},
	"maxTaskSize": {
	"type": "keyword"
	},
	"minTaskSize": {
	"type": "keyword"
	},
	"mode": {
	"type": "keyword"
	},
	"now": {
	"type": "date"
	},
	"rescheduleDroppedTasks": {
	"type": "boolean"
	},
	"taskHeartbeatPeriod": {
	"type": "long"
	},
	"timeRangeType": {
	"type": "keyword"
	},
	"timestampTimezone": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"schedule": {
	"properties": {
	"cronSchedule": {
	"type": "keyword"
	},
	"enabled": {
	"type": "boolean"
	},
	"maxConcurrentRuns": {
	"type": "long"
	},
	"resumeMissed": {
	"type": "boolean"
	},
	"run": {
	"type": "keyword"
	},
	"skippable": {
	"type": "boolean"
	}
	}
	},
	"ttl": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	},
	"workerAffinity": {
	"type": "boolean"
	}
	}
	},
	"id": {
	"type": "keyword"
	},
	"stats": {
	"properties": {
	"collectedBytes": {
	"type": "long"
	},
	"collectedEvents": {
	"type": "long"
	},
	"discoveredEvents": {
	"type": "long"
	},
	"discoveryComplete": {
	"type": "long"
	},
	"filteredEvents": {
	"type": "long"
	},
	"flushedBuffers": {
	"type": "long"
	},
	"state": {
	"properties": {
	"cancelled": {
	"type": "long"
	},
	"finished": {
	"type": "long"
	},
	"initializing": {
	"type": "long"
	},
	"pending": {
	"type": "long"
	},
	"running": {
	"type": "long"
	}
	}
	},
	"tasks": {
	"properties": {
	"cancelled": {
	"type": "long"
	},
	"count": {
	"type": "long"
	},
	"failed": {
	"type": "long"
	},
	"finished": {
	"type": "long"
	},
	"inFlight": {
	"type": "long"
	},
	"maxExecutionTime": {
	"type": "long"
	},
	"minExecutionTime": {
	"type": "long"
	},
	"orphaned": {
	"type": "long"
	},
	"totalExecutionTime": {
	"type": "long"
	}
	}
	},
	"totalResults": {
	"type": "long"
	}
	}
	},
	"status": {
	"properties": {
	"state": {
	"type": "long"
	}
	}
	}
	}
	},
	"host": {
	"properties": {
	"name": {
	"type": "keyword"
	}
	}
	},
	"hostname": {
	"type": "keyword"
	},
	"hosts": {
	"properties": {
	"host": {
	"type": "keyword"
	},
	"port": {
	"type": "long"
	},
	"tls": {
	"type": "keyword"
	},
	"weight": {
	"type": "long"
	}
	}
	},
	"http": {
	"properties": {
	"response": {
	"properties": {
	"status_code": {
	"type": "short"
	}
	}
	}
	}
	},
	"id": {
	"type": "keyword"
	},
	"idleTimeout": {
	"type": "long"
	},
	"inBytes": {
	"type": "long"
	},
	"inEvents": {
	"type": "long"
	},
	"includeUnidentifiableBinary": {
	"type": "boolean"
	},
	"index": {
	"type": "keyword"
	},
	"indexerDiscovery": {
	"type": "boolean"
	},
	"indexerDiscoveryConfigs": {
	"properties": {
	"authToken": {
	"type": "keyword"
	},
	"authType": {
	"type": "keyword"
	},
	"masterUri": {
	"type": "keyword"
	},
	"refreshIntervalSec": {
	"type": "long"
	},
	"site": {
	"type": "keyword"
	}
	}
	},
	"info": {
	"properties": {
	"architecture": {
	"type": "keyword"
	},
	"cpus": {
	"type": "long"
	},
	"cribl": {
	"properties": {
	"version": {
	"type": "keyword"
	}
	}
	},
	"hostname": {
	"type": "keyword"
	},
	"node": {
	"type": "keyword"
	},
	"platform": {
	"type": "keyword"
	},
	"release": {
	"type": "keyword"
	},
	"totalmem": {
	"type": "long"
	}
	}
	},
	"infoFile": {
	"type": "keyword"
	},
	"input": {
	"type": "keyword"
	},
	"inputId": {
	"type": "keyword"
	},
	"instanceId": {
	"type": "keyword"
	},
	"interval": {
	"type": "long"
	},
	"intervalsSecs": {
	"type": "long"
	},
	"ip": {
	"type": "keyword"
	},
	"ipWhitelistRegex": {
	"type": "keyword"
	},
	"isConfigHelper": {
	"type": "boolean"
	},
	"isForkedProcess": {
	"type": "boolean"
	},
	"isLeader": {
	"type": "boolean"
	},
	"isManaged": {
	"type": "boolean"
	},
	"isRunning": {
	"type": "boolean"
	},
	"isSessionStale": {
	"type": "boolean"
	},
	"isStale": {
	"type": "boolean"
	},
	"isStandalone": {
	"type": "boolean"
	},
	"isV4": {
	"type": "boolean"
	},
	"job": {
	"properties": {
	"collector": {
	"properties": {
	"conf": {
	"properties": {
	"__scheduling": {
	"properties": {
	"stateTracking": {
	"type": "keyword"
	}
	}
	},
	"app_id": {
	"type": "keyword"
	},
	"authHeaderExpr": {
	"type": "keyword"
	},
	"authHeaderKey": {
	"type": "keyword"
	},
	"authentication": {
	"type": "keyword"
	},
	"clientSecretParamName": {
	"type": "keyword"
	},
	"client_secret": {
	"type": "keyword"
	},
	"collectBody": {
	"type": "keyword"
	},
	"collectMethod": {
	"type": "keyword"
	},
	"collectRequestHeaders": {
	"type": "keyword"
	},
	"collectRequestParams": {
	"type": "keyword"
	},
	"collectScript": {
	"type": "keyword"
	},
	"collectUrl": {
	"type": "keyword"
	},
	"connectionId": {
	"type": "keyword"
	},
	"content_type": {
	"type": "keyword"
	},
	"disableTimeFilter": {
	"type": "boolean"
	},
	"discoverScript": {
	"type": "keyword"
	},
	"discovery": {
	"properties": {
	"discoverDataField": {
	"type": "keyword"
	},
	"discoverMethod": {
	"type": "keyword"
	},
	"discoverRequestParams": {
	"type": "keyword"
	},
	"discoverType": {
	"type": "keyword"
	},
	"discoverUrl": {
	"type": "keyword"
	}
	}
	},
	"earliest": {
	"type": "keyword"
	},
	"endpoint": {
	"type": "keyword"
	},
	"ingestionLag": {
	"type": "long"
	},
	"latest": {
	"type": "keyword"
	},
	"loginBody": {
	"type": "text"
	},
	"loginUrl": {
	"type": "keyword"
	},
	"outputMode": {
	"type": "keyword"
	},
	"pagination": {
	"properties": {
	"attribute": {
	"type": "keyword"
	},
	"limit": {
	"type": "long"
	},
	"limitField": {
	"type": "keyword"
	},
	"maxPages": {
	"type": "long"
	},
	"offset": {
	"type": "long"
	},
	"offsetField": {
	"type": "keyword"
	},
	"totalRecordField": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	},
	"zeroIndexed": {
	"type": "boolean"
	}
	}
	},
	"password": {
	"type": "keyword"
	},
	"plan_type": {
	"type": "keyword"
	},
	"query": {
	"type": "keyword"
	},
	"rejectUnauthorized": {
	"type": "boolean"
	},
	"search": {
	"type": "keyword"
	},
	"searchHead": {
	"type": "keyword"
	},
	"shell": {
	"type": "keyword"
	},
	"tenant_id": {
	"type": "keyword"
	},
	"timeout": {
	"type": "long"
	},
	"tokenRespAttribute": {
	"type": "keyword"
	},
	"useRoundRobinDns": {
	"type": "boolean"
	},
	"username": {
	"type": "keyword"
	}
	}
	},
	"destructive": {
	"type": "boolean"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"groupId": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"input": {
	"properties": {
	"breakerRulesets": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"metadata": {
	"properties": {
	"name": {
	"type": "keyword"
	},
	"value": {
	"type": "keyword"
	}
	}
	},
	"output": {
	"type": "keyword"
	},
	"pipeline": {
	"type": "keyword"
	},
	"preprocess": {
	"properties": {
	"disabled": {
	"type": "boolean"
	}
	}
	},
	"sendToRoutes": {
	"type": "boolean"
	},
	"staleChannelFlushMs": {
	"type": "long"
	},
	"throttleRatePerSec": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"resumeOnBoot": {
	"type": "boolean"
	},
	"schedule": {
	"properties": {
	"cronSchedule": {
	"type": "keyword"
	},
	"enabled": {
	"type": "boolean"
	},
	"maxConcurrentRuns": {
	"type": "long"
	},
	"resumeMissed": {
	"type": "boolean"
	},
	"run": {
	"properties": {
	"discoverToRoutes": {
	"type": "boolean"
	},
	"earliest": {
	"type": "keyword"
	},
	"expression": {
	"type": "keyword"
	},
	"filterTimeField": {
	"type": "keyword"
	},
	"jobTimeout": {
	"type": "keyword"
	},
	"latest": {
	"type": "keyword"
	},
	"logLevel": {
	"type": "keyword"
	},
	"maxTaskReschedule": {
	"type": "long"
	},
	"maxTaskSize": {
	"type": "keyword"
	},
	"minTaskSize": {
	"type": "keyword"
	},
	"mode": {
	"type": "keyword"
	},
	"rescheduleDroppedTasks": {
	"type": "boolean"
	},
	"timeRangeType": {
	"type": "keyword"
	},
	"timestampTimezone": {
	"type": "keyword"
	}
	}
	},
	"skippable": {
	"type": "boolean"
	}
	}
	},
	"ttl": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	},
	"workerAffinity": {
	"type": "boolean"
	}
	}
	},
	"jobId": {
	"type": "keyword"
	},
	"jobIds": {
	"type": "keyword"
	},
	"jobTimeout": {
	"type": "keyword"
	},
	"jobs": {
	"type": "text"
	},
	"jobsStore": {
	"type": "boolean"
	},
	"journals": {
	"type": "text"
	},
	"keep": {
	"type": "boolean"
	},
	"keepAliveTime": {
	"type": "long"
	},
	"keepAliveTimeout": {
	"type": "long"
	},
	"keepFor": {
	"type": "long"
	},
	"key": {
	"type": "keyword"
	},
	"keyExpr": {
	"type": "keyword"
	},
	"killed": {
	"type": "long"
	},
	"lastCursor": {
	"properties": {
	"offset": {
	"type": "long"
	},
	"sliceId": {
	"type": "long"
	}
	}
	},
	"lastIndexer": {
	"type": "keyword"
	},
	"lastModifiedTime": {
	"type": "long"
	},
	"latest": {
	"type": "long"
	},
	"leaderId": {
	"type": "keyword"
	},
	"lease": {
	"properties": {
	"guid": {
	"type": "keyword"
	},
	"missedHBLimit": {
	"type": "long"
	},
	"period": {
	"type": "keyword"
	},
	"url": {
	"type": "keyword"
	}
	}
	},
	"leaseFile": {
	"type": "keyword"
	},
	"level": {
	"type": "keyword"
	},
	"licenseLimit": {
	"type": "long"
	},
	"loadBalanceStatsPeriodSec": {
	"type": "long"
	},
	"loadBalanced": {
	"type": "boolean"
	},
	"loadBalancingAlgorithm": {
	"type": "keyword"
	},
	"logGroupName": {
	"type": "keyword"
	},
	"logLevel": {
	"type": "keyword"
	},
	"logNames": {
	"type": "text"
	},
	"logStreamEnv": {
	"type": "keyword"
	},
	"logStreamName": {
	"type": "keyword"
	},
	"lookups": {
	"type": "long"
	},
	"matchMode": {
	"type": "keyword"
	},
	"matchType": {
	"type": "keyword"
	},
	"maxActiveCxn": {
	"type": "long"
	},
	"maxActiveReq": {
	"type": "long"
	},
	"maxBufferSize": {
	"type": "long"
	},
	"maxBytes": {
	"type": "long"
	},
	"maxBytesPerPartition": {
	"type": "long"
	},
	"maxCacheSize": {
	"type": "long"
	},
	"maxConcurrentFileParts": {
	"type": "long"
	},
	"maxConcurrentSenders": {
	"type": "long"
	},
	"maxEventBytes": {
	"type": "long"
	},
	"maxFailedHealthChecks": {
	"type": "long"
	},
	"maxFileIdleTimeSec": {
	"type": "long"
	},
	"maxFileOpenTimeSec": {
	"type": "long"
	},
	"maxFileSize": {
	"type": "keyword"
	},
	"maxFileSizeMB": {
	"type": "long"
	},
	"maxJobs": {
	"type": "long"
	},
	"maxMessages": {
	"type": "long"
	},
	"maxMetrics": {
	"type": "long"
	},
	"maxMissedKeepAlives": {
	"type": "long"
	},
	"maxOpenFiles": {
	"type": "long"
	},
	"maxPayloadEvents": {
	"type": "long"
	},
	"maxPayloadSize": {
	"type": "long"
	},
	"maxPayloadSizeKB": {
	"type": "long"
	},
	"maxProcs": {
	"type": "long"
	},
	"maxQueueSize": {
	"type": "keyword"
	},
	"maxRecordSizeKB": {
	"type": "long"
	},
	"maxResults": {
	"type": "long"
	},
	"maxRetries": {
	"type": "long"
	},
	"maxS2Sversion": {
	"type": "keyword"
	},
	"maxSize": {
	"type": "keyword"
	},
	"maxVersion": {
	"type": "keyword"
	},
	"mem": {
	"properties": {
	"ext": {
	"type": "long"
	},
	"heap": {
	"type": "long"
	},
	"rss": {
	"type": "long"
	}
	}
	},
	"memberId": {
	"type": "keyword"
	},
	"memory": {
	"properties": {
	"free": {
	"type": "long"
	},
	"size": {
	"type": "long"
	},
	"total": {
	"type": "long"
	}
	}
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"meta": {
	"properties": {
	"name": {
	"type": "keyword"
	}
	}
	},
	"metadata": {
	"properties": {
	"name": {
	"type": "keyword"
	},
	"value": {
	"type": "text"
	}
	}
	},
	"method": {
	"type": "keyword"
	},
	"metric": {
	"properties": {
	"dimensions": {
	"type": "keyword"
	},
	"name": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"metricsAddress": {
	"properties": {
	"path": {
	"type": "keyword"
	}
	}
	},
	"metricsFieldsBlacklist": {
	"type": "keyword"
	},
	"metricsGCPeriod": {
	"type": "keyword"
	},
	"metricsMaxCardinality": {
	"type": "long"
	},
	"metricsNeverDropList": {
	"type": "keyword"
	},
	"metricsWorkerIdBlacklist": {
	"type": "keyword"
	},
	"minEvents": {
	"type": "long"
	},
	"minVersion": {
	"type": "keyword"
	},
	"minimizeDuplicates": {
	"type": "boolean"
	},
	"missed": {
	"type": "long"
	},
	"mode": {
	"type": "keyword"
	},
	"modelName": {
	"type": "keyword"
	},
	"msg": {
	"properties": {
	"__cloneCount": {
	"type": "long"
	},
	"__criblEventType": {
	"type": "keyword"
	},
	"__final": {
	"type": "boolean"
	},
	"__raw": {
	"type": "text"
	},
	"__socketAddr": {
	"type": "keyword"
	},
	"__srcIpPort": {
	"type": "keyword"
	},
	"_time": {
	"type": "float"
	},
	"body": {
	"type": "text"
	},
	"opts": {
	"properties": {
	"_dst": {
	"type": "keyword"
	},
	"_src": {
	"type": "keyword"
	},
	"timeout": {
	"type": "long"
	}
	}
	},
	"req": {
	"type": "keyword"
	},
	"reqId": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	},
	"workerId": {
	"type": "keyword"
	}
	}
	},
	"name": {
	"type": "keyword"
	},
	"nestedFields": {
	"type": "text"
	},
	"newCursor": {
	"properties": {
	"offset": {
	"type": "long"
	},
	"sliceId": {
	"type": "long"
	}
	}
	},
	"notifications": {
	"properties": {
	"condition": {
	"type": "keyword"
	},
	"conf": {
	"properties": {
	"name": {
	"type": "keyword"
	},
	"timeWindow": {
	"type": "keyword"
	},
	"usageThreshold": {
	"type": "long"
	}
	}
	},
	"disabled": {
	"type": "boolean"
	},
	"group": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"targets": {
	"type": "text"
	}
	}
	},
	"nullFieldVal": {
	"type": "keyword"
	},
	"numFlushed": {
	"type": "long"
	},
	"numInFlight": {
	"type": "long"
	},
	"numMetrics": {
	"type": "long"
	},
	"numProcs": {
	"type": "long"
	},
	"numReceivers": {
	"type": "long"
	},
	"numRows": {
	"type": "long"
	},
	"numToAllow": {
	"type": "long"
	},
	"numToFlush": {
	"type": "long"
	},
	"objectACL": {
	"type": "text"
	},
	"octetCounting": {
	"type": "boolean"
	},
	"offset": {
	"type": "long"
	},
	"oldPid": {
	"type": "long"
	},
	"onBackpressure": {
	"type": "keyword"
	},
	"openCxn": {
	"type": "long"
	},
	"openFiles": {
	"type": "long"
	},
	"openFilesCount": {
	"type": "long"
	},
	"os": {
	"type": "keyword"
	},
	"outBytes": {
	"type": "long"
	},
	"outEvents": {
	"type": "long"
	},
	"output": {
	"type": "keyword"
	},
	"outputId": {
	"type": "keyword"
	},
	"overwrite": {
	"type": "boolean"
	},
	"pack": {
	"type": "keyword"
	},
	"packDir": {
	"type": "keyword"
	},
	"packNames": {
	"type": "text"
	},
	"packageFile": {
	"type": "keyword"
	},
	"packs": {
	"type": "text"
	},
	"parquetChunkDownloadTimeout": {
	"type": "long"
	},
	"parquetChunkSizeMB": {
	"type": "long"
	},
	"partition": {
	"type": "long"
	},
	"partitionExpr": {
	"type": "keyword"
	},
	"password": {
	"type": "keyword"
	},
	"path": {
	"type": "keyword"
	},
	"pathsToRemove": {
	"type": "text"
	},
	"payloadFormat": {
	"type": "keyword"
	},
	"pendingDuration": {
	"type": "long"
	},
	"period": {
	"type": "long"
	},
	"periodsToWait": {
	"type": "long"
	},
	"persistence": {
	"properties": {
	"enable": {
	"type": "boolean"
	},
	"maxDataSize": {
	"type": "keyword"
	}
	}
	},
	"pid": {
	"type": "long"
	},
	"pipe": {
	"type": "keyword"
	},
	"pipeId": {
	"type": "keyword"
	},
	"pipeline": {
	"type": "keyword"
	},
	"pipelines": {
	"type": "keyword"
	},
	"planType": {
	"type": "keyword"
	},
	"pointsPerInterval": {
	"type": "long"
	},
	"policies": {
	"type": "keyword"
	},
	"policy": {
	"properties": {
	"args": {
	"type": "keyword"
	},
	"description": {
	"type": "keyword"
	},
	"template": {
	"type": "text"
	},
	"title": {
	"type": "keyword"
	}
	}
	},
	"pollTimeout": {
	"type": "long"
	},
	"poolSize": {
	"properties": {
	"isAuto": {
	"type": "boolean"
	},
	"role": {
	"type": "keyword"
	},
	"value": {
	"type": "long"
	}
	}
	},
	"port": {
	"type": "long"
	},
	"pq": {
	"properties": {
	"commitFrequency": {
	"type": "long"
	},
	"compress": {
	"type": "keyword"
	},
	"maxBufferSize": {
	"type": "long"
	},
	"maxFileSize": {
	"type": "keyword"
	},
	"maxSize": {
	"type": "keyword"
	},
	"mode": {
	"type": "keyword"
	},
	"path": {
	"type": "keyword"
	}
	}
	},
	"pqBufferedEvents": {
	"type": "long"
	},
	"pqCompress": {
	"type": "keyword"
	},
	"pqEnabled": {
	"type": "boolean"
	},
	"pqInBytes": {
	"type": "long"
	},
	"pqInEvents": {
	"type": "long"
	},
	"pqIsEngaged": {
	"type": "boolean"
	},
	"pqMaxFileSize": {
	"type": "keyword"
	},
	"pqMaxSize": {
	"type": "keyword"
	},
	"pqOnBackpressure": {
	"type": "keyword"
	},
	"pqOutBytes": {
	"type": "long"
	},
	"pqOutEvents": {
	"type": "long"
	},
	"pqPath": {
	"type": "keyword"
	},
	"pqSize": {
	"type": "long"
	},
	"pqStrictOrdering": {
	"type": "boolean"
	},
	"pqTotalBytes": {
	"type": "long"
	},
	"pqTotalEvents": {
	"type": "long"
	},
	"prefix": {
	"type": "keyword"
	},
	"preprocess": {
	"properties": {
	"disabled": {
	"type": "boolean"
	}
	}
	},
	"previousState": {
	"type": "keyword"
	},
	"processorId": {
	"type": "keyword"
	},
	"procs": {
	"type": "long"
	},
	"query": {
	"type": "keyword"
	},
	"queueName": {
	"type": "keyword"
	},
	"queueSize": {
	"type": "long"
	},
	"queueType": {
	"type": "keyword"
	},
	"r": {
	"type": "long"
	},
	"readMode": {
	"type": "keyword"
	},
	"reapableBackups": {
	"properties": {
	"createdDate": {
	"type": "date"
	},
	"path": {
	"type": "keyword"
	}
	}
	},
	"reapedJobs": {
	"type": "long"
	},
	"reapedTasks": {
	"type": "long"
	},
	"reason": {
	"properties": {
	"code": {
	"type": "keyword"
	},
	"errno": {
	"type": "long"
	},
	"host": {
	"type": "keyword"
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"method": {
	"type": "keyword"
	},
	"name": {
	"type": "keyword"
	},
	"path": {
	"type": "keyword"
	},
	"port": {
	"type": "keyword"
	},
	"reason": {
	"properties": {
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"name": {
	"type": "keyword"
	},
	"reason": {
	"properties": {
	"code": {
	"type": "keyword"
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"stack": {
	"type": "text"
	}
	}
	},
	"stack": {
	"type": "keyword"
	}
	}
	},
	"req": {
	"properties": {
	"body": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"opts": {
	"properties": {
	"_dst": {
	"type": "keyword"
	},
	"_src": {
	"type": "keyword"
	},
	"service": {
	"type": "keyword"
	},
	"workerProcessFilter": {
	"type": "keyword"
	}
	}
	},
	"req": {
	"type": "text"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"stack": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"syscall": {
	"type": "keyword"
	}
	}
	},
	"reauthenticationThreshold": {
	"type": "long"
	},
	"rebalanceTimeout": {
	"type": "long"
	},
	"received": {
	"type": "long"
	},
	"reconnectInMs": {
	"type": "long"
	},
	"reconnectTime": {
	"type": "long"
	},
	"redirectUri": {
	"type": "keyword"
	},
	"redirect_uri": {
	"type": "keyword"
	},
	"refCount": {
	"type": "long"
	},
	"region": {
	"type": "keyword"
	},
	"rejectCxn": {
	"type": "long"
	},
	"rejectUnauthorized": {
	"type": "boolean"
	},
	"remainingRefs": {
	"type": "long"
	},
	"removeEmptyDirs": {
	"type": "boolean"
	},
	"removed": {
	"type": "long"
	},
	"reqId": {
	"type": "long"
	},
	"requestId": {
	"type": "keyword"
	},
	"requestTimeout": {
	"type": "long"
	},
	"requested": {
	"type": "long"
	},
	"res": {
	"properties": {
	"__cloneCount": {
	"type": "long"
	},
	"__criblEventType": {
	"type": "keyword"
	},
	"__final": {
	"type": "boolean"
	},
	"__raw": {
	"type": "text"
	},
	"__socketAddr": {
	"type": "keyword"
	},
	"__srcIpPort": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"body": {
	"type": "text"
	},
	"req": {
	"type": "text"
	},
	"reqId": {
	"type": "long"
	},
	"status": {
	"type": "long"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"response_time": {
	"type": "long"
	},
	"resiliency": {
	"type": "keyword"
	},
	"restartDelay": {
	"type": "long"
	},
	"result": {
	"properties": {
	"availableVersions": {
	"properties": {
	"architecture": {
	"type": "keyword"
	},
	"build": {
	"type": "keyword"
	},
	"downloadUrl": {
	"type": "keyword"
	},
	"fullVersion": {
	"type": "keyword"
	},
	"major": {
	"type": "long"
	},
	"minor": {
	"type": "long"
	},
	"platform": {
	"type": "keyword"
	},
	"point": {
	"type": "long"
	},
	"preRelease": {
	"type": "keyword"
	}
	}
	},
	"canUpgrade": {
	"type": "boolean"
	},
	"installedVersion": {
	"properties": {
	"architecture": {
	"type": "keyword"
	},
	"build": {
	"type": "keyword"
	},
	"fullVersion": {
	"type": "keyword"
	},
	"major": {
	"type": "long"
	},
	"minor": {
	"type": "long"
	},
	"platform": {
	"type": "keyword"
	},
	"point": {
	"type": "long"
	},
	"preRelease": {
	"type": "keyword"
	}
	}
	},
	"isSuccess": {
	"type": "boolean"
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"result": {
	"properties": {
	"author": {
	"type": "keyword"
	},
	"description": {
	"type": "text"
	},
	"displayName": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"source": {
	"type": "keyword"
	},
	"tags": {
	"properties": {
	"streamtags": {
	"type": "keyword"
	}
	}
	},
	"version": {
	"type": "keyword"
	}
	}
	},
	"upgradedToVersion": {
	"properties": {
	"architecture": {
	"type": "keyword"
	},
	"build": {
	"type": "keyword"
	},
	"fullVersion": {
	"type": "keyword"
	},
	"major": {
	"type": "long"
	},
	"minor": {
	"type": "long"
	},
	"platform": {
	"type": "keyword"
	},
	"point": {
	"type": "long"
	},
	"preRelease": {
	"type": "keyword"
	}
	}
	}
	}
	},
	"retryCount": {
	"type": "long"
	},
	"retryMs": {
	"type": "long"
	},
	"retryTime": {
	"type": "long"
	},
	"reuseConnections": {
	"type": "boolean"
	},
	"reverseResults": {
	"type": "boolean"
	},
	"role": {
	"type": "keyword"
	},
	"rowCount": {
	"type": "long"
	},
	"ruleType": {
	"type": "keyword"
	},
	"rules": {
	"properties": {
	"description": {
	"type": "text"
	},
	"filter": {
	"type": "keyword"
	},
	"final": {
	"type": "boolean"
	},
	"output": {
	"type": "keyword"
	},
	"rate": {
	"type": "long"
	}
	}
	},
	"running": {
	"type": "long"
	},
	"s2sVersion": {
	"type": "keyword"
	},
	"sampleId": {
	"type": "keyword"
	},
	"samplePeriodMS": {
	"type": "long"
	},
	"samples": {
	"properties": {
	"eventsPerSec": {
	"type": "long"
	},
	"sample": {
	"type": "keyword"
	}
	}
	},
	"scheduled": {
	"type": "keyword"
	},
	"scheduledJob": {
	"type": "keyword"
	},
	"schedulingPolicy": {
	"type": "keyword"
	},
	"sendToRoutes": {
	"type": "boolean"
	},
	"senderUnhealthyTimeAllowance": {
	"type": "long"
	},
	"sent": {
	"type": "long"
	},
	"sentAt": {
	"type": "long"
	},
	"servername": {
	"type": "keyword"
	},
	"service": {
	"type": "keyword"
	},
	"serviceId": {
	"type": "long"
	},
	"serviceInterval": {
	"type": "long"
	},
	"serviceName": {
	"type": "keyword"
	},
	"servicePeriodMS": {
	"type": "long"
	},
	"servicePeriodMin": {
	"type": "float"
	},
	"servicePeriodSecs": {
	"type": "long"
	},
	"sessionTimeout": {
	"type": "long"
	},
	"severity": {
	"type": "keyword"
	},
	"sfdcAccountName": {
	"type": "keyword"
	},
	"shardExpr": {
	"type": "keyword"
	},
	"shardIteratorType": {
	"type": "keyword"
	},
	"shouldMarkCriblBreaker": {
	"type": "boolean"
	},
	"shouldStartConsumer": {
	"type": "keyword"
	},
	"signal": {
	"type": "keyword"
	},
	"signature": {
	"type": "keyword"
	},
	"signatureVersion": {
	"type": "keyword"
	},
	"since": {
	"type": "long"
	},
	"singleMsgUdpPackets": {
	"type": "boolean"
	},
	"size": {
	"type": "long"
	},
	"skipOnError": {
	"type": "boolean"
	},
	"socketTimeout": {
	"type": "long"
	},
	"source": {
	"type": "keyword"
	},
	"sourceDir": {
	"type": "keyword"
	},
	"sourcetype": {
	"type": "keyword"
	},
	"splunkHecAPI": {
	"type": "keyword"
	},
	"splunkHecAcks": {
	"type": "boolean"
	},
	"src": {
	"type": "keyword"
	},
	"srcId": {
	"type": "keyword"
	},
	"ssl": {
	"type": "keyword"
	},
	"stack": {
	"type": "keyword"
	},
	"stagePath": {
	"type": "keyword"
	},
	"staleChannelFlushMs": {
	"type": "long"
	},
	"started": {
	"type": "long"
	},
	"starting": {
	"type": "long"
	},
	"starttime": {
	"type": "date"
	},
	"state": {
	"type": "keyword"
	},
	"status": {
	"properties": {
	"code": {
	"type": "long"
	},
	"error": {
	"properties": {
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	}
	}
	},
	"health": {
	"type": "keyword"
	},
	"timestamp": {
	"type": "long"
	}
	}
	},
	"stopped": {
	"type": "long"
	},
	"streamName": {
	"type": "keyword"
	},
	"subscriptions": {
	"properties": {
	"batchTimeout": {
	"type": "long"
	},
	"compress": {
	"type": "boolean"
	},
	"contentFormat": {
	"type": "keyword"
	},
	"heartbeatInterval": {
	"type": "long"
	},
	"querySelector": {
	"type": "keyword"
	},
	"readExistingEvents": {
	"type": "boolean"
	},
	"sendBookmarks": {
	"type": "boolean"
	},
	"subscriptionName": {
	"type": "keyword"
	},
	"targets": {
	"type": "text"
	},
	"version": {
	"type": "keyword"
	},
	"xmlQuery": {
	"type": "keyword"
	}
	}
	},
	"supressionMs": {
	"type": "long"
	},
	"sync": {
	"type": "boolean"
	},
	"systemFields": {
	"type": "keyword"
	},
	"tableIsNull": {
	"type": "boolean"
	},
	"tags": {
	"properties": {
	"system": {
	"type": "boolean"
	}
	}
	},
	"tailOnly": {
	"type": "boolean"
	},
	"task": {
	"properties": {
	"collectibles": {
	"properties": {
	"collectorId": {
	"type": "keyword"
	},
	"compression": {
	"type": "keyword"
	},
	"contentCreated": {
	"type": "date"
	},
	"contentExpiration": {
	"type": "date"
	},
	"contentId": {
	"type": "keyword"
	},
	"contentUri": {
	"type": "keyword"
	},
	"earliest": {
	"type": "long"
	},
	"host": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"latest": {
	"type": "long"
	},
	"offset": {
	"type": "long"
	},
	"source": {
	"type": "keyword"
	},
	"taskId": {
	"type": "keyword"
	}
	}
	},
	"conf": {
	"properties": {
	"app_id": {
	"type": "keyword"
	},
	"authHeaderExpr": {
	"type": "keyword"
	},
	"authHeaderKey": {
	"type": "keyword"
	},
	"authentication": {
	"type": "keyword"
	},
	"client_secret": {
	"type": "keyword"
	},
	"collectMethod": {
	"type": "keyword"
	},
	"collectRequestHeaders": {
	"properties": {
	"name": {
	"type": "keyword"
	},
	"value": {
	"type": "keyword"
	}
	}
	},
	"collectRequestParams": {
	"properties": {
	"name": {
	"type": "keyword"
	},
	"value": {
	"type": "keyword"
	}
	}
	},
	"collectUrl": {
	"type": "keyword"
	},
	"collectorId": {
	"type": "keyword"
	},
	"content_type": {
	"type": "keyword"
	},
	"disableTimeFilter": {
	"type": "boolean"
	},
	"discoverToRoutes": {
	"type": "boolean"
	},
	"discovery": {
	"properties": {
	"discoverType": {
	"type": "keyword"
	}
	}
	},
	"earliest": {
	"type": "keyword"
	},
	"filter": {
	"type": "keyword"
	},
	"ingestionLag": {
	"type": "long"
	},
	"latest": {
	"type": "keyword"
	},
	"loginBody": {
	"type": "text"
	},
	"loginUrl": {
	"type": "keyword"
	},
	"metadata": {
	"properties": {
	"name": {
	"type": "keyword"
	},
	"value": {
	"type": "keyword"
	}
	}
	},
	"pagination": {
	"properties": {
	"attribute": {
	"type": "keyword"
	},
	"maxPages": {
	"type": "long"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"password": {
	"type": "keyword"
	},
	"plan_type": {
	"type": "keyword"
	},
	"tenant_id": {
	"type": "keyword"
	},
	"timeout": {
	"type": "long"
	},
	"tokenRespAttribute": {
	"type": "keyword"
	},
	"useRoundRobinDns": {
	"type": "boolean"
	},
	"username": {
	"type": "keyword"
	}
	}
	},
	"destructive": {
	"type": "boolean"
	},
	"dir": {
	"type": "keyword"
	},
	"executor": {
	"properties": {
	"collectStep": {
	"type": "keyword"
	},
	"collectibles": {
	"properties": {
	"collectorId": {
	"type": "keyword"
	},
	"compression": {
	"type": "keyword"
	},
	"contentCreated": {
	"type": "date"
	},
	"contentExpiration": {
	"type": "date"
	},
	"contentId": {
	"type": "keyword"
	},
	"contentTime": {
	"type": "float"
	},
	"contentType": {
	"type": "keyword"
	},
	"contentUri": {
	"type": "keyword"
	},
	"earliest": {
	"type": "long"
	},
	"guid": {
	"type": "keyword"
	},
	"host": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"latest": {
	"type": "long"
	},
	"source": {
	"type": "keyword"
	},
	"taskId": {
	"type": "keyword"
	}
	}
	},
	"heartbeatPeriod": {
	"type": "long"
	},
	"input": {
	"properties": {
	"breakerRulesets": {
	"type": "keyword"
	},
	"filter": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"sendToRoutes": {
	"type": "boolean"
	},
	"staleChannelFlushMs": {
	"type": "long"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"jobId": {
	"type": "keyword"
	},
	"keep": {
	"type": "boolean"
	},
	"logLevel": {
	"type": "keyword"
	},
	"reschedule": {
	"type": "long"
	},
	"task": {
	"properties": {
	"conf": {
	"properties": {
	"app_id": {
	"type": "keyword"
	},
	"client_secret": {
	"type": "keyword"
	},
	"collectorId": {
	"type": "keyword"
	},
	"content_type": {
	"type": "keyword"
	},
	"discoverToRoutes": {
	"type": "boolean"
	},
	"earliest": {
	"type": "keyword"
	},
	"filter": {
	"type": "keyword"
	},
	"ingestionLag": {
	"type": "long"
	},
	"latest": {
	"type": "keyword"
	},
	"plan_type": {
	"type": "keyword"
	},
	"tenant_id": {
	"type": "keyword"
	},
	"timeout": {
	"type": "long"
	}
	}
	},
	"destructive": {
	"type": "boolean"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"taskId": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"taskId": {
	"type": "keyword"
	},
	"tasksCompleted": {
	"type": "long"
	},
	"tasksStarted": {
	"type": "long"
	},
	"tcpPort": {
	"type": "long"
	},
	"tcpRouting": {
	"type": "keyword"
	},
	"tenantId": {
	"type": "keyword"
	},
	"text": {
	"type": "keyword"
	},
	"textSecret": {
	"type": "keyword"
	},
	"throttleRatePerSec": {
	"type": "keyword"
	},
	"time": {
	"type": "date"
	},
	"timeout": {
	"type": "long"
	},
	"timeoutSec": {
	"type": "long"
	},
	"timestamp": {
	"type": "long"
	},
	"timestampAnchorRegex": {
	"type": "keyword"
	},
	"timestampEarliest": {
	"type": "keyword"
	},
	"title": {
	"type": "keyword"
	},
	"tls": {
	"properties": {
	"caPath": {
	"type": "keyword"
	},
	"certPath": {
	"type": "keyword"
	},
	"certificateName": {
	"type": "keyword"
	},
	"commonNameRegex": {
	"type": "keyword"
	},
	"disabled": {
	"type": "boolean"
	},
	"enabled": {
	"type": "boolean"
	},
	"maxVersion": {
	"type": "keyword"
	},
	"minVersion": {
	"type": "keyword"
	},
	"ocspCheck": {
	"type": "boolean"
	},
	"passphrase": {
	"type": "keyword"
	},
	"privKeyPath": {
	"type": "keyword"
	},
	"rejectUnauthorized": {
	"type": "boolean"
	},
	"requestCert": {
	"type": "boolean"
	}
	}
	},
	"toCommit": {
	"properties": {
	"offset": {
	"type": "keyword"
	},
	"partition": {
	"type": "keyword"
	},
	"topic": {
	"type": "keyword"
	}
	}
	},
	"toDelete": {
	"type": "long"
	},
	"toDo": {
	"properties": {
	"author": {
	"type": "keyword"
	},
	"description": {
	"type": "text"
	},
	"displayName": {
	"type": "keyword"
	},
	"id": {
	"type": "keyword"
	},
	"tags": {
	"properties": {
	"streamtags": {
	"type": "keyword"
	}
	}
	},
	"version": {
	"type": "keyword"
	}
	}
	},
	"token": {
	"type": "keyword"
	},
	"tokenTTLMinutes": {
	"type": "long"
	},
	"tokenUrl": {
	"type": "keyword"
	},
	"topic": {
	"type": "keyword"
	},
	"topicPartitions": {
	"properties": {
	"numPartitions": {
	"type": "long"
	},
	"topic": {
	"type": "keyword"
	}
	}
	},
	"topics": {
	"type": "text"
	},
	"total": {
	"type": "long"
	},
	"totalBytes": {
	"type": "long"
	},
	"totalEvents": {
	"type": "long"
	},
	"totalGiB": {
	"type": "keyword"
	},
	"totalMessages": {
	"type": "long"
	},
	"totalSize": {
	"type": "long"
	},
	"totalSkips": {
	"type": "long"
	},
	"trigger": {
	"type": "keyword"
	},
	"type": {
	"type": "keyword"
	},
	"udpPort": {
	"type": "long"
	},
	"updated": {
	"type": "long"
	},
	"url": {
	"type": "keyword"
	},
	"urls": {
	"properties": {
	"url": {
	"type": "keyword"
	},
	"weight": {
	"type": "long"
	}
	}
	},
	"useAck": {
	"type": "boolean"
	},
	"useFwdTimezone": {
	"type": "boolean"
	},
	"useRoundRobinDns": {
	"type": "boolean"
	},
	"useToken": {
	"type": "boolean"
	},
	"user": {
	"type": "keyword"
	},
	"user_agent": {
	"properties": {
	"name": {
	"type": "keyword"
	}
	}
	},
	"username": {
	"type": "keyword"
	},
	"version": {
	"type": "keyword"
	},
	"waitPeriod": {
	"type": "long"
	},
	"warning": {
	"properties": {
	"count": {
	"type": "long"
	},
	"message": {
	"type": "text",
	"fields": {
	"keyword": {
	"type": "keyword",
	"ignore_above": 256
	}
	}
	},
	"name": {
	"type": "keyword"
	},
	"stack": {
	"type": "text"
	},
	"type": {
	"type": "keyword"
	}
	}
	},
	"weight": {
	"type": "long"
	},
	"worker": {
	"type": "keyword"
	},
	"workerCount": {
	"type": "long"
	},
	"workerId": {
	"type": "keyword"
	},
	"writeTimeout": {
	"type": "long"
	}
	}
	},
	"settings": {
	"index": {
	"mapping": {
	"total_fields": {
	"limit": 1300
	}
	}
	}
	}
	}
	}

4. Index Templates

Metrics:

PUT _index_template/metrics-cribl-internal
{
  "priority": 500,
  "index_patterns": [
    "metrics-cribl-internal"
  ],
  "composed_of": [
    "metrics-cribl-internal"
  ],
  "data_stream": { },
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.routing_path": ["host.name", "cribl_wp", "_metric"]
    }
  }
}

Logs:

PUT _index_template/logs-cribl-internal
{
  "index_patterns": [
    "logs-cribl-internal"
  ],
  "composed_of": [
    "logs-cribl-internal"
  ],
  "priority": 101,
  "data_stream": {}
}

2. Cribl Stream Configuration

Repeat the following instructions for every Worker Group in your distributed environment:

1. Import the pipeline to remove mapping conflicts:

• Create a new pipeline with the name cribl-internal_rm_conflicts and wait to save it
• Open the Advanced JSON editor.
• Copy and paste the following JSON and then save the pipeline:

Pipeline cribl-internal_rm_conflicts

elastic-cribl-monitoring/ecm_pipeline_config.json

Lines 1 to 656 in b6f4ead

	{
	"id": "cribl-internal_rm_conflicts",
	"conf": {
	"output": "default",
	"streamtags": [],
	"groups": {
	"NwzcZU": {
	"name": "Parity between Worker and Leader Logs",
	"index": 0,
	"disabled": false,
	"description": "Worker logs are through Internal Logs source, Leader are through File Monitor Source. This group will ensure parity"
	},
	"j4Eqzf": {
	"name": "Remove Conflicts on logs",
	"description": "Prevent conflicts with Elasticsearch mappings",
	"index": 1,
	"disabled": false
	},
	"Xci0x7": {
	"name": "Clean data",
	"description": "Place events at the right index and make sure to remove any unnecessary fields",
	"index": 2
	}
	},
	"asyncFuncTimeout": 1000,
	"functions": [
	{
	"filter": "_raw",
	"conf": {
	"mode": "extract",
	"type": "json",
	"srcField": "_raw"
	},
	"id": "serde",
	"groupId": "NwzcZU",
	"disabled": false,
	"description": "Parse _raw if it exists"
	},
	{
	"filter": "source",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "sourcetype",
	"value": "source.match(/(\\/cribl\\.log$)|(\\/cribl\\d\\.log$)/) ? \"cribl:log\" : sourcetype"
	},
	{
	"disabled": false,
	"value": "source.match(/(\\/access\\.log$)|(\\/access\\d\\.log$)/) ? \"cribl:access\" : sourcetype",
	"name": "sourcetype"
	},
	{
	"disabled": false,
	"name": "sourcetype",
	"value": "source.match(/(\\/ui-access\\.log$)|(\\/ui-access\\d\\.log$)/) ? \"cribl:ui-access\" : sourcetype"
	},
	{
	"disabled": false,
	"name": "sourcetype",
	"value": "source.match(/(\\/notifications\\.log$)|(\\/notifications\\d\\.log$)/) ? \"cribl:notifications\" : sourcetype"
	},
	{
	"disabled": false,
	"name": "sourcetype",
	"value": "source.match(/(\\/audit\\.log$)|(\\/audit\\d\\.log$)/) ? \"cribl:audit\" : sourcetype"
	},
	{
	"disabled": false,
	"name": "sourcetype",
	"value": "source.match(/cribl_stderr\\.log$/) ? \"cribl:stderr\" : sourcetype"
	}
	]
	},
	"id": "eval",
	"groupId": "NwzcZU"
	},
	{
	"filter": "_raw",
	"conf": {
	"remove": [
	"_raw"
	]
	},
	"id": "eval",
	"groupId": "NwzcZU",
	"disabled": false
	},
	{
	"id": "comment",
	"filter": "true",
	"conf": {
	"comment": "Conflicts found in the same logging channel"
	},
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "error && typeof error !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "error",
	"value": "JSON.parse(`{\"message\": \"${error}\"}`)"
	}
	]
	},
	"description": "error field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "reason && typeof reason !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "reason",
	"value": "JSON.parse(`{\"name\": \"${reason}\"}`)"
	}
	]
	},
	"description": "reason field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "__e[\"tls\"] !== undefined && typeof tls !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "tls",
	"value": "JSON.parse(`{\"enabled\": ${tls}}`)"
	}
	]
	},
	"description": "TLS field: sometimes a boolean and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "channel == \"func:aggregation\" && __e[\"fields2add\"] !== undefined",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "fields2add",
	"value": "fields2add.filter(Boolean)"
	}
	]
	},
	"groupId": "j4Eqzf",
	"description": "fields2add: sometimes an array with different values, strings or booleans. This removes the booleans and preserves the strings",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "channel.startsWith(\"output:\") && __e[\"excludeFields\"] !== undefined && Array.isArray(excludeFields) && typeof excludeFields[0] === \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "excludeFields",
	"value": "JSON.parse(`{\"fieldname\": ${excludeFields}}`)"
	}
	]
	},
	"groupId": "j4Eqzf",
	"description": "excludeFields: an array of objects but sometimes a string",
	"disabled": false
	},
	{
	"id": "comment",
	"filter": "true",
	"conf": {
	"comment": "Conflicts found in different logging channels"
	},
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "response && typeof response !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "response",
	"value": "JSON.parse(`{\"message\": ${response}}`)"
	}
	]
	},
	"description": "response field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "data && typeof data !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "data",
	"value": "JSON.parse(`{\"message\": ${data}}`)"
	}
	]
	},
	"description": "data field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "errors && typeof errors == \"number\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "errors",
	"value": "JSON.parse(`{\"number\": ${errors}}`)"
	}
	]
	},
	"description": "errors field: sometimes a number and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "all && typeof all == \"number\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "all",
	"value": "JSON.parse(`{\"values\": ${all}}`)"
	}
	]
	},
	"description": "Prevent mapping conflicts for the all field",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "__e[\"ancestors\"] !== undefined && typeof ancestors == \"boolean\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "ancestors",
	"value": "ancestors ? 1 : 0"
	}
	]
	},
	"description": "ancestors field: sometimes a boolean and sometimes a number",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "__e[\"dropped\"] !== undefined && typeof dropped == \"boolean\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "dropped",
	"value": "dropped ? 1 : 0"
	}
	]
	},
	"description": "dropped field: sometimes a boolean (preview channel) and sometimes a number",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "__e[\"enabled\"] !== undefined && typeof enabled == \"boolean\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "enabled",
	"value": "enabled ? 1 : 0"
	}
	]
	},
	"description": "enabled field: sometimes a boolean and sometimes a number",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "__e[\"keepAlive\"] !== undefined && typeof keepAlive == \"boolean\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "keepAlive",
	"value": "keepAlive ? 1 : 0"
	}
	]
	},
	"description": "keepAlive field: sometimes a boolean and sometimes a number",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "__e[\"toDo\"] !== undefined && typeof toDo !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "toDo",
	"value": "JSON.parse(`{\"name\": ${toDo}}`)"
	}
	]
	},
	"description": "toDo field: sometimes null and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "err && err.reason && typeof err.reason == \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "err.reason",
	"value": "JSON.parse(`{\"message\": ${err.reason}}`)"
	}
	]
	},
	"description": "err.reason field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "err && err.req && typeof err.req == \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "err.req",
	"value": "JSON.parse(`{\"body\": ${err.req}}`)"
	}
	]
	},
	"description": "err.req field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "error && error.reason && typeof error.reason == \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "error.reason",
	"value": "JSON.parse(`{\"message\": ${error.reason}}`)"
	}
	]
	},
	"description": "error.reason field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "result && result.error && typeof result.error == \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "result.error",
	"value": "JSON.parse(`{\"message\": ${result.error}}`)"
	}
	]
	},
	"description": "result.error field: sometimes object, sometimes string",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "reason && reason.req && typeof reason.req == \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "reason.req",
	"value": "JSON.parse(`{\"name\": ${reason.req}}`)"
	}
	]
	},
	"description": "sometimes this value is text and sometimes its an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "endpoint && typeof endpoint !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "endpoint",
	"value": "JSON.parse(`{\"url\": ${endpoint}}`)"
	}
	]
	},
	"description": "endpoint field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "pack && typeof pack !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "pack",
	"value": "JSON.parse(`{\"name\": ${pack}}`)"
	}
	]
	},
	"description": "pack field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "reason && reason.reason && typeof reason.reason != \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "reason.reason",
	"value": "JSON.parse(`{\"message\": ${reason.reason}}`)"
	}
	]
	},
	"description": "sometimes this value is text and sometimes its an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"filter": "status && typeof status == \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "status",
	"value": "JSON.parse(`{\"name\": ${status}}`)"
	}
	]
	},
	"id": "eval",
	"groupId": "j4Eqzf",
	"description": "status field: sometimes object, sometimes string, sometimes number. when string, we want an object. numbers will be changed later",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "first && typeof first !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "first",
	"value": "JSON.parse(`{\"__offset\": ${first}}`)"
	}
	]
	},
	"description": "first field: sometimes a string and sometimes an object, but would benefit from being a number",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "memory && typeof memory !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "memory",
	"value": "JSON.parse(`{\"size\": ${memory}}`)"
	}
	]
	},
	"description": "memory field: sometimes a number and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "req && typeof req !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "req",
	"value": "JSON.parse(`{\"name\": ${req}}`)"
	}
	]
	},
	"description": "req field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "file && typeof file !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "file",
	"value": "JSON.parse(`{\"name\": ${file}}`)"
	}
	]
	},
	"description": "file field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "opts && opts.period && typeof opts.period == \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "opts.period",
	"value": "Number(opts.period.match(/\\d*/)[0])"
	}
	]
	},
	"description": "sometimes this value is \"60\" and sometimes its \"60s\". would be good to turn this into a number",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "metricsGCPeriod && typeof metricsGCPeriod == \"string\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "metricsGCPeriod",
	"value": "Number(metricsGCPeriod.match(/\\d*/)[0])"
	}
	]
	},
	"groupId": "j4Eqzf",
	"description": "sometimes this value is \"60\" and sometimes its \"60s\". would be good to turn this into a number",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "provider && typeof provider !== \"object\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "provider",
	"value": "JSON.parse(`{\"name\": ${provider}}`)"
	}
	]
	},
	"description": "provider field: sometimes a string and sometimes an object",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "eval",
	"filter": "timestamp && typeof timestamp == \"object\" && timestamp.length && timestamp.type",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "timestampLength",
	"value": "timestamp.length"
	},
	{
	"disabled": false,
	"name": "timestampType",
	"value": "timestamp.type"
	}
	],
	"remove": [
	"timestamp"
	]
	},
	"description": "timestamp field: sometimes an object and sometimes a string. These same events also contain other timestamp* fields in different formats.",
	"groupId": "j4Eqzf",
	"disabled": false
	},
	{
	"id": "comment",
	"filter": "true",
	"conf": {
	"comment": "Change fields to become compatible with ECS: https://github.com/elastic/ecs/blob/main/generated/csv/fields.csv"
	},
	"groupId": "Xci0x7"
	},
	{
	"filter": "status && typeof status == \"number\"",
	"conf": {
	"add": [
	{
	"disabled": false,
	"name": "http",
	"value": "JSON.parse(`{\"response\":{\"status_code\": ${status}}}`)"
	}
	],
	"remove": [
	"status"
	]
	},
	"id": "eval",
	"groupId": "Xci0x7"
	},
	{
	"id": "comment",
	"filter": "true",
	"conf": {
	"comment": "Set index/remove fields"
	},
	"groupId": "Xci0x7"
	},
	{
	"filter": "true",
	"conf": {
	"remove": [
	"cribl_*"
	]
	},
	"id": "eval",
	"groupId": "Xci0x7"
	}
	]
	}
	}

2. Create a new Elasticsearch Destination named cribl_elasticsearch:

• Set the appropriate Bulk API URL or Cloud ID
• Set the appropriate authentication settings
• Set the DataStream to metrics-cribl-internal
• Configure backpressure behaviour to be persistent queueing
• Ensure the Include document _id advanced setting is disabled

3. Configure the Cribl Internal Metrics Source and from the source configuration page:

• Navigate to Processing Settings > Fields
  • Set an __index field with value metrics-cribl-internal
  • (Optional) Add a custom_id field with a custom value, for example a data centre name
• Navigate to Connected Destinations
  • Set the Pipeline/Pack to cribl_metrics_rollup
  • Set the Destination to elastic:cribl_elasticsearch
• Enable the source

4. Configure the Cribl Internal Logs Source and from the source configuration page:

• Navigate to Processing Settings > Fields
  • Set an __index field with value logs-cribl-internal
  • (Optional) Add a custom_id field with a custom value, for example a data centre name
• Navigate to Connected Destinations
  • Set the Pipeline/Pack to cribl-internal_rm_conflicts
  • Set the Destination to elastic:cribl_elasticsearch
• Enable the source

5. Commit & deploy if your Stream is in a distributed environment

Importing Saved Objects

Dashboards as well as Rules can be imported by uploading the ecm_saved_objects.ndjson file to Kibana.

They can be imported through the Managed Saved Objects Interface.

Note that the setting xpack.encryptedSavedObjects.encryptionKey may need to be set.

Optional Configuration

Leader logs:

Leader logs are currently NOT sent via the Internal Logs source, so for the leader dashboard to work, additional steps need to be performed so leader logs are forwarded to the Elasticsearch destination as well:

1. Install a Cribl Edge node on your Leader node and configure local log collection via a File Monitor source. An easy way to do this would be by using the bootstrap method described here.

ℹ️ When deploying the edge node to your leader node, we recommend having a separate fleet just for this node. Be sure to disable all other inputs on that edge node except for file monitor inputs.

2. Configure the File Monitor source to collect logs by setting the Filename Allowlist modal to /opt/cribl/log/*.log
3. Then you’ll also need to import the cribl-internal_rm_conflicts pipeline from step 2.1 above into the edge environment.
4. And configure the File Monitor Source like the Cribl Internal Logs source in step 2.4.

Additional Considerations

Mappings:

The existing field mapping for logs is set to static and this has been deliberately the chosen for the following reasons:

• prevent excessive mapping conflicts and explosions
• to keep in line with shard sizing across versions
• to have a solid set of known fields as a starting point to choose from

The mappings specified in the component templates may need further adjustments going forward. This is because the list of existing fields will continue to grow as more features are either enabled or added.

General:

1. Due to the static mapping on the logs data streams, the size of the created Logs indices can grow quickly.

   • Consider adjusting the index templates and add an Index Lifecycle Management (ILM) policy with a rollover action

2. Some mapping conflicts could still arise for specific unaccounted for logs

   • The index mapping or pre-processing pipeline attached to the Elasticsearch destination may have to be continuously updated

3. If some interesting fields cannot be searched or aggregated

   • You can add the field to the component template in the index template. You may have to reindex the existing index and/or do a rollover to have new data come in with the updated mapping.

4. The metrics index uses TSDS under the hood, so you can use Elasticsearch’s Downsample ILM action, to reduce storage over time as metrics become less relevant.

Available Dashboards

Name	Description
Cribl Logs - Home	Starting point for troubleshooting issues in a Cribl environment
Cribl Logs - Stats	Minutely Stats created from Internal Logs
Cribl Logs - Stats by Worker Process	Minutely Stats created from Internal Logs but split by Worker Process
Cribl Leader - Overview	Overview of important leader related Statistics and metrics created from the logs of the leader node
Cribl Metrics - Overview	Dashboard based on general metrics on a host level
Cribl Metrics - Worker Processes	The overview of workers broken down by worker process
Cribl Metrics - Source & Destination	The overview of incoming and outgoing data broken down by source and destination

Available Rules

Name	Description
Thruput Threshold	200GB Total Thruput per worker process per day exceeded
Memory Threshold	2GB Memory Usage per worker process exceeded in the last 5 minutes
CPU Threshold	CPU Utilization by Worker Process averages above 95% in the last 5 minutes
PQ Threshold	Persistent Queue size is engaged or exceeded threshold in the last 5 minutes
Source Unhealthy	Destination has been marked as unhealthy in the last 5 minutes
Destination Unhealthy	Destination has been marked as unhealthy in the last 5 minutes

Searching Leader Logs

You can use the Discover App to find leader logs. The included pipeline splits the logs by sourcetype field, allowing you to quickly search for a specific type of log:

sourcetype value	Description
cribl:access	API calls
cribl:audit	Actions pertaining to files
cribl:log	Principal logs
cribl:notifications	Messages that appear in the notification list
cribl:stderr	Garbage collections and OOM killers
cribl:ui-access	Interactions with different UI components

Navigating the available Dashboards

Start by applying the available controls. Every Dashboard can have its own set of Controls. Controls are filters to help narrow down the searches that are sent to the Elasticsearch infrastructure. These are filters to help let you set context on the dashboards. That context can then be carried over to the other dashboards as well by pinning filters.

Controls

ℹ️ Narrowing down the time range can help prevent infrastructure overhead.

• The available values in the controls depend on the data which has been found in the time range.
• Every dashboard will have different controls based on the underlying searched data
• You can easily switch dashboards while keeping context with pinned filters

Drilldowns

Most dashboards included here contain drill downs, allowing you to go into more details.

For example:

Visualizations on the Cribl Metrics - Overview dashboard, allows you to drilldown into the Cribl Metrics - Worker Processes dashboard after observing that one of the workers shows signs of high CPU utilization.

About

These dashboards should help you get started to monitor Cribl Products with Elastic. Feel free to contribute and shoot us a pull request. If you notice any issues, feel free to create an issue in this repository.

You’re welcome to share feedback and ideas in our community slack channel. Are you not a member of our Slack Community? Head over to https://cribl-community.slack.com/

Author: Robbert Hink

Honorable Mentions:

Ben Marcus - General Testing

Jordyn Short - General Testing and contributor