elastic-agent: failed to parse field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]\ "}}, dropping event!
davidg-datascene opened this issue · comments
Error reported in elastic-agent log file with debug logging:
Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2024, time.May, 14, 11, 21, 18, 572370253, time.Local), Meta:{\"input_id\":\"system-metrics\",\"raw_index\":\"metrics-system.process_summary-default\"}
failed to parse field [event.dataset] of type [constant_keyword] in document with id 'LaXWdo8B3mT0fKffBUgR'. Preview of field's value: 'system.process_summary'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"[constant_keyword] field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]\"}}, dropping event!If we change the index template to be the generic metrics as per screen shot then data is sent to the index. However changing it to metrics-system.process.summary index template which should be the correct one then fails with the mentioned error.
Using metrics index template
Should be able to use this index template but errors as above
For confirmed bugs, please report:
- Version:
Elastic Agent: 8.13.2 running on Kubernetes as daemon set
Elastic deployment: 8.11.3 - Steps to Reproduce:
Install elastic agent standalone onto Kubernetes using yaml as provided.
Install k8s integration in Kibana
Check elastic-agents for 400 errors
Snippet showing DataStream
- data_stream:
dataset: system.process_summary
type: metrics
period: 10s
metricsets:
- process_summaryElastic agent standalone yaml
agentNodeDatastreams:
agentYml: |
outputs:
default:
type: elasticsearch
hosts:
- >-
${ES_HOST}
username: ${ES_USERNAME}
password: ${ES_PASSWORD}
agent:
monitoring:
enabled: false
use_output: default
logs: false
metrics: false
providers.kubernetes:
node: ${NODE_NAME}
scope: node
#Uncomment to enable hints' support
#hints.enabled: true
inputs:
- id: kubernetes-cluster-metrics
condition: ${kubernetes_leaderelection.leader} == true
type: kubernetes/metrics
use_output: default
meta:
package:
name: kubernetes
version: 1.52.0
data_stream:
namespace: default
streams:
- data_stream:
dataset: kubernetes.apiserver
type: metrics
metricsets:
- apiserver
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${env.KUBERNETES_SERVICE_HOST}:${env.KUBERNETES_SERVICE_PORT}'
period: 30s
ssl.certificate_authorities:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.event
type: metrics
metricsets:
- event
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
add_metadata: true
- data_stream:
dataset: kubernetes.state_container
type: metrics
metricsets:
- state_container
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_cronjob
type: metrics
metricsets:
- state_cronjob
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_daemonset
type: metrics
metricsets:
- state_daemonset
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_deployment
type: metrics
metricsets:
- state_deployment
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_job
type: metrics
metricsets:
- state_job
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_namespace
type: metrics
metricsets:
- state_namespace
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_node
type: metrics
metricsets:
- state_node
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_persistentvolume
type: metrics
metricsets:
- state_persistentvolume
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_persistentvolumeclaim
type: metrics
metricsets:
- state_persistentvolumeclaim
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_pod
type: metrics
metricsets:
- state_pod
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_replicaset
type: metrics
metricsets:
- state_replicaset
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_resourcequota
type: metrics
metricsets:
- state_resourcequota
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_service
type: metrics
metricsets:
- state_service
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_statefulset
type: metrics
metricsets:
- state_statefulset
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.state_storageclass
type: metrics
metricsets:
- state_storageclass
add_metadata: true
hosts:
- 'prometheus-operator-kube-state-metrics.monitoring.svc.cluster.local:8080'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- id: system-logs
type: logfile
use_output: default
meta:
package:
name: system
version: 1.20.4
data_stream:
namespace: default
streams:
- data_stream:
dataset: system.auth
type: logs
paths:
- /var/log/auth.log*
- /var/log/secure*
exclude_files:
- .gz$
multiline:
pattern: ^s
match: after
processors:
- add_locale: null
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
ignore_older: 72h
- data_stream:
dataset: system.syslog
type: logs
paths:
- /var/log/messages*
- /var/log/syslog*
exclude_files:
- .gz$
multiline:
pattern: ^s
match: after
processors:
- add_locale: null
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
ignore_older: 72h
- id: system-metrics
type: system/metrics
use_output: default
meta:
package:
name: system
version: 1.20.4
data_stream:
namespace: default
streams:
- data_stream:
dataset: system.cpu
type: metrics
period: 10s
cpu.metrics:
- percentages
- normalized_percentages
metricsets:
- cpu
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.diskio
type: metrics
period: 10s
diskio.include_devices: null
metricsets:
- diskio
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.filesystem
type: metrics
period: 1m
metricsets:
- filesystem
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.fsstat
type: metrics
period: 1m
metricsets:
- fsstat
processors:
- drop_event.when.regexp:
system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.load
type: metrics
condition: '${host.platform} != ''windows'''
period: 10s
metricsets:
- load
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.memory
type: metrics
period: 10s
metricsets:
- memory
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.network
type: metrics
period: 10s
network.interfaces: null
metricsets:
- network
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.process
type: metrics
period: 10s
processes:
- .*
process.include_top_n.by_cpu: 5
process.include_top_n.by_memory: 5
process.cmdline.cache.enabled: true
process.cgroups.enabled: false
process.include_cpu_ticks: false
metricsets:
- process
process.include_cpu_ticks: false
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.process_summary
type: metrics
period: 10s
metricsets:
- process_summary
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
dataset: system.socket_summary
type: metrics
period: 10s
metricsets:
- socket_summary
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- data_stream:
type: metrics
dataset: system.uptime
metricsets:
- uptime
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- id: kubernetes-node-metrics
type: kubernetes/metrics
use_output: default
meta:
package:
name: kubernetes
version: 1.52.0
data_stream:
namespace: default
streams:
- data_stream:
dataset: kubernetes.controllermanager
type: metrics
metricsets:
- controllermanager
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${kubernetes.pod.ip}:10257'
period: 10s
ssl.verification_mode: none
condition: ${kubernetes.labels.component} == 'kube-controller-manager'
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.scheduler
type: metrics
metricsets:
- scheduler
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${kubernetes.pod.ip}:10259'
period: 10s
ssl.verification_mode: none
condition: ${kubernetes.labels.component} == 'kube-scheduler'
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.proxy
type: metrics
metricsets:
- proxy
hosts:
- 'localhost:10249'
period: 10s
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.container
type: metrics
metricsets:
- container
add_metadata: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${env.NODE_NAME}:10250'
period: 10s
ssl.verification_mode: none
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.node
type: metrics
metricsets:
- node
add_metadata: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${env.NODE_NAME}:10250'
period: 10s
ssl.verification_mode: none
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.pod
type: metrics
metricsets:
- pod
add_metadata: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${env.NODE_NAME}:10250'
period: 10s
ssl.verification_mode: none
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.system
type: metrics
metricsets:
- system
add_metadata: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${env.NODE_NAME}:10250'
period: 10s
ssl.verification_mode: none
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:
- data_stream:
dataset: kubernetes.volume
type: metrics
metricsets:
- volume
add_metadata: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
hosts:
- 'https://${env.NODE_NAME}:10250'
period: 10s
ssl.verification_mode: none
processors:
- add_fields:
target: orchestrator.cluster
fields:
name: "tools-prod"
- drop_fields:
fields: [/kubernetes.apiserver.*/, /kubernetes.namespace_labels.*/]
- add_kubernetes_metadata:Snippet from the metrics-system.process.summary Managed index template
failed to parse field [event.dataset] of type [constant_keyword] Preview of field's value: 'system.process_summary'","caused_by":{"type":"illegal_argument_exception","reason":"[constant_keyword] field [event.dataset] only accepts values that are equal to the value defined in the mappings [system.process.summary], but got [system.process_summary]"}}, dropping event!
"event": {
"properties": {
"agent_id_status": {
"type": "keyword",
"ignore_above": 1024
},
"dataset": {
"type": "constant_keyword",
"value": "system.process.summary" <----- **_Shouldn't this be system.process_summary_**
},
"ingested": {
"type": "date",
"format": "strict_date_time_no_millis||strict_date_optional_time||epoch_millis"
},
"module": {
"type": "constant_keyword",
"value": "system"
}
}
},
This issue doesn't have a Team:<team> label.
The system integration is using system.process.summary so that is the correct one:
title: System process_summary metrics
dataset: system.process.summaryIt appears that our reference YAML is out of sync: