telegraf inputs.influxdb_v2_listener plugin not working

Question

telegraf inputs.influxdb_v2_listener plugin not working

nkurn opened this issue 2 years ago · comments

nkurn commented 2 years ago

I am running the latest docker image

I am getting page not found when doing curl

curl -v http://localhost:8086/

Trying 127.0.0.1:8086...
Connected to localhost (127.0.0.1) port 8086 (#0)

GET / HTTP/1.1
Host: localhost:8086
User-Agent: curl/7.74.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Tue, 24 May 2022 22:51:29 GMT
< Content-Length: 19
<
404 page not found
Connection #0 to host localhost left intact

curl -i -XPOST 'http://localhost:8086/api/v2/write' --data-binary 'cpu_load_short,host=test,region=us-east value=0.64 1434055562000000000'
HTTP/1.1 204 No Content
Date: Tue, 24 May 2022 22:57:09 GMT

netstat -anlp

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 172.18.222.37:34570 10.202.25.1:31182 ESTABLISHED 7/telegraf
tcp6 0 0 :::8086 :::* LISTEN 7/telegraf
tcp6 0 0 172.18.222.37:8086 172.18.15.21:39586 ESTABLISHED 7/telegraf
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path

config file

# # Accept metrics over InfluxDB 2.x HTTP API
[[inputs.influxdb_v2_listener]]
#   ## Address and port to host InfluxDB listener on
#   ## (Double check the port. Could be 9999 if using OSS Beta)
   service_address = ":8086"
#
#   ## Maximum allowed HTTP request body size in bytes.
#   ## 0 means to use the default of 32MiB.
#   # max_body_size = "32MiB"
#
#   ## Optional tag to determine the bucket.
#   ## If the write has a bucket in the query string then it will be kept in this tag name.
#   ## This tag can be used in downstream outputs.
#   ## The default value of nothing means it will be off and the database will not be recorded.
#   # bucket_tag = ""
#
#   ## Set one or more allowed client CA certificate file names to
#   ## enable mutually authenticated TLS connections
#   # tls_allowed_cacerts = ["/etc/telegraf/clientca.pem"]
#
#   ## Add service certificate and key
#   # tls_cert = "/etc/telegraf/cert.pem"
#   # tls_key = "/etc/telegraf/key.pem"
#
#   ## Optional token to accept for HTTP authentication.
#   ## You probably want to make sure you have TLS configured above for this.
#   # token = "some-long-shared-secret-token"
#
#   ## Influx line protocol parser
#   ## 'internal' is the default. 'upstream' is a newer parser that is faster
#   ## and more memory efficient.
#   # parser_type = "internal"

logs

2022-05-24T22:37:34Z I! Starting Telegraf 1.22.2
2022-05-24T22:37:34Z I! Loaded inputs: cpu disk diskio influxdb_v2_listener kernel mem processes swap system
2022-05-24T22:37:34Z I! Loaded aggregators:
2022-05-24T22:37:34Z I! Loaded processors:
2022-05-24T22:37:34Z I! Loaded outputs: influxdb_v2
2022-05-24T22:37:34Z I! Tags enabled: host=storage-telegraf-be-0
2022-05-24T22:37:34Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"storage-telegraf-be-0", Flush Interval:10s
2022-05-24T22:37:34Z I! [inputs.influxdb_v2_listener] Started HTTP listener service on :8086

nkurn · Answer 1 · Wed May 25 2022 06:59:12 GMT+0800 (China Standard Time)

can someone please tell me if I am missing something for this plugin to work

thanks,
Naga.

Joshua Powers · Answer 2 · Wed May 25 2022 22:01:40 GMT+0800 (China Standard Time)

Hi,

curl -i -XPOST 'http://localhost:8086/api/v2/write' --data-binary 'cpu_load_short,host=test,region=us-east value=0.64 1434055562000000000'
HTTP/1.1 204 No Content

The 204 return code means the data was successfully processed and there is no content to return. This means that the endpoint is working as expected. If you enable some debugging in your logs or use the [[inputs.file]] output, then you can also see the metric get processed:

❯ curl -i -XPOST 'http://localhost:8086/api/v2/write' --data-binary 'cpu_load_short,host=test,region=us-east value=0.64 1434055562000000000'
HTTP/1.1 204 No Content
Date: Wed, 25 May 2022 13:59:59 GMT

2022-05-25T13:59:47Z I! [inputs.influxdb_v2_listener] Started HTTP listener service on :8086
2022-05-25T13:59:57Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
cpu_load_short,host=test,region=us-east value=0.64 1434055562000000000
2022-05-25T14:00:07Z D! [outputs.file] Wrote batch of 1 metrics in 44.85µs
2022-05-25T14:00:07Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

Hopefully, that helps clear up how the plugin is working.

nkurn · Answer 3 · Wed May 25 2022 23:38:51 GMT+0800 (China Standard Time)

thanks fort that, if I had specified my outputs as influxdb

[[outputs.influxdb_v2]]

should I have seen this in influxdb ?

my goal here is to see this data in influxdb, how do I do that?

thanks a lot for your time

Naga.

Joshua Powers · Answer 4 · Wed May 25 2022 23:40:55 GMT+0800 (China Standard Time)

should I have seen this in influxdb ?

Assuming there were no other error messages, yes. I would again suggest running with debug mode if you are concerned about it and see what the messages show.

nkurn · Answer 5 · Thu May 26 2022 00:00:14 GMT+0800 (China Standard Time)

how do I enable debugging

this is my entrypoint file

Also I am getting other stats from this host to influxdb, just not seeing this cpu_load_short

#!/bin/bash
telegraf --config /etc/telegraf/telegraf.conf &
while true ; do sleep 24h ; done

thanks,
Naga.

nkurn · Answer 6 · Thu May 26 2022 00:01:38 GMT+0800 (China Standard Time)

nvm, I see

Log at debug level.

debug = false

Log only error level messages.

quiet = false

I will enable this

nkurn · Answer 7 · Thu May 26 2022 00:14:41 GMT+0800 (China Standard Time)

I do see it sent the extra metric (60 instead of 59) than normal when I ran the curl POST, but how do I see this influxdb I am filtering by host

curl -i -XPOST 'http://storage-telegraf-be:8086/api/v2/write' --data-binary 'cpu_load_short,host=storage-telegraf-be-0,region=us-east value=0.64 1434055562000000000'

I filter by host storage-telegraf-be-0 and measurement cpu_load_short, but I do not see it in influxdb UI

2022-05-25T16:10:16Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-05-25T16:10:26Z D! [outputs.influxdb_v2] Wrote batch of 59 metrics in 87.983629ms
2022-05-25T16:10:26Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-05-25T16:10:36Z D! [outputs.influxdb_v2] Wrote batch of 59 metrics in 104.083835ms
2022-05-25T16:10:36Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-05-25T16:10:46Z D! [outputs.influxdb_v2] Wrote batch of 60 metrics in 108.415584ms
2022-05-25T16:10:46Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics
2022-05-25T16:10:56Z D! [outputs.influxdb_v2] Wrote batch of 59 metrics in 95.398646ms
2022-05-25T16:10:56Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

nkurn · Answer 8 · Thu May 26 2022 00:17:51 GMT+0800 (China Standard Time)

nkurn commented 2 years ago

nkurn · Answer 9 · Thu May 26 2022 00:18:48 GMT+0800 (China Standard Time)

I do see other data

Joshua Powers · Answer 10 · Thu May 26 2022 00:23:22 GMT+0800 (China Standard Time)

If you look at the query on the right it says "Past 1m", which stands for last minute. The timestamp of that example metric you added: 1434055562000 translate to June 11, 2015(?) I think, so it will not show up in that query.

nkurn · Answer 11 · Thu May 26 2022 00:24:07 GMT+0800 (China Standard Time)

awesome !! that makes sense :))

thanks, I will close this