Errors in InfluxDB logs when using HTTPS connection from NiFi
clever-trevor opened this issue · comments
I'm using the PutInfluxDatabaseRecord processor in a NiFi flow to pull metrics from a Kafka topic and then put into InfluxDB.
The InfluxDB instance is configured to with both SSL and LDAP, and in the InfluxDB Controller Service, I have used :
SSL Conext Service : No Value set
InfluxDB connection URL : https://myserver:8086
InfluxDB Max Connection Time Out : 0 seconds
Username : myuser
Password : mypassword
The data is flowing into influx, but it's incredibly slow, and if I look in the Influx DB logs, I see it getting hammered with these messages
ts=2021-03-25T14:54:48.075630Z lvl=info msg="{\"error\":\"unable to parse authentication credentials\"}" log_id=0T6H~k~1000 service=subscriber
If I disable the NiFi flow, the messages stop immediately.
So why does Influx complain about the data if it's getting inserted to the database, and are these messages slowing down the ingestion rate?
By comparison, I wrote a Python script to do the same thing (same user + endpoint) and that is way faster and doesn't generate the Info messages above
@schmorgs, thanks for using our Apache NiFi bundle, We will take a look.
I'm using the PutInfluxDatabaseRecord processor in a NiFi flow to pull metrics from a Kafka topic and then put into InfluxDB.
The InfluxDB instance is configured to with both SSL and LDAP, and in the InfluxDB Controller Service, I have used :
SSL Conext Service : No Value set InfluxDB connection URL : https://myserver:8086 InfluxDB Max Connection Time Out : 0 seconds Username : myuser Password : mypassword
Is your certificate in NiFi keystore? If not you have to create and configure SSL Context Service. Something like:
The data is flowing into influx, but it's incredibly slow, and if I look in the Influx DB logs, I see it getting hammered with these messages
Do you use aggregations before push data into InfluxDB? The MergeRecord could be useful to create a proper batch before writing into InfluxDB.
Thanks for the feedback there.
As suggested, I created a new keystore and added the InfluxDB certificate into that, and the logs are clean now, and throughput back to where I expected it to be.
It's odd because prior to this, and despite the Influx log messages, the data was still being ingested by INfluxDB, but clearly, it didn't like the fact it was being sent as HTTP (because of no cert) and so caused some performance issue
Anyway, this seems to have resolved the issue so can be closed.