Giters

influxdata / influxdb-client-ruby

InfluxDB 2.0 Ruby Client

Home Page:https://influxdata.github.io/influxdb-client-ruby/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Delete data points

DevExplorer101 opened this issue · comments

commented

I am trying to delete data points from the db but got this error.
Error: Failed to delete data: Not implemented.

Steps to reproduce:
influx write --bucket 'test' 'mem,host=host1 used_percent=23.43234543 1556892576842902000'
Then from the console,
client.create_delete_api.delete(5.years.ago, DateTime.now, predicate: "host=host1")
I have also tried running from the cli and see the same error.

The bucket retention policy is set to forever. Does that has any effect on deletion?

Appreciate any help on this.

commented

Hi @adriany3m,

thanks for using our client.

The delete API works only for the Influx Cloud instance.

In InfluxDB OSS 2.0rc, the delete with predicate API (/api/v2/delete) has been disabled and returns a 501 Not implemented message.

Related issue: influxdata/influxdb#19635

Regards

commented

Thank you for the reply @bednar

Is the delete api going to be implemented in the near future?

I have another question. I have a use case to import data from few thousands large csv files into influxdb. The current implementation is to basically loop through all files, read and convert the contents to the accepted protocol, and batch writing into the db. During the writing process, these errors occasionally appear and the db server crashes.
The retriable error occurred during writing of data. Reason: 'unexpected error w riting points to database: timeout

I am sure the problem is with my implementation. Appreciate any insight on this. Thanks!

commented

Is the delete api going to be implemented in the near future?

The delete api is implemented in the client, but we are waiting for - influxdata/influxdb#19635.

During the writing process, these errors occasionally appear and the db server crashes.
The retriable error occurred during writing of data. Reason: 'unexpected error writing points to database: timeout

This looks like as hardware sizing problem, try to check this document: https://docs.influxdata.com/influxdb/v1.8/guides/hardware_sizing/#data-nodes

commented

Thank you for the reply @bednar

I have increased our hardware requirements but still facing issue with completing the entire import. Now while importing the csv, the job gets terminated half way with this error can't create Thread: Resource temporarily unavailable (ThreadError) and the db is also running on very high load.

commented

Hi @adriany3m,

How your import looks like? Do you use LineProtocol or Point? Do you use synchronous or batching mode?

There is also CLI tool for import CSV - https://docs.influxdata.com/influxdb/cloud/write-data/developer-tools/csv/. Maybe it could be useful for initial import.

commented

Basically, i am using a ruby script to open and parse the csv files and convert it to a line protocol format. Quite similar to the one provided in the documentation but i am looping through more than 10000 csv files with a combined 50M lines approximately.

hash = { name: 'h2o', tags: { host: 'aws', region: 'us' }, fields: { level: 5, saturation: '99%' }, time: 123 }

write_api = client.create_write_api
write_api.write(data: ['h2o,location=west value=33i 15', point, hash])

I am using batching mode with the following configuration.

InfluxDB2::WriteOptions.new( write_type: InfluxDB2::WriteType::BATCHING, batch_size: 100_000, flush_interval: 10_000, max_retries: 1, max_retry_delay: 180_000, exponential_base: 2 )

I have tried using the csv tool you suggested and also experimented with telegraf but it seems like they are not any faster than the current implementation so I opted for a simple way.

Thank you very much for your insight.

commented

There is a document how to optimize writes into InfluxDB: https://docs.influxdata.com/influxdb/cloud/write-data/best-practices/optimize-writes/

I think you could try following settings:

1/ Use lower batch_size maybe 10_000
2/ Be sure that tags are sorted
3/ Use a proper precision... probably seconds: write_api.write(data: ['h2o,location=west value=33i 15', point, hash], precision: InfluxDB2::WritePrecision::SECOND)