write_points does not write data
danarnold opened this issue · comments
When upgrading to influxdb-ruby v0.3.8 from v0.3.0, we realized after many hours that no data was being written.
The client is being initialized like this:
INFLUXDB = InfluxDB::Client.new('database_name', host: 'host', username: 'username', retry: 10, password: 'password', use_ssl: true)
The data being written, where data is a valid array of points with timestamps:
2.2.5 :001 > INFLUXDB.write_points(data, 's')
{
"content-type" => [
[0] "application/json"
],
"date" => [
[0] "Fri, 16 Sep 2016 05:46:10 GMT"
],
"request-id" => [
[0] "df1bf0d9-7bd0-11e6-b600-000000000000"
],
"x-influxdb-version" => [
[0] "1.0.0-c1.0.0"
],
"connection" => [
[0] "Close"
]
}
No data is written, which is easily noticeable by querying on the series that was just written to. Downgrading back to v0.3.0 fixed this immediately.
How does data look like?
Here is an example:
[{
:series => "example",
:tags => {
:id => "5678c9c86361735fc6010000"
},
:values => {
:value => 1
},
:timestamp => 1474056205
},
{
:series => "example",
:tags => {
:id => "5678c9c86361735fc6010000"
},
:values => {
:value => 1
},
:timestamp => 1474056227
}]Okay, thanks. I'll have a look.
Did you try other versions beside 0.3.0 and 0.3.8?
No, I went straight to 0.3.8. If you can't reproduce it, I could try the other versions. Let me know if that's something that would be helpful.
No, it's OK. I need to create reproduction code anyway and the number of Gem versions I run this against doesn't matter that much.
Would you tell me the Ruby version you're using?
Sure. ruby 2.2.5p319 (2016-04-26 revision 54774)
Hm... I cannot reproduce this, my test case (Gist) passes:
$ ./repro.sh
[PASS] using Gem version 0.3.0
[PASS] found empty database
[PASS] writing successful
[PASS] got expected result
[PASS] using Gem version 0.3.1
[PASS] found empty database
[PASS] writing successful
[PASS] got expected result
[PASS] using Gem version 0.3.2
[PASS] found empty database
[PASS] writing successful
[PASS] got expected result
[PASS] using Gem version 0.3.4
[PASS] found empty database
[PASS] writing successful
[PASS] got expected result
[PASS] using Gem version 0.3.5
[PASS] found empty database
[PASS] writing successful
[PASS] got expected result
[PASS] using Gem version 0.3.6
[PASS] found empty database
[PASS] writing successful
[PASS] got expected result
[PASS] using Gem version 0.3.7
[PASS] found empty database
[PASS] writing successful
[PASS] got expected result
[PASS] using Gem version 0.3.8
[PASS] found empty database
[PASS] writing successful
[PASS] got expected result
[PASS] all 0.3.x versions passed
Can you try this locally?
Looking at the Changelog, I found that 0.3.1 introduced better handling of Integer values, which might be the problem here.
The gem upto (and including) v0.3.0 did not encode Integer values as <value>i on the line protocol (see this diff), so InfluxDB had converted those fields to float64 (Go's 64bit unsigned float type). I believe, writing int64's into a float64 field (or vice versa) silently (?) fails.
To verify this, you can try to insert this point:
{
series: "example",
tags: { id: "5678c9c86361735fc6010000" },
values: { value: 1.to_f } # [!] convert to float
timestamp: 1474056227
}(AFAIK there's currently no way to determine the type of a field, see influxdata/influxdb#6063)
With my local database running InfluxDB 0.13.0-1, it completed successfully. I'm upgrading to 1.0.0-1 to see if that changes things. What version did you try against?
I have 1.0.0 running (on a Debian system).
Completing your script with version 1.0.0-1 did not change anything, the test still completed successfully in the test database. (By the way, I had to change your count method to get it to work, I'll comment that on the gist.)
Trying again with 0.3.8 and writing into our existing series, it failed with an integer value and then succeeded if I converted the value to a float. I think you've nailed the cause there.
Here's the session showing that this works:
(main)> data = {
:series => "series_name",
:tags => {
:id => "57aa0842f2195443ea00033c"
},
:values => {
:value => 1
},
:timestamp => 1474061514
}
(main)> INFLUXDB.write_points([data], 's')
{
"content-type" => [
[0] "application/json"
],
"date" => [
[0] "Fri, 16 Sep 2016 21:32:01 GMT"
],
"request-id" => [
[0] "0110d12c-7c55-11e6-8d18-000000000000"
],
"x-influxdb-version" => [
[0] "1.0.0-c1.0.0"
],
"connection" => [
[0] "Close"
]
}
(main)> INFLUXDB.query("select count(value) from series_name where id='#{i.id.to_s}' and time > now() - 1m")
[]
(main)> data[:values][:value] = 1.to_f
1.0
(main)> INFLUXDB.write_points([data], 's')
{
"content-type" => [
[0] "application/json"
],
"date" => [
[0] "Fri, 16 Sep 2016 21:32:39 GMT"
],
"request-id" => [
[0] "17e6802e-7c55-11e6-8db6-000000000000"
],
"x-influxdb-version" => [
[0] "1.0.0-c1.0.0"
],
"connection" => [
[0] "Close"
]
}
(main)> INFLUXDB.query("select count(value) from series_name where id='#{i.id.to_s}' and time > now() - 1m")
[
[0] {
"name" => "series_name",
"tags" => nil,
"values" => [
[0] {
"time" => "2016-09-16T21:31:41.817049749Z",
"count" => 1
}
]
}
]Glad to see that we've found a workaround. I hope you didn't lose too much valuable data...
We actually did lose quite a bit of data. 😢
I'm wondering, for the sake of other users - are there any plans to fix this and raise an error if the data isn't written because the field type is different? I can't see how silently failing to write data could be seen as acceptable behavior. At the very least, I feel the Changelog and/or README should be updated to reflect this change, since there will certainly be users who are upgrading from versions <=0.3.0 who will also be affected by this and won't be checking closed issues before they upgrade.
Well, on the client side, we can't detect the field type, and the #write* methods return a Net::HTTPNoContent in both cases.
I will add a note to the README, though.
Great. I think I'll look into opening an issue on the server project, then, since I feel it's reasonable to expect that a server return an error when it fails to write data.
Actually, the server does return an error, which is already handled in the gem. Changing the first value in the Gist to a float (l. 33) creates an InfluxDB::Error in line 42:
InfluxDB::Error: {"error":"field type conflict"}
.../gems/influxdb-0.3.2/lib/influxdb/client/http.rb:84:in `resolve_error'
.../gems/influxdb-0.3.2/lib/influxdb/client/http.rb:35:in `block in post'
.../gems/influxdb-0.3.2/lib/influxdb/client/http.rb:53:in `connect_with_retry'
.../gems/influxdb-0.3.2/lib/influxdb/client/http.rb:26:in `post'
.../gems/influxdb-0.3.2/lib/influxdb/query/core.rb:62:in `write'
.../gems/influxdb-0.3.2/lib/influxdb/query/core.rb:48:in `write_points'
Yeah, you're right about that. I'm confused - why wouldn't I be getting a field type conflict error, then?
That is indeed confusing... I will think about this and check back with you tomorrow (it was a long week so far and I need to catch up some sleep ;-))
OK, sounds good. Thanks for your help!
After some experimentation, I couldn't get no error when there's a type mismatch... However, maybe the server log might be the right place to look next.
[write] 2016/09/17 10:11:41 write failed for shard 387: field type conflict: input field "value" on measurement "example" is type int64, already exists as type float
[httpd] ::1 - username [17/Sep/2016:10:11:41 +0200] "POST /write?db=issue164&p=%5BREDACTED%5D&precision=s&u=username HTTP/1.1" 400 123 "-" "Ruby" 799be58d-7cbc-11e6-8001-000000000000 1533
Can you check whether your server logs contain similar entries? If your OS runs systemd, you can narrow your log window with
[sudo] journalctl --unit influxdb.service --since="2016-09-16 15:00" --until="2016-09-16 18:00"Otherwise you should find the logs in /var/log/influxdb/influxd.log{,-*.gz}.
(At this point, I'd suggest to escalate your issue to the server project.)
We're hosted on InfluxDB cloud, so I'll have to ask them if they can check the logs for us.
Well... if that doesn't work, you could try this monkey patch to get low level debug logs (cf. original):
require "influxdb"
DEBUG_LOGGER = File.open("./debug.log", "w")
module InfluxDB::HTTP
def do_request(http, req, data = nil)
req.basic_auth config.username, config.password if basic_auth?
req.body = data if data
http.set_debug_output DEBUG_LOGGER
http.request(req)
ensure
# http.set_debug_output(nil) creates a warning
http.instance_variable_set :@debug_ouput, nil
end
end
INFLUXDB = InfluxDB::Client.new # ...
# your code here
DEBUG_LOGGER.close