Kibana in `dev` returns duplicate hits for a single log record

Question

Kibana in `dev` returns duplicate hits for a single log record

hannes-ucsc opened this issue 6 years ago · comments

Hannes Schmidt commented 6 years ago

One log record:

Nine hits:

Matt Weiden · Answer 1 · Tue Dec 04 2018 01:52:56 GMT+0800 (China Standard Time)

@hannes-ucsc are you still seeing this behavior? I can't seem to replicate this.

Hannes Schmidt · Answer 2 · Tue Dec 04 2018 02:34:27 GMT+0800 (China Standard Time)

Yes, occasionally. How did you attempt to reproduce it?

Matt Weiden · Answer 3 · Tue Dec 04 2018 03:09:34 GMT+0800 (China Standard Time)

@hannes-ucsc I dug through some log lines similar to the ones you showed and tried to find duplicates

Hannes Schmidt · Answer 4 · Wed Dec 05 2018 05:28:50 GMT+0800 (China Standard Time)

@mweiden I saw this at considerable scale yesterday. I was expecting 130k log messages for a particular Azul operation but got 160k. Here is a query showing such a message that was indexed in quadruplicate:

The key takeaway is that we have four distinct values for _id (the ES document identifier) but the same value for @id (the CloudWatch log record identifier IIRC). Can we use @id to populate _id?

Matt Weiden · Answer 5 · Tue Dec 11 2018 05:32:47 GMT+0800 (China Standard Time)

I think this may have something to do with an OOM issue we're seeing in the dev deployment...

OOM report

REPORT RequestId: 04f76c37-fca4-11e8-a7ec-e52492b51ce1	Duration: 5983.41 ms	Billed Duration: 6000 ms Memory Size: 1024 MB	Max Memory Used: 1024 MB

This is retried

RequestId: 04f76c37-fca4-11e8-a7ec-e52492b51ce1 Process exited before completing request
...
RequestId: 04f76c37-fca4-11e8-a7ec-e52492b51ce1 Process exited before completing request
...
RequestId: 04f76c37-fca4-11e8-a7ec-e52492b51ce1 Process exited before completing request