Randomly breaking messages under load

Question

Randomly breaking messages under load

giner opened this issue 6 years ago · comments

Stanislav German-Evtushenko commented 6 years ago

I found a bug - here are some steps to recreate it.

Versions of loggregator prior to 101.4 (tested versions 99, 100, 101 and 101.3) are randomly breaking messages under load.

Here is how to reproduce:

mkdir -p /tmp/testlog && cd /tmp/testlog && touch .testlog & cf push testlog -u process -c 'data=$(seq -s " " 1 20000); while true; do echo "$data"; done' -b https://github.com/cloudfoundry/binary-buildpack

# Wait for a few minutes and check doppler logs, you are likely to see this:
...
2018/11/21 01:03:24 Received bad envelope: proto: wrong wireType = 1 for field SourceInstance
2018/11/21 01:06:24 Received bad envelope: proto: wrong wireType = 0 for field SourceInstance
2018/11/21 01:07:24 Received bad envelope: proto: wrong wireType = 6 for field SourceInstance
2018/11/21 01:07:37 Received bad envelope: proto: wrong wireType = 7 for field SourceInstance
2018/11/21 01:07:37 Received bad envelope: proto: wrong wireType = 1 for field SourceInstance
...

cf stop testlog

Notes:

We cannot reproduce the issue with loggregator 101.4
We could not reproduce the issue on bosh-lite

cf-gitbot · Answer 1 · Thu Nov 22 2018 07:59:54 GMT+0800 (China Standard Time)

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/162144887

The labels on this github issue will be updated when the story is started.

Stanislav German-Evtushenko · Answer 2 · Thu Nov 22 2018 09:07:24 GMT+0800 (China Standard Time)

One more piece: upgrading only metron without upgrading doppler to the version from loggregator-release v101.4 also solves the issue.

Travis Patterson · Answer 3 · Wed Dec 05 2018 01:15:54 GMT+0800 (China Standard Time)

Hello, thanks for the bug report.

Do you have a use case for a Loggregator release older than 101.4? If not, we'd suggest you upgrade.

Johannes Tuchscherer · Answer 4 · Wed Dec 05 2018 02:42:09 GMT+0800 (China Standard Time)

Closing for now, since this seems to be fixed in newer versions. @giner Please let us know if you have any concerns or problems around updating to a newer version of loggregator.

Thanks for reporting this, though. It is good to know and I will put add a line to the release notes for the affected version.

Stanislav German-Evtushenko · Answer 5 · Wed Dec 05 2018 05:49:43 GMT+0800 (China Standard Time)

@jtuchscherer, we will upgrade to the fixed version soon. There seem to be no real blockers. Though another problem (#387) was introduced in the newer releases but we will reduce the chance of this to happen by lowering number of lines buffered in doppler.

Johannes Tuchscherer · Answer 6 · Wed Dec 05 2018 05:56:24 GMT+0800 (China Standard Time)

@giner But even that issue should be fixed in a more recent version. I hope you would be able to upgrade to at least version 103.2. If not, please let us know. We would be happy to work with you on that.

Stanislav German-Evtushenko · Answer 7 · Wed Dec 05 2018 06:02:37 GMT+0800 (China Standard Time)

@jtuchscherer if I remember correctly the issue with recent logs was fixed after (or with) introduction of logcache. I'm not sure we can safely upgrade to that one yet. Though it's lesser of an issue, I think we can live with it for a while until we can upgrade.