Backpressure handling seems incorrect

[Piotr Kołaczkowski]

I'm doing:
StringObservable.byLine(StringObservable.decode(observableOfArrays, "UTF-8"))

The source observable handles backpressure and emits items only on demand, when they are requested. If lines of text span multiple arrays, processing of the stream stops after some time.

This is probably because the end subscriber requests 128 items on subscribe and this request gets passed upstream to the source observable without any modification. It seems no code in RxJava / RxJavaString takes into account the fact that the number of items produced by the source observable may be different than the number of lines finally produced at the end of the pipeline. In my case, because the source observable has to emit more arrays than 128 to build 128 lines, soon the end subscriber gets starved and processing stops.

[George Campbell]

thanks for the report. I'll double check the logic. Do you have a test case that I could use to validate?

[Piotr Kołaczkowski]

Unfortunately this is happening in our closed-source part of the code which I can't share. I disabled backpressure on the source observable by requesting MAX_LONG items and everything works fine now, so the issue is not very critical. I'll try to create a minimized example once I have some spare time.

[Dave Moten]

Hi, StringObservable.byLine uses StringObservable.split which does not support backpressure yet. You can either apply .onBackpressureBuffer() to StringObservable.byLine (which you've done) or try Strings.split from rxjava-extras which does support backpressure.

I've talked about submitting the code for Strings.split to here but it requires an operator called Transformers.stateMachine that didn't attract enough interest to make it into the core library (which is where it should live as opposed to here).

[Jan Zajic]

I made PR #44 to fix this issue. I think it's that simple. It's work fine in out internal source codes, but if someone find some issue with this solution I am open to discussion.

[Tom Wieczorek]

I added a simple test case that tests backpressure. It succeeds with jan-zajic's patch and fails without it: 2071573.

Any chance that this gets merged and we get a new release? 😄

[George Campbell]

Sure. I'll make some time today to get this in.

[George Campbell]

new version of split has been released.