flatMapConcat does not buffer

[kraf]

flatMapConcat does not seem to buffer the events of the other stream(s). Am I understanding the docs wrong here?

var s1 = Bacon.sequentially(100, [1,2,3]);
var s2 = Bacon.sequentially(100, [4,5,6,7,8,9,10]);
var s = Bacon.fromArray([s1, s2]).flatMapConcat(function(stream) { return stream; });

s2.onValue(function(x) {
    console.log('x', x);
});

s.onValue(function(d) {
    console.log(d);
});
s.onEnd(function() {
    console.log('<END>');
});

This outputs

x 4
1
x 5
2
x 6
3
x 7
7
x 8
8
x 9
9
x 10
10
<END>

I would like something like

...
x 6
3
4
5
6
x 7
7
x 8
8
...

Background: I am trying to read from a database up to the current point in time and seamlessly switch over to a live stream of data getting pushed into the store. I can do the buffering manually but was wondering if this can be accomplished using the primitives.

[Oleg Grenrus]

This works as expected. flatMapConcat doesn't buffer s2: it doesn't subscribe for it before s1 ends. As s2 is hot (produces element regardless whether there are subscribers) the events are lost.

You can implement your behaviour as:

var Bacon = require('baconjs');

var s1 = Bacon.sequentially(100, [1,2,3]);
var s2 = Bacon.sequentially(100, [4,5,6,7,8,9,10]);

// buffer s2
// there should be shortcut for this
var bufferedS2 = s2.holdWhen(s1.map(true).startWith(true).mapEnd(false));

// now events are timed properly, so we can just merge the streams
// no need to flatmap, hooray!
var s = Bacon.mergeAll([s1, bufferedS2]);

s1.log('s1');
s2.log('s2');
bufferedS2.log('buf s2');
s.log('s');

Which outputs:

s1 1
s 1
s2 4
s1 2
s 2
s2 5
s1 3
s 3
s1 <end>
buf s2 4
s 4
buf s2 5
s 5
s2 6
buf s2 6
s 6
s2 7
buf s2 7
s 7
s2 8
buf s2 8
s 8
s2 9
buf s2 9
s 9
s2 10
buf s2 10
s 10
s2 <end>
buf s2 <end>
s <end>

[Juha Paananen]

@phadej that's correct. FlatMapConcat only subscribes to the next stream after the first one has ended. A bufferingFlatMapConcat combinator would be nice!

[Juha Paananen]

Actually you're looking for a bufferingConcat method, which could be made by applying @phadej's approach recursively. That wouldn't be very efficient way of course, performance-wise.

But it could be the first implementation. With good tests, we may replace the implementation with a more efficient one later.

[kraf]

Thanks, that works! An efficient bufferingConcat would be a great addition.

[Juha Paananen]

If you want to contribute, please submit PR for the "naive" version with tests.

[kraf]

I do want to contribute, but I'm a little stuck. I did the straight forward implementation like you suggested:

Bacon.EventStream :: bufferingConcat = (right) ->
  left = this
  new EventStream describe(left, "bufferingConcat", right), (sink) ->
    bufferedRight = right.holdWhen(left.map(true).startWith(true).mapEnd(false))
    unsubStream = left.merge(bufferedRight).subscribe sink
    -> unsubStream()

I copied the tests from concat as a starting point and they fail at Bacon.once. I traced this back to holdWhen not working with Bacon.once. I tried to fix it but I can't get this test I wrote to work:

#specs/holdwhen.coffee
describe "Works with Bacon.once()", -> 
  expectStreamEvents(
    ->
      Bacon.once(2).
        holdWhen(Bacon.later(1000, false).startWith(true))
    [2])

I've run into the issue with Bacon.once behaving differently due to calling subscribers synchronously so I tried the test with Bacon.once(2).delay(0) which then worked. I still can't figure out how to fix holdWhen. Do you have an idea?

[Juha Paananen]

You're correct in that holdWhen doesn't play ball with synchronously responding sources. The problem is that here it first subscribes to the source once and then subscribes later on lines 19, 21. This causes the events from the synchronously responding source to be missed by the later subscriptions. Haven't thought about how to fix this yet. This is one more problem related to #367

[Juha Paananen]

Ok, I re-implemented holdWhen in the fix/589 branch to work with synchronous sources. Does this help?

[kraf]

Thanks! I merged the branch and it's almost working now, but this one test fails:

describe "provides values from streams in given order and ends when both are exhausted", ->
  expectStreamEvents(
    ->
      left = series(2, [1, error(), 2, 3])
      right = series(1, [4, 5, 6])
      left.bufferingConcat(right)
    [1, error(), 2, 3, 4, 5, 6], semiunstable)

because only [ 1, error(), 2, 3, 4 ] is returned.

In this case holdWhen is actually sending [ 4, <end>, 5, 6 ] to the subscriber. When I move the end to the next tick it works:

  #holdwhen.coffee:29
  else if event.isEnd()
    setTimeout ( ->
      endIfBothEnded(unsubMe)
    ), 0

This doesn't feel right but I don't know the internals well enough yet to suggest something real. I don't get why this is happening tbh. Shouldn't the flushing iteration in holdWhen call sink synchronously three times?

[Juha Paananen]

Could you point me to your actual or, or even more prefreably, try and write a failing test against holdWhen?

[kraf]

I seem to only get the exact use case of bufferingConcat to fail holdWhen, maybe it's related to mapEnd. I pushed the test to my copy of the branch: https://github.com/kraf/bacon.js/tree/fix/589

[Juha Paananen]

I added your test to master and it's green. 06df359

I did change the expected output from [1,2,3,4,5] to [1,2,3,4,5,6,7,8,9,10] though.

[kraf]

Apologies for the bad commit. I updated the test on my branch and it's failing now.

[Juha Paananen]

Ah, ok. Got it. This indeed produces unexpected results. Thanks.

[Juha Paananen]

Found it. I had mistakenly used subscribe instead of subscribeInternal, which caused the End event to be processed while flushing the buffered values [4,5,6]. The subscribe method is meant for actual side-effects while the subscribeInternal method is a more direct subscribe for internal use.

Releasing 0.7.65 with the fix.

[kraf]

I created a pull request with bufferingConcat added.