scramjetorg / scramjet

Public tracker for Scramjet Cloud Platform, a platform that bring data from many environments together.

Home Page:https://www.scramjet.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Reading from subject or result of Datastream.tee() appears to fail silently beyond 32nd item.

alexdunncs opened this issue · comments

Basics
Affects master and teed streams when tee() is called on the master.

Describe the bug
The process silently halts, ostensibly when attempting to read the 33rd item from either stream.

To Reproduce (self-contained except node/scramjet)

const scramjet = require("scramjet");

async function test()  {
    try {

        const minimaltestArr = [{x: 1}, {x: 2}, {x: 3}, {x: 4}, {x: 5}]; // Succeeds
        let thirtyTwoElements = Array.from(Array(32).keys()); //Succeeds
        let thirtyThreeElements = Array.from(Array(33).keys()); //Silently fails/exits, ostensibly after 32nd element (index 31)
        let fiftyElements = Array.from(Array(50).keys()); //Also fails after 32nd element


        const masterStream = scramjet.DataStream.from(fiftyElements);
        const teedStream = masterStream.tee((stream) => stream);

        let masterCount = 0;
        let teedCount = 0;

        ////Uncomment one of the following three blocks to test a scenario

        // Exits after hitting 32 elements from each
        const results = await Promise.all([
            masterStream.each((chunk => {console.log(`p ${masterCount}`);masterCount++;})).toArray(),
            teedStream.each((chunk => {console.log(`c ${teedCount}`);teedCount++;})).toArray()
        ])

        // // Exits after hitting 32 elements of masterStream
        // const masterResults = await masterStream.each((chunk => {console.log(`M ${masterCount}`);masterCount++;})).toArray();
        // const teedResults = await teedStream.each((chunk => {console.log(`T ${teedCount}`);teedCount++;})).toArray();

        // // Exits after hitting 32 elements of teedStream
        // const teedResults = await teedStream.each((chunk => {console.log(`T ${teedCount}`);teedCount++;})).toArray();
        // const masterResults = await masterStream.each((chunk => {console.log(`M ${masterCount}`);masterCount++;})).toArray();

        console.log("Done!");
        const breakpoint = ''; 
    } catch (e) {
    console.log(e);
    const breakpoint = ''
    }   
}

test();

Expected behavior
results should be an array with two elements, each consisting of an array of fifty integers (when using fiftyElements)

Test case
See code, above

System:

  • OS: Ubuntu 19.10
  • Node version: v10.15.2
  • Scramjet Version: 4.28.5

Additional context
Add any other context about the problem here.

@MichalCz The fault appears to be in signicode/scramjet-core/

Closing here and reopening there with additional info.