Node.js streams support for .pipe()

[Param Aggarwal]

Currently getMessages() reads all the messages as fast as possible. I have a use case of processing this data and I would like to pause the reading of the bag while I process the messages that are already read.

By having a stream based API, I will be able to handle the entire processing pipeline as a Node.js stream.

[JP Posma]

That seems reasonable to me. cc @jtbandes @brianc

[Brian C]

Yeah sounds reasonable - what type of API are you thinking? We'd still likely need the same config object passed to getMessages passed to the stream constructor to make it an idiomatic node stream I believe.

For example:

// bagStream would be in objectMode: true
const bagStream = new RosbagStream({ topics: ['/foo', '/bar'], decompress }) // could support optional startTime & endTime as well

bagStream.pipe(through2(function (message) {

}))

[Param Aggarwal]

Yes, I think that's what the API would look like. Currently it seems to read the entire bag to memory if readMessages() is called (without any filter).

rosbag.js/src/bag.js

Lines 113 to 120 in 28b4c6b

 const messages = await this.reader.readChunkMessagesAsync( 
   info, 
   filteredConnections, 
   startTime, 
   endTime, 
   decompress 
 ); 
 messages.forEach((msg) => callback(parseMsg(msg, i))); 

which internally does:

rosbag.js/src/BagReader.js

Lines 204 to 206 in 28b4c6b

 const messages = entries.map((entry) => { 
   return this.readRecordFromBuffer(chunk.data.slice(entry.offset), chunk.dataOffset, MessageData); 
 }); 

This would change to becoming on demand once we use a NodeStream.