Continue on 'finish' instead of 'end'

[Guilherme Ventura]

I'm trying to figure streams out.
To help with that, I'm changing the way my tiny videos download helper works, to deal with streams.

Here is the desired workflow:

1. user passes a list of videos to be downloaded
2. get the first video from the list
3. search for the video (from Youtube API, returns a Promise)
4. get the video URL
5. download its content (returns a stream, which is piped to a file stream)
6. on finish, get the next video and go back to step 1, until list is over

The problem is that step 6 is run before the file is completely written (before the write stream's finish event), so the program "finishes" before the files are completely downloaded.

I would like the calls to be queued, somehow.
Here is the (simplified) code:

highland(tracks) // tracks = array of videos requested by the user
  .flatMap(track => highland(spotify.getTrack(track))) // getTrack returns a promise, which resolves with a track object
  .flatMap(track => downloadTrack(track, { path: output, logger })) // downloadTrack calls downloadYoutubeVideo that does the heavy stream work (see below)
  .on('end', resolve) // this code is wrapped in a Promise, irrelevant
  .resume();

function downloadTrack(track, { path = './', logger } = {}) {
  info(logger, chalk.bold.blue("   [Downloading track]"), track.name);

  let fileName = `${track.artists[0].name} - ${track.name}`;
  return downloadYoutubeVideo(fileName, path, { logger });
}

function downloadYoutubeVideo(name, location = './', { logger } = {}) {
  let fullPath = fsPath.join(location, `${createFolderName(name)}.mp4`);
  // setup folders
  if (!fs.existsSync(location))
    mkdirp.sync(location);

  let videoSearchPromise = youtube.searchMusicVideo(name);
  let writeStream = fs.createWriteStream(fullPath);

  return highland(videoSearchPromise) // search the video
    .map(video => {
      if (!video) {
        throw new Error("Video not found");
        return;
      }

      let downloadUrl = `https://www.youtube.com/watch?v=${video.id.videoId}`;
      debug(logger, `Downloading video from url: ${downloadUrl}`);

      let videoStream = ytdl(downloadUrl, { quality: 18 /* 360p */ });

      return highland(videoStream)
        .pipe(writeStream)
        .on('finish', () => console.log('Finish write:', name));
    });
}

(The complete code can be found here.)

This is the current output:

[spotivy v0.4.0]
Saving media to 'C:\Users\Guilherme\git\danguilherme\spotivy\media'

   [Downloading track] Don't Hold the Wall
   [Downloading track] Broken-Hearted Girl
   [Downloading track] Break the Ice
   [Downloading track] DJ Got Us Fallin' In Love

[spotivy v0.4.0] Finished successfuly
Finish write: Justin Timberlake - Don't Hold the Wall
Finish write: Beyoncé - Broken-Hearted Girl
Finish write: Usher - DJ Got Us Fallin' In Love
Finish write: Britney Spears - Break the Ice

And this is the desired output:

[spotivy v0.4.0]
Saving media to 'C:\Users\Guilherme\git\danguilherme\spotivy\media'

   [Downloading track] Don't Hold the Wall
Finish write: Justin Timberlake - Don't Hold the Wall
   [Downloading track] Broken-Hearted Girl
Finish write: Beyoncé - Broken-Hearted Girl
   [Downloading track] Break the Ice
Finish write: Britney Spears - Break the Ice
   [Downloading track] DJ Got Us Fallin' In Love
Finish write: Usher - DJ Got Us Fallin' In Love

[spotivy v0.4.0] Finished successfuly

How could I achieve that?

[Victor Vu]

I think the problem is in this code

  return highland(videoSearchPromise) // search the video
    .map(video => {
      if (!video) {
        throw new Error("Video not found");
        return;
      }

      let downloadUrl = `https://www.youtube.com/watch?v=${video.id.videoId}`;
      debug(logger, `Downloading video from url: ${downloadUrl}`);

      let videoStream = ytdl(downloadUrl, { quality: 18 /* 360p */ });

      return highland(videoStream)
        .pipe(writeStream)
        .on('finish', () => console.log('Finish write:', name));
    });

Specifically, you do a map instead of a flatMap. map doesn't wait on its results the way that flatMap does, so the stream that you create completes as soon as the download is started, and not when it's done.

I think the following will work

  return highland(videoSearchPromise) // search the video
    .flatMap(video => highland((push, next) => {
      ...

      videoStream
        .pipe(writeStream)
        .on('finish', () => {
          push(null, highland.nil); // Notify Highland that the download has completed.
          console.log('Finish write:', name)
        });
    }});

By the way, using the EventListener events are kind of deprecated (they're not well-supported, and may interact poorly with the rest of the API). So for this part

  .on('end', resolve) // this code is wrapped in a Promise, irrelevant
  .resume();

you are better off using done instead, which does the same thing, but doesn't rely on events.

  .done(resolve); // No need to manually resume the stream.

[Guilherme Ventura]

It works perfectly!

Thank you for the response and for the tips about events too. Will use errors for errors as well.
(Updated code for the curious.)