README

Project: ToyScrape

This app is a Gallery downloader for the character creating/sharing site Toyhouse. It uses a custom built API to fetch and then download galleries for characters.

Check it out here!

Usage

Using this downloader app is quite simple! You only need the link of the character that you wish to download the gallery of. After pasting it in the input, simply clicking "Download" will fetch all the images for the user and download them in a zip folder for you! The app also includes credits for each artists and important data for each image (tagged characters, description, etc.)!

Sample Links:

Since Toyhouse is currently an invite only app, sample links are necessary to test the app without an account. Here's a few:

Challenges

The main challenge that stumped me for a while was the way file-saver.js worked. As it simply waits for the fetch call to end rather than the blob object, it would download the zip file before actually getting all of the images for the character. This was a perfect opportunity to practice using Promises.

I scrapped the original code, and rewrote it to use Promise.all():

response.gallery.forEach(async (link, idx) => {
  // We iterate through the gallery 
   // and create a promise for each item
  
  const linkPromise = new Promise(async (resolve, reject) => { 
  
  let response = null;
  try {
    response = await fetch(link);
  } catch(err) {
    setHasError(err);
  }
  
  const blob = await response.blob();
  
  // Get datatype from image link
  let dataType = link.split(".")[3]
  if(dataType.length > 4) {
  
    // Sometimes, the link includes extra characters,
     // so we check to make sure we dont get the datatype wrong
    dataType = dataType.split("?")[0]
  }
    resolve({data: blob, type: dataType });
  });
  
// Push the promise to the promises array,
 // so that we can use it in Promise.all()
promises.push(linkPromise);

When all of the images resolve, we use Promise.all() to save the images to a zip folder.

// If we're at the last item,

if(idx === response.gallery.length - 1 ) {
  // Set a loading indicator,
  props.setLoading("Saving files...")
  // And call Promise.all()
  Promise.all(promises)
    .then(data => {
      // And we save each file to a zip
      data.forEach((blob, idx) => zip.file(`${idx}.${blob.type}`, blob.data))
    })
    .then(data => {
      props.setLoading(null);
      
      // And we finally generate our zip and download it.
      zip.generateAsync({type:"blob"})
        .then(content => {
          setQueryStr("");
          saveAs(content, `${response.name}-gallery.zip`)
      })
   })
 }

Screenshots

Technologies used

This app is written in React and plain Javascript for the fetch code and uses Bootstrap for styling. To save the files to memory, I used file-saver.js and JSZip to zip everything up and download it as a zip file. My own Toyhouse API was used to gather all the necessary data for the characters.

zelta01 / toyhouse_downloader