Giters

mwpenny / kijiji-scraper

A lightweight node.js module for retrieving and scraping ads from Kijiji

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

es6 notation

aemc opened this issue · comments

commented

Would you be open to updating the js syntax to es6+? I could help with this.

commented

Sure, I am open to the idea. Most of the code was written quite a while ago. Did you have any particular language features/constructs in mind?

commented

For starters, replacing var w/ const/let following this guideline https://eslint.org/docs/rules/no-var. And maybe less reliance on jquery. I think it would be more of a refactor than anything lol.

commented

This module is already pretty small, but I agree that some refactoring would be beneficial. The code is currently very procedural and could stand to improve by becoming more object-oriented. The most glaring example that I can think of is ad-parser.js. An Ad object could be created and the functions defined in ad-parser.js could instead be defined as member functions of Ad.

Additionally, the new Ad object could be returned by both the RSS feed scraper (kijiji-query.js) and regular scraper (ad-scraper.js), increasing consistency. No more ad/innerAd weirdness (the scapeInnerAd option could still be supported via lazy initialization upon first access to ad information).

As for the reliance on cheerio (jQuery-like library), I'm not sure how this would work well without it. We'd be re-implementing all of their great work. Use of that library provides an easy way to read Kijiji's markup and is one of the reasons the code for this module isn't huge.

If you'd like to open a PR for some of this, I'm all for it. I will work on improving this project it in my spare time either way, but if we coordinate then we won't duplicate our efforts. 😄

commented

Good news!

I've had a lot of ideas for ways to improve this module for a while which were on the back burner due to life, etc. You gave me some motivation to get back to it. I've refactored this module to be more object-oriented, fixing some things that have bugged me and adding features while I was at it.

ES6 features now being used:

  • Replaced var with let/const
  • Arrow functions
  • Template literals
  • Default parameters
  • ES6-style class definitions
  • Native promises

Refactored:

  • New Ad class to handle scraping and converting ads to strings
  • Ad objects returned from searching are now the same as the ones created by scraping a single ad (consistency at last!)

New features:

  • Can now specify minimum/maximum number of search results. The scraper will automatically get search result pages until the number has been reached
  • Helper objects have been defined to make using locationId and categoryId easier (see the readme and lib/locations.js / lib/categories.js)
  • Promise support (callbacks are still supported as optional parameters)

Check the readme for a more detailed explanation :)