IonicaBizau / scrape-it

🔮 A Node.js scraper for humans.

Home Page:http://ionicabizau.net/blog/30-how-to-write-a-web-scraper-in-node-js

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Async convert functions

jdalrymple opened this issue · comments

Would it be possible to support async convert functions?

That'd be cool! ✨

I spent some time to try and implement it, but it was turning out to be more difficult then i thought lol

Hi, thanks for this great scraper !

Do you have any news on this ? I tried on my own firstly with your assured library then with other workarounds, but with no success

You can find part of my tries at https://github.com/jon-ht/scrape-it/tree/async-convert-function
However, it seems to totally break Promise's based interface.

This is what I get when I run node example/index.js

Callback interface

{ articles: 
   [ Promise { [Object] },
     Promise { [Object] },
     Promise { [Object] } ],
  pages: 
   [ Promise { [Object] },
     Promise { [Object] },
     Promise { [Object] },
     Promise { [Object] },
     Promise { [Object] },
     Promise { [Object] },
     Promise { [Object] } ],
  title: 'Ionică Bizău',
  desc: 'Programmer,  Pianist,  Jesus follower',
  avatar: '/@/bloggify/public/images/logo.png' }

Promise interface

Status Code: 200
{ title: 'Ionică Bizău',
  desc: 'Programmer,  Pianist,  Jesus follower',
  avatar: '/@/bloggify/public/images/logo.png' }

Hello @IonicaBizau, an update is planned to make it possible to use async functions in the convert?

It would be a great step forward and I think you will spend less time developing this than the community for a better result.

I am very busy with other projects right now, but contributions to make this better are always appreciated.

I guess if we implement a series promise option, that would not be difficult since we can use async-await, so syntax won't change much.

I am going to close this for now, but thank you very much for the idea. We will see if anyone else needs such an interface. Cheers!