bda-research / node-crawler

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

Home Page:http://node-crawler.org

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Consideration to ditch Request dependency?

sbr2567 opened this issue · comments

Since Request has been deprecated for some time now, have you considered switching out the dependency for something more stable like got, node-fetch or axios? I imagine this would be a desirable move for the long run.

The author of request said it is one of first modules uploaded in npmjs, and in the last ten years nodejs core has changed so much that it cannot keep it up to date. crawler was also published about ten years ago, as we can see the code style, new features of es6 even es7 make developers' code differ.

With so many years development, I believe request is more stable than any other 'new generation' module. We also have hundreds of crawler instances running in the production environment where stability is really counted. That's why I don't want to simply replace request with modules you mentioned. So no need to worry about the stability if it's your concern, crawler is always reliable.

If we are talking about the API or code style like promise, calling chain, or even async, await, I agree with you that we should do something to make it look like 'modern'. So the problem is whether the code style deserves the risks of instability?

My answer is no. I prefer to develop a totally new library or framework for next ten years in a modern way.