Crawler Add Fetch Condition

Question

Crawler Add Fetch Condition

guga-grigolia opened this issue 6 years ago · comments

Feature.

Hi I want to use simplecrawler's Add Fetch Condition functionality and I was not able to pass my own exclusions or override existing ones. If there is some possibility I'd be really thankful if you will point it out for me, or I can create PR for that.

Regards,
Guga

TheCodeholic commented 6 years ago

Thank you

Lars Graubner · Answer 1 · Wed Apr 18 2018 03:00:25 GMT+0800 (China Standard Time)

At the moment this is not possible, but can be easily added.

I'm thinking of something like generator.getCrawler() to expose the simplecrawler instance. That way you can easily add new fetch conditions. How does this sound?

Guga Grigolia · Answer 2 · Wed Apr 18 2018 03:19:00 GMT+0800 (China Standard Time)

Yes, it sounds quite promising. Thanks a lot.

…

On Tue, Apr 17, 2018, 23:00 Lars Graubner ***@***.***> wrote: At the moment this is not possible, but can be easily added. I'm thinking of something like generator.getCrawler() to expose the simplecrawler instance. That way you can easily add new fetch conditions. How does this sound? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#31 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD3q0fVSDkLHt2vDZRU7ffsdx_9TrVGwks5tpjvKgaJpZM4TXEdl> .

Lars Graubner · Answer 3 · Wed Apr 18 2018 03:25:59 GMT+0800 (China Standard Time)

Just released v8.1.0 which provides a getCrawler() method. I hope it helps :)

TheCodeholic · Answer 4 · Wed Apr 18 2018 20:59:42 GMT+0800 (China Standard Time)

I think the tag is not published on npmjs.com. I can not install that tag through npm.

Lars Graubner · Answer 5 · Fri Apr 20 2018 01:14:39 GMT+0800 (China Standard Time)

My bad, seems to have been errored without me noticing. Should be out now for real.