adding proxy to x-ray node js
bassemAmous opened this issue · comments
bassemAmous commented
Subject of the issue
Describe your issue here.
Your environment
- version of node: run node --version
- version of npm: run npm --version
Steps to reproduce
Tell us how to reproduce this issue.
Expected behaviour
Tell us what should happen.
Actual behaviour
Tell us what happens instead.
bassemAmous commented
I am developing a node js application and I am scraping a web site using x-ray and I think the blocked my ip address so how can I configure a proxy or hide my Ip address for that ? here is my code:
var Xray = require('x-ray');
var x = Xray()
x('https://www.myurl.com', {
title: x('#cm_cr-review_list .a-section.review', [{,
blogs:"..a-text-bold"
}]),
})
.paginate('li.a-l a@href')
.write('result.json')
Geert commented
1. x-ray allows for custom [drivers](
https://github.com/matthewmueller/x-ray#xraydriverdriver) .. E.g.: see prev
link to a [request](https://github.com/request/request) driver.
2. adapt request driver to point to a [proxy](
https://github.com/request/request#proxies)
3. You can choose proxies yourself randomly/ round-robin, etc. (you need to
build that into your driver) or choose a managed rotating-proxy. Advantage
of the latter is that you just need to point the `proxy` argument of your
request-driver to 1 static url and be done with it. There's paid ones,
there's also [this](https://github.com/mattes/rotating-proxy) which sets up
a socks5 proxy using Haproxy on your own server using TOR.
Hth
…On Thu, Jan 11, 2018 at 12:34 PM, bassemAmous ***@***.***> wrote:
I am developing a node js application and I am scraping a web site using
x-ray and I think the blocked my ip address so how can I configure a proxy
or hide my Ip address for that ? here is my code:
var Xray = require('x-ray');
var x = Xray()
x('https://www.myurl.com', {
title: x('#cm_cr-review_list .a-section.review', [{,
blogs:"..a-text-bold"
}]),
})
.paginate('li.a-l ***@***.***')
.write('result.json')
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#299 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAYAK5zpthH16iRYTmRyxD5vnsQ0dsgOks5tJfGvgaJpZM4RatV6>
.
bassemAmous commented
@gebrits Thank you for your answer but I didn't really understand the second part could you please send me an example for that ?
Geert commented
Something like this. Untested: (second option in the comments)
https://gist.github.com/gebrits/57689768eceaec43ae0ddd17949d7503
bassemAmous commented
Thank you I just resolved the problem.