international craigslist URL

Question

international craigslist URL

jonpolak opened this issue 8 years ago · comments

for cities outside the US craigslist redirects away from the .org domain. e.g. vancouver.craigslist.ca - any way to configure this?

Ross K. · Answer 1 · Wed Aug 31 2016 08:08:28 GMT+0800 (China Standard Time)

@jonpolak I was able to get it to work outside of the US. Without knowing what you have as your arguments, can't really see where you're going wrong.

Jonathan · Answer 2 · Thu Sep 01 2016 00:30:01 GMT+0800 (China Standard Time)

what were the settings you used in your test? @WIRUT

Jonathan · Answer 3 · Thu Sep 01 2016 00:36:39 GMT+0800 (China Standard Time)

@WIRUT are you using digitalocean as your cloud provider? I find craigslist super slow when I use my own vpn server hosted at digitalocean. Maybe they are limiting digitalocean connections since lots of scrapers host their bots there?

Jonathan · Answer 4 · Thu Sep 01 2016 01:41:04 GMT+0800 (China Standard Time)

@VikParuchuri CONFIRMED craigslist blocks EC2 and DigitalOcean IPs

Ross K. · Answer 5 · Thu Sep 01 2016 17:24:06 GMT+0800 (China Standard Time)

@jonpolak I was able to test it with Calgary as well as using Amazon's EC2 services. I didn't bother using digital ocean. I'm not sure which settings you want. I did set SITE = "Calgary", and I think I bypassed the area and just hardcode the AREA in for this test instance.

Amazon EC2 was nothing different either.