kleinanzeige vorübergehen gesperrt
czrty opened this issue · comments
Hi everyone,
it is great work, thank you.
i made a mistake and used it in two different folders with different urls, both filters (urls) are for kleinanzeige. Since two program runs simultaneously, although time limit was 600, it called website more often than this....
I got the feedback, IP area is temporary blocked. Problem is, although i changed the IP( used my phone as hotspot got different IP), it keeps blocking. I am not IT expert, but, can we assume that, website doesnt block the IP-Bereich but the some parameters crawler has; header, fingerprinting or some other informations? If this is the case, how can it be fixed?
Thank you.
Hi @czrty ,
Thanks for your comment and the issue. I don't know exactly how the blocking works. We're using the requests-random-user-agent library to change the user agent for each request, and if you're changing your IP address then the only common elements are the request headers and of course the URL. You could experiment to see if a different search URL gets around the blocking, or you could check to see if you can access the page normally in your web browser.
It is not about user-agents or proxies. Manually accessing the page is working, so it is probably something with the request headers. Other repos are facing the same issue atm.
Okay. Then it might be time to switch Kleinanzeigen over to use the headless browser that we have for Immoscout. That should make it much harder to detect.
it is still blocked :)
do you know, is it planned to make changes in kleinanzeige.py?
Yes, I mean - I would love to do that, and I don't think it takes very long. I am very busy right now and don't have time to make the change, but anyone who wants to look at the Immoscout scraper and try and implement the same functionality in the Kleinanzeigen scraper is very welcome to do so.
is it still recommended to have 600 seconds wait time for kleinanzeige, iif t uses headless browser like scout?
Hi Artur,
i will not take the risk, they dont block only for few days but permanently :)
i ljust earnt python last week to see if i can make update the code, i fitted it like on the scout but, i am not sure if it is good code to merge.
Beside that, i spent days to learn python and make update, i dont want someone be greedy and set time so low get us blocked again, it would be painful after investing that much time:)
Hey @czrty ,
I see you closed this ticket - can you share your modified code?