After learning about web-scraping I found a video that caught my attention. It comes from a really nice content creator called NeuralNine that works a lot with Python.
The idea of this system is really simple, when we normally scrape a site we can either have no problems at all or getting disconnected after requesting multiple times from a single proxy. Therefore, to solve this problem we will use a proxy list that will constantly rotate to let us scrape without getting blocked from the site.
-
I used the same page as the video to get the proxy list: here
-
As for the proxy status code trials I used this ip page but choosing a California IP: here
DISCLAIMER: I made this repository for the sake of learning.