DEPRECATED: even with tor, the rate limiting of gsmarena is too sensitive for the script to complete properly. Use it at your own risk.
This script extract the mobile specs from all the phones available in gsmarena.com to a csv file (+ one for each brand).
To avoid spam detection, run with TOR (see below).
- python + pip
- docker + docker-compose
Clone the repository:
git clone https://github.com/dbeley/gsmarena-scraper
cd gsmarena-scraper
Install the python dependencies:
pip install requests beautifulsoup4 lxml pandas pysocks stem
If you prefer, you can also install the requirements in a virtual environment with pipenv (in order to run the python script, you will need to use pipenv run python gsmarena-scraper.py instead of python gsmarena-scraper.py):
pipenv install
Run the docker container containing the tor proxy (you can tweak the torrc configuration file if you want, but the defaults should be good):
docker-compose up -d
Run the script:
python gsmarena-scraper.py
After completion, you can stop the docker container with docker-compose down.
python gsmarena_scraper.py -h
usage: gsmarena-scraper.py [-h] [--debug]
Scraper gsmarena.
optional arguments:
-h, --help show this help message and exit
--debug Display debugging information
The exported files will be placed in a folder named Exports. The all-brands_exports.csv will contain the data of all brands.