Santhin / TorScrapy

Simple crawler made with scrapy and TorIpChanger package

---

Simple app for scrapping data from gumtree.

🧐 About

The project was created for learning purposes to know how to combine scrapy framework with TorIp changer.

🏁 Getting Started

Prerequisites

• Docker desktop

Project structure

.
├── docker-compose.yml
├── LICENSE
├── README.md
└── src
    ├── crawler
    │   ├── __init__.py
    │   ├── items.py
    │   ├── middlewares.py
    │   ├── pipelines.py
    │   ├── settings.py
    │   └── spiders
    │       ├── __init__.py
    │       ├── mieszkania2.py
    │       └── quotes_spider.py
    ├── Dockerfile
    ├── go_spider.py
    ├── scrapy.cfg
    └── tests
        └── ipchanger_works.py

Installing

Clone repository:

git clone https://github.com/Santhin/TorScrapy.git

To run the crawler type:

docker-compose up

🔧 Running the tests

Simple check if tor ip changer is working unmark commented test in dockerfile.
The exemplary output:

🛠️ Todo

• add control startup for TorIpChanger container in docker-compose

⛏️ Built Using

• Scrapy - Crawler
• TorIpChanger - Privoxy + Tor

🎉 Acknowledgements

• Hat tip to DusanMadar for amazing framework and tutorial step by step https://github.com/DusanMadar/TorIpChanger https://gist.github.com/DusanMadar/8d11026b7ce0bce6a67f7dd87b999f6b