ShahinZeynalov / nemlig

README

Scrape Flta.com with Selenium & Celery &RabbitMQ & BeautifulSoup

https://www.flta.org/IndMemberDirectory?&tab=1

First you need to clone this repo to your local computer.

$ git clone https://github.com/ShahinZeynalov/scrape_flta_with_selenium_celery_rabbitmq
$ cd scrape_flta_with_selenium_celery_rabbitmq/

--- Download requirements

After successfuly installed project you need to download requirements.

$ pip install -r requirements.txt

Or download via virtualenv

--- Build Docker

After successfuly download requirements you need to Build docker bellow command:

$ docker-compose up -d --build

For more info about docker visit to docker hub

--- Run celery

After successfuly build docker you need to run celery with below command:

$ celery -A tasks worker --loglevel=INFO --autoscale=AUTOSCALE

For more info go to offical documentation.

--- Run scaper

$ python scrape.py

---

--- Used modules & Apps

1. Selenium: selenium offical
2. Celery: celery
3. BeautifulSoup: bs4