dockerized-scraper
Description
This software scrapes Honolulu county permitting data and stores it in the MongoDB instance.
How to use
-
`git clone https://github.com/alestainer/dockerized-scraper'
-
Run
npm installin both hnl_property_scraper and screst folders -
Install Docker from https://www.docker.com/get-docker
-
Start Docker and run
docker-machine ip -
Create environment variables MONGO_URL, MONGO_USER, MONGO_PASS, and DOCKER_SERVER using your own credentials, address of your MongoDB instance, and docker machine ip you got from the previous step.
-
In hnl-property-scraper folder run
docker build -t scrimage . -
In screst folder run
docker build -t restimage . -
In docker console
docker swarm init --advertise-addr $DOCKER_SERVER -
docker stack deploy -c docker-compose.yml scaledScraper -
To check the workers
docker service logs scaledScraper_scraper