ielove

ielove.co.jp scraper

Howto

Start a celery worker

. ./secret.env
export MONGO_USER=$MONGO_INITDB_ROOT_USERNAME
export MONGO_PASSWORD=$MONGO_INITDB_ROOT_PASSWORD
celery -A ielove.tasks worker --loglevel=INFO

Scrape a property page

python3 -m ielove get-property https://www.ielove.co.jp/chintai/c1-397758400

If commiting to database:

. ./secret.env
export MONGO_USER=$MONGO_INITDB_ROOT_USERNAME
export MONGO_PASSWORD=$MONGO_INITDB_ROOT_PASSWORD
python3 -m ielove get-property --commit https://www.ielove.co.jp/chintai/c1-397758400

Start the webui

python3 -m ielove.webui

Contributing

Dependencies

python3.10 or newer;
requirements.txt for runtime dependencies;
requirements.dev.txt for development dependencies.
make (optional);

Simply run

virtualenv venv -p python3.10
. ./venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install -r requirements.dev.txt

Documentation

Simply run

make docs

This will generate the HTML doc of the project, and the index file should be at docs/index.html. To have it directly in your browser, run

make docs-browser

Code quality

Don't forget to run

make

to format the code following black, typecheck it using mypy, and check it against coding standards using pylint.

altaris / ielove-scraper