Scraping FMEL Listings with Selenium

Setup

Dependencies

selenium conda install selenium or pip install selenium
python-telegram-bot conda install -c conda-forge python-telegram-bot or pip install python-telegram-bot
pandas conda install pandas or pip install pandas

Config file

Rename config_sample.ini to config.ini and fill in your login credentials.

Chromium Browser Driver

Install Chromium Web Browser (not the standard snap version of Chromium)
Download a ChromeDriver corresponding to your version of Chrome (https://chromedriver.chromium.org/downloads)
Copy it to the same directory as fmel.py
Make it executable with chmod +x chromedriver?
Add the ChromeDriver to PATH (PATH=$PATH:~/path/to/chromedriver)

Building Blocks

FMEL Webscraper

The scraper works just like a human would to read all room offers from the FMEL website. It opens a browser driver, logs in with the credentials provided in the config.ini file and consecutively clicks every move-in date and house. If this software runs permanently on a server, you can set the headless flag to run the Chrome driver without actually opening a browser window.

jo-ruether / fmel_housing