Last Updated: 2020-01-01
PurchaseAnalysis is a general-purpose web scraper and visualiser for analysing prices. Currently this is applied to housing prices and the second-hand car market
- Running the script
update_cars_db.sh
will:- Pull any changes from the github repo
- Run
project_files/cars/scrape_cars.py
which collects the most up-to-date information about the cars from various websites and stores various data about them inproject_files/cars/items.db
and the links to those cars inproject_files/cars/car_links.txt
- Push these changes to github in a commit named
Beep Boop: Automatic update from raspberry pi
update_cars_db.sh
is intended to be run fromcrontab
automatically and is set up on a raspberry pi to run every night- Note that
PurchaseAnalysis/main.py
is deprecated and not in use at all.
- Data analysis isn't automated at the moment, however there are
various cells in
Scratchpad.ipynb
which are usefull for quick graphs of the data
- Scrape and parse 20+ variables (engine size, price, number of doors, etc) from cars.co.za
- Automatically scrape and store the car data every night
- Store the changes in price per vehicle over time