Room exploder is a python scrapy project which fetch and prase the HKUST course catalogue to extra various course-related data. From which one can generate a list of empty classroom at each 30-minute timeslot of a weekday.
- Python 2.7
- Scrapy 0.24.4
-
Install Python 2.7 and pip
sudo apt-get install python2.7 python2.7-dev pip
-
Install Scrapy
pip install Scrapy
3 spiders are created to grab course information, section information and timeslot information respectively. You should change the URLs listed within each spider if you are parsing data for another semester from the course catalog.
-
To grab course information, run in the application folder
scrapy crawl course
-
To grab section information, run in the application folder
scrapy crawl section
-
To grab timeslot information, run in the application folder
scrapy crawl timeslot
There is an extra Python script created to convert results from timeslot spider into JSON format that the UST爆房 needs.
python timeslotsSorter.py
After executing that, there is a database.json
created that works with UST爆房's data seeder class to feed the data into MySQL database.