Crawler
=======
Weibo Content Crawler
2015-01-27:
- All crawling modules completed
- Revisions to error completed
- Need to put them in json format
2015-02-02:
- Changed the target URL to only crawl the actual weibo excluding the ads and sponsored pages (this can be easily implemented later depending on the circumstances)
- Need to surmount the issue of logging in to view the entire list of tweet and resolve the issue of crawling frequency
2015-02-03:
- can log in the weibo through input ID and PASS
- Can crawl the page after logging in and continue to move to next pages
2015-02-08:
Things to Tweak Around
- How to stop this crawler? When to stop this crawler?
- doesn't take long time to compare the latest (one-by-one) prior to making it in json file
- What happens after page 50? (It repeats)
- Repeat of page 1
- Stopped at max 50
- Duplicate prevention method to prevent the crawler from crawling duplicate posts compared with previous crawling session
- Exception handling for weird posts without names & urls (Resolved)
- possibly exception handling for index error
- Or is there a better way to handle this without loss of posts?
2015-02-24:
- How to stop this crawler? When to stop this crawler? (Resolved)
- Stopped when click button doesn't exit
- consequently move to different time period
- perhaps better naming?