Webscraping is the process of automatically mining data or collecting information from the World Wide Web. This process has many applications
1. Scrape product details (price, images, rating, reviews etc.) from retailer/manufacturer/eCommerce websites (Ex: Amazon, eBay, AliExpress, Alibaba etc.) to show on own websites, to provide price comparisons, to perform a price watch on competing sellers etc.
2. Scrape property details and agent contact details from real estate websites
3. Scrape contact details of businesses as well as individuals from yellow pages websites
4. Scrape people profiles from social networks like Facebook, LinkedIn etc. for tracking online reputation.
5. Scrape hospital/clinic websites to build a catalog of health physicians including their contact details
6. Scrape reviews of products and places/hotels/restaurants
In this tutorial we will build a web scraping program that will scrape YouTube and Twitter.
We will leverage these Resources
Python 3 / Anaconda
Use the package manager pip to install the package needed.
pip install PackageName