Photo by Christin Hume on Unsplash
In this first masterclass we introduce some basics on web scraping. This technique allows us to retrieve data from websites. This can be especially very handy when there are no datasets available or/and it is not possible to obtain such data via APIs.
Through hands-on examples it is possible to see how to obtain data from different websites having a basic knowledge on HTML and using some Python, and Python packages.
The notebook Intro & examples webscraping introduces the basics to start webscraping.
Try what you have learn in another set of examples in Masterclass Hands-on-Questions. The answer can be found in Masterclass Hands-on-Complete.
Extra: It is also possible to work on HTML files previously saved using BeautifulSoup. Notebook Hands-On Example 2 for Saved HTML File shows how to save a HTML page with help of requests and work on the HTML using BeatifulSoup.
Visit our repository Basics on Web Scraping if you want to follow a example with even more details.
-
requests: HTTP library for Python that allows us to send HTTP requests in a simple way. -
Beautiful Soup: Python library for pulling data out of HTML and XML files. It allows us to access easily the information we need to retrieve.
Some knowledge on lists, list comprehension, and string methods is also very helpful.
- Install requirements using
pip install -r requirements.txt.- Make sure you use Python 3.
- You may want to use a virtual environment for this.