This Python library is designed to facilitate the common workflow of web scraping, particularly for e-commerce websites. It provides a structured framework where users can define their own logic for gathering product URLs, parsing individual product pages, and selecting the next page. The URLs and product info are saved directly to a database. It supports various databases such as SQLite and MySQL.
Install webscraperr with pip
pip install webscraperr
The configurations of the scraper is stored in a config dictionary. The config must be prepared, modified and validated before passing it to the scraper.
from webscraperr.config import get_default_config, validate_config, DBTypes
config = get_default_config()
config['DATABASE']['TYPE'] = DBTypes.SQLITE
config['DATABASE']['DATABASE'] = 'mydatabase.db'
config['DATABASE']['TABLE'] = 'products' # If TABLE is not set "items" will be the defaut table name
config['SCRAPER']['REQUEST_DELAY'] = 1.6
validate_config(config) # Will raise an error if config is not properly set
After preparing and validating the config, you must initialize the database
from webscraperr.db import init_sqlite
init_sqlite(config['DATABASE'])
# This will create the database and the table
For this example we are going to use WebScraperRequest
. This scrapper will be using requests
library for the http requests. You will need to define the functions for parsing the html. There is also WebScraperChrome
that uses selenium-wire
and undetected-chromedriver
.
from webscraperr import WebScraperRequest
from urllib.parse import urljoin
import parsel
urls = ["https://webscraper.io/test-sites/e-commerce/static/computers/tablets"]
# The `get_next_page_func` must return a url or None. If it returns None it means there is no next page
def get_next_page_func(response):
selector = parsel.Selector(text=response.text) # in this example `parsel` is used for parsing the html
next_page_url = selector.css('a[rel="next"]::attr(href)').get()
if next_page_url is not None:
return urljoin(BASE_URL, next_page_url)
return None
# The `parse_info_func` must return a `dict`.
def parse_info_func(response):
selector = parsel.Selector(text=response.text)
info = {
'name': selector.css(".caption h4:nth-child(2)::text").get(),
'price': selector.css(".caption .price::text").get()
}
return info
with WebScraperRequest(config) as scraper:
scraper.get_items_urls_func = lambda selector : [urljoin(BASE_URL, i) for i in selector.css(".thumbnail a::attr(href)").getall()]
scraper.get_next_page_func = get_next_page_func
scraper.parse_info_func = parse_info_func
scraper.scrape_items_urls(urls) # This will start the scraping of products urls
scraper.scrape_items_infos() # This will navigate to the product page and parse the html
Please note that this library is still under development and may be subject to changes. I am constantly working on improving its functionality, flexibility and performance. Your patience, feedback, and contributions are much appreciated.