web-scraping-python
There are 28 repositories under web-scraping-python topic.
- scrapyLanguage:Python 58897
- SeleniumBase
seleniumbase / SeleniumBase
Python APIs for web automation, testing, and bypassing bot-detection with ease.
Language:Python 11845 - Scrapling
D4Vinci / Scrapling
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
Language:Python 8110 - botasaurus
omkarcloud / botasaurus
The All in One Framework to Build Undefeatable Scrapers
anti-botanti-detectioncloudflare-bypasscloudflare-scrapeanti-detectanti-detect-browserantidetect-browserundetectedundetected-chromedriverbypass-cloudflarepython-web-scraperpython-web-scrapingscraping-frameworkscraping-toolundetectableweb-scraping-pythonbot-detectionscraping-pythonweb-crawlingpython-scraperLanguage:Python 3193 - oxylabs-ai-studio-py
oxylabs / oxylabs-ai-studio-py
Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio python SDK for intelligent web data gathering.
Language:Python 1241 - how-to-scrape-amazon-product-data
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
- agentql
tinyfish-io / agentql
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
Language:Python 999 - Language:Rust 991
scrapfly / scrapfly-scrapers
Scalable Python web scraping scripts for +40 popular domains
Language:Python 746davidteather / everything-web-scraping
Learn everything web scraping with David Teather Codes on YouTube
Language:HTML 433- Python-Web-Scraping-Tutorial
In this Python Web Scraping Tutorial, we will outline everything needed to get started with web scraping. We will begin with simple examples and move on to relatively more complex.
Language:Python 298 drshahizan / python-web
This topic explains how to implement web scraping and python web development. Web scraping topics such as scrapy, beautiful soup, and others will be covered. A case study based on a Malaysian website.
Language:Jupyter Notebook 128thewebscraping / tls-requests
TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS client, fingerprinting, anti-bot page bypass, and high performance.
tls-clientcloudflare-scraperweb-scraping-apicloudflare-bypasscf-clearanceanti-botanti-bot-detectionanti-bot-pagepython-tls-clientpython-crawlerpython-spiderweb-scraping-pythonpython-web-scraperpython-web-scrapingpython-web-crawlerweb-spiderscraping-pythonweb-crawler-pythonpython-scrapercrawling-pythonLanguage:Python 108- web-scraping-google-sheets
Guide to Using Google Sheets for Basic Web Scraping
DataCrawl-AI / datacrawl
A simple and easy to use web crawler for Python
Language:Python 64vishwajeetdabholkar / eGet-Crawler-for-ai
Web scraping framework built for AI applications. Extract clean, structured content from any website with dynamic content handling, markdown conversion, and intelligent crawling capabilities. Perfect for RAG applications and AI training data pipelines. Features async processing, browser management, and Prometheus monitoring.
Language:Python 47mike-gee / webtranspose
Web scraping API for building AI applications.
Language:Python 40GoncaloMark / CobWeb-lnx
CobWeb is a Python library for web scraping. The library consists of two classes: Spider and Scraper.
Language:Python 37mhmdkardosha / CAT-Reloaded-2025-Data-Science-Roadmap
Roadmap for Data Science circle associated with CAT Reloaded.
Decodo / Python-scraper-tutorial
A short introduction to scraping with Python with given steps and an example scraper script.
Language:Python 32raymelon / tagalog-dictionary-scraper
Builds a Tagalog dictionary by collecting Tagalog words from tagalog.pinoydictionary.com
Language:Python 30kvcops / Deep-Research-using-Gemini-api
AI-powered deep research tool leveraging web scraping for cost-effective, comprehensive analysis. Open-source and API-cost free!
Language:HTML 21Narenpradhan / WatchTower
WatchTower - A platform to save your valuable time while staying updated in the Cyber realm.
Language:Python 18PB2204 / Covid-19
This Is A Web Scraping Projects With Covid-19 Data From 2 Very Popular & Authentic Websites
Language:Jupyter Notebook 18Elmehdi9 / web-scraping-projects
This repository provides various web scraping projects in Jupyter notebooks for both learning and data-related workshopes
Language:Jupyter Notebook 13FirasKahlaoui / news-headlines-tracker
The News Headlines Tracker application collects the latest news headlines from major news sources such as CNN, BBC, and The New York Times.
Language:Python 12- Language:Python 11
irfanalidv / trustpilot_scraper
A Python library for scraping Trustpilot reviews.
Language:Python 11sarperavci / kick-unofficial-api
🛡️ Unofficial Kick.com API wrapper with automatic bypass protection.
Language:Python 10boo283 / Facebook_comment_crawler
The Facebook Comments Crawler is an unofficial tool for extracting comments from Facebook posts using Selenium in Python. It's designed to aid in academic and personal research. #Facebook comments scaper #Facebook comments crawler
Language:Python 9JaydeepAgravat / SmartCode
Scraping LeetCode data, analyzing for insights, crafting a user-friendly dashboard, and building a problem recommender for optimized problem-solving.
Language:Jupyter Notebook 8- Web-Scraping-Starter-Kit
gayanukabulegoda / Web-Scraping-Starter-Kit
Repository designed to help freshers easily grasp the basics of web scripting, offering simple guides and examples to build a strong foundation.
Language:Python 7 Mindful-AI-Assistants / SP2024-Election-Analysis
📊 An analysis of voting patterns in São Paulo's 2024 elections, focusing on voter behavior, absenteeism, and geographic trends.
Language:HTML 7odevjorge / instagram-post-fetcher
"instagram-post-fetcher" is a Python module leveraging Selenium to extract Instagram post details, including account username, descriptions, media URLs, and post timestamps. Simplifying access to Instagram data for analytics and research.
Language:Python 7A comparison of asynchronous and synchronous web scraping methods with practical examples.
Language:Python 7Karthick-840 / Crawl4ai-RAG-with-Local-LLM
A tool for scraping web documentation using Crawl4AI, converting it to Markdown, and preparing it for integration with local LLMs (like Ollama) to enhance their knowledge for learning and "vibe coding" workflows.
Language:Python 6
