web-scraping

There are 176 repositories under web-scraping topic.

• scrapy

  scrapy / scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  crawlercrawlingframeworkhacktoberfestpythonscrapingweb-scrapingweb-scraping-python
  Language:Python 53221
• changedetection.io

  dgtlmoon / changedetection.io

  The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification

  back-in-stockchange-alertchange-detectionchange-monitoringchangedetectionmonitoringnotificationsrestock-monitorself-hostedurl-monitorweb-scrapingwebsite-change-detectionwebsite-change-detectorwebsite-change-monitorwebsite-change-notificationwebsite-change-trackerwebsite-defacement-monitoringwebsite-monitorwebsite-monitoringwebsite-watcher
  Language:Python 19431
• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
  Language:TypeScript 15709
• Douyin_TikTok_Download_API

  Evil0ctal / Douyin_TikTok_Download_API

  🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

  pythonpywebiotiktokdouyinapiscraperfastapino-watermarkonline-parsingasyncdouyin-tiktok-apidouyin-tiktok-downloadcrawlerspiderweb-scrapingtiktok-scraperdouyin-scraperdouyin-apitiktok-apitiktok-signature
  Language:Python 9407

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  web-scrapingcaptcha-bypasscaptcha-recaptchacrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolwebscrapingcrawlerspider
  Language:Makefile 6701
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  aiartificial-intelligenceautomationcrawlermachine-learningpythonscrapescraperscrapingweb-scrapingwebautomationwebscraping
  Language:Python 6480
• SeleniumBase

  seleniumbase / SeleniumBase

  📊 Blazing fast Python framework for web crawling, scraping, testing, and reporting. Supports pytest. Stealth abilities: UC Mode and CDP Mode.

  anti-detectionbehavebot-detectioncdpchromedrivercloudflare-bypasse2e-testingpytestpytest-pluginpythonpython-scraperseleniumselenium-pythonseleniumbasetest-automationweb-automationweb-scrapingweb-scraping-pythonwebdriverwebkit
  Language:Python 5418

• go-rod / rod

  A Chrome DevTools Protocol driver for web automation and scraping.

  cdpchrome-headlesschrome-devtoolschrome-devtools-protocolheadlessweb-scrapingautomationscraperdevtoolsdevtools-protocolrodgogolangtestingwebgorodcrawling
  Language:Go 5409

• getmaxun / maxun

  🔥 Open-source no-code web data extraction platform. Turn websites to APIs and spreadsheets with no-code robots in minutes! [In Beta]

  apiautomationbrowserbrowser-automationlow-codemaxunno-codeno-code-web-scraperplaywrightrobotic-process-automationrpascraperself-hostedspreadsheetweb-automationweb-scraperweb-scrapingwebsite-to-apiwebsite-to-databasewebsite-to-spreadsheet
  Language:TypeScript 4980

• mherrmann / helium

  Lighter web automation with Python

  chromefirefoxheliumpythonpython3seleniumselenium-pythonweb-automationweb-scrapingwebdriver
  Language:Python 4956

• apify / crawlee-python

  Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationbeautifulsoupcrawlercrawlinghacktoberfestheadlessheadless-chromepipplaywrightpythonscraperscrapingweb-crawlerweb-crawlingweb-scraping
  Language:Python 4628

• adbar / trafilatura

  Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

  article-extractorcorpuscorpus-buildercorpus-toolscrawlerhtml-to-markdownhtml2textnewsnews-aggregatornews-crawlernlpreadabilityrss-feedscrapingteitext-cleaningtext-extractiontext-miningtext-preprocessingweb-scraping
  Language:Python 3656

• php-curl-class / php-curl-class

  PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

  phpcurlclassapiapi-clientclientframeworkhttphttp-clienthttp-proxyjsonphp-curlphp-curl-libraryproxyrequestsrestfulweb-scraperweb-scrapingweb-servicexml
  Language:PHP 3262
• snoop

  snooppr / snoop

  Snoop — инструмент разведки на основе открытых данных (OSINT world)

  osinttermuxusername-searchusername-checkerpentestweb-scrapingctfscannerredteamblueteaminfosecsecuritynicknameipgeopoliceparserscrapinggeocoderusername
  Language:Python 3000

• lexiforest / curl_cffi

  Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

  akamai-fingerprintcurlcurl-impersonatefingerprintinghttphttp-clienthttp2-fingerprinthttpsja3ja3-fingerprinttls-fingerprintweb-scraping
  Language:Python 2457

• lorien / grab

  Web Scraping Framework

  asynchronouscrawlercrawlingframeworkhttp-clientnetworkpycurlpythonpython-librarypython3scrapingspiderurllib3web-scraping
  Language:Python 2394

• codingforentrepreneurs / 30-Days-of-Python

  Learn Python for the next 30 (or so) Days.

  apiautomationcsvfastapiflaskjupyterpandaspythonpython3rest-apiseleniumselenium-webdrivertutorialweb-scraping
  Language:HTML 2145

• x4nth055 / pythoncode-tutorials

  The Python Code Tutorials

  pythonpython3scapyethical-hackingnetwork-programmingnetwork-securitynetwork-analysispython-tutorialstutorialsscapy-tutorialsmachine-learningtext-classificationsocket-programmingface-detectioncomputer-visionprogramming-tutorialnatural-language-processingweb-scraping
  Language:Jupyter Notebook 2137

• justmarkham / DAT8

  General Assembly's 2015 Data Science course in Washington, DC

  data-sciencemachine-learningscikit-learndata-analysispandasjupyter-notebookpythoncourselinear-regressionlogistic-regressionmodel-evaluationnaive-bayesnatural-language-processingdecision-treesensemble-learningclusteringregular-expressionsweb-scrapingdata-visualizationdata-cleaning
  Language:Jupyter Notebook 1608

• tidyverse / rvest

  Simple web scraping for R

  htmlrweb-scraping
  Language:R 1492
• Scrapling

  D4Vinci / Scrapling

  Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python

  automationcrawlercrawlingcrawling-pythoncssdom-manipulationhacktoberfestlxmlplaywrightpythonpython3scrapingselectorsseleniumstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
  Language:Python 1454

• roach-php / core

  The complete web scraping toolkit for PHP.

  crawlingphpweb-scraping
  Language:PHP 1362

• A9T9 / RPA

  Ui.Vision Open-Source RPA Software - Robotic Process Automation with Computer Vision and OCR, Selenium IDE compatible.

  autohotkeyautomationbrowser-automationbrowser-extensiondata-driven-testsimacrosopencvselenium-idesikulixui-testsuipathvisual-recognitionweb-automationweb-scraping
  Language:JavaScript 1285

• rushter / selectolax

  Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).

  pythonhtml5cssmodest-engineweb-scrapingparser
  Language:Cython 1162

• spider-rs / spider

  A web crawler and scraper for Rust

  ai-scrapingcrawlerheadless-chromeindexerllm-crawlerrustspiderweb-crawlerweb-scraperweb-scraping
  Language:Rust 1154

• juancarlospaco / faster-than-requests

  Faster requests on Python 3

  pythonpython3python-libraryhttp-requestspython-requestsweb-scrapingdownload-filespeedcurlcythonurllibfaster-than-requestsopen-dataurllib3requests3scrapyndjsonhigh-performancerequests-toolbeltweb-scraper
  Language:Nim 1107

• Smartproxy / Smartproxy

  HTTP(S)/SOCKS5 rotating residential proxies - code examples & general information.

  proxyproxieshttps-proxyproxy-serverpythondata-collectionproxy-integrationproxy-listpython-scraperscrapingurl-scraperweb-scrapingweb-scraping-apiresidential-proxiesdata-gatheringhttp-proxyip-rotationproxy-rotatorproxy-scrapersocks5-proxy
  Language:Java 1105
• amazon-scraper

  oxylabs / amazon-scraper

  Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.

  amazon-apiamazon-scraperamazon-scrapinge-commerce-apiweb-scrapingpythonamazon-scraper-apiamazon-scraping-libraryamazon-price-trackerscraping-apiamazon-aipamazon-api-dataamazon-products-apiamazon-scrape-apihow-to-scrape-amazonscrape-amazonscraping-amazonprice-scraperprice-scraper-apiamazon-product-scraper
  Language:Python 1055

• intoli / user-agents

  A JavaScript library for generating random user agents with data that's updated daily.

  user-agentuser-agent-spooferrandomrandomizationjavascriptnavigatorbrowsersbrowser-automationweb-scraping
  Language:TypeScript 982
• google-maps-scraper

  gosom / google-maps-scraper

  scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

  distributed-scraperdistributed-scrapinggolanggoogle-mapsgoogle-maps-scrapingweb-scraperweb-scraping
  Language:Go 952

• vprusso / youtube_tutorials

  Collection of scripts corresponding to LucidProgramming YouTube tutorials

  ctci-solutionslucidprogrammingpythonpython-tutorialpython3python3-tutorialtechnical-interviewweb-scrapingyoutube-tutorial
  Language:Python 923

• DataHenHQ / till

  DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

  crawlerman-in-the-middlemitmproxy-serverscraperscrapingweb-scraping
  Language:Go 813

• postmodern / spidr

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  crawlerrubyscraperspiderspider-linkswebweb-crawlerweb-scraperweb-scrapingweb-spider
  Language:Ruby 806

• platonai / PulsarRPA

  Automate webpages at scale, scrape web data completely and accurately with high performance, distributed AI-RPA.

  crawlerdata-miningdata-sciencerpascraperscrapingweb-automationweb-crawlerweb-miningweb-scrapingweb-sql
  Language:Kotlin 773
• web-scraping

  je-suis-tm / web-scraping

  Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist

  bloombergdata-scraperdata-scrapingfinancial-datafinancial-timesfuturesfutures-historical-datanews-scrapernews-websitesnewsletteroptions-datapython-web-scraperreutersscrappersrapingwall-street-journalwallstreetbetsweb-scraperweb-scrapersweb-scraping
  Language:Python 742

• alecxe / scrapy-fake-useragent

  Random User-Agent middleware based on fake-useragent

  pythonscrapyweb-scraping
  Language:Python 687