web-scraping

There are 227 repositories under web-scraping topic.

• firecrawl

  firecrawl / firecrawl

  🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

  aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
  Language:TypeScript 66915
• scrapy

  scrapy / scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  pythonscrapingcrawlingframeworkcrawlerhacktoberfestweb-scrapingweb-scraping-python
  Language:Python 58897
• anything-llm

  Mintplex-Labs / anything-llm

  The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.

  ai-agentscustom-ai-agentsdeepseekkimillama3llmlmstudiolocal-llmlocalaimcpmcp-serversmoonshotmultimodalno-codeollamaqwen3ragvector-databaseweb-scraping
  Language:JavaScript 50841
• changedetection.io

  dgtlmoon / changedetection.io

  Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!

  website-monitorwebsite-monitoringchange-detectionmonitoringself-hostedchange-alertchange-monitoringwebsite-change-monitorurl-monitorwebsite-change-detectorwebsite-change-detectionwebsite-change-trackerwebsite-change-notificationnotificationsweb-scrapingrestock-monitorwebsite-defacement-monitoringback-in-stockwebsite-watcherrss
  Language:Python 28421

• ScrapeGraphAI / Scrapegraph-ai

  Python scraper based on AI

  scrapingscraping-pythonautomated-scraperllmweb-crawlerweb-scrapingai-scrapingcrawlermarkdownragweb-crawlersai-crawlerai-searchlarge-language-modelweb-data-extractionweb-searchweb-scraperdata-extractionweb-datawebscraping
  Language:Python 21730
• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  web-scrapingweb-crawlingnpmheadless-chromepuppeteerautomationapifyscrapingcrawlingcrawlerheadlessscraperweb-crawlerjavascriptnodejsplaywrighttypescript
  Language:TypeScript 20477
• Douyin_TikTok_Download_API

  Evil0ctal / Douyin_TikTok_Download_API

  🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

  pythonpywebiotiktokdouyinapiscraperfastapino-watermarkonline-parsingasyncdouyin-tiktok-apidouyin-tiktok-downloadcrawlerspiderweb-scrapingtiktok-scraperdouyin-scraperdouyin-apitiktok-apitiktok-signature
  Language:Python 14836

• getmaxun / maxun

  ⚡ Easiest no code web data extraction platform • Instantly turn any website into API or spreadsheet ⚡

  automationno-codescraperweb-automationweb-scraperweb-scrapingapibrowserbrowser-automationplaywrightself-hostedrobotic-process-automationrpano-code-web-scraperagentsdata-extractionwebscrapinghacktoberfesthacktoberfest-acceptednocode
  Language:TypeScript 13826
• SeleniumBase

  seleniumbase / SeleniumBase

  Python APIs for web automation, testing, and bypassing bot-detection with ease.

  pythonseleniumwebdriverselenium-pythone2e-testingseleniumbasepytest-pluginweb-automationpytestwebkitchromedriveranti-detectionbot-detectioncloudflare-bypassweb-scraping-pythonpython-scraperweb-scrapingtest-automationcdpbehave
  Language:Python 11845
• Scrapling

  D4Vinci / Scrapling

  🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

  crawlercrawlingcrawling-pythonplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpathautomationaiai-scrapingdatadata-extractionmcpmcp-server
  Language:Python 8110

• mherrmann / helium

  Lighter web automation with Python

  seleniumselenium-pythonpythonpython3webdriverchromefirefoxweb-automationweb-scrapinghelium
  Language:Python 8082

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  web-scrapingcaptcha-bypasscaptcha-recaptchacrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolwebscrapingcrawlerspider
  Language:Makefile 7421

• apify / crawlee-python

  Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationbeautifulsoupcrawlercrawlingheadlessheadless-chromepipplaywrightpythonscraperscrapingweb-crawlerweb-crawlingweb-scrapinghacktoberfest
  Language:Python 7129
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  aiartificial-intelligenceautomationcrawlermachine-learningpythonscrapescraperscrapingweb-scrapingwebautomationwebscraping
  Language:Python 7021

• go-rod / rod

  A Chrome DevTools Protocol driver for web automation and scraping.

  cdpchrome-headlesschrome-devtoolschrome-devtools-protocolheadlessweb-scrapingautomationscraperdevtoolsdevtools-protocolrodgogolangtestingwebgorodcrawling
  Language:Go 6400

• autoscrape-labs / pydoll

  Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.

  cdpchromiumplaywrightpuppeteerrecaptcha-v3seleniumwebdriverbrowser-automationanti-detectionautomationcrawlere2e-testsheadlessscrapingtestingtesting-toolsturnstile-solverfingerprintingweb-scraping
  Language:Python 5988

• firecrawl / firecrawl-mcp-server

  🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

  batch-processingclaudecontent-extractiondata-collectionfirecrawlfirecrawl-aillm-toolsmcp-servermodel-context-protocolsearch-apiweb-crawlerweb-scrapingjavascript-renderingmcp
  Language:JavaScript 4887

• adbar / trafilatura

  Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

  web-scrapingtext-extractionnlphtml2texttext-miningcrawlertext-cleaningtext-preprocessingarticle-extractorreadabilityscrapingnews-crawlerteihtml-to-markdowncorpus-buildercorpus-toolsrss-feednews-aggregatorragllm
  Language:Python 4879

• jaypyles / Scraperr

  Self-hosted webscraper.

  opensourceself-hostedwebscraperdockerhelmkubernetesplaywrightpythonscrapingweb-scraperweb-scrapersweb-scrapingwebscraping
  Language:TypeScript 4681

• lexiforest / curl_cffi

  Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.

  curlhttp-clientcurl-impersonatehttphttpsja3ja3-fingerprinttls-fingerprintfingerprintingweb-scrapingakamai-fingerprinthttp2-fingerprint
  Language:Python 4410
• snoop

  snooppr / snoop

  Snoop — инструмент разведки на основе открытых данных (OSINT world)

  osinttermuxusername-searchusername-checkerpentestweb-scrapingctfscannerredteamblueteaminfosecsecuritynicknameipgeopoliceparserscrapinggeocoderusername
  Language:Python 3493

• php-curl-class / php-curl-class

  PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs

  phpcurlclassapiapi-clientclientframeworkhttphttp-clienthttp-proxyjsonphp-curlphp-curl-libraryproxyrequestsrestfulweb-scraperweb-scrapingweb-servicexml
  Language:PHP 3300

• x4nth055 / pythoncode-tutorials

  The Python Code Tutorials

  pythonpython3scapyethical-hackingnetwork-programmingnetwork-securitynetwork-analysispython-tutorialstutorialsscapy-tutorialsmachine-learningtext-classificationsocket-programmingface-detectioncomputer-visionprogramming-tutorialnatural-language-processingweb-scraping
  Language:Jupyter Notebook 2938

• lorien / grab

  Web Scraping Framework

  web-scrapinghttp-clientframeworkpythonpycurlasynchronousnetworkurllib3spidercrawlercrawlingscrapingpython-librarypython3
  Language:Python 2428
• google-maps-scraper

  gosom / google-maps-scraper

  scrape data data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

  golanggoogle-maps-scrapingweb-scraperweb-scrapingdistributed-scraperdistributed-scrapinggoogle-maps
  Language:Go 2409

• Virtual-Browser / VirtualBrowser

  Free anti fingerprint browser, 指纹浏览器, 隐私浏览器, 防识别浏览器, 反识别浏览器, 防关联浏览器, 免费的web3空投专用指纹浏览器 https://virtualbrowser.cc/?src=github

  chromium-basedmulti-accountprivacy-toolsproxy-supportweb-scrapinganti-detectionfingerprint-browserzhiwenliulanqi
  Language:Vue 2254

• codingforentrepreneurs / 30-Days-of-Python

  Learn Python for the next 30 (or so) Days.

  pythonapipython3tutorialjupyterpandasseleniumselenium-webdriverweb-scrapingautomationcsvrest-apiflaskfastapi
  Language:HTML 2213
• amazon-scraper

  oxylabs / amazon-scraper

  Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.

  amazon-apiamazon-scraperamazon-scrapinge-commerce-apiweb-scrapingpythonamazon-scraper-apiamazon-scraping-libraryamazon-price-trackerscraping-apiamazon-aipamazon-api-dataamazon-products-apiamazon-scrape-apihow-to-scrape-amazonscrape-amazonscraping-amazonprice-scraperprice-scraper-apiamazon-product-scraper
  Language:Python 1857
• patchright

  Kaliiiiiiiiii-Vinyzu / patchright

  Undetected version of the Playwright testing and automation library.

  automationbotbotsbottingbrowserchromechromedriverchromiumcloudflarecloudflare-bypassplaywrightstealthundetectableundetectedweb-automationweb-scrapingwebautomationwebdriverwebscraping
  Language:JavaScript 1780

• A9T9 / RPA

  Ui.Vision Open-Source RPA Software with Computer Vision, OCR, Anthropic Computer Use/LLM. Selenium IDE import/export.

  web-automationselenium-ideimacrosbrowser-extensionbrowser-automationdata-driven-testsweb-scrapinganthropicanthropic-claudecomputer-use
  Language:JavaScript 1766

• justmarkham / DAT8

  General Assembly's 2015 Data Science course in Washington, DC

  data-sciencemachine-learningscikit-learndata-analysispandasjupyter-notebookpythoncourselinear-regressionlogistic-regressionmodel-evaluationnaive-bayesnatural-language-processingdecision-treesensemble-learningclusteringregular-expressionsweb-scrapingdata-visualizationdata-cleaning
  Language:Jupyter Notebook 1615

• brightdata / brightdata-mcp

  A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.

  ai-agentsai-integrationsanti-bot-detectionbrowser-automationdata-collectiondata-extractionllmmcpmcp-servermodelcontextprotocolscrapingscraping-toolsstructured-dataweb-crawlingweb-dataweb-scraping
  Language:JavaScript 1558

• tidyverse / rvest

  Simple web scraping for R

  rweb-scrapinghtml
  Language:R 1507

• rushter / selectolax

  Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

  crawlercsshtmlhtml5parserpythonscraperweb-scraping
  Language:Cython 1458

• roach-php / core

  The complete web scraping toolkit for PHP.

  crawlingphpweb-scraping
  Language:PHP 1433
• free-proxy-list

  oxylabs / free-proxy-list

  Claim Free proxy list with United States IP addresses and use it for your projects.

  free-proxiesfree-proxies-for-web-scrapingfree-proxyfree-proxy-ipfree-proxy-listproxiesproxies-httpproxies-httpsproxies-listproxies-socks5proxyproxy-listproxy-serverproxypoolweb-scrapingproxy-scrapeproxy-scraperproxy-scrapers
  1297