webscraping

There are 151 repositories under webscraping topic.

• firecrawl

  firecrawl / firecrawl

  🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

  aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
  Language:TypeScript 67381

• huginn / huginn

  Create agents that monitor and act on your behalf. Your agents are standing by!

  agentautomationfeedfeedgeneratorhuginnmonitoringnotificationsrssscrapertwittertwitter-streamingwebscraping
  Language:Ruby 47969
• gpt-researcher

  assafelovic / gpt-researcher

  An LLM agent that conducts deep research (local and web) on any given topic and generates a long report with citations.

  agentaiautomationdeepresearchllmsmcpmcp-serverpythonresearchsearchwebscraping
  Language:Python 24112

• ScrapeGraphAI / Scrapegraph-ai

  Python scraper based on AI

  ai-crawlerai-scrapingai-searchautomated-scrapercrawlerdata-extractionlarge-language-modelllmmarkdownragscrapingscraping-pythonweb-crawlerweb-crawlersweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
  Language:Python 21748

• getmaxun / maxun

  ⚡ Easiest no code web data extraction platform • Instantly turn any website into API or spreadsheet ⚡

  agentsapiautomationbrowserbrowser-automationdata-extractionhacktoberfesthacktoberfest-acceptedno-codeno-code-web-scrapernocodeplaywrightrobotic-process-automationrpascraperself-hostedweb-automationweb-scraperweb-scrapingwebscraping
  Language:TypeScript 13839

• pystardust / ani-cli

  A cli tool to browse and play anime

  shellclianimeposixsteamdecksyncplaytermuxwebscrapingfzflinuxmacrofiterminalwindows
  Language:Shell 10071
• Scrapling

  D4Vinci / Scrapling

  🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!

  aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionmcpmcp-serverplaywrightpythonscrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
  Language:Python 8120

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  web-scrapingcaptcha-bypasscaptcha-recaptchacrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolwebscrapingcrawlerspider
  Language:Makefile 7425
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  scrapingscraperscrapewebscrapingcrawlerweb-scrapingaiartificial-intelligencepythonwebautomationautomationmachine-learning
  Language:Python 7022

• niespodd / browser-fingerprinting

  Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

  botdetectionchromiumstealthpuppeteerscraperwebscrapingwebautomationchromium-browserbot-detectionchromedriverfingerprintingcrawlerrecaptchaspiderbrowser-fingerprinting
  Language:JavaScript 4905

• jaypyles / Scraperr

  Self-hosted webscraper.

  dockerhelmkubernetesopensourceplaywrightpythonscrapingself-hostedweb-scraperweb-scrapersweb-scrapingwebscraperwebscraping
  Language:TypeScript 4688

• daijro / camoufox

  🦊 Anti-detect browser

  antidetectantidetect-browserfingerprintfirefoxplaywrightwebscrapingnetworkingscraping
  Language:C++ 4047

• scrapoxy / scrapoxy

  Scrapoxy is a super proxies manager that orchestrates all your proxies into one place, rather than spreading management across multiple scrapers. It manages IP rotation and fingerprinting, and smartly routes traffic to avoid bans.

  antibotblacklistingproxieswebscraping
  Language:TypeScript 2395

• anaskhan96 / soup

  Web Scraper in Go, similar to BeautifulSoup

  beautifulsoupgogolanghtml-nodeweb-scraperwebscraperwebscraping
  Language:Go 2218

• itsOwen / CyberScraper-2077

  A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

  ai-scrapingllmopenaiscraperwebscrapinggemini-apillm-scraperweb-scraper
  Language:Python 1888
• patchright

  Kaliiiiiiiiii-Vinyzu / patchright

  Undetected version of the Playwright testing and automation library.

  automationbotbotsbottingbrowserchromechromedriverchromiumcloudflarecloudflare-bypassplaywrightstealthundetectableundetectedweb-automationweb-scrapingwebautomationwebdriverwebscraping
  Language:JavaScript 1792

• reworkd / tarsier

  Vision utilities for web interaction agents 👀

  gpt4vllmsocrplaywrightpypi-packagepythonseleniumwebscraping
  Language:Jupyter Notebook 1740

• TheWebScrapingClub / webscraping-from-0-to-hero

  The web scraping open project repository aims to share knowledge and experiences about web scraping with Python

  playwrightpythonscrapyscrapy-spiderscrapysplashwebscraping
  1687
• requests-cache

  requests-cache / requests-cache

  Persistent HTTP cache for python requests

  cachedynamodbhttpmongodbperformanceredisrequestssqlitewebwebscraping
  Language:Python 1460

• jamesturk / scrapeghost

  👻 Experimental library for scraping websites using OpenAI's GPT API.

  gptwebscrapingopenai-api
  Language:Python 1443

• m8sec / CrossLinked

  LinkedIn enumeration tool to extract valid employee names from an organization through search engine scraping

  webscrapingpython3osintenumerationusername-generatorpentest-toolpentest-scriptslinkedin-scraper
  Language:Python 1441

• raznem / parsera

  Lightweight library for scraping web-sites with LLMs

  aiai-scrapingdata-extractionllmopensourceplaywrightpythonscrapingwebscraping
  Language:Python 1235

• holgerd77 / django-dynamic-scraper

  Creating Scrapy scrapers via the Django admin interface

  pythondjangoscraperscrapingscrapyspiderwebscraping
  Language:Python 1160
• mov-cli

  mov-cli / mov-cli

  Watch everything from your terminal.

  androidclihacktoberioslinuxscrapingwebscrapingwindows
  Language:Python 1049

• GodsScion / Auto_job_applier_linkedIn

  Make your job hunt easy by automating your application process with this Auto Applier

  pythonseleniumwebscrapingautomationjob-applicationjob-searchpython3linkedin-job-scraperlinkedin-jobs-scraperautomation-seleniumlinkedinselenium-pythonautomatic-job-applierundetected-chromedriverauto-apply
  Language:Python 1031
• patchright-python

  Kaliiiiiiiiii-Vinyzu / patchright-python

  Undetected Python version of the Playwright testing and automation library.

  automationbotbotsbottingbrowserchromechromedriverchromiumcloudflarecloudflare-byplaywrightstealthundetectableundetectedweb-automationweb-scrapingwebautomationwebdriverwebscraping
  Language:Python 951

• cdpdriver / zendriver

  A blazing fast, async-first, undetectable webscraping/web automation framework based on ultrafunkamsterdam/nodriver. Now with Docker support!

  anti-botasyncbot-detectionbrowserbrowser-automationcaptchachromechrome-devtools-protocolchromedrivercloudflarecloudflare-bypasspythonseleniumwebdriverwebscraping
  Language:Python 867

• benibela / xidel

  Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.

  xqueryxmlhtmljsonxpathclicommand-linehttpwebrestcss-selectorwgetcurlhttpiexmlstarletwebscraperwebscrapingscraperdatascrapingdata-processing
  Language:Pascal 817
• suckit

  Skallwar / suckit

  Suck the InTernet

  hacktoberfestrustwebscraping
  Language:Rust 791

• maxhumber / gazpacho

  🥫 The simple, fast, and modern web scraping library

  gazpachowebscrapingscraping
  Language:Python 771

• scrapfly / scrapfly-scrapers

  Scalable Python web scraping scripts for +40 popular domains

  crawlingpythoncrawlerscrapingweb-scrapingweb-scraping-pythonantibotautomationcaptcha-bypasscrawling-pythondatascrapingproxiespython-scraperscraperscraping-pythonspidertwitter-scraperweb-crawlerwebscraperwebscraping
  Language:Python 749
• Uscrapper

  z0m31en7 / Uscrapper

  Uscrapper Vanta: Dive deeper into the web with this powerful open-source tool. Extract valuable insights with ease and efficiency, from both surface and deep web sources. Empower your data mining and analysis with Vanta's advanced capabilities. Fast, reliable, and user-friendly, Uscrapper Vanta is the ultimate choice for researchers and analysts.

  osintpythonweb-scrapingwebsite-scraperinformation-extractioninformation-gatheringosint-pythonosint-toolreconnaissancewebscrapingwebsitesseleniumselenium-webscraperwebcrawebcrawlerdarkwebdarkweb-crawlertor
  Language:Python 719

• wodsuz / EasyApplyJobsBot

  A python bot to automatically apply all Linkedin,Glassdoor, etc Easy Apply jobs based on your preferences. Auto login, auto fill additional questions, apply automatically!

  automationpython3botseleniumlinkedinaichallengechatgptjobsapply-jobsautomatedfind-jobsglassdoorglassdoor-scraperjobwebscrapinglist-jobsindeedziprecruiter
  Language:Python 691
• instascrape

  chris-greening / instascrape

  Powerful and flexible Instagram scraping library for Python, providing easy-to-use and expressive tools for accessing data programmatically

  pythoninstagramwebscrapingdata-mininginstagram-scraperlightweightpython3data-sciencepython-scraperinstagram-databeginner-friendly
  Language:Python 659

• openzim / zimit

  Make a ZIM file from any Web site and surf offline!

  dockerscraperwebscrapingzim
  Language:Python 639
• H4X-Tools

  vil / H4X-Tools

  Open source toolkit for scraping, OSINT and more.

  pythonpython-scriptpython3toolshackinglinuxh4x-toolshacking-toolhacktoolsip-scannerphone-numberwebhook-spammerdata-gatheringosintwebscrapingwebsearchdirbusterport-scanneremail-osintigscraper
  Language:Python 597