scraping

There are 196 repositories under scraping topic.

• scrapy

  scrapy / scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  crawlercrawlingframeworkhacktoberfestpythonscrapingweb-scrapingweb-scraping-python
  Language:Python 54165

• feder-cr / Jobs_Applier_AI_Agent_AIHawk

  Jobs_Applier_AI_Agent_AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in a tailored way.

  agentapplication-resumeartificial-intelligenceautomateautomationbotchatgptchromegpthuman-resourcesjobjobsjobsearchjobseekeropeaipythonresumescraperscrapingselenium
  Language:Python 27132

• mendableai / firecrawl

  🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.

  aiai-scrapingcrawlerdatahtml-to-markdownllmmarkdownragscraperscrapingweb-crawlerwebscraping
  Language:TypeScript 25825

• gocolly / colly

  Elegant Scraper and Crawler Framework for Golang

  crawlercrawlingframeworkgogolangscraperscrapingspider
  Language:Go 23734

• ScrapeGraphAI / Scrapegraph-ai

  Python scraper based on AI

  aiautomated-scrapergpt-3gpt-4llama3llmmachine-learningscscrapingscraping-pythonscrapingwebwebscraping
  Language:Python 18044
• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
  Language:TypeScript 16792
• maigret

  soxoj / maigret

  🕵️‍♂️ Collect a dossier on a person by username from thousands of sites

  blueteamcybersecurityidentificationinvestigationnamecheckeropen-sourceosintosint-frameworkosint-pythonparsingpythonpython3redteamscrapingsherlocksocial-networksocmintusernameusername-checkerusername-search
  Language:Python 14657

• psf / requests-html

  Pythonic HTML Parsing for Humans™

  beautifulsoupcss-selectorshtmlhttpkennethreitzlxmlpyquerypythonrequestsscraping
  Language:Python 13779

• code4craft / webmagic

  A scalable web crawler framework for Java.

  crawlerframeworkjavascraping
  Language:Java 11492

• ultrafunkamsterdam / undetected-chromedriver

  Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

  anti-botanti-detectionautomationbot-detectionbrowsercaptchachromechromedrivercloudflarecloudflare-bypassdistilnavigatorpython3scrapingseleniumtestingwebdriver
  Language:Python 10662

• tabulapdf / tabula

  Tabula is a tool for liberating data tables trapped inside PDF files

  csvexcelpdfscrapingtables
  Language:CSS 6908

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  captcha-bypasscaptcha-recaptchacrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolspiderweb-scrapingwebscraping
  Language:Makefile 6882
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  aiartificial-intelligenceautomationcrawlermachine-learningpythonscrapescraperscrapingweb-scrapingwebautomationwebscraping
  Language:Python 6630
• ferret

  MontFerret / ferret

  Declarative web scraping

  cdpchromeclicrawlercrawlingdata-miningdslgogolanghacktoberfestlibraryquery-languagescraperscrapingscraping-websitestool
  Language:Go 5771

• yujiosaka / headless-chrome-crawler

  Distributed crawler powered by Headless Chrome

  chromechromiumcrawlercrawlingheadless-chromejquerypromisepuppeteerscraperscraping
  Language:JavaScript 5546

• apify / crawlee-python

  Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationbeautifulsoupcrawlercrawlinghacktoberfestheadlessheadless-chromepipplaywrightpythonscraperscrapingweb-crawlerweb-crawlingweb-scraping
  Language:Python 5272

• sparklemotion / mechanize

  Mechanize is a ruby library that makes automated web interaction easy.

  rubyscrapingweb
  Language:Ruby 4407

• khuyentran1401 / Data-science

  Collection of useful data science topics along with articles, videos, and code

  articlesartificial-intelligencedata-analysisdata-sciencedata-visualizationmachine-learningnatural-language-processingpythonscrapingtime-series
  Language:Jupyter Notebook 4077

• adbar / trafilatura

  Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

  article-extractorcorpus-buildercorpus-toolscrawlerhtml-to-markdownhtml2textllmnews-aggregatornews-crawlernlpragreadabilityrss-feedscrapingteitext-cleaningtext-extractiontext-miningtext-preprocessingweb-scraping
  Language:Python 3941

• fake-useragent / fake-useragent

  Up-to-date simple useragent faker with real world database

  agentfakefakerpythonpython3scrapinguseruser-agentuser-agent-spooferuseragentuseragent-scraper
  Language:Python 3780
• snoop

  snooppr / snoop

  Snoop — инструмент разведки на основе открытых данных (OSINT world)

  blueteamctfgeogeocoderinfosecipnicknameosintparserpentestpoliceredteamscannerscrapingsecuritytermuxusernameusername-checkerusername-searchweb-scraping
  Language:Python 3201

• aapatre / Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE

  Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

  pythonpython3scraperscrapingselenium
  Language:Python 3187
• panther

  symfony / panther

  A browser testing and web crawling library for PHP and Symfony

  chromedrivere2e-testinghacktoberfestphpscrapingseleniumselenium-webdriversymfonywebdriver
  Language:PHP 2974

• bjesus / pipet

  Swiss-army tool for scraping and extracting data from online assets, made for hackers

  csscurlgjsonjsonplaywrightscraperscraping
  Language:Go 2678

• geziyor / geziyor

  Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

  crawlergoscraperscrapingspider
  Language:Go 2664

• NikolaiT / GoogleScraper

  A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

  search-enginepythonscrapingcrawlersearch-enginessearch-engine-optimization
  Language:HTML 2664
• Scrapling

  D4Vinci / Scrapling

  🕷️ Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python

  aiai-scrapingautomationcrawlercrawlingcrawling-pythondatadata-extractionhacktoberfestplaywrightpythonpython3scrapingselectorsstealthweb-scraperweb-scrapingweb-scraping-pythonwebscrapingxpath
  Language:Python 2571

• kevinzg / facebook-scraper

  Scrape Facebook public pages without an API key

  facebookfacebook-scraperfacebook-scrapinghacktoberfestscraping
  Language:Python 2563

• transitive-bullshit / awesome-puppeteer

  A curated list of awesome puppeteer resources.

  automationawesomeawesome-listcrawlingheadless-chromepuppeteerscraping
  2437

• lorien / grab

  Web Scraping Framework

  asynchronouscrawlercrawlingframeworkhttp-clientnetworkpycurlpythonpython-librarypython3scrapingspiderurllib3web-scraping
  Language:Python 2400

• emadehsan / thal

  Getting started with Puppeteer and Chrome Headless for Web Scraping

  chrome-headlessmongodbmongoosenodejspuppeteerscraping
  Language:JavaScript 2358

• oscarotero / Embed

  Get info from any web service or page

  embedsoembedopengraphscrapingtwitter-cards
  Language:PHP 2103
• twikit

  d60 / twikit

  Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

  pythonbotclientpython3scraperscrapingsearchtwitterwrappertwitter-apitwitter-scraperscrapetwitter-bottwitter-clienttwitter-internal-apixx-apitweepypython-web-scraper
  Language:Python 2094
• shot-scraper

  simonw / shot-scraper

  A command-line utility for taking automated screenshots of websites

  playwrightplaywright-pythonscrapingscreenshot-utilityscreenshots
  Language:Python 1789

• spider-rs / spider

  A web crawler and scraper for Rust

  crawlerdataheadless-chromeindexerrustscrapingspider
  Language:Rust 1467

• claffin / cloudproxy

  Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.

  cloudproxyproxy-serverscraping
  Language:Python 1428