scraping

There are 164 repositories under scraping topic.

• scrapy

  scrapy / scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  crawlercrawlingframeworkhacktoberfestpythonscrapingweb-scrapingweb-scraping-python
  Language:Python 51107

• gocolly / colly

  Elegant Scraper and Crawler Framework for Golang

  golangscraperframeworkcrawlerscrapingcrawlingspidergo
  Language:Go 22302

• psf / requests-html

  Pythonic HTML Parsing for Humans™

  beautifulsoupcss-selectorshtmlhttpkennethreitzlxmlpyquerypythonrequestsscraping
  Language:Python 13597
• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
  Language:TypeScript 12328

• code4craft / webmagic

  A scalable web crawler framework for Java.

  crawlerframeworkjavascraping
  Language:Java 11265

• ultrafunkamsterdam / undetected-chromedriver

  Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

  chromedriverseleniumwebdriverchromeanti-detectionanti-botdistilbrowserautomationscrapingpython3captchanavigatortestingcloudflarecloudflare-bypassbot-detection
  Language:Python 8581

• VinciGit00 / Scrapegraph-ai

  Python scraper based on AI

  automated-scrapergpt-3gpt-4llama3llmmachine-learningscscrapingscraping-pythonscrapingweb
  Language:Python 8146

• tabulapdf / tabula

  Tabula is a tool for liberating data tables trapped inside PDF files

  pdfcsvexceltablesscraping
  Language:CSS 6543

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  captcha-bypasscaptcha-recaptchacrawlercrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolspiderweb-scrapingwebscraping
  Language:Makefile 6357
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  scrapingscraperscrapewebscrapingcrawlerweb-scrapingaiartificial-intelligencepythonwebautomationautomationmachine-learning
  Language:Python 5972
• ferret

  MontFerret / ferret

  Declarative web scraping

  cdpchromeclicrawlercrawlingdata-miningdslgogolanghacktoberfestlibraryquery-languagescraperscrapingscraping-websitestool
  Language:Go 5631

• yujiosaka / headless-chrome-crawler

  Distributed crawler powered by Headless Chrome

  headless-chromepuppeteerjquerycrawlercrawlingscraperscrapingchromechromiumpromise
  Language:JavaScript 5479

• sparklemotion / mechanize

  Mechanize is a ruby library that makes automated web interaction easy.

  scrapingwebruby
  Language:Ruby 4358

• khuyentran1401 / Data-science

  Collection of useful data science topics along with articles, videos, and code

  data-sciencemachine-learningnatural-language-processingpythondata-visualizationdata-analysisarticlesartificial-intelligencetime-seriesscraping
  Language:Jupyter Notebook 3966

• fake-useragent / fake-useragent

  Up-to-date simple useragent faker with real world database

  pythonpython3useragentfakefakerscrapinguser-agentuser-agent-spooferuseragentuseragent-scraper
  Language:Python 3487

• aapatre / Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE

  Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

  pythonpython3scraperscrapingselenium
  Language:Python 3064

• adbar / trafilatura

  Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

  web-scrapingtext-extractionnlphtml2textnewstext-miningcrawlertext-cleaningtext-preprocessingarticle-extractorreadabilityscrapingnews-crawlerteihtml-to-markdowncorpuscorpus-buildercorpus-toolsrss-feednews-aggregator
  Language:Python 2932
• panther

  symfony / panther

  A browser testing and web crawling library for PHP and Symfony

  scrapinge2e-testingwebdriverseleniumselenium-webdriversymfonyphpchromedriverhacktoberfest
  Language:PHP 2893

• mendableai / firecrawl

  🔥 Turn entire websites into LLM-ready markdown

  aiai-scrapingcrawlerdatahtml-to-markdownllmmarkdownragscraperscrapingweb-crawler
  Language:TypeScript 2859
• snoop

  snooppr / snoop

  Snoop — инструмент разведки на основе открытых данных (OSINT world)

  blueteamctfgeogeocoderinfosecipnicknameosintparserpentestpoliceredteamscannerscrapingsecuritytermuxusernameusername-checkerusername-searchweb-scraping
  Language:Python 2713

• NikolaiT / GoogleScraper

  A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

  search-enginepythonscrapingcrawlersearch-enginessearch-engine-optimization
  Language:HTML 2584

• geziyor / geziyor

  Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

  goscrapingscrapercrawlerspider
  Language:Go 2490

• emadehsan / thal

  Getting started with Puppeteer and Chrome Headless for Web Scraping

  puppeteerchrome-headlessnodejsscrapingmongoosemongodb
  Language:JavaScript 2359

• lorien / grab

  Web Scraping Framework

  asynchronouscrawlercrawlingframeworkhttp-clientnetworkpycurlpythonpython-librarypython3scrapingspiderurllib3web-scraping
  Language:Python 2359

• transitive-bullshit / awesome-puppeteer

  A curated list of awesome puppeteer resources.

  puppeteerheadless-chromeawesomeawesome-listscrapingcrawlingautomation
  2330

• kevinzg / facebook-scraper

  Scrape Facebook public pages without an API key

  facebookfacebook-scraperfacebook-scrapinghacktoberfestscraping
  Language:Python 2214

• oscarotero / Embed

  Get info from any web service or page

  opengraphtwitter-cardsembedsscrapingoembed
  Language:PHP 2065
• shot-scraper

  simonw / shot-scraper

  A command-line utility for taking automated screenshots of websites

  playwrightplaywright-pythonscrapingscreenshot-utilityscreenshots
  Language:Python 1541

• claffin / cloudproxy

  Hide your scrapers IP behind the cloud. Provision proxy servers across different cloud providers to improve your scraping success.

  cloudproxyproxy-serverscraping
  Language:Python 1360
• mlscraper

  lorey / mlscraper

  🤖 Scrape data from HTML websites automatically by just providing examples

  scrapingcrawlinghtmlmachine-learningextraction-enginescrapercrawlercrawler-python
  Language:Python 1231

• istresearch / scrapy-cluster

  This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

  pythonscrapykafkaredisscrapingdistributed
  Language:Python 1163

• holgerd77 / django-dynamic-scraper

  Creating Scrapy scrapers via the Django admin interface

  pythondjangoscraperscrapingscrapyspiderwebscraping
  Language:Python 1141

• medialab / artoo

  artoo.js - the client-side scraping companion.

  scraping
  Language:JavaScript 1101

• scrapy / parsel

  Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

  pythonlxmlxpathxmlselectorscssscrapinghacktoberfest
  Language:Python 1089

• damklis / DataEngineeringProject

  Example end to end data engineering project.

  big-datascrapingmongodbelasticsearchdata-engineeringkafkakafka-connectdebeziumdjango-rest-frameworkredisairflowminios3pythondata-pipelinehacktoberfest
  Language:Python 1025
• querido-diario

  okfn-brasil / querido-diario

  📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.

  data-sciencemachine-learningpoliticsartificial-intelligenceopen-datacivic-techgovernments-gazettesgovtechspiderscrapinghacktoberfesthacktoberfest2023
  Language:Python 1010