scraping

There are 96 repositories under scraping topic.

• scrapy

  scrapy / scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  pythonscrapingcrawlingframeworkcrawlerhacktoberfest
  Language:Python 43888

• gocolly / colly

  Elegant Scraper and Crawler Framework for Golang

  crawlercrawlingframeworkgogolangscraperscrapingspider
  Language:Go 16884

• psf / requests-html

  Pythonic HTML Parsing for Humans™

  beautifulsoupcss-selectorshtmlhttpkennethreitzlxmlpyquerypythonrequestsscraping
  Language:Python 12601

• code4craft / webmagic

  A scalable web crawler framework for Java.

  crawlerframeworkjavascraping
  Language:Java 10413

• tabulapdf / tabula

  Tabula is a tool for liberating data tables trapped inside PDF files

  csvexcelpdfscrapingtables
  Language:CSS 5732

• curlconverter / curlconverter

  Generate code from cURL commands

  ansiblecurldartdeveloper-toolsgohttpjavajavascriptmatlabphppythonpython-requestsrrustscrapingtree-sitter
  Language:TypeScript 5641

• yujiosaka / headless-chrome-crawler

  Distributed crawler powered by Headless Chrome

  headless-chromepuppeteerjquerycrawlercrawlingscraperscrapingchromechromiumpromise
  Language:JavaScript 5247
• ferret

  MontFerret / ferret

  Declarative web scraping

  golangquery-languagedata-miningscrapingscraping-websitesdslcdpcrawlingscrapercrawlergochromeclitoollibraryhacktoberfesthacktoberfest2021
  Language:Go 5024
• autoscraper

  alirezamika / autoscraper

  A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  aiartificial-intelligenceautomationcrawlermachine-learningpythonscrapescraperscrapingweb-scrapingwebautomationwebscraping
  Language:Python 4483

• sparklemotion / mechanize

  Mechanize is a ruby library that makes automated web interaction easy.

  rubyscrapingweb
  Language:Ruby 4222
• apify-js

  apify / apify-js

  Apify SDK — The scalable web scraping and crawling library for JavaScript/Node.js. Enables development of data extraction and web automation jobs (not only) with headless Chrome and Puppeteer.

  web-scrapingweb-crawlingjavascript-librarynpmheadless-chromepuppeteerautomationapifyscrapingcrawlingrpa
  Language:JavaScript 3375

• aapatre / Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE

  Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

  pythonpython3scraperscrapingselenium
  Language:Python 2672

• khuyentran1401 / Data-science

  Collection of useful data science topics along with code and articles

  articlesartificial-intelligencedata-analysisdata-sciencedata-visualizationmachine-learningnatural-language-processingpythonscrapingtime-series
  Language:Jupyter Notebook 2621
• panther

  symfony / panther

  A browser testing and web crawling library for PHP and Symfony

  scrapinge2e-testingwebdriverseleniumselenium-webdriversymfonyphpchromedriverhacktoberfest
  Language:PHP 2586

• NikolaiT / GoogleScraper

  A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

  search-enginepythonscrapingcrawlersearch-enginessearch-engine-optimization
  Language:HTML 2407

• emadehsan / thal

  Getting started with Puppeteer and Chrome Headless for Web Scraping

  puppeteerchrome-headlessnodejsscrapingmongoosemongodb
  Language:JavaScript 2351

• ultrafunkamsterdam / undetected-chromedriver

  Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)

  anti-botanti-detectionautomationbot-detectionbrowsercaptchachromechromedrivercloudflarecloudflare-bypassdistilnavigatorpython3scrapingseleniumtestingwebdriver
  Language:Python 2251

• oscarotero / Embed

  Get info from any web service or page

  embedsoembedopengraphscrapingtwitter-cards
  Language:PHP 1863

• transitive-bullshit / awesome-puppeteer

  A curated list of awesome puppeteer resources.

  automationawesomeawesome-listcrawlingheadless-chromepuppeteerscraping
  1842

• geziyor / geziyor

  Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

  crawlergoscraperscrapingspider
  Language:Go 1729
• snoop

  snooppr / snoop

  Snoop — инструмент разведки на основе открытых данных (OSINT world)

  osinttermuxusername-searchusername-checkerpentestweb-scrapingctfscannerredteamblueteaminfosecsecuritynicknamegeolocationipgeopoliceparserscrapinggeocoder
  Language:Python 1469

• kevinzg / facebook-scraper

  Scrape Facebook public pages without an API key

  facebookfacebook-scraperfacebook-scrapinghacktoberfestscraping
  Language:Python 1222

• holgerd77 / django-dynamic-scraper

  Creating Scrapy scrapers via the Django admin interface

  pythondjangoscraperscrapingscrapyspiderwebscraping
  Language:Python 1088

• medialab / artoo

  artoo.js - the client-side scraping companion.

  scraping
  Language:JavaScript 1078

• istresearch / scrapy-cluster

  This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

  pythonscrapykafkaredisscrapingdistributed
  Language:Python 1046

• meetmangukiya / instagram-scraper

  Scrape the Instagram frontend. Inspired from twitter-scraper by @kennethreitz.

  instagramno-authenticationjavascriptclientscrapingpython-3-6requests-htmlinstagram-scraper
  Language:Python 921

• iawia002 / Lulu

  [Unmaintained] A simple and clean video/music/image downloader 👾

  crawlercrawlingdownloaderpythonpython3scraperscrapingvideo
  Language:Python 807

• DataHenHQ / till

  DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

  crawlerman-in-the-middlemitmproxy-serverscraperscrapingweb-scraping
  Language:Go 784

• scrapy / parsel

  Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

  pythonlxmlxpathxmlselectorscssscrapinghacktoberfest
  Language:Python 780

• online-judge-tools / oj

  Tools for various online judges. Downloading sample cases, generating additional test cases, testing your code, and submitting it.

  competitive-programmingautomationscrapingatcodercodeforcesprogramming-conteststesting
  Language:Python 724

• sananth12 / ImageScraper

  :scissors: High performance, multi-threaded image scraper

  command-linecommandline-toolpypipythonscraperscrapingterminal
  Language:Python 700

• MorvanZhou / easy-scraping-tutorial

  Simple but useful Python web scraping tutorial code.

  asynciobeautifulsoupcrawlercrawlingdistributed-scraperregexrequestsscrapingscrapyurllib
  Language:Jupyter Notebook 666

• maxhumber / gazpacho

  🥫 The simple, fast, and modern web scraping library

  gazpachoscrapingwebscraping
  Language:Python 660

• jfilter / clean-text

  🧹 Python package for text cleaning

  pythonnatural-language-processingtext-cleaningtext-normalizationtext-preprocessingpython-packagenlpuser-generated-contentscraping
  Language:Python 642

• iiab / iiab

  Internet-in-a-Box - Build your own LIBRARY OF ALEXANDRIA with a Raspberry Pi !

  learninghotspotlibraryscrapingraspberry-piknowledgemedicalprisoners-rightshuman-rightscurriculum-designdiycommunity-networksmesh-networksprivacycivic-techofflinehome-schoolinternational-developmentdistraction-freeeducation
  Language:Jinja 612

• AmmeySaini / Edu-Mail-Generator

  Generate Free Edu Mail(s) within minutes

  seleniumpythonedumailscrapingscraping-websitesinstall-webdriverpython3edumailstudent-mailedu-accountedu-generatorselenium-pythonauto-install-webdriver
  Language:Python 593