crawling

There are 67 repositories under crawling topic.

• scrapy

  scrapy / scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  pythonscrapingcrawlingframeworkcrawlerhacktoberfestweb-scrapingweb-scraping-python
  Language:Python 51104

• gocolly / colly

  Elegant Scraper and Crawler Framework for Golang

  golangscraperframeworkcrawlerscrapingcrawlingspidergo
  Language:Go 22300

• codelucas / newspaper

  newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

  pythonnewscrawlercrawlingscrapernews-aggregator
  Language:Python 13781
• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  web-scrapingweb-crawlingnpmheadless-chromepuppeteerautomationapifyscrapingcrawlingcrawlerheadlessscraperweb-crawlerjavascriptnodejsplaywrighttypescript
  Language:TypeScript 12326

• lorien / awesome-web-scraping

  List of libraries, tools and APIs for web scraping and data processing.

  web-scrapingcaptcha-bypasscaptcha-recaptchacrawlingcrawling-frameworkcrawling-pythoncrawling-toolscrapingscraping-frameworkscraping-pythonscraping-toolwebscrapingcrawlerspider
  Language:Makefile 6357
• ferret

  MontFerret / ferret

  Declarative web scraping

  golangquery-languagedata-miningscrapingscraping-websitesdslcdpcrawlingscrapercrawlergochromeclitoollibraryhacktoberfest
  Language:Go 5631

• yujiosaka / headless-chrome-crawler

  Distributed crawler powered by Headless Chrome

  headless-chromepuppeteerjquerycrawlercrawlingscraperscrapingchromechromiumpromise
  Language:JavaScript 5479

• go-rod / rod

  A Devtools driver for web automation and scraping

  cdpchrome-headlesschrome-devtoolschrome-devtools-protocolheadlessweb-scrapingautomationscraperdevtoolsdevtools-protocolrodgogolangtestingwebgorodcrawling
  Language:Go 4841
• hakrawler

  hakluke / hakrawler

  Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

  bugbountycrawlinghackingosintpentestingreconreconnaissance
  Language:Go 4250

• hardkoded / puppeteer-sharp

  Headless Chrome .NET API

  puppeteerchromechromiumautomationcrawlercrawlingcsharpe2ee2e-testingwebautomation
  Language:C# 3185

• apache / nutch

  Apache Nutch is an extensible and scalable web crawler

  javanutchweb-crawlercrawlinghadoopapache
  Language:Java 2821

• lorien / grab

  Web Scraping Framework

  web-scrapinghttp-clientframeworkpythonpycurlasynchronousnetworkurllib3spidercrawlercrawlingscrapingpython-librarypython3
  Language:Python 2359

• transitive-bullshit / awesome-puppeteer

  A curated list of awesome puppeteer resources.

  puppeteerheadless-chromeawesomeawesome-listscrapingcrawlingautomation
  2330

• zorlan / skycaiji

  蓝天采集器是一款开源免费的爬虫系统，仅需点选编辑规则即可采集数据，可运行在本地、虚拟主机或云服务器中，几乎能采集所有类型的网页，无缝对接各类CMS建站程序，免登录实时发布数据，全自动无需人工干预！是网页大数据采集软件中完全跨平台的云端爬虫系统

  crawlercrawlingspiderwebcrawlerphp
  Language:PHP 1867

• edoardottt / cariddi

  Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

  endpointsendpoint-discoverybugbountycrawlersecret-keyssecrets-detectioninfosecreconnaissancereconcrawlinggolangpentestinggosecurityosintpenetration-testingsecurity-toolsscraperhacktoberfestredteam
  Language:Go 1368

• roach-php / core

  The complete web scraping toolkit for PHP.

  phpweb-scrapingcrawling
  Language:PHP 1322
• mlscraper

  lorey / mlscraper

  🤖 Scrape data from HTML websites automatically by just providing examples

  scrapingcrawlinghtmlmachine-learningextraction-enginescrapercrawlercrawler-python
  Language:Python 1231
• holiday-cn

  NateScarlet / holiday-cn

  📅🇨🇳**法定节假日数据 自动每日抓取国务院公告

  datanatural-language-processingcrawlingholidaychina
  Language:Python 1117

• needleworm / bhban_rpa

  <6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

  rpaautomationcrawlingdesignpythoneducation
  Language:Python 991

• clemfromspace / scrapy-selenium

  Scrapy middleware to handle javascript pages using selenium

  scrapyseleniumcrawling
  Language:Python 892

• elixir-crawly / crawly

  Crawly, a high-level web crawling & scraping framework for Elixir.

  elixirerlangscraperscrapingscraping-websitesextract-dataspidercrawlercrawling
  Language:Elixir 845

• iawia002 / Lulu

  [Unmaintained] A simple and clean video/music/image downloader 👾

  downloadervideopythonpython3crawlerscrapercrawlingscraping
  Language:Python 817

• scrapinghub / scrapyrt

  HTTP API for Scrapy spiders

  crawlercrawlinghacktoberfesthacktoberfest2021pythonscraperscrapytwistedwebcrawlerwebcrawling
  Language:Python 816

• MorvanZhou / easy-scraping-tutorial

  Simple but useful Python web scraping tutorial code.

  beautifulsoupscrapyregexurllibscrapingcrawlercrawlingrequestsasynciodistributed-scraper
  Language:Jupyter Notebook 765

• bluet / proxybroker2

  The New (auto rotate) Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

  proxyproxy-serverproxiesproxy-listproxypoolproxy-checkerproxychainscrawlercrawlinghttp-proxyhttps-proxyanonymityprivacyanonymoussockshacktoberfest
  Language:Python 666

• slotix / dataflowkit

  Extract structured data from web sites. Web sites scraping.

  golanggolang-libraryextract-datachrome-fetcherscraping-websitescrawlingscraperscrapingcdpgoheadless
  Language:Go 645
• AdminHack

  mishakorzik / AdminHack

  today we will hack the admin panel of the site.

  termuxlinuxadmin-hackwebsitewebsitehackingadmin-website-hackwebsite-hackingwebsite-hacking-methodsadmin-panelcpanl-finderdirectory-bruteforcecpanelkali-linuxcrawlingadmin-finderhacking-toolallhackingtoolstermux-hackingtermux-tool
  Language:Shell 637

• essandess / isp-data-pollution

  ISP Data Pollution to Protect Private Browsing History with Obfuscation

  webcrawlingdataobfuscationprivacy-enhancing-technologiesdata-analyticsprivacy
  Language:Python 581

• webrecorder / browsertrix-crawler

  Run a high-fidelity browser-based crawler in a single Docker container

  crawlercrawlingwaczwarcweb-archivingweb-crawlerwebrecorder
  Language:TypeScript 552

• scrapinghub / spidermon

  Scrapy Extension for monitoring spiders execution.

  scrapinghubscrapingmonitoringspiderscrawlingtestingmonitoring-toolhacktoberfest
  Language:Python 511

• zhuyingda / webster

  a reliable high-level web crawling & scraping framework for Node.js.

  scraping-frameworkcrawlercrawlingheadless-chromechromiumspiderautomation-uiautomation-testnodejsnodejs-frameworkjavascriptjavascript-frameworkpuppeteer
  Language:JavaScript 505

• crawljax / crawljax

  Crawljax

  crawlingcrawlerdomdynamictest-generationweb-analysisweb-testingevent-driven-crawlingjavascript
  Language:Java 504

• josephlimtech / linkedin-profile-scraper-api

  🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON.

  puppeteernodejsscraperscrapingscraping-websiteswebsite-scraperjsonlinkedin-profile-scraperlinkedinscraperscrawlercrawlingspiderexpressjslinkedin-scraperlinkedin-scrapinglinkedin-botlinkedin-crawlerprofile-datalinkedin-profile
  Language:TypeScript 482

• Florents-Tselai / WarcDB

  WarcDB: Web crawl data as SQLite databases.

  crawlingsqlitewarccliweb-datadatabaseweb-archiving
  Language:Python 384

• l4rm4nd / LinkedInDumper

  Python 3 script to dump/scrape/extract company employees from LinkedIn API

  osintpython3employeeslinkedinspidercrawlingextractingscraping
  Language:Python 371

• mhmdiaa / second-order

  Second-order subdomain takeover scanner

  securitysecurity-toolswordlistwordlist-generatorpenetration-testingpentestinginfosecreconreconnaissancemappingweb-application-securitypenetration-testing-toolscrawlercrawling
  Language:Go 359