web-crawler

There are 77 repositories under web-crawler topic.

• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
  Language:TypeScript 12067
• crawlab

  crawlab-team / crawlab

  Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

  webcrawlerscrapycrawlabspiders-managementgoscrapyd-uispidercrawlerwebspiderweb-crawlerdockerplatformcrawling-tasks
  Language:Go 10783

• ssssssss-team / spider-flow

  新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

  crawlerjsoupspiderspider-flowweb-crawlerweb-spiderwebcrawlerwebspiderxpath
  Language:Java 9032

• BruceDone / awesome-crawler

  A collection of awesome web crawler,spider in different languages

  awesomecrawlernode-crawlerscraperspiderweb-crawlerweb-scraper
  6066

• apache / nutch

  Apache Nutch is an extensible and scalable web crawler

  apachecrawlinghadoopjavanutchweb-crawler
  Language:Java 2807

• sjdirect / abot

  Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

  c-sharpabotabot-nugetcrawlerweb-crawlerparsingspiderspiderspluggableunit-testingcsharpnetcorenetcore2netcore3netstanetstandard20netstandard21csharp-libraryjavascript-renderercross-platform
  Language:C# 2202

• xianhu / PSpider

  简单易用的Python爬虫框架，QQ交流群：597510560

  crawlerspiderpythonproxiesweb-spidermulti-threadingweb-crawlerpython-spidermultiprocessing
  Language:Python 1817

• DigitalPebble / storm-crawler

  A scalable, mature and versatile web crawler based on Apache Storm

  web-crawlerapache-stormdistributedjavacrawlerstormcrawler
  Language:HTML 847

• MarginaliaSearch / MarginaliaSearch

  Internet search engine for text-oriented websites. Indexing the small, old and weird web.

  alt-searchindexerinternet-searchlanguage-processingno-ai-usedno-cloudsearch-engineself-hostablesmall-webweb-crawler
  Language:HTML 821

• postmodern / spidr

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  crawlerrubyscraperspiderspider-linkswebweb-crawlerweb-scraperweb-scrapingweb-spider
  Language:Ruby 792

• platonai / PulsarRPA

  Automate webpages at scale, scrape web data completely and accurately with high performance, distributed RPA.

  crawlerdata-miningdata-sciencerpascraperscrapingweb-automationweb-crawlerweb-miningweb-scrapingweb-sql
  Language:Kotlin 607

• Algebra-FUN / WeReadScan

  扫描“微信读书”已购图书并下载本地PDF的爬虫

  book-downloaderseleniumweb-crawlerweread
  Language:Python 550

• webrecorder / browsertrix-crawler

  Run a high-fidelity browser-based crawler in a single Docker container

  crawlercrawlingwaczwarcweb-archivingweb-crawlerwebrecorder
  Language:TypeScript 538

• hyunwoongko / kochat

  Opensource Korean chatbot framework

  chatbotdeeplearningdeep-learningkoreankorean-chatbotsentence-classificationsequance-taggingweb-crawler
  Language:Python 443

• VIDA-NYU / ache

  ACHE is a web crawler for domain-specific search.

  web-crawlerfocused-crawlerdomain-specific-searchweb-scrapingweb-spiderweb-searchhacktoberfest
  Language:Java 433

• USCDataScience / sparkler

  Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

  big-datadistributed-systemsinformation-retrievalnutchsearchsearch-enginesolrsparktikaweb-crawler
  Language:Java 409

• brendonboshell / supercrawler

  A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

  crawlerdistributed-crawlerrobotsitemapweb-crawler
  Language:JavaScript 369

• rivermont / spidy

  The simple, easy to use command line web crawler.

  web-crawlerweb-spiderpythonpython3crawlingcrawler
  Language:Python 323

• infinilabs / crawler

  🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

  spidercrawlerlightweightelasticsearchweb-crawlercrawlingweb-spiderweb-scrapingscraping
  Language:Go 299

• crwlrsoft / crawler

  Library for Rapid (Web) Crawler and Scraper Development

  crawlercrawlinghacktoberfestphpscraperscrapingscraping-websitesweb-crawlerweb-crawlingweb-scraperweb-scraping
  Language:PHP 298

• microfisher / Strong-Web-Crawler

  基于C#.NET+PhantomJS+Sellenium的高级网络爬虫程序。可执行Javascript代码、触发各类事件、操纵页面Dom结构。

  web-crawlercrawlerphantomjssellenium
  Language:C# 278

• yields / ant

  A web crawler for Go

  gogolangscraperspiderweb-crawler
  Language:Go 276

• lucasxlu / LagouJob

  Data Analysis & Mining for lagou.com

  lagoudata-analysisweb-crawlermachine-learningdata-miningpython3nlp
  Language:Python 260

• antchfx / antch

  Antch, a fast, powerful and extensible web crawling & scraping framework for Go

  golangweb-crawlercrawlingscrapingframeworkcrawlerweb-spider
  Language:Go 255

• commoncrawl / news-crawl

  News crawling with StormCrawler - stores content as WARC

  crawlernewswarcweb-crawlerapache-stormcommon-crawlcommoncrawlstorm-crawler
  Language:Java 249

• duyet / awesome-web-scraper

  A collection of awesome web scaper, crawler.

  awesomeawesome-listgouttephantomjsphpscrapyslimerjsspiderstorageweb-crawlerweb-scraper
  236
• InfinityCrawler

  TurnerSoftware / InfinityCrawler

  A simple but powerful web crawler library for .NET

  crawlerrobots-txtspiderweb-crawlerweb-crawling
  Language:C# 236

• crawler-commons / crawler-commons

  A set of reusable Java components that implement functionality common to any web crawler

  web-crawlerjavasitemapsrobots-txtopen-sourcelibrary
  Language:Java 228

• s0rg / crawley

  The unix-way web crawler

  clicrawlergogolanggolang-applicationpentestpentest-toolpentestingunix-wayweb-crawlerweb-scrapingweb-spider
  Language:Go 225

• crawlab-team / crawlab-lite

  Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

  crawlabscrapycrawlerscrapy-uiweb-crawlerspidercrawling-tasksplatformcrawler-managementscrapyd-uiscrapyd
  Language:Vue 217

• xiayouran / Musicer

  旨在将网易云、酷狗、QQ、酷我等各音乐平台集于一体

  kugou-musicmusic-playerpythonwangyiyunmusicweb-crawlerweb-spidermusic-download-scriptkuwo-musicmusic-downloaderqq-musicmusic-robot
  Language:Python 216

• Hecate2 / Ignareo-ISML-auto-voter

  Ignareo the Carillon, a web crawler/spider template of ultimate high concurrency built for leprechauns. Carillons as the best web spiders; Long live the golden years of leprechauns! (ISML=international saimoe; 2022 ISML is last ISML)

  web-spiderconcurrencyhttpismldistributedgeventasynciosukasukasukamokaignareotiatchthollyhigh-performancepythonweb-crawlermicroservice
  Language:Python 189

• elliotxx / zhihu-crawler-people

  A simple distributed crawler for zhihu && data analysis

  pythoncrawlerspiderweb-spiderweb-crawlerpython-crawler
  Language:Python 188
• google-news-scraper

  lewisdonovan / google-news-scraper

  Lightweight scraper for Google News

  google-newsgoogle-news-scrapernewsnews-scrapernews-articlesweb-scrapercrawlerweb-crawlernews-crawlergoogle-crawler
  Language:JavaScript 176

• saeeddhqan / evine

  Interactive CLI Web Crawler

  clicrawlerdata-miningfuzzinggogolangosintscraperweb-crawler
  Language:Go 171

• Norconex / crawlers

  Norconex Crawlers (or spiders) are flexible web and filesystem crawlers for collecting, parsing, and manipulating data from the web or filesystem to various data repositories such as search engines.

  search-engineweb-crawlerjavacollector-httpflexiblecrawlercrawlersfilesystem-crawlercollector-fs
  Language:Java 168