web-crawler

There are 118 repositories under web-crawler topic.

• firecrawl

  firecrawl / firecrawl

  🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

  aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
  Language:TypeScript 66741

• ScrapeGraphAI / Scrapegraph-ai

  Python scraper based on AI

  ai-crawlerai-scrapingai-searchautomated-scrapercrawlerdata-extractionlarge-language-modelllmmarkdownragscrapingscraping-pythonweb-crawlerweb-crawlersweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
  Language:Python 21725
• crawlee

  apify / crawlee

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationcrawlercrawlingheadlessheadless-chromejavascriptnodejsnpmplaywrightpuppeteerscraperscrapingtypescriptweb-crawlerweb-crawlingweb-scraping
  Language:TypeScript 20469
• crawlab

  crawlab-team / crawlab

  Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

  crawlabcrawlercrawling-tasksdockergoplatformscrapyscrapyd-uispiderspiders-managementweb-crawlerwebcrawlerwebspider
  Language:Go 12046

• ssssssss-team / spider-flow

  新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

  crawlerjsoupspiderspider-flowweb-crawlerweb-spiderwebcrawlerwebspiderxpath
  Language:Java 11049

• apify / crawlee-python

  Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

  apifyautomationbeautifulsoupcrawlercrawlinghacktoberfestheadlessheadless-chromepipplaywrightpythonscraperscrapingweb-crawlerweb-crawlingweb-scraping
  Language:Python 7125

• BruceDone / awesome-crawler

  A collection of awesome web crawler,spider in different languages

  awesomecrawlernode-crawlerscraperspiderweb-crawlerweb-scraper
  6993
• omniparse

  adithya-s-k / omniparse

  Ingest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworks

  ingestion-apiocromniparserparse-serverparser-libraryvision-transformerweb-crawlerwhisper-api
  Language:Python 6725

• firecrawl / firecrawl-mcp-server

  🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

  batch-processingclaudecontent-extractiondata-collectionfirecrawlfirecrawl-aijavascript-renderingllm-toolsmcpmcp-servermodel-context-protocolsearch-apiweb-crawlerweb-scraping
  Language:JavaScript 4880

• apache / nutch

  Apache Nutch is an extensible and scalable web crawler

  apachecrawlinghadoopjavanutchweb-crawler
  Language:Java 3087

• jasonxtn / Argus

  The Ultimate Information Gathering Toolkit

  cms-detectiondirectory-finderdns-lookupinformation-gatheringosintrecon-toolsreconnaissanceserver-infossl-analitcstxt-recordsvirustotalweb-crawlerwhois-lookuppastebin-monitoringemail-harvester
  Language:Python 2421

• sjdirect / abot

  Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.

  abotabot-nugetc-sharpcrawlercross-platformcsharpcsharp-libraryjavascript-renderernetcorenetcore2netcore3netstanetstandard20netstandard21parsingpluggablespiderspidersunit-testingweb-crawler
  Language:C# 2289

• xianhu / PSpider

  简单易用的Python爬虫框架，QQ交流群：597510560

  crawlerspiderpythonproxiesweb-spidermulti-threadingweb-crawlerpython-spidermultiprocessing
  Language:Python 1837

• MarginaliaSearch / MarginaliaSearch

  Internet search engine for text-oriented websites. Indexing the small, old and weird web.

  search-engineno-cloudsmall-webinternet-searchindexerlanguage-processingweb-crawleralt-searchself-hostablejavaweb-scale
  Language:HTML 1558

• JustinBeckwith / linkinator

  Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.

  link-checkertypescriptbroken-links404broken-link-checkerseo-toolsseodead-linkslink-validatorurl-validatorwebsite-crawlerweb-crawlerhtmlci-cdtestingnodejs
  Language:TypeScript 1105

• oxylabs / ai-crawler-py

  Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.

  aiai-agentsai-crawlerai-studioweb-crawlerai-web-crawlercrawl-agentai-scraping
  1054

• gildas-lormeau / single-file-cli

  CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

  clinodejssingle-fileweb-archivingweb-scraperweb-scrapingarchivingscraping-websitescrawlerweb-crawlerdenodockerfile
  Language:JavaScript 1032

• Algebra-FUN / WeReadScan

  扫描“微信读书”已购图书并下载本地PDF的爬虫

  seleniumwereadweb-crawlerbook-downloader
  Language:Python 976

• apache / stormcrawler

  A scalable, mature and versatile web crawler based on Apache Storm

  web-crawlerapache-stormdistributedjavacrawlerstormcrawler
  Language:Java 946

• webrecorder / browsertrix-crawler

  Run a high-fidelity browser-based web archiving crawler in a single Docker container

  crawlercrawlingwaczwarcweb-archivingweb-crawlerwebrecorder
  Language:TypeScript 910

• postmodern / spidr

  A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.

  spiderrubyspider-linkscrawlerwebscraperweb-scrapingweb-spiderweb-crawlerweb-scraper
  Language:Ruby 827

• scrapfly / scrapfly-scrapers

  Scalable Python web scraping scripts for +40 popular domains

  crawlingpythoncrawlerscrapingweb-scrapingweb-scraping-pythonantibotautomationcaptcha-bypasscrawling-pythondatascrapingproxiespython-scraperscraperscraping-pythonspidertwitter-scraperweb-crawlerwebscraperwebscraping
  Language:Python 744

• cxcscmu / Craw4LLM

  Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

  crawlercrawlinglarge-language-modelsllmpre-trainingpretrainingweb-crawlerweb-crawling
  Language:Python 641

• firecrawl / firecrawl-app-examples

  🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.

  aiai-scrapingdataexampleshtml-to-markdownllmmarkdownragscrapersweb-crawlertemplates
  Language:Jupyter Notebook 593

• devflowinc / firecrawl-simple

  ➖ Stripped down, stable version of firecrawl optimized for self-hosting and ease of contribution. Billing logic and AI features are completely removed. Crawl and convert any website into LLM-ready markdown.

  aiai-scrapingcrawlerdataembeddingshtml-to-markdownllmmarkdownragscraperscrapingsearchweb-crawlerwebscraping
  Language:TypeScript 536

• VIDA-NYU / ache

  ACHE is a web crawler for domain-specific search.

  web-crawlerfocused-crawlerdomain-specific-searchweb-scrapingweb-spiderweb-searchhacktoberfest
  Language:Java 475

• hyunwoongko / kochat

  Opensource Korean chatbot framework

  chatbotdeep-learningdeeplearningkoreankorean-chatbotsentence-classificationsequance-taggingweb-crawler
  Language:Python 457

• lefterisloukas / edgar-crawler

  The only open-source toolkit that can download SEC EDGAR financial reports and extract textual data from specific item sections into nice & clean structured JSON files. Presented at WWW 2025 @ Sydney, Australia (https://dl.acm.org/doi/10.1145/3701716.3715289)

  pythonfinancesecedgaredgar-crawlerbusinessnatural-language-processingnlpdata-miningweb-crawler
  Language:Python 450

• USCDataScience / sparkler

  Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.

  solrweb-crawlersparknutchtikabig-datainformation-retrievalsearch-enginesearchdistributed-systems
  Language:Java 418

• brendonboshell / supercrawler

  A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

  crawlerdistributed-crawlerrobotsitemapweb-crawler
  Language:JavaScript 380

• graphlit / graphlit-mcp-server

  Model Context Protocol (MCP) Server for Graphlit Platform

  claudecontent-extractioncontent-ingestiondata-collectionllm-toolsmcp-servermodel-context-protocolsearch-apiunstructured-dataweb-crawlerweb-scraping
  Language:TypeScript 369
• crawler

  crwlrsoft / crawler

  Library for Rapid (Web) Crawler and Scraper Development

  crawlingphpscraperscrapingscraping-websitesweb-crawlerweb-crawlingweb-scrapinghacktoberfestcrawlerweb-scraper
  Language:PHP 366

• commoncrawl / news-crawl

  News crawling with StormCrawler - stores content as WARC

  crawlernewswarcweb-crawlerapache-stormcommon-crawlcommoncrawlstorm-crawler
  Language:Java 358

• rivermont / spidy

  The simple, easy to use command line web crawler.

  web-crawlerweb-spiderpythonpython3crawlingcrawler
  Language:Python 348
• google-news-scraper

  lewisdonovan / google-news-scraper

  Lightweight scraper for Google News

  google-newsgoogle-news-scrapernewsnews-scrapernews-articlesweb-scrapercrawlerweb-crawlernews-crawlergoogle-crawler
  Language:TypeScript 347

• internetarchive / Zeno

  State-of-the-art web crawler 🔱

  web-crawlerzenoarchiving
  Language:Go 343