warc

There are 10 repositories under warc topic.

• ArchiveBox

  ArchiveBox / ArchiveBox

  🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

  archiveboxbackupsbookmark-archiverbrowser-bookmarkschromiumdigipresfirefoxheadless-browserinternet-archivingpinboardpocketpythonrssself-hostedsinglefilewarcwayback-machineweb-archivingwgetyoutube-dl
  Language:Python 19720

• internetarchive / heritrix3

  Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

  heritrixjavawarcwebcrawling
  Language:Java 2675
• conifer

  Rhizome-Conifer / conifer

  Collect and revisit web pages.

  webrecorderweb-archivingarchivespywbpythondockerwaybackwarc
  Language:Python 1459

• ArchiveTeam / grab-site

  The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

  archivingcrawlcrawlerspiderwarc
  Language:Python 1257

• webrecorder / replayweb.page

  Serverless replay of web archives directly in the browser

  web-archivingweb-archivereplay-web-pageweb-replaywayback-machinewarcservice-workerwacz
  Language:TypeScript 605
• ipwb

  oduwsdl / ipwb

  InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

  dockeripfsmementomemento-rfcpythonservice-workerwarcwaybackweb-archiving
  Language:Python 589

• webrecorder / browsertrix-crawler

  Run a high-fidelity browser-based crawler in a single Docker container

  crawlercrawlingwaczwarcweb-archivingweb-crawlerwebrecorder
  Language:TypeScript 531

• webrecorder / webrecorder-player

  Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)

  webrecorderwarcpywbelectronweb-archiving
  Language:JavaScript 421

• Florents-Tselai / WarcDB

  WarcDB: Web crawl data as SQLite databases.

  clicrawlingdatabasesqlitewarcweb-archivingweb-data
  Language:Python 383
• wail

  machawk1 / wail

  :whale2: Web Archiving Integration Layer: One-Click User Instigated Preservation

  web-archivingwaybackpythonheritrixguiwarcopenwaybackpyinstaller
  Language:Roff 341

• webrecorder / warcio

  Streaming WARC/ARC library for fast web archive IO

  web-archivesweb-archivingwarcpywbpython
  Language:Python 340
• bitextor

  bitextor / bitextor

  Bitextor generates translation memories from multilingual websites

  document-alignerapertiumdictionariescrawlerwgethunalignsentence-segmentationtokenizerbicleanertmxwarccorpus-toolscorpus-processingcorpus-generatorparallel-corporamachine-translationneural-machine-translationstatistical-machine-translationbitextorbleualign
  Language:Python 278

• commoncrawl / news-crawl

  News crawling with StormCrawler - stores content as WARC

  crawlernewswarcweb-crawlerapache-stormcommon-crawlcommoncrawlstorm-crawler
  Language:Java 248
• warcreate

  machawk1 / warcreate

  Chrome extension to "Create WARC files from any webpage"

  chrome-extensionwarcweb-archiving
  Language:JavaScript 192

• cocrawler / cocrawler

  CoCrawler is a versatile web crawler built using modern tools and concurrency.

  crawlerpython3async-pythonwarcpluggable-modulesscreenshotconcurrencyaiohttpaiohttp-client
  Language:Python 176

• cocrawler / cdx_toolkit

  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

  web-archivingweb-archiveswarccdxcdx-apicommoncrawlpython
  Language:Python 150

• helgeho / ArchiveSpark

  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

  archivesparkspark-frameworksparkweb-archivingwebarchiveinternet-archivewarc
  Language:Scala 140

• crissyfield / troll-a

  Drill into WARC web archives

  command-line-toolcommon-crawlinternet-archivesecuritysecurity-toolswarc
  Language:Go 129

• N0taN3rd / wail

  :whale2: One-Click User Instigated Preservation

  electronweb-archivingwarcbrowser-based-presrevationhigh-fidelity-preservation
  Language:JavaScript 119

• webrecorder / browsertrix-cloud

  Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

  archivingcloudwarcweb-archiveweb-archivingwebrecorderwaczkubernetes
  Language:TypeScript 118

• maxcountryman / warc-parquet

  🗄️ A simple CLI for converting WARC to Parquet.

  crawlingduckdbparquetwarcweb-archiving
  Language:Rust 99

• N0taN3rd / node-warc

  Parse And Create Web ARChive (WARC) files with node.js

  webarchivewebarchivingweb-archiveswarc-fileswarcweb-archivingpupeteerchrome-remote-interface
  Language:JavaScript 90

• CGamesPlay / chronicler

  Offline-first web browser

  electronbrowserwarc
  Language:JavaScript 82

• ArchiveTeam / wget-lua

  Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

  webarchivingwarcwgetluaarchivingcrawlercrawlcrawlingspiderarchiveteamwget-luazstdftpscraperscrapingcrawlersdownloader
  Language:C 81

• mikwielgus / forum-dl

  Scrape posts, threads from forums, news aggregators, mail archives, export to JSONL, mailbox, WARC

  pythonscraperforumdiscoursephpbbsimplemachinesdata-fetchinginternet-archivingwarc
  Language:Python 59

• centic9 / CommonCrawlDocumentDownload

  A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika

  cdx-filescommoncrawlmime-typeswarcjava
  Language:Java 58

• archivesunleashed / warclight

  A Rails engine supporting the discovery of web archives.

  blacklightrubydiscoverywebarchive-discoverysolrrailswebarchiveswarcrails-engine
  Language:Ruby 48

• pirate / internet-archiving-talk

  🎭 An introduction to the Internet Archiving ecosystem, tooling, and some of the ethical dilemmas that the community faces.

  internet-archivingtalksslideshowweb-archivingwgetwarcarchiveboxcensorshipethics
  Language:JavaScript 47
• cdx-summary

  internetarchive / cdx-summary

  Summarize web archive capture index (CDX) files.

  archivecdxcollectionnodejspythonreportstatisticssummarywarcweb-archivewebcomponents
  Language:Python 43

• PromyLOPh / crocoite

  Web archiving using Google Chrome

  warcchrome-browserarchivingdevtools
  Language:Python 42

• chatnoir-eu / chatnoir-resiliparse

  A robust web archive analytics toolkit

  pythonwebwarcbigdatacythoncppextractionwebarchivehtmlparser
  Language:Cython 40

• jedireza / warc

  :gear: A Rust library for reading and writing WARC files

  warcrust-libraryrust
  Language:Rust 39

• harvard-lil / warc-gpt

  WARC + AI - Experimental Retrieval Augmented Generation Pipeline for Web Archive Collections.

  airagwarcwebarchiving
  Language:Python 34

• openzim / warc2zim

  Command line tool to convert a file in the WARC format to a file in the ZIM format

  warczimscraper
  Language:Python 33

• datatogether / warc

  Golang WARC (Web ARChive) Library

  golangarchivingiipcwarcpackage
  Language:Go 29

• datacoon / metawarc

  metawarc: a command-line tool for metadata extraction from files from WARC (Web ARChive)

  warcwarc-fileswebarchivingmetadataosintosint-python
  Language:Python 23