web-archiving

There are 51 repositories under web-archiving topic.

• ArchiveBox

  ArchiveBox / ArchiveBox

  🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

  archiveboxbackupsbookmark-archiverbrowser-bookmarkschromiumdigipresfirefoxheadless-browserinternet-archivingpinboardpocketpythonrssself-hostedsinglefilewarcwayback-machineweb-archivingwgetyoutube-dl
  Language:Python 25490

• webrecorder / pywb

  Core Python Web Archiving Toolkit for replay and recording of web archives

  pythonpywbwaybackweb-archivesweb-archiving
  Language:JavaScript 1576
• conifer

  Rhizome-Conifer / conifer

  Collect and revisit web pages.

  webrecorderweb-archivingarchivespywbpythondockerwaybackwarc
  Language:Python 1525

• webrecorder / archiveweb.page

  A High-Fidelity Web Archiving Extension for Chrome and Chromium based browsers!

  chromiumextensionweb-archivingwebrecorderarchivingwaczbrowser-extensionwarc
  Language:TypeScript 1086

• gildas-lormeau / single-file-cli

  CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)

  clinodejssingle-fileweb-archivingweb-scraperweb-scrapingarchivingscraping-websitescrawlerweb-crawlerdenodockerfile
  Language:JavaScript 1034

• bellingcat / auto-archiver

  Automatically archive links to videos, images, and social media content from Google Sheets (and more).

  archivedockeropen-source-researchpythonscrapingserviceweb-archiving
  Language:Python 976

• webrecorder / browsertrix-crawler

  Run a high-fidelity browser-based web archiving crawler in a single Docker container

  crawlercrawlingwaczwarcweb-archivingweb-crawlerwebrecorder
  Language:TypeScript 910

• Ray-D-Song / web-archive

  Free web archiving and sharing service based on Cloudflare. 跑在 Cloudflare 上的免费网页归档和分享工具。

  cloudflarecloudflare-pagesd1freehonoself-hostedserverlessweb-archiveweb-archiving
  Language:TypeScript 892

• webrecorder / replayweb.page

  Serverless replay of web archives directly in the browser

  web-archivingweb-archivereplay-web-pageweb-replaywayback-machinewarcservice-workerwacz
  Language:TypeScript 854
• ipwb

  oduwsdl / ipwb

  InterPlanetary Wayback: A distributed and persistent archive replay system using IPFS

  ipfswarcwaybackweb-archivingpythonservice-workermementomemento-rfcdocker
  Language:Python 647
• waybackpy

  akamhy / waybackpy

  Wayback Machine API interface & a command-line tool

  internet-archivewayback-machineinternet-archivingarchive-webpagearchive-webpageswayback-machine-apicdx-apiwayback-machine-pythonsavepagenowweb-archivingwebarchivingosint
  Language:Python 552

• harvard-lil / perma

  Indelible links

  web-archivinglibraries
  Language:JavaScript 488

• eclaire-labs / eclaire

  Local-first, open-source AI assistant for your data. Unify tasks, notes, docs, photos, and bookmarks. Private, self-hosted, and extensible via APIs.

  aiai-assistantautomationbookmark-managerbookmarksdata-extractiondocument-processingllmlocal-firstnote-takingocron-device-aiopen-sourcepersonal-knowledge-managementprivacyrest-apiself-hostedtask-managementweb-archiving
  Language:TypeScript 484

• rahiel / archiveror

  Archiveror will help you preserve the webpages you love. 💾

  archivingwebextensionlinkrotmhtmlbrowser-extensionweb-archivingfirefox-extensionchrome-extensionjavascriptbookmark
  Language:JavaScript 448

• webrecorder / webrecorder-player

  Webrecorder Player for Desktop (OSX/Windows/Linux). (Built with Electron + Webrecorder)

  webrecorderwarcpywbelectronweb-archiving
  Language:JavaScript 446

• webrecorder / warcio

  Streaming WARC/ARC library for fast web archive IO

  web-archivesweb-archivingwarcpywbpython
  Language:Python 438

• oduwsdl / archivenow

  A Tool To Push Web Resources Into Web Archives

  web-archivinginternet-archive
  Language:Python 423

• Florents-Tselai / WarcDB

  WarcDB: Web crawl data as SQLite databases.

  crawlingsqlitewarccliweb-datadatabaseweb-archiving
  Language:Python 406
• wail

  machawk1 / wail

  :whale2: Web Archiving Integration Layer: One-Click User Instigated Preservation

  web-archivingwaybackpythonheritrixguiwarcopenwaybackpyinstaller
  Language:Roff 381

• ArchiveBox / archivebox-browser-extension

  Official ArchiveBox browser extension: automatically/manually preserve your browsing history using ArchiveBox.

  archiveboxarchivingbrowser-extensionchrome-extensiondigipresdigital-preservationfirefox-extensioninternet-archivingsvelteweb-archiving
  Language:JavaScript 374

• webrecorder / browsertrix

  Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

  archivingcloudwarcweb-archiveweb-archivingwebrecorderwaczkubernetes
  Language:TypeScript 350
• warcreate

  machawk1 / warcreate

  Chrome extension to "Create WARC files from any webpage"

  chrome-extensionwarcweb-archiving
  Language:JavaScript 224

• cocrawler / cdx_toolkit

  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

  web-archivingweb-archiveswarccdxcdx-apicommoncrawlpython
  Language:Python 186

• ArchiveBox / electron-archivebox

  Desktop Electron app for ArchiveBox internet archiver. (ALPHA: not ready for general use)

  archiveboxelectrondockerinternet-archivingdigipresweb-archivingdesktopdesktop-electronmacoswindowslinuxgui
  Language:JavaScript 178

• gwu-libraries / sfm-ui

  Social Feed Manager user interface application.

  code4libsocial-feed-managerweb-archivingsocial-media
  Language:Python 156

• helgeho / ArchiveSpark

  An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.

  archivesparkspark-frameworksparkweb-archivingwebarchiveinternet-archivewarc
  Language:Scala 153

• programminghistorian / ph-submissions

  The repository and website hosting the peer review process for new Programming Historian lessons

  apidata-managementdhdigital-historydigital-humanitiesdistant-readinglinked-open-datamappingmulti-lingualnetwork-analysisopen-educational-resourcesopen-sourcepedagogyprogramming-historianpythonweb-archivingweb-scrapingr-studio
  Language:HTML 147

• N0taN3rd / wail

  :whale2: One-Click User Instigated Preservation

  electronweb-archivingwarcbrowser-based-presrevationhigh-fidelity-preservation
  Language:JavaScript 129

• internetarchive / fatcat

  Perpetual Access To The Scholarly Record

  rustweb-archivingscholarly-communicationdigital-librarypythonopen-accesspostgresql
  Language:Python 119

• maxcountryman / warc-parquet

  🗄️ A simple CLI for converting WARC to Parquet.

  crawlingduckdbparquetwarcweb-archiving
  Language:Rust 113

• N0taN3rd / node-warc

  Parse And Create Web ARChive (WARC) files with node.js

  webarchivewebarchivingweb-archiveswarc-fileswarcweb-archivingpupeteerchrome-remote-interface
  Language:JavaScript 102
• hoardy-web

  Own-Data-Privateer / hoardy-web

  Passively capture, archive, and hoard your web browsing history, including the contents of the pages you visit, for later offline viewing, replay, mirroring, data scraping, and/or indexing. Your own personal private Wayback Machine that can also archive HTTP POST requests and responses, as well as most other HTTP-level data.

  archivebackupsinternetinternet-archivingself-hostedwayback-machineweb-archivingweb-archivearchiverarchivingweb-browsingwebsite-archiveauto-saveoffline-readingsnapshotbrowser-extensioncli
  Language:Python 100

• oduwsdl / warrick

  Recover lost websites from the Web Infrastructure

  web-archivingmemento-rfcmementorecovery
  Language:HTML 89

• xarantolus / Collect

  A server to collect & archive websites that also supports video downloads

  self-hostedwebinterfacearchivewebsite-archivevideo-downloaderwebsite-scraperweb-archiving
  Language:TypeScript 86
• website-downloader

  PKHarsimran / website-downloader

  Website-downloader is a powerful and versatile Python script designed to download entire websites along with all their assets. This tool allows you to create a local copy of a website, including HTML pages, images, CSS, JavaScript files, and other resources. It is ideal for web archiving, offline browsing, and web development.

  automationbeautifulsoupdata-mininghtmlinternet-toolsoffline-browsingopen-sourcepythonpython-scriptsrequestsweb-archivingweb-scrapingwebsite-clonerwebsite-downloaderwget
  Language:Python 78
• MemGator

  oduwsdl / MemGator

  A Memento Aggregator CLI and Server in Go

  web-archivingmementotimemapmemento-rfc
  Language:Go 70