Data Together

research

📚 A compilation of research relevant to Data Together's efforts tackling the general problem of data resilience & interactivity

datatogether

:checkered_flag: Start here! Discussion for Data Together: Building a better future for data

reading_datatogether

📚 Monthly reading group for Data Together

warc

Golang WARC (Web ARChive) Library

sentry

Parallelized web crawler written in Golang

webapp

Web application to allow users to add content metadata about crawled resources

website

A static-generated website to introduce the Data Together project, built with Hugo.

xmp

Golang package for parsing Extensible Metadata Platform (XMP) documents

dt

command line tool for storing data together on the distributed web

roadmap

Coordinating technical work & roadmapping additional services

archivertools

Python package for scraping websites into the Data Together pipeline via morph.io

core

Core Archive Model Definitions

cdxj

Golang package implementing the CDXJ file format used by OpenWayback 3.0.0+ to index web archive contents

extract_href

Command line tool for extracting urls from a HTML web page using a jquery-style selector

ffi

Golang package for making sensible guesses about file formats from Url strings

linked_data

Golang package for working with linked data structures, an implementation of the W3C Data Catalog Vocabulary (DCAT)

patchbay

Websockets-based API backend for our react and redux webapp.

pdf

Library that extracts xmp metadata from a PDF document.

sql_datastore

Experimental Golang implementation of the ipfs datastore interface for sql databases.

artwork

Open-licensed artwork related to Data Together

sql_util

Golang package that provides utils for working with dotsql and postgres

archive

golang package for creating/working with warc & cdxj archives

content

Service for serving archived content stored on amazon S3.

coverage

Project for visualizing the status of digital data archiving efforts across various data repositories

resources

extract urls of dependant files for displaying a web page

task_mgmt

Service for managing & executing archiving tasks written in Golang

api

Serves the Data Together JSON API

learning

Learning materials for Data Together

OS_deployment

rewrite

modify the contents of web-related content types for archival purposes