Transparency Toolkit

LookingGlass

Intuitive and configurable search interface for document archives.

ICWATCH-Data

Resume data and scripts for managing it

NSA-Data

NSA documents in machine readable form

Harvester

Web crawling and document processing through a usable interface.

TransparencyToolkit

Main repository for Transparency Toolkit

DocManager

Universal backend for indexing, storing, and querying documents.

generalscraper

Scrapes all pages on any site you specify for keywords.

UtilityScripts

Scripts for managing scrapers

LinkedinCrawler

Crawls public LinkedIn profiles

Catalyst

Text mining framework.

Surveillance-Research-Data

Raw data and scripts for Surveillance Research Archive

TwitterCrawler

A crawler for Twitter

DesignAssets

A collection of branding, interfaces, and other visual resources!

CrawlerManager

API for calling crawlers

ParseFile

OCRs document and extracts metadata

TSJobCrawler

Collects listings for jobs that require security clearance.

DirCrawl

Runs block of code on every file in directory

ArchiveAdministrator

Archive administration system. Handles archive creation and user authentication.

DocUpload

Upload application for documents in archiving service.

OCRServer

OCR server for hosted archiving service

Test-Data

Test data for Transparency Toolkit development

IndexServer

Receives, decrypts, and verifies data, then indexes with DocManager

catalyst_test_scripts

Test scripts for all Catalyst methods.

dataspec-TsjobCrawl

Dataspec for cleared job listings

dataspec-TwitterCrawl

LookingGlass dataspec for tweets

DocIntegrityCheck

Methods for encrypting and verifying documents. Utility gem for document processing pipeline.

DocSuggestions

Backend for processing document suggestions from LookingGlass

gj-docs

HarvesterReporter

Incremental crawler result reporting for Transparency Toolkit

tt-ansible

Ansible roles for deployment. In development, expect problems.