Transparency Toolkit's repositories
LookingGlass
Intuitive and configurable search interface for document archives.
ICWATCH-Data
Resume data and scripts for managing it
TransparencyToolkit
Main repository for Transparency Toolkit
DocManager
Universal backend for indexing, storing, and querying documents.
generalscraper
Scrapes all pages on any site you specify for keywords.
UtilityScripts
Scripts for managing scrapers
LinkedinCrawler
Crawls public LinkedIn profiles
Surveillance-Research-Data
Raw data and scripts for Surveillance Research Archive
TwitterCrawler
A crawler for Twitter
DesignAssets
A collection of branding, interfaces, and other visual resources!
CrawlerManager
API for calling crawlers
TSJobCrawler
Collects listings for jobs that require security clearance.
ArchiveAdministrator
Archive administration system. Handles archive creation and user authentication.
IndexServer
Receives, decrypts, and verifies data, then indexes with DocManager
catalyst_test_scripts
Test scripts for all Catalyst methods.
dataspec-TsjobCrawl
Dataspec for cleared job listings
dataspec-TwitterCrawl
LookingGlass dataspec for tweets
DocIntegrityCheck
Methods for encrypting and verifying documents. Utility gem for document processing pipeline.
DocSuggestions
Backend for processing document suggestions from LookingGlass
HarvesterReporter
Incremental crawler result reporting for Transparency Toolkit
tt-ansible
Ansible roles for deployment. In development, expect problems.