jaredgorski / data.civdocs.us

Data utilities for api.civdocs.us, an open API for founding documents of the United States of America.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

🇺🇸 data.civdocs.us

This repository contains the necessary materials for scraping and compiling data from civdocs.us sources into JSON data and then building the remote SQL database.

src/
  compiledJson/           // compiled JSON files with merged data from scraped JSON
    documents.json
    paragraphs.json
    sections.json
  scrapedJson/            // individual JSON files from scraping, stored for repeated use
    ...
  scripts/                // individual scripts for scraping documents into JSON files for compilation
    ...
  compile.js              // utility logic yielding compiled JSON from scraped JSON
  db-builder.js           // contains logic for building remote SQL database from compiled JSON
  index.js                // top-level logic
exec.sh                   // main script

Usage

$ ./exec.sh [ -s | -c | -p ]
Command Info
-s SCRAPE Execute scrape scripts currently uncommented in index.js DOCUMENT_NAMES
-c COMPILE Execute compilation from scrapedJson
-p PRODUCTION Build production database (defaults to local credentials)

Note: this program assumes src/.env src/.env-local, and src/.env-production files containing the required database credentials.

Repo links

About

Data utilities for api.civdocs.us, an open API for founding documents of the United States of America.


Languages

Language:JavaScript 96.1%Language:Shell 3.9%