🇺🇸 data.civdocs.us
This repository contains the necessary materials for scraping and compiling data from civdocs.us sources into JSON data and then building the remote SQL database.
src/
compiledJson/ // compiled JSON files with merged data from scraped JSON
documents.json
paragraphs.json
sections.json
scrapedJson/ // individual JSON files from scraping, stored for repeated use
...
scripts/ // individual scripts for scraping documents into JSON files for compilation
...
compile.js // utility logic yielding compiled JSON from scraped JSON
db-builder.js // contains logic for building remote SQL database from compiled JSON
index.js // top-level logic
exec.sh // main script
Usage
$ ./exec.sh [ -s | -c | -p ]
Command | Info |
---|---|
-s SCRAPE | Execute scrape scripts currently uncommented in index.js DOCUMENT_NAMES |
-c COMPILE | Execute compilation from scrapedJson |
-p PRODUCTION | Build production database (defaults to local credentials) |
Note: this program assumes src/.env
src/.env-local
, and src/.env-production
files containing the required database credentials.