e pluribus unum
Using Python to bring transparency to the financial disclosures of Senators. Offering insights from capturing, transforming, and reporting on available public data.
A detailed walkthrough each step of creating a robust data gathering and analysis application.
Python 3.7 and a virtual environment allows us to contain dependencies with the venv module. Required packages are split between production and development conerns (linting, testing, documentation).
$ python --version
Python 3.7.3
$ python -m venv env
# Create a virtual environment within ./env/
$ source env/scripts/activate
# Activate the virtual environment. You will now see
# "(env)" appearing in front of your terminal.
$ pip install -r devrequirements.txt
# Installing packages from devrequirements.txt.
$ mkdir data
# Create the folder for the SQLite db file - This is by default you can change the location in the Storage module
# Now your editor will recognize the virtual environment.
$ python main.py
$ deactivate
# Exits the virtual environment once you are done.United States Senitors are required to file Financial Disclosure Reports detailing assets, liabilities, travel, agreements, and transactions for both themselves and their spouse. Access to this data is governed by the Ethics in Government Act of 1978 stating:
- It shall be unlawful for any person to obtain or use a report:
1. for any unlawful purpose;
2. for any commercial purpose, other than by news and communications media for dissemination to the general public;
3. for determining or establishing the credit rating of any individual; or
4. for use, directly or indirectly, in the solicitation of money for any political, charitable, or other purpose.
- The Attorney General may bring a civil action against any person who obtains or uses a report for any purpose prohibited in paragraph (1) of this subsection. The court in which such action is brought may assess against such person a penalty in any amount not to exceed $10,000. Such remedy shall be in addition to any other remedy available under statutory or common law.
Reports from 2012 to present are available for search at the Senate's Electronic Financial Disclosure. Document appear in html for those submitted electronically and as images for those submitted on paper.
- Fetch Electronic Filing Links
- Store Document Links
- Parse Document Links into Data
- Relational Storage
- Data Integration
- Fetch Electronic Filing Document
- Store
- Parse
- Storage
- Data Transformation
- Visualization
- Robust Fetching (Rate Limiting, Pagination, Data Discovery, Error Handling)