USDR: Grants One Stop Tool (GOST)

GOST is a platform that hosts tools to enable state and local government officials to more easily apply for and report on their federal grants. Feel free to make a contribution, or if you'd like to volunteer with us in a more in-depth manner you can sign up at usdigitalresponse.org/volunteer.

This application is currently hosted at grants.usdigitalresponse.org. Changes made to the main branch will be reflected immediately on this site.

Project structure

├── packages                                    # Yarn workspace
│   ├── client                                  # Vue App
│   └── server                                  # Node server
│       ├── migrations                          # db migrations
│       └── seeds                               # db seeds
│       └── knexfile                            # knex configuration

Each folder inside packages/ is considered a workspace. To see a list of all worskpaces, run the following

yarn workspaces info

Setup

These steps are for an install on a Mac. The Windows instructions are here.

Instructions for using Docker to manage development environments can be found here.

1). Ensure using the correct version of NODE Version

First, check the .nvmrc file to make sure you have the correct version of Node.js installed. If you are using Nodenv or NVM, it should pick up on the correct version.

To setup your workspace run the following commands at the root of the project

1.1). Setup nvm

Follow the install directions at https://github.com/nvm-sh/nvm#install--update-script

Make sure to use new terminals after completing install

> nvm install v16.14.0
> nvm use v16.14.0

2). Install dependencies

The scripts will install yarn and download npm dependencies for all yarn workspaces.

> cd usdr-gost/
> npm i yarn@^1.22.4 -g
> yarn run setup

3). Create database(s)

Install postgres DB. I personally used https://postgresapp.com/

psql -h localhost -p 5432
CREATE DATABASE usdr_grants;
CREATE DATABASE usdr_grants_test;

4). Setup ENVs

Copy packages/client & packages/server .env.example to .env

> cp packages/client/.env.example packages/client/.env
> cp packages/server/.env.example packages/server/.env

Then export the environment variables

> cd packages/client && export $(cat .env)
> cd packages/server && export $(cat .env)

Note: In order to login, the server must be able to send email. Set the relevant environment variables under # Email Server: in .env to credentials for a personal email account (e.g. for Gmail, see (4.1)here).

4.1). Setup Gmail

Visit: https://myaccount.google.com/apppasswords and set up an "App Password" (see screenshot below). Note: Select "Mac" even if you're not using a Mac.

In packages/server/.env, set NODEMAILER_EMAIL to your email/gmail and set your NODEMAILER_EMAIL_PW to the new generated PW.

NOTE: In order to enable App Password MUST turn on 2FA for gmail.

If running into Error: Invalid login: 535-5.7.8 Username and Password not accepted. then "Allow Less Secure Apps" - source

NOTE: Much more reliable and preferable to go the App Password route vs Less Secure Apps.

5). Run DB Migrations & Seed

In server workspace, run migrations:

NOTE: In server/seeds/dev/01_main.js, update the adminList by replacing CHANGEME@GMAIL.COM with your email to be able to login to the system. Use lower-case email address. Then run seeds:

> cd packages/server
> export $(cat .env) #delete all comment lines in .env file
> yarn db:migrate
> yarn db:seed

6). Run Client (Terminal 1)

Now you should be able to serve the frontend.

Ensure you are using the correct node version and are in the project root directory

> yarn start:client

6.1). Run Server (Terminal 2)

Now you should be able to serve the backend.

NOTE: update WEBSITE_DOMAIN in .env to your client endpoint from Step 6 else When you get the login email link, change the redirected path from localhost:8000/api/sessions/... to your client_url e.g localhost:8080/api/sessions/

Ensure you are using the correct node version and are in the project root directory

> yarn start:server

NOTE: if error references AWS (see screenshot below) then run > unset AWS_ACCESS_KEY_ID. The application will try to use AWS Simple Email Service (SES) if AWS_ACCESS_KEY_ID is found as an env var.

7). Visit client_url/login (e.g http://localhost:8080/#/login) and login w/ user set in Step 5.

NOTE: if you only see a blank screen then ensure you've set up the packages/client/.env

NOTE: if you get Error: Invalid login: 534-5.7.9 Application-specific password required. then you'll need to set an App Password (https://myaccount.google.com/apppasswords) (See Step 4)

Additional Info:

Yarn Workspaces

Workspaces optimizes our repo by hoisting all of our separate node_modules/ to the root level meaning that a single yarn install command installs the NPM modules for all services

https://classic.yarnpkg.com/en/docs/workspaces/

Lerna

Lerna helps us manage our packages, publish them, and keeps track of the dependencies between them. For example, it is used to run linting, deploy, and test scripts for each package from the root of the project.

Example usages

npx lerna bootstrap - recursive yarn install

npx lerna run --scope server --stream start

Adding dependencies

To add dependencies to one workspace

npx lerna add {modules} --scope="{workspace_name}" - where workspace_name is the name in package.json. Run yarn workspaces info to see a list of packages

Example `npx lerna add uuid --scope="server" --dev`

Or you can run yarn inside the workspace

yarn add {modules}

Add dependencies to multiple packages

npx lerna add {modules}

NOTE: yarn complains about incompatibility of some node modules with our node version. When using yarn, pass --ignore-engines when doing yarn add/remove. I have not been able to pass this argument when running npx lerna add... After running lerna do a yarn run bootstrap at the root of the project to get your dependency correctly installed.

Linting

VSCode

Install Vue plugin: jcbuisson.vue, Vue.volar

Install Vue Dev Tools: https://chrome.google.com/webstore/detail/vuejs-devtools/nhdogjmejiglipccpnnnanhbledajbpd?hl=en

Install the eslint plugin https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint

After that you should be able to see eslint prompts in js files

For linting on auto save:

Go to VSCode settings
- Shift + Command + P
- Search for settings
- Select "Open Settings (JSON)"
Paste the following snippet

"editor.formatOnSaveMode": "modifications",
  "editor.formatOnSave": true,
  "editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
}

Note: Before Pasting check if there are any conflicting settings regarding esling or formatOnSave

Sharing my complete VSCode Setting

{
  "terminal.integrated.shell.osx": "/bin/zsh",
  "azureFunctions.showProjectWarning": false,
  "window.zoomLevel": 0,
  "diffEditor.ignoreTrimWhitespace": false,
  "azureFunctions.showCoreToolsWarning": false,
  "editor.columnSelection": false,
  "editor.find.cursorMoveOnType": true,
  "editor.formatOnSaveMode": "modifications",
  "editor.formatOnSave": true,
  "editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
  }
}

IntelliJ

After installing depedencies, IntelliJ should start using eslint automatically:

By default, IntelliJ IDEA marks the detected errors and warnings based on the severity levels from the ESLint configuration https://www.jetbrains.com/help/idea/eslint.html#ws_js_linters_eslint_install

Testing

Server

> cd packages/server
> yarn test && yarn test:apis
  ...

OR

> yarn test:db
  ...
> yarn test:apis
  ...
> yarn test:email
  ...

Client

> yarn test
...
> yarn test:e2e

Deployment

Render

Create web service

Create database

Update web service environment variables

NOTE: Don't set NODE_ENV=production else NPM dev deps will not be installed and prod deployments will fail (source)

POSTGRES_URL=<POSTGRE_CONNECTION_STRING> # Render Internal connection string ie postgres://cares_opportunity_user:<pass>@<domain>/cares_opportunity_1e53

COOKIE_SECRET=<RANDOM_ALPHANUMERIC_SECRET>

WEBSITE_DOMAIN=<WEB_SERVICE_URL> # Render web service url ie. https://cares-grant-opportunities-qi8i.onrender.com

NODE_ENV=development or production or test

NOTIFICATIONS_EMAIL="grants-identification@usdigitalresponse.org"
SES_REGION="us-east-1"
AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID>
AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>

ENABLE_GRANTS_SCRAPER=true
GRANTS_SCRAPER_DATE_RANGE=7 # date range of grants that will be scraped
GRANTS_SCRAPER_DELAY=1000 # delay in milliseconds for scraper

NODE_OPTIONS=--max_old_space_size=1024 # increase node max memory, had problems with node not using all of renders server memory. This will depend on the plan

DB Migrations

Get the postgres external connection string from render. Set it as an environment variable

export POSTGRES_URL="postgres://user:{pass}@{domain}/{db}?ssl=true"

NOTE: must add ?ssl=true

Change directory to packages/server
Update seeds/dev files accordingly
- seeds/dev/ref/agencies.js - list of agencies to be created. You can update this with the state provided agency. Note: We add a special USDR agency for our accounts in the system
- seeds/dev/index.js - Update the admin list variable accordingly
Run the following commands

npx knex migrate:latest
npx knex seed:run

After that you should be able to access the site and login with the users set in the migration.

Debugging

Sometimes lerna seems to hang w/ no output. By adding --stream you can get more information about the error.

Code of Conduct

This repository falls under U.S. Digital Response’s Code of Conduct, and we will hold all participants in issues, pull requests, discussions, and other spaces related to this project to that Code of Conduct. Please see CODE_OF_CONDUCT.md for the full code.

Contributing

This project wouldn’t exist without the hard work of many people. Thanks to the following for all their contributions! Please see CONTRIBUTING.md to find out how you can help.

Lead Maintainer: Rafael Pol (@Rapol)

License & Copyright

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at:

LICENSE in this repository or http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

albertlieyingadrian / usdr-gost