agrc / deq-enviro

DEQ Environmental Data Viewer

deq-enviro

DEQ Environmental Data Viewer

Generic application for searching, viewing and downloading DEQ GIS data and related tables.

Requirements

Scope of Work

Original Mockup

Config Spreadsheet

Staging Config Spreadsheet

Master Plan

Stage - test.mapserv.utah.gov/deqenviro/

Production - enviro.deq.utah.gov

Query Layer Data Requirements

In order for a dataset to be used as a query layer within the application, it must satisfy all of the following requirements:

• Accessible from our server (usually requires some firewall requests)
• For tables that need to be translated into points:
  • Coordinates stored in either LATITUDE & LONGITUDE or EASTING & NORTHING fields.
• A row in the configuration spreadsheet that defines mappings for the five main fields (ID, NAME, ADDRESS, CITY, & TYPE). If there is no mapping for a specific field a value of n/a should be used in the config.
• A unique id field. If the field is not automatically recognized by ArcGIS Pro, then the OID Field column in the config spreadsheet can be used to define it.

Testing

Unit tests are run via intern.

To run tests:

1. grunt
2. npx intern serveOnly
3. Go to: http://localhost:9000/__intern/

Nightly Script

Runs nightly on test and prod servers.

Builds DEQEnviro.json which the web app uses to configure itself. Part of building this json file is getting all of the map service layer indices so it needs to be rerun manually after adding, removing or reordering any of the map service layers.

Make sure that you have a latest version of pip before pip install -r requirements.txt.

This script requires settings/oauth2key.json. Check out the oauth2 gspread docs to learn how to generate it. Make sure to grant read permission to the email address in client_email to the config spreadsheets.

Updates related data in SGID10. Reads sources from the config spreadsheet.

Data Schema Changes

Most updates are taken care of via the config spreadsheet and updating the schema of data.

Adding a new field

1. Add the field to the "Identify Attributes" column in the config spreadsheet. This will make it show up in the identify pane in the app.
2. Add the field to the data in SGID10 (prod & staging).
3. Add the field to the data in staging/deqquerylayers.gdb.
4. Delete the associated dataset in staging/sgid_stage.gdb if it's there.

Adding a new query layer

1. Add the new row in the config spreadsheet
2. Run forklift pallet.
3. Add new layer to maps/MapService.mxd or maps/Secure.mxd and republish.
4. Manually run build_json.py to get the layer number from the map service of the newly added layer.

Deploy Steps

1. Set up and install ArcGisServerPermissionsProxy.

   • Import RavenDB and web.config from previous server.
   • Use configs/permissionproxy.json to create a new application
   • May need to set the AccessRules.EndDate to 5000000000000 for the initial user so that you can log in successfully the first time.

2. Create a deqnightly user in ArcGIS Server and assign it to the deq_admin role.

   • Fill in the credentials in the settings for the pallet.

3. Build and deploy (using web deploy) api/Search.Api/Search.Api.sln to the web server (<root>/deqenviro/api).

   • Register SOE from the same project with ArcGIS Server.

4. Publish maps/MapService.mxd and maps/Secure.mxd to a DEQEnviro folder in ArcGIS Server.

   • Secure should be locked down to just the deq_admin and deq_water roles.
   • Add the SOE to each of these services:
     • sitename: NAME
     • maxrecords: 25000
     • returnFields: ID,NAME,ADDRESS,CITY,TYPE,OBJECTID,ENVIROAPPLABEL,ENVIROAPPSYMBOL
     • facilityust: FACILITYUST
     • programid: ID

5. Publish ExportWebMap service to the DEQEnviro folder using maps/PrintTemplates/Portrait.mxd as the default template.

   • Make sure that the server can resolve the domain name that the app is hosted on (e.g. test.mapserv.utah.gov). If it can't you will need to edit the hosts file. This is required for the ExportWebMap service.
   • synchronous

6. Run and publish scripts/download/DeqEnviro.pyt/download as Toolbox/download in the same DEQEnviro folder.

   • pip install xlsxwriter on the hosting server fom the python installation that ArcGIS Server uses (x64).
   • pip install xlsxwriter on the publishing server for the python installation that ArcGIS Desktop uses (x32).
   • You can use these inputs as a test:

   {"BFNONTARGETED":["Pre5","Pre9","Pre8","Pre4","Pre7","Pre10","Pre12","Pre13","Pre14","Pre11","13","14"],"BFTARGETED":["2A","3","5","6","4","8","9","10","11","12","1","2","7"]}
   shp
   C:\forklift\data\production\deqquerylayers.gdb
   

7. Add repo to forklift.

   1. Copy scripts/nightly/databases & scripts/nightly/settings/__init__.py from old server.
   2. Download and install the latest oracle instant client.
   3. From within the forklift environment: pip install -r .\scripts\nightly\requirements.txt

8. Build and deploy the application by running grunt build-prod && grunt deploy-prod.

   • You will need to run scripts/nightly/build_json.py to generate DEQEnviro.json before you can load the application for the first time.