Replicate the emodnet_web_api schema in the mpa_web_api database
LynnHeeley opened this issue · comments
changes will be completed by 27th. schedule 28/29th
Helen would like to try and replicate the mapper for MPA this Thursday (28/4/19). So Simon could you work with Helen and Paul in order to help them:
- Replicate the API db
- Replicate the API and point at new DB
- Replicate the mapper files to the web server
- Ensure that everything is joined up and talking to correct geoserver etc with CORs enabled
Can someone clarify for me please?
Is the MPA mapper and backend database running on the same hosts we have already or will they be on different servers?
If they are on different servers then can we have the same access from our staging server? Presumably if you are just copying the existing VM that hosts the web-api and web-mapper then the SSH keys should already be set up to allow us access (I will need to know the new hostname).
We will also need to know about the database login credentials so we can configure the appsettings.json file for you that the web-api uses.
If it's the same server then we will need to add more configuration to Apache so as to proxy to a different web-api instance and mapper instance.
MPA mapper website is on a different server I believe
It will use a different API database - but on the same copy of postgres
I imagine that needs a separate API installation - which I guess will be on the same server as the existing API
I think at this point we should be stepping back from doing the actual deployment (and therefore not needing access) but rather assisting JNCC staff to do their own deployment. They may in the future want any number of mapper instances setup so they need to be confident doing it themselves.
@SimonAnnetts @JamesPe The back-end MPA api database is on the same host as the Emodnet one, it just exists as a separate database. The staging MPA mapper website will be a different server. @paul-gilbertson can you comment to confirm and provide the hostname etc as required for the staging version of the MPA mapper?
See also #41
Given the new hostname for the MPA mapper/web-api server, I could easily deploy the current mapper and api to the server. It would just be a small tweak to the deploy script we use.
On first run on the new host, the web-api would auto generate the schema in the database (assuming the database is setup with right permissions and I have been given the database name, the web-api's username and password for db access). The generated schema may need to have permissions tweaked on it (like we had to do when running migrations on the schema with the existing database #60 ).
Alternatively, as the new MPA host will probably be a clone, things would almost work out of the box. The appsettings.json file in /home/esdm/dotnet-published/jncc-web-api would need to be edited to provide the new connection parameters for the database.
The advantage of us having access to push the code to this MPA host as well as the JNCC host would mean that we could still push changes for bug fixes and enhancements, without further work required by JNCC/MPA staff to keep the two in sync. These two servers would become your reference servers.
We would however not want to do this for any more mappers that you want to deploy! These could be just clones of these reference two, and you could just rsync/copy the /home/esdm/dotnet-published and /home/esdm/web-published folders if ever the code changes.
@SimonAnnetts A set of MPA credentials were passed to you with the EMODnet ones. The mpa_web_api database is ready for you to use. Can you try deploying from your deploy script?
Hi @paul-gilbertson
I have managed to deploy the mapper and the web-api to the MPA server, however systemd has not started the web-api (probably gave up because it wasn't there until now). Can you check that the service is enabled?
Also I can't find the database credentials I need for the MPA web-api !
@paul-gilbertson was there a specific user account set up to allow exegesis access to the mpa_web_api database? or did you just want to provide the mpa web api admin role to there existing user set up?
@SimonAnnetts do you have everything you need to set up the staging version of the MPA mapper? Or is there more info you need from @paul-gilbertson?
I need confirmation that I can still use the same username to access the mpa_web_mapper database that is used to access to emodnet_web_mapper database as I don't think I have any other credentials on file.
I also need @paul-gilbertson to restart the web-api service via systemd as it is not running currently and systemd is not restarting it after I deploy to the server so we get a service unavailable error when the mapper tried to access the web-api.
This is now complete
Thanks Paul. I can confirm that the mpa web-api is now functioning and has populated the database.
I have filled it with test data initially, @HelenwoodsJNCC do you want the test data in there or shall I clear it? The mpa mapper is looking for a mapinstance called MPA and the test data only provided emodnet so the mpa mapper is blank currently anyway.
@SimonAnnetts I have real data ready to upload so it doesn't matter too much about the test data. I will get this data loaded in asap. However I am now having trouble with permissions in the database I get the 'ERROR: permission denied for relation BaseLayer, SQL state: 42501' error when trying to view/edit the tables - @LynnHeeley could you get someone to grant me read/write permissions for the db.
@SimonAnnetts can you grant permissions for the mpa_web_api tables etc. to the mpa_mapper_web_api_admin role? Currently they are locked to the mpa_web_api user.
I do not have the permissions to do that. Unfortunately, when the web-api creates the database tables on first run, it can only give it permissions for itself to use, it can't grant permissions for others. I had this problem with the emodnet database too - see issue #60 .
I have removed the test data from the database.
@SimonAnnetts no worries. @mattdebont can you help with getting the permissions sorted?
@SimonAnnetts @JamesPe I am still having permissions issues this end, but hopefully we should be sorted early next week to allow user testing to happen asap.
@SimonAnnetts @JamesPe @andyb-esdm we now have data in the MPA api database but nothing is showing up on the mapper, I get
Failed to load resource: net::ERR_NAME_NOT_RESOLVED https://api.mpa.jncc.gov.uk/api/mapinstance/MPA
Could not load map instance config from API https://mapper.mpa.jncc.gov.uk/main.d33c277b31422467f677.js
messages in the console. Any ideas why this might be?
This works for me - https://mapper.mpa.jncc.gov.uk/
Is that what you mean?
@HelenwoodsJNCC are you not seeing this ?
@JamesPe, no strangely enough we don't see any of the data layers - glad to see it's working for you though. Must be an IT issue at our end.
@HelenwoodsJNCC @JamesPe
I was getting unauthorised response when I first connected but it worked when I connected to the exegesis vpn. Is this IP locked?
By the way, nice to see some different data!
I think the active layers container needs a bit more thought though. I'm not keen on that scroll ...
@JamesPe @andyb-esdm I think it's just taking a while for the changes IT made to work their way up to Aberdeen, seemingly it is working fine in Peterborough. Agreed it's nice to look at a slightly different map for a change!
I think the API is unlocked - as you can call it I believe from your location I believe.
I would guess that maybe in the mapper you see the layer control but none of the layers on the map itself
That would suggest that geoserver is locked down to known IP addresses - and yours isn't one of them Helen - clearly for a public mapper this needs to be open
It works from my home fixed IP - but JNCC does know that so maybe included. Andy said from home (unknown IP) - API worked - but he got CORS error until he ran via ESDM VPN which would give him a known IP again
P.S. Layers look nice ;-)
@HelenwoodsJNCC - seeing as Andy isn't seeing it at home either - I would be suspicious that Peterborough is another known IP and Geoserver is locked.
This is a direct call to geoserver that should work (and return an image) - https://staging.ows.emodnet-seabedhabitats.eu/mpa_mapper/wms?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng8&TRANSPARENT=true&LAYERS=sac_mc&TILED=TRUE&WIDTH=256&HEIGHT=256&CRS=EPSG%3A3857&STYLES=&BBOX=-2504688.542848654%2C7514065.628545966%2C1.3969838619232178e-9%2C10018754.171394622
GetFeatureInfo a bit difficult with unfilled polygons - you have to click on the boundaries - in the past I have given them a very low opacity fill so they can be selected when you click on their interiors. Might be worth a try on one layer?
Yes, I had to turn on ESDM VPN
I can access the EMODnet prelive site using my Peterborough VM but seemingly not the MPA mapper site. I will speak to IT. @andyb-esdm good tip about the polygons I will tweak the styling. Good to see the group styles seem to be working ok.
@HelenwoodsJNCC try it on one first to see if it will work. Yes, really liked what you did with the sand eels point data :)
data look great against bing background too
GetFeatureInfo a bit difficult with unfilled polygons - you have to click on the boundaries - in the past I have given them a very low opacity fill so they can be selected when you click on their interiors. Might be worth a try on one layer?
(Quick update from @andyb-esdm to this captured below for referencing elsewhere (thanks andy!)
The relevant code is here:
https://github.com/jncc/web-mapper-core/blob/master/web-map/src/app/map/map.component.ts
beginning on line 144
The offending function is on line 150 map.forEachLayerAtPixel
The documentation is here:
https://openlayers.org/en/v4.6.5/apidoc/ol.Map.html#forEachLayerAtPixel
and says:
“Detect layers that have a color value at a pixel on the viewport, and execute a callback with each matching layer”
My understanding is that this was a ‘standard’ way of getting feature info urls from a map:
I guess the idea is that OL can’t distinguish between inside a feature and large areas of the layer that have no features since it’s just an image. So this prevents unnecessary getfeatureinfo requests when you are clicking nowhere near a feature.
But I can see how this isn’t ideal with 0 opacity fills.
I’ve found that very low opacity fills work okay and isn’t distracting. But we could potentially investigate other strategies e.g. all clicks within the extent of any visible layer raises a getfeatureinfo request. That wouldn’t be a difficult function to write. You would lose the ‘optimisation’ of a lot of feature misses on a layer but perhaps that’s irrelevant?