Virtual Machine / Deployment Setup

[Quinn Hart]

The plan is to replicate the setup for aggie experts, and the dams in order
to run a zero down-time system. This requires the library website going to
a router that switches between two potential backend systems.

Because the website is editable, the setup in this case will be

Required Virtual Machines

We require 3 virtual machines in order to run the setup.

machine	specs
blue.library.ucdavis.edu	8Gb, 250Gb, 8cpu
gold.library.ucdavis.edu	8Gb, 250Gb, 8cpu
router.library.ucdavis.edu	4Gb, 25Gb, 8cpu

Apache Router

We are currently using apache as our router. Although there are some more modern alternatives to this setup; nginx and traffic server, my inclination is to continue using apache. The reasons being 1) we are most used to this setup and 2) It appears that the nginx load balance requires a commercial setup for the type of load balancing we use, and 2a) I found trafficserver very confusing.

<VirtualHost *:80>
  ServerName library.ucdavis.edu
  ServerAlias lib.ucdavis.edu
  ServerAlias www.library.ucdavis.edu
  ServerAlias www.lib.ucdavis.edu
  ServerAdmin [e-mail]
  RewriteEngine on
  RewriteRule ^ https://library.ucdavis.edu%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>
<VirtualHost *:443>
  ServerName lib.ucdavis.edu
  ServerAlias www.library.ucdavis.edu
  ServerAlias www.lib.ucdavis.edu
  ServerAdmin [e-mail]
  RewriteEngine on
  RewriteRule ^ https://library.ucdavis.edu%{REQUEST_URI} [END,QSA,R=permanent]

  SSLCertificateFile    /etc/ssl/certs/__library_ucdavis_edu_cert.cer
  SSLCertificateKeyFile /etc/ssl/private/__library.ucdavis.edu.key
  SSLCertificateChainFile /etc/ssl/certs/incommon_interm.cer

</VirtualHost>

<VirtualHost *:443>
  ServerName library.ucdavis.edu
  Header add Set-Cookie "ROUTEID=d.%{BALANCER_WORKER_ROUTE}e; path=/" env=BALANCER_ROUTE_CHANGED
  SSLProxyEngine on

  <Proxy "balancer://prod">
    BalancerMember "http://blue.web.library.ucdavis.edu:80" route=blue
    BalancerMember "http://gold.web.library.ucdavis.edu:80" route=gold
    ProxySet stickysession=ROUTEID
    ProxySet failonstatus=503
    ProxySet lbmethod=byrequests
  </Proxy>
  AllowEncodedSlashes On
  ProxyPreserveHost On

#  ForensicLog ${APACHE_LOG_DIR}/forensic.log

  ProxyRequests off

  ProxyPass "/" "balancer://dev/" nocanon
  ProxyPassReverse "/" "balancer://dev/" nocanon

  # IMPORTANT!
  RequestHeader unset X-Forwarded-Host
  RequestHeader unset X-Forwarded-For
  RequestHeader unset X-Forwarded-Server

  SSLCertificateFile    /etc/ssl/certs/__library_ucdavis_edu_cert.cer
  SSLCertificateKeyFile /etc/ssl/private/__library.ucdavis.edu.key
  SSLCertificateChainFile /etc/ssl/certs/incommon_interm.cer
</VirtualHost>

Routing setup

The router will continuously monitor a standard port (80?) on both blue and gold, and route to whichever is currently working. In our setup, the standard practice will be that only one computer will be running at any time. After an update, one computer or the other will be turned off.

The same blue and gold machines can also run stage and dev machines. In fact, this is considered best form, as we can more easily upgrade from stage to production in that setup. blud and gold use different ports to identify the version they are running.

Update Path

All stage and production deployment configurations will be located at /etc/library-website/{version} where version is a number that corresponds to the current data setup. This is setup, so that the volumes are identified with data updates, and not with something else (dns eg.). This allows us to move an implementation from stage to production with a simple docker compose up/down without needing a seperate copy of the data. dev and other setups should be in /etc/library-website/dev

The typical update path then is for the new setup to be tested in stage. For switching, stage is brought down, the production setup is set to read-only; a final database / fs update is performed, the .env file is switched to production, and the machine is brought back up. At that point two servers respond to library requests, the old server in read-only, the new server ready for updates. Once the new server is verified to be working, the old server is brought down.

Hot Spare

The previous server configuration can act as a ready to go spare, Without modification, this spare would bring up a slightly outdated website (both functionality and data). It is possible to keep a more up-to-date spare, aggie experts uses rsync techniques between docker volumnes for this task; the library website may have a different setup.

[Quinn Hart]

@spelkey-ucd / @jrmerz Let me know what you think about these notes. Questions that I have:

• I choose above to 301 everything to library.ucdavis.edu
• we will use *.library.ucdavis.edu, a subset in the staff VPN
• I would suggest we use dev.library.ucdavis.edu for testing
• Let me know about the hot spare

[Quinn Hart]

@jniffenegger and others suggested using *.library as the subdomain. I've edited the above to show that.

I've added a diagram that combines this with the production site, and the special collection redirect issues.