djtfmartin / generator-living-atlas

An Yeoman Generator for Living Atlas Ansible Inventories

generator-living-atlas

A Yeoman Generator for Living Atlas Ansible Inventories

• Intro
• Installation
• Options
• Screens
• Rerunning the generator
• Ansible Wrapper
• TODO
• Caveats
• Any problem with this generator?
• But why this?
• Can these quick-start inventories solve all this?
• About Yeoman
• License

Intro

This is a simple yeoman generator, so, just asking some simple questions about your Living Atlases Node, you will get a functional configuration for setup your Virtual Machines (VMs) using ansible.

The main goal of this utility is to allow you to configure, install & maintain a Living Atlas infrastructure in a fast & smoothly way.

In others words, with the ansible inventories produced by this yeoman generator and the ala-install ansible playbooks you should deploy the main LA services quickly and without too much pain.

Also, thanks to yeoman, you can rerun this generator, for instance, enabling some new module in the future, or enabling SSL, etc and reusing your previous settings and comparing the differences (see the screenshots).

Installation

First, install Yeoman and generator-living-atlas using npm (we assume you have pre-installed node.js).

npm install -g yo

Wait!, as this generator-living-atlas is still somehow experimental, it's not published in npm. So if you want to test it, you can clone this repo and just use it like:

cd this-cloned-repo
npm link
cd /tmp # or other directory
yo living-atlas

As this is a yeoman development environment you can get some EPERM or EACCESS errors, so check the yeoman setup guide for proper configuration.

If we decide to publish this generator in the future, the normal use will be like:

npm install -g yo
npm install -g generator-living-atlas

Then generate your new project:

yo living-atlas

Options

• Use --replay to reuse all the previous responses and regenerate the inventories with some new modification (or improvements if you update this repo). More info below.
• Also, you can use --debug to see some verbose debug info.

Screens

A screen recording re-running the generator but using another solr hostname and switching to use SSL (see the differences in the generated inventories):

After running the inventories following the instructions from the generated README over two VMs (like in the Paris 2019 Workshop):

Differences beween two runs:

We can use these inventories as a base for extracting documentation via doxygen (or similar):

Rerunning the generator

You can rerun the generator with the option yo living-atlas --replay to reuse all the previous responses and regenerate the inventories with some modification (if for instance you want to add a new service, or using a new version of this generator with improvements).

We recommend to override and set variables adding then to quick-start-local-extras.yml and quick-start-spatial-local-extras.yml without modify the generated quick-start-inventory.yml and quick-start-spatial-inventory.yml, so you can rerun the generator in the future without lost local changes. The *-local-extras.sample files will be updated with future versions of this generator, so you can compare from time to time these samples with your *-local-extras.yml files to add new vars, etc.

Ansible wrapper

Furthermore the utility ansiblew is generated, an ansible-playbook wrapper that can help you to use the inventories and can be easily modificable by you to your needs. It depends on python-docopt package (sudo apt install python-docopt in debian and derivatives).

Help output:

$ ./ansiblew --help

This is an ansible wrapper to help you to exec the different playbooks with your
inventories.

By default don't exec nothing only show the commands. With --nodryrun you can exec
the real commands.

With 'main' only operates over your main host.

Usage:
   ansiblew --alainstall=<dir_of_ala_install_repo> [options] [ main | collectory | ala_hub | biocache_service | ala_bie | bie_index | images | lists | regions | logger | solr | cas | biocache_backend | biocache_cli | spatial |  all ]
   ansiblew -h | --help
   ansiblew -v | --version

Options:
  --nodryrun             Exec the ansible-playbook commands
  -p --properties        Only update properties
  -l --limit=<hosts>     Limit to some inventories hosts
  -s --skip=<tags>       Skip tags
  -h --help              Show help options.
  -d --debug             Show debug info.
  -v --version           Show ansiblew version.
----
ansiblew 0.1.0
Copyright (C) 2019 living-atlases.gbif.org
Apache 2.0 License

So you can install the CAS service or the spatial service with simple commands like:

./ansiblew --alainstall=../ala-install cas --nodryrun

and

./ansiblew --alainstall=../ala-install spatial --nodryrun

or all the services with something like:

./ansiblew --alainstall=../ala-install all --nodryrun

Without --nodryrun you will get only the full ansible-playbook command to use these inventories .

TODO

• Add basic services (collectory, ala-hub, etc).
• Add domain/context and service subdomains support
• Add http/https urls support
• Add regions service
• Add species-list service
• Add spatial service
• Disable caches when using the same host for collectory & biocache
• use --limit with hostnames
• Improve README hosts
• Improve README playbook commands
• Add ansible wrapper
• Add inventories to add extra vars without modify the generated inventories
• Store = true if you running it for the first time
• Add other administration info (technical_contact, orgEmail, etc)
• Recommend to use git to track your inventories changes
• Add biocache-backend and biocache-cli playbook and cassandra host selection
• Add CAS 5 service
• Fix apikey_db_dump location relative to ala-install
• Add ssl self signed certs support when ssl enabled
• Document letsencrypt usage
• Improve generator for only a domain and /context use
• Add local-extras.sample files that can be updated in the future without lost local changes in local-extras.yml files
• Generation of passwords on first run and increase security by default
• Use a different playbook than demo that do not install by default all main services so we can choose a different host for them
• Document --skip
• Demo index.html uses biocache_hub_context_path etc that does not works with multi hosts
• Add --tags to ansible wrapper
• Better testing

Caveats

• Currently running several services in the same machine and the same domain doesn't configure correctly nginx vhosts. See this enhancement issue in ala-install. As a workaround you can copy and configure the generated vhosts in /etc/nginx/sites-available with different names so they are not overwritten by ansible.
• As we are using the https://github.com/AtlasOfLivingAustralia/ala-install/blob/master/ansible/ala-demo.yml playbook, if you are using more that two hosts (the minimal is one for main services, and another for spatial services), some services will be also configured in the main machine. As a workaround, while we fix this, you can use a different playbook for your main machine with the correct limited roles.

Any problem with this generator?

Please add an issue give us details of your problem and we'll try to help you and to improve the generator. Thanks!

But why this?

Ansible is a fantastic tool for manage LA node infrastructures. But nowadays we have some problems with our current ala-install inventories/roles & documentation that make their use difficult by newcomers nodes, but also for other non-Australian nodes maintenance:

• There are many config properties that are not documented and this makes quite difficult to tune a LA node
• Sometimes the /data/*/config/*properties are well commented but not their source inventories
• From time to time new properties appears but others not ALA nodes don’t notice
• We end using several self-made inventories, duplicate properties (like orgName, urls, etc), and many times this is a source of problems (like code duplication). Some minor sample:
• There is a lack of more real production-ready inventories (you have to ask for then).
• ala-install has a structure that sometimes is difficult to find some samples or how to start to deploy some service (think in CAS)
• Many times you have to check the ansible generated config/*properties for non configured or default properties, search the ansible role code for know how to configure these variables, re-run ansible with these new variables and re-check the ansible /data/*/config/*properties.
• The demo inventory (or the last Paris workshop inventories) are a good sample but not enough for a new production LA node (lack of inventory variables documentation, missing of important services like CAS)
• We have a lack of info about services versions compatibility
• Sometimes default module versions in inventories or LATEST nexus packages are not the correct/latest ones.
• It would be nice to choose strategies that was usable by ALA too (not only LA nodes) - that might help maintenance.

Can these quick-start inventories solve all this?

No. This is only a Proof of Concept that tries to mitigate some of the previous problems.

If we think that some tool like this can be useful, we can add more functionality like:

• Better comments of properties in generated inventories for easy operation & tuning
• CAS deployment done ✓
• Focus in have well maintained these generator and their inventories
• Autogenerate documentation from the inventories comments with tools like doxygen
• etc

or maybe this experiment suggests us other path to follow.

About Yeoman

• Learn more about Yeoman.

License

Apache-2.0 © Living Atlases