Design: Customizable Options for Provisioner with Documentation

[Joel Anderson]

@JustinProminic talked today about a design for allowing the user to further customize the properties for the server or provisioner. For example, this would allow the user to add additional genesis_packages on the Vagrant side.

Requirements:

1. This should not involve editing Hosts.yml. This file is already large enough that it would be confusing for casual users.
2. This should not involve editing a large configuration file (i.e. a file with commented options where the user is expected to uncomment and modify the current lines). We could like something more modular than this, probably based on the default properties for individual tasks (example).
3. Ideally, we should be able to manage the customizations from a GUI. The GUI may just be a wrapper around editing a YML file
4. The possible customizations should be configured and documented in the provisioner
   • We should have metadata file(s) in the provisioner to document the available properties
   • The metadata should also define a "priority", so that we can order them by how likely (we think) the user will want to modify the properties.
   • Use a standard format like YML or JSON for the metadata files
5. We should be able to copy the customizations to other versions of the provisioner, so that we don't need to manually figure out what changed.

We'll discuss this more in the meeting tomorrow.

[Joel Anderson]

@MarkProminic, would it be possible to override the properties (similar to this) in separate files in a "custom" directory, and then import them programatically after loading Hosts.yml? I can explain this more tomorrow, but I wanted to start brainstorming for this.

[Mark Gilbert]

The Whole point of the Hosts.yml was so that it could be copied from one persons machine to another as the Source of Truth.

This defeats that purpose and why Hosts.yml was designed.

I could refactor Hosts.rb to include extra directories to load additional variables from, however this is a major refactor and affects other projects like spinning up these exact instances on Bhyve in privately Hosted environments.

A role is created to provision a service, But a Role will always defer to the any of the variables overridden via the Hosts.yml.

This will take some time for me to fully grok what you are asking, but it does not seem easy.

My Solution is to override variables in the Hosts.yml as I designed it.

I would argue that splitting up things in a bunch of different directories rather than defining all the versions, names, passwords variables, etc in one file, would be more confusing to end users.

[Mark Gilbert]

The Hosts.yml is designed around the exact premise you made:

Ideally, we should be able to manage the customizations from a GUI. The GUI may just be a wrapper around editing a YML file

In regards to the point about:

The possible customizations should be configured and documented in the provisioner

We should have metadata file(s) in the provisioner to document the available properties
The metadata should also define a "priority", so that we can order them by how likely (we think) the user will want to modify the properties.
Use a standard format like YML or JSON for the metadata files

That's precisely what the /meta/main.yml files are for in the Ansible roles . .

and more specifically about:

The metadata should also define a "priority", so that we can order them by how likely (we think) the user will want to modify the properties.

That is what we define as our default, via the /defaults/main.yml files, IE; the versions that we build it with on release. If a user wants to override this, they simply redefine the variable to override it in the Hosts.yml, if not defined it defaults to to this.

In regards to:

We should be able to copy the customizations to other versions of the provisioner, so that we don't need to manually figure out what changed.

That's why roles are created, to be modular, simply copy a Role Folder of the task sequence you want, and make sure that your Hosts.yml has it defined in it to have Ansible call that Role.

[Joel Anderson]

Unfortunately, we ran out of time to discuss this in today's meeting. I think some things were not clear from my writeup.

Yes, What I am trying to do with these separate files could also be done in Hosts.yml. The scope for this proposal is limited to overriding properties in existing roles. I'm not opposed to keeping the overriding properties in Hosts.yml and then having a GUI to manage them. @JustinProminic was oppsed to having users work with Hosts.yml at this point, since it has gotten large, but it might not be a problem if this were abstracted.

The design @JustinProminic proposed was update the properties in the role directories instead, so that we could be editing this in a more modular way. We were talking about editing the defaults file here, but I think there may be a more appropriate file here. I was asking about handling the overrides in seaprate files as an idea to manage the properties in smaller files like Justin wanted, but without needing to modify (and later merge) the existing files in the template. I'm not sure this is a great idea, but I wanted to at least explore it.

Adding new roles or modifying existing roles would be outside the scope of this feature, at least for the GUI support. We could still let the user modify a copy of the local template as desired (or fork the repository and use it as a custom provisioner (#83).