This README document has the intent to explain the logic behind the HA Procedure that it uses the combination of Heat Scaling group, Heat Resource Group, Ceilometer Alarm and Nova metadata.
Background and requirements: this solution has been designed with the absolutely requirement to be agentless and to relay only on the OpenStack environment, such as Nova and Ceilometer. It is also important to keep consistent the number of VM created.
- Just make sure to have template
- Write down the image ID in the env.yaml file
- In order to have 60 seconds for the "cpu" Ceilometer meters you need to load the pipeline file:
- "Ceilometer/pipeline_kilo.yaml" Ceilometer Pipeline for OpenStack Kilo, load into the compute node (usually at /etc/ceilometer/pipeline.yaml) and restart the Ceilometer Compute Agent (usually ceilometer-compute-agent)
- "Ceilometer/pipeline_since_liberty.yaml" Ceilometer Pipeline for OpenStack Liberty, Mitaka and Newton, load into the compute node (usually at /etc/ceilometer/pipeline.yaml) and restart the Ceilometer Compute Agent (usually ceilometer-compute-agent)
$ source <environment file>
$ heat stack-create --template-file ./init.yaml --environment-file ./env.yaml LifeCycleStack
At Time0 the template has been loaded into Heat. The heat engine processed it and the result respects what has been written in the HOT files. E.g. Let's assume that has been created:
- One network, one related subnet, one router
- One ResourceGroup policy that has defined 20...
- ...20 "OS::Scaling::Fedora". This resource have the following content
- One Randon String
- One AutoScalingGroup with "OS::VM::Fedora" (min 1, max 2/3)
- The VM "OS::VM::Fedora" have a custom metadata "metering.stack"
- One ScalingPolicy (+1 in size)
- One BackToNormalPolicy (-1 in size)
- Three OS::Ceilometer::Alarm:
- MissingSample: meter cpu, evaluation periods 3x 60s, threshold 1, comparison_operator: lt. "insufficient_data_actions:" -> Call ScalingPolicy
- NoExecution: meter cpu, evaluation periods 3x 60s, threshold 1, comparison_operator: lt. "alarm_actions:" -> Call ScalingPolicy
- BackToNormal: meter cpu, evaluation periods 3x 60s, threshold 1, comparison_operator: gt. "alarm_actions:" -> Call BackToNormalPolicy
At Time1 the compute host that was hosting a portion of the previously running VM is now down.
At Time1b the Ceilometer alarm for each VM (3x Ceilometer alarm for VM) will report a fault situation. These alarms are called: NoExecution and MissingSamples. Also a special alarm for the Time2 is now in OK status.
At Time1c the previously triggered alarm have created one VM in replacement of the deaths one.
At Time2 the special alarm BackToNormal triggered the deletion of the non-running VM (actually of the oldest one)
The meaning to use the Heat AutoScalingGroup is to provide a kind of VM life-cicle. The AutoScalingGroups are generated by the ResourceGroup, which just create a pre-defined number of resources. In order to provide reliable Ceilometer alarms (which by designed look for the whole tenant resources) each ResourceGroup will create a defined number of AutoScalingGroups. Each AutoScalingGroup will manage one VM and the related Ceilometer Alarm:
- Will create a new VM if the "cpu" samples will disapper
- Will delete the oldest VM when the "cpu" samples are presents
- In order to have Ceilometer Alarms per VM a random string will be used, segregating other VMs and other AutoScalingGroup VMs.
- In order to have a reliable Ceilometer samples the "cpu" meters has to be generated every 60 seconds
- A lost-quorum/split-brain controller situation (at least two out of three) will prevent this system to work correrly since the OpenStack API will not answer anymore
- Implement Anti-Affinity Policy, to be erditated by the main stack to all of the VMs
- Better testing of host down (really down) scenario