Giters

Luxoft / fabric-skeleton

A tool for devops automation of Hyperledger projects.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Killing cluster behaves unexpectedly

fedyiv opened this issue · comments

commented

Steps to reproduce:

  1. Start cluster
    ./ops-cli -i ~/.ssh/Blockchain-controller.pem -c single_sample
  2. Kill cluster
    ./ops-cli -i ~/.ssh/Blockchain-controller.pem -c single_sample -k
  3. Start cluster again
    ./ops-cli -i ~/.ssh/Blockchain-controller.pem -c single_sample

Expected result:
Cluster starts normally
Actual result:
Cluster fails to start
Root cause
When cluster is starting it checks whether it was already started or not by checking availablility of AWS hosts. When we execute kill and hosts are in progress of termination, it thinks that cluster is alive and this leads to errors.
Workaround
After killing cluster, open AWS EC2 website and manually check that everything is down.
Proposed solution:
Enhance killing script so that it waits until actual cluster stop synchronously

commented

I guess, the better way is to check host state on startup and wait for appropriate.
It would be noce to implement that is a reasonably uniform way - taking into account that eventually this might be run at least on: local hosts, docker hosts, aws ec2 hosts, openstack hosts, etc...