The knife-topo plugin allows you to create and update topologies consisting of multiple nodes using single knife commands, based on a JSON definition of the topology. The plugin:
- creates a data bag for the topology
- generates attribute file(s) in a topology-specific cookbook
- sets and updates the run list, chef environment and properties of nodes
- uploads the topology-specific cookbook and bootstraps nodes
You may find this plugin useful if you are regularly updating a system consisting of multiple nodes, and would like to manage its dynamic configuration (e.g. changing software versions) through a single (json) configuration file. It may also be useful if you are regularly bringing up multi-node systems with similar topologies but differences in their configuration details.
This plugin can be used in conjunction with the topo cookbook to configure dynamically deployed topologies of nodes (e.g. in AWS). In this approach, use knife-topo to set up the topology details. Use the 'topo' cookbook to let nodes pull their own detailed configuration from the topology data bag on the Chef server.
V2 introduces the notion of an attribute setting strategy. Instead of specifying node attributes and cookbook attributes separately, you specify one set of attributes and the method by which they should be set on the node (e.g. 'direct_to_node' or 'via_cookbook').
The purpose of this change is to make it easier to support new methods of setting attributes such as policy files, and also making it easier to switch between methods.
A node type can be specified for a node. The node type is used by the 'topo' cookbook to identify the right configuration to use. It is also used in the 'via cookbook' method to support attributes that vary by node type.
To reduce complexity, V2 no longer supports multiple topologies in a JSON file.
V1 topology JSON files can be converted to V2 by importing them and
then exporting them. knife-topo will auto-detect V1 format files on
import. You can also explicitly specify the input format using
knife topo import sometopo.json --input-format 'topo_v1'
Install knife-topo as a gem
$ chef gem install knife-topo
Define each topology in a topology file. Import that file into your Chef workspace using knife topo import, then create the topology knife topo create, specifying the '--bootstrap' option if you want to bootstrap all of the nodes.
Update the topology file as the configuration changes, import those changes knife topo import and run knife topo update to update the topology.
Try out this plugin using a test-repo provided in the knife-topo github repository. Download the latest knife-topo release and unzip it, then follow the Instructions for the example.
The instructions assume you have chefDK installed and working with Vagrant and VirtualBox.
If you're the sort of person who just wants to jump in and try it, here's some hints.
Generate a topology file for a topology called test1 from existing nodes node1 and node2:
knife topo export node1 node2 --topo test1 > test1.json
Import a topology json file, generating all of the necessary artifacts in your workspace:
knife topo import test1.json
Create the topology using existing nodes:
knife topo create test1
Create the topology, bootstrapping new nodes in vagrant (you will need to add the host details for bootstrap to the file before importing):
knife topo create test1 --bootstrap -xvagrant -Pvagrant --sudo
See the example topology file
The topology file contains a single topology. Each topology has some overall properties, an array of nodes and an array defining topology cookbook attributes.
{
"name": "test1",
"node_type": "appserver",
"chef_environment": "test",
"tags": [ "testsys" ],
"normal": {
"owner": {
"name": "Christine Draper"
}
},
"nodes" : [
...
]
}
The name is how you will refer to the topology in the
knife topo subcommands.
The chef-environment and normal attributes defined
here will be applied to all nodes in the topology, unless alternative
values are provided for a specific node. The tags
will be added to each node.
The default strategy for setting attributes is direct_to_node.
In this strategy, normal attributes are set directly on the nodes when
the topology is created, bootstrapped or updated. Attributes with
other priorities are ignored.
The strategy field can be set to 'via_cookbook', in which case
additional strategy_data can be provided to specify a cookbok
and attribute filename.
{
"name": "test1",
...
"strategy" : "via_cookbook",
"strategy_data": {
"cookbook": "topo_test1",
"filename": "softwareversion",
}
}
In this strategy, the cookbook and attribute file are generated in the local workspace when the topology is imported, and uploaded to the server when the topology is created or updated. Attributes can have any valid priority.
Each topology contains a list of nodes.
{
"name": "test1",
...
"nodes": [
{
"name": "buildserver01",
"node_type" : "buildserver",
"ssh_host": "192.168.1.201",
"ssh_port": "2224",
"chef_environment": "dev",
"run_list": [
"role[base-ubuntu]",
"ypo::db",
"recipe[ypo::appserver]"
],
... node attributes, see below ...,
"tags": [ "build" ]
},
...
]
}
Within nodes, the name field is the node name that will be used in Chef.
The fields chef_environment, run_list and tags
will also be applied to the node in Chef. All of these
fields are optional.
The node_type sets the node attribute normal['topo']['node_type'].
This attribute is used in the 'via_cookbook' strategy to specify
attributes that apply to only nodes of that type.
The ssh_host and ssh_port fields are used to
bootstrap a node.
Each topology may have attributes that are set on each node according to the attribute setting strategy. The attribute names and values are specified by priority ('default', 'normal', 'override').
"nodes": [
{
"name": "buildserver01",
"normal":
{
"nodejs":
{
"version": "0.10.40"
},
"testapp":
{
"version": "0.0.3"
},
"mongodb":
{
"package_version": "2.6.9"
}
}
}
]
The main subcommands for knife topo are:
- knife topo import - Import one or more into your workspace/local repo
- knife topo create - Create and optionally bootstrap a topology of nodes
- knife topo update - Update a topology of nodes
The additional subcommands can also be useful, depending on your workflow:
- knife topo bootstrap- Bootstraps a topology of nodes
- knife export - Export data from a topology (or from nodes that you want in a topology)
- knife topo list - List the topologies
- knife topo search - Search for nodes that are in a topology, or in no topology
- knife topo delete - Delete a topology, but not the nodes in the topology
The topologies are data bag items in the 'topologies' data bag, so you can also use knife commands such as:
knife data bag show topologies test1- Show details of the test1 topology data bag
The knife topo subcommands support the following common options.
| Option | Description |
|---|---|
| -D, --data-bag DATA_BAG | The data bag to use for the topologies. Defaults to 'topologies'. |
knife topo bootstrap TOPOLOGY
Runs the knife bootstrap command for each node in the topology that
has the ssh_host attribute. Specified options will be passed through
to knife bootstrap and applied to each node.
The knife topo bootstrap subcommand supports the following additional options.
| Option | Description |
|---|---|
| --overwrite | Re-bootstrap existing nodes |
| See knife bootstrap | Options supported by knife bootstrap are passed through to the bootstrap command |
The following will bootstrap nodes in the test1 topology, using a user name of vagrant, password of vagrant, and running using sudo.
$ knife topo bootstrap test1 -x vagrant -P vagrant --sudo
knife topo create TOPOLOGY
Creates the specified topology in the chef server as an item in the topology data bag. Creates the chef environment associated with the topology, if it does not already exist. Uploads the topology cookbook, if using the 'via_cookbook' method. Updates existing nodes based on the topology information. New nodes will be created if the bootstrap option is specified.
The knife topo create subcommand supports the following additional options.
| Option | Description |
|---|---|
| --bootstrap | Bootstrap the topology (see topo bootstrap) |
| See knife bootstrap | Options supported by knife bootstrap are passed through to the bootstrap command |
| --disable-upload | Do not upload topology cookbooks |
| --overwrite | Re-bootstrap existing nodes |
The following will create the 'test1' topology, and bootstrap it.
$ knife topo create test1 --bootstrap
The following will create the 'test1' topology but will not bootstrap it or upload topology cookbooks.
$ knife topo create test1 --disable-upload
knife topo delete TOPOLOGY
Deletes the specified topology. Does not delete the nodes in the topology, but does
remove them from the topology by removing the ['topo']['name'] attribute
which is used by knife topo search.
knife topo export NODE [ NODE ... ]
Exports the nodes into a topology JSON.
If the topology does not already exist, an outline for a new topology will be exported. The exported JSON can be used as the basis for a new topology definition.
If nodes are specified, these will be exported in addition to any nodes that are in the topology.
If no topology is specified, all defined topologies will be exported.
The knife topo export subcommand supports the following additional options.
| Option | Description |
|---|---|
| --topo | Name of the topology to export (defaults to 'topo1') |
| --min-priority | Only export attributes with a priority equal or above this priority. |
The following will export the data for nodes n1 and n2 as part of a topology called 'my_topo':
$ knife topo export n1 n2 --topo my_topo > my_topo.json
The following will create an outline for a new topology called
'christine_test', or export the current details if it already exists:
$ knife topo export --topo christine_test > christine_test.json
knife topo import [ TOPOLOGY_FILE ]
Imports data bag items containing the topologies from a topology file into the local repo. If no topology file is specified, attempts to read from a file called 'topology.json' in the current directory. Generates additional artifacts (e.g. topology cookbook attribute file) where needed.
The following will import the topology or topologies defined in the 'topology.json' file.
$ knife topo import topology.json
The following will import the 'test1' topology defined in the 'topology.json' file.
$ knife topo import topology.json test1
knife topo list
Lists the topologies that have been created on the server.
knife topo search [ QUERY ]
Searches for nodes that are in a topology and satisfy the query. With no options,
this searches for nodes in any topology. Use --topo=topo_name to search
within a specific topology. Use --no-topo to search for nodes in no topology.
knife topo search uses the ['topo']['name'] attribute to identify which nodes
are in which topology.
The following will search for nodes in any topology that have a name starting with "tst".
$ knife topo search "name:tst*"
The following will search for nodes in the "prod" chef environment that are not in a topology.
$ knife topo search "chef_environment:prod" --no-topo
The following will search for all nodes in the "systest" topology.
$ knife topo search --topo=systest
The knife topo search subcommand supports the following additional options.
| Option | Description |
|---|---|
| --topo | Search for nodes in the specified topology |
| --no-topo | Search for nodes that are not in any topology |
| See knife search | Options supported by knife search are passed through to the search command |
knife topo update [ TOPOLOGY ]
Updates the specified topology. Creates or updates nodes identified in the topology, using information specified in the topology for the specific node.
If no topology is specified, all existing topologies will be updated.
| Option | Description |
|---|---|
| --bootstrap | Bootstrap the topology (see topo bootstrap) |
| See knife bootstrap | Options supported by knife bootstrap are passed through to the bootstrap command |
| --disable-upload | Do not upload topology cookbooks |
The following will update the 'test1' topology.
$ knife topo update test1
The following will update all topologies in the 'topologies' data bag.
$ knife topo update
Author:: Christine Draper (christine_draper@thirdwaveinsights.com)
Copyright:: Copyright (c) 2014-2016 ThirdWave Insights, LLC
License:: Apache License, Version 2.0
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.