This is my quick reference of commands, configuations and procedures to setup a local environment on Mac OS X during my learning process for infrastructure and cloud automation with Chef, Knife, Cookbooks, Vagrant, and other automation modules with Ruby. The references and commands documented are the results of my learning experience and the findings following along the Chef documentation and other books.
Books and resources I'm learning from:
Chef Cookbook 3ed, by Matthias Marschall
Learn Ruby The Hard Way, 3rd Edition, by Zed A. Shaw
Automating Infrastructure - Chef for DevOps, by Whizlabs.com
Chef documentation
- Chef Overview
- Install Chef Workstation (formerly known as ChefDK)
- Setup Chef Workstation
- About Attributes
- All Infra Resources
- Ohai & Automatic Attributes
Install Ruby on Mac OS X and update RubyGems (Ruby package manager)
brew install ruby
gem update --system
Install Chef Workstation on Mac OS X
brew install --cask chef-workstation
In case we need to uninstall and start a clean Chef Workstation installation, execute the uninstall script
uninstall_chef_workstation
rm -fr ~/.chef*
rm -fr ~/Library/Application\ Support/Chef\ Workstation\ App/
rm -fr /etc/chef
gem uninstall chef chef-config chef-telemetry chef-utils chef-vault chef-zero
Configure Ruby Environment for Chef
# BASH
echo 'eval "$(chef shell-init bash)"' >> ~/.bashrc
chef shell-init bash
#ZSH
echo 'eval "$(chef shell-init zsh)"' >> ~/.zshrc
chef shell-init zsh
Create a Chef account at Chef Manage (Hosted Chef https://manage.chef.com). Download and extract the Chef Starter Kit which contains your credentials, keys (.pem) and initial Chef repo diretory (chef-repo).
# To download the Starter Kit go to:
# Administration > Select your Organization ID > Starter Kit > Download Starter Kit
# Extract the StarterKit.zip and go into the chef-repo directory to get started
# If you are using
Alternatively to a Hosted Chef (https://managed.chef.io), if you are using Chef Server hosted in your private cloud/premises, then setup your Chef repo & credentials by following the instructions from https://docs.chef.io/workstation/getting_started/
# Download and extract the StarterKit.zip from your
chef generate repo chef-repo
Starting to Work at Chef Working Directory (the chef-repo directory)
cd chef-repo/
Create a new cookbook named my_cookbook with the chef tool
chef generate cookbook cookbooks/my_cookbook
Create a new cookbook named my_cookbook with the knife tool
knife generate cookbook my_cookbook
Upload a cookbook to Chef Server
knife cookbook upload my_cookbook
Download a cookbook previously uploaded to a Chef Server or to a Hosted Server
knife cookbook download my_cookbook
Delete a cookbook from Chef Server
knife cookbook delete <cookbook name>
Add a cookbook to your node's runlist
knife node run_list add server 'recipe[my_cookbook]'
Remove a server from a node's runlist
knife node run_list delete server 'recipe[my_cookbook]'
Set a runlist or a recipe for a specific node
knife node run_list set test-node 'recipe[iptables]'
knife node run_list set <my-server> role[custom_role], recipe[my_cookbook]
Run the Chef client on your node to associate the host, add it as a node, and bootstrap it to the Chef Server or Hosted Chef
sudo chef-client
Show the list of the nodes that has been bootstrapped to Chef Server or to Hosted Chef
(hosts with the chef-client
installed and bootstrapped)
knife node show server
Show the list of the chef-clients association of the hosts that has been bootstrapped to Chef Server or to Hosted Chef with the chef-client
agent.
knife client list
Test a cookbook with Test Kitchen
cat << EOF >> .kitchen.yml
---
driver:
name: vagrant
provisioner:
name: chef_zero
#verifier:
# name: inspec
platforms:
- name: ubuntu-18.04
suites:
- name: default
run_list:
- recipe[my_cookbook::default]
attributes:
EOF
kitchen verify
kitchen test
Search the Marketplace for Cookbooks
knife supermarket search iptables
knife supermarket search ntp
knife supermarket search mysql
Shows details about the Cookbook searched in the Supermarket
knife supermarket show iptables
knife supermarket show ntp
knife supermarket show mysql
Download a Cookbook from the Marketplace
knife supermarket download iptables
tar zxvf iptables.tar.gz
Upload a cookbook on your Chef server
knife cookbook upload iptables --include-dependencies
Find the current version of a cookbook
knife cookbook show iptables
Inspect a cookbook syntax with Cookstyle tool (syntax and style-cheking tool for Ruby and cookbooks)
cookstyle cookbooks/my_cookbook
cookstyle -C true cookbooks/my_cookbook
Downloads all the dependencies you defined recursively with Berkshelf and upload the cookbook
cat << EOF >>> cookbooks/my_cookbook/metadata.rb
depends 'chef-client'
depends 'apt'
depends 'ntp'
EOF
cat << EOF >>> cookbooks/my_cookbook/recipes/default.rb
include_recipe 'chef-client'
include_recipe 'apt'
include_recipe 'ntp'
EOF
cd cookbooks/my_cookbook
chef generate cookbook -b ./
berks install
berks upload
Integrate Berkshelf with Vagrant to install and uploads all the required cookbooks on your Chef server whenever you execute vagrant up or vagrant provision
cat << EOF >> Vagrantfile
# -*- mode: ruby -*-
# vi: set ft=ruby :
box_image = 'ubuntu/bionic64'
your_org = 'https://api.chef.io/organizations/mydatacenter'
your_user_key = '.chef/dx-zone.pem'
your_user = 'dx-zone'
node_name = 'server'
Vagrant.configure("2") do |config|
config.vm.box = box_image
config.vm.box_url = 'http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-16.04_chef-provisionerless.box'
config.omnibus.chef_version = :latest
config.berkshelf.enabled = true
config.vm.provision :chef_client do |chef|
chef.provisioning_path = '/etc/chef'
chef.chef_server_url = your_org
chef.validation_key_path = your_user_key
chef.validation_client_name = your_user
chef.node_name = node_name
end
end
EOF
cat << EOF >> Berksfile
source 'https://supermarket.chef.io'
cookbook 'my_cookbook', path: 'cookbooks/my_cookbook'
EOF
vagrant up
vagrant reconfigure
List the knife plugins that are sipped as Ruby gems using the chef command-line tool
chef gem search -r knife-
Install EC2 plugin to manage servers in the Amazon AWS Cloud
chef gem install knife-ec2
List all the available instance types in AWS using the knife ec2
plugin
knife ec2 flavor list --aws-access-key-id <YOUR_AWS_KEY_ID> --aws-secret-access-key <YOUR_AWS_ACCESS_KEY>
List the nodes (chef-clients) registered on a Chef Server
knife node list
Delete a node (chef client) registered on a Chef Server
knife node delete my_node
Delete the client object of a node (chef-client) registered on a Chef Server
knife client delete my_node
Delete both the client and the node from Chef Hosted/Server (I.E. node named server in this case)
knife node delete server -y && knife client delete server -y
Install a command-line tool that allows you to do both deletions, delete the node and the client object at once.
chef gem install knife-playground
Delete both, the node and the client object simultaneously with the knife-playground
tool
knife pg clientnode delete my_node
Run a cookbook and deploy a recipe with local mode (without a Chef Server or hosted Chef) and validate that the Chf client run creates the desired temporary file on you local workstation
chef generate cookbook cookbooks/my_local_cookbook
cat << EOF >>> cookbooks/my_local_cookbook/recipes/default.rb
file '/tmp/local_mode.txt' do
content 'creaed by chef client local mode'
action :create
end
EOF
chef-client --local-mode -o my_local_cookbook
cat /tmp/local_mode.txt
Run a cookbook locally on a chef client server/node without pulling the cookbook from the Chef Server (assuming the cookbook has been created locally in the chef client/node server).
chef-client --local-mode -o my_local_cookbook
Override a Chef server run-list and specify the role, cookbook, and/or recipe to be run from a host client directly. This is for situations were you need to bypass what has been assigned in a run_list for a node and execute a specific role, cookbook, and/or recipe outside the run_list from the host client directly. In other words, even when a node is not assigned to that run_list where that particular role, cookbook/recipe may reside to, there is a way that a node can call the Chef Infra Server and ask for specific role/cookbook/recipe to run (hence override a run_list and call to execute whatever role, cookbook/recipe has been created in other run_list in Chef Infra). This will skip any cookbook/recipe assigned to the node and will run the specified items (role, cookbook, recipe, etc.) for a single run where the following command is executed from:
chef-client -o 'recipe[cookbook::recipe]'
chef-client -o 'role[us-region-a],role[pci],role[base_os],recipe[my-cookbook::my-recipe]'
chef-client --override-runlist 'recipe[cookbook::recipe]'
Run chef-client
as dry run to find what will be changed in the node
chef-client -W
Using knife
in local mode just like running a cookbook with chef-client
in local mode.
knife node run_list add -z laptop 'recipe[my_cookbook]'
Working with cached cookbooks/recipes on a node.
After chef-client
runs, it will download and store the cookbooks and recipes locally on a node at /var/chef/cache/cookbooks
and then will execute them. This can be useful to troubleshoot and test cookbooks at a local level in the node when something either failed or needs to be tweak for troubleshooting purposes without affecting or changing the cookbooks/recipes already in the Chef Infra server.
ls /var/chef/cache/cookbooks/
vi /var/chef/cache/cookbooks/my_cookbook_01/default.rb
# Make some modifications as needed
# Use cached cookbooks without overwriting local differences from the Chef Infra Server.
chef-client --skip-cookbook-sync
Group nodes with similar configuration with roles.
Create a role and upload the role to Chef Server
cat << EOF >> my_cookbook/roles/web_servers.rb
name 'web_servers'
description 'This role contains nodes, which act as web servers'
run_list 'recipe[ntp]'
default_attributes 'ntp' => {
'ntpdate' => {
'disable' => true
}
}
knife role from file web_servers.rb
Upload a role to the Chef Server
knife role from file web_servers.rb
Assign/Append a role to an existing run_list for a system named server
knife node run_list add server 'role[web_servers]'
Overwrite a role assigned in a run_list for a system named server
knife node run_list set server 'role[web_servers]'
Remove a node from the role to a system named server
knife node run_list remove server 'role[web_servers]'
Creating a separated Chef environments for development named dev (but it could be for development, testing, and/or production) and list the environment variables
export EDITOR=$(which vi) # for vim
knife environment create dev
knife environment list
List the environment of the nodes
knife node list
knife node list -E dev
Change the environment of the node named server to dev using knife
knife node environment set server dev
Use specific cookbook version and override certain attributes for the environment
knife environment edit dev
{
"name": "dev",
"description": "",
"cookbook_versions": {
"ntp": "1.6.8"
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {
},
"override_attributes": {
"ntp": {
"servers": ["0.europe.pool.ntp.org", "1.europe.pool.ntp.org", "2.europe.pool.ntp.org", "3.europe.pool.ntp.org"]
}
}
}
Alternatively, you can create a new environment with with a new Ruby file in he environments diretory inside your Chef repository.
cat << EOF >> ./chef-repo/environments/dev.rb
{
"name": "dev",
"description": "",
"cookbook_versions": {
"ntp": "1.6.8"
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {
},
"override_attributes": {
"ntp": {
"servers": [
"0.europe.pool.ntp.org",
"1.europe.pool.ntp.org",
"2.europe.pool.ntp.org",
"3.europe.pool.ntp.org"
]
}
}
}
EOF
Create the environment on the Chef server from the newly created file using knife
knife environment from file dev.rb
Sync your local changes to your Chef server for both, knife and Berkshelf simultaneously
knife exec -E 'nodes.transform("chef_environment:_default") { |n| n.chef_environment("dev") }'
Search for nodes in a specific environment (dev in this example)
knife search node 'chef_environment:dev'
Freezing a cookbook version to avoid someone to overwrite the same version with a broken code
knife cookbook upload ntp --freeze
Force overwrite a cookbook that has been already frozen
knife cookbook upload ntp --freeze --force
berks upload --force
Start the Chef client in daemon mode so that it runs automatically every 30 minutes and validate it is running
sudo chef-client -i 1800
ps auxw | grep chef-client
You can use the chef-client
cookbook to install the Chef client as a service
knife supermarket download chef-client
tar zxvf chef-client.tar.gz
mv chef-client ./chef-repo/cookbooks && rm chef-client.tar.gz
knife cookbook upload chef-client
berks upload
Using Cronjob to run the Chef client as a daemon every 15 minutes
cat << EOF >> /etc/cron.d/chef_client
PATH=/usr/local/bin:/usr/bin:/bin
# m h dom mon dow user command
*/15 * * * * root chef-client -l warn | grep -v 'retrying [1234]/5 in'
EOF
Write unit tests to test fails a recipe and inspect it with Chef Spec
cat << EOF >> cookbooks/my_cookbook/spec/default_spec.rb
require 'chefspec'
describe 'my_cookbook::default' do
let(:chef_run) {
ChefSpec::ServerRunner.new(
platform:'ubuntu', version:'16.04'
).converge(described_recipe)
}
it 'creates a greetings file, containing the platform name' do
expect(chef_run).to render_file('/tmp/greeting.txt').with_content('Hello! ubuntu!')
end
end
EOF
chef exec rspec cookbooks/my_cookbook/spec/default_spec.rb
cat << EOF >> cookbooks/my_cookbook/recipes/default.rb
template '/tmp/greeting.txt' do
variables greeting: 'Hello!'
action :create
end
EOF
mkdir cookbooks/my_cookbook/templates
cat << EOF >> cookbooks/my_cookbook/templates/greeting.txt.erb
<%= @greeting %> <%= node['platform'] %>!
EOF
chef exec rspec cookbooks/my_cookbook/spec/default_spec.rb
Create a profile for compliance auditing and testing with Chef InSpec
# Create a new profile for your InSpec tests
inspec init profile my_profile
# Create a test ensuring that there is only one account called root with UID 0 in your /etc/passwd file
cat << EOF >> my_profile/controls/passwd.rb
describe passwd.uids(0) do
its('users') { should cmp 'root' }
its('entries.length') { should eq 1 }
end
EOF
# Run the test
inspec exec my_profile/controls/passwd.rb
Integration-testing Chef cookbooks and creating default attributes
# Edit your cookbook's default recipe
cat << EOF >> cookbooks/my_cookbook/recipes/default.rb
file "/tmp/greeting.txt" do
content node['my_cookbook']['greeting'] # Resolve to 'content "Ohai, Chefs!"' as defined in the attributes/default.rb
end
EOF
# Edit your cookbook's default attributes
mkdir -p cookbooks/my_cookbook/attributes
cat << EOF >> cookbooks/my_cookbook/attributes/default.rb
default['my_cookbook']['greeting'] = "Ohai, Chefs!"
EOF
# Change to your cookbook directory
cd cookbooks/my_cookbook
# Edit the default Test Kitchen configuration file to only test against Ubuntu 18.04
cat << EOF >> ./chef-repo/.kitchen.yml
platforms:
- name: ubuntu-18.04
# - name: ubuntu-20.04
# - name: centos-8
EOF
# Create your test, defining what you expect your cookbook to do:
mkdir test/recipes/
cat << EOF >> test/recipes/default_test.rb
describe file('/tmp/greeting.txt') do
its('content') { should match 'Ohai, Chefs!' }
end
EOF
# Run Test Kitchen
kitchen test
# Alternative, Test Kitchen against a particular platform/version
kitchen test default-ubuntu-20.04
kitchen test 20
# List the status of the various VMs managed by Test Kitchen
kitchen list
Pre-Flight checks before executing a cookbook to find which nodes will be affected and how
# PENDING TO COMPLETE THIS PART: Showing affected nodes before uploading cookbooks
chef gem install knife-preflight
knife preflight ntp
knife search node recipes:ntp -a name
knife search node roles:ntp -a name
Try parts of a cookbook interactively with chef-shell
chef-shell