#Packaging
This is a repository for packaging automation for Puppet software. The goal is to abstract and automate packaging processes beyond individual software projects to a level where this repo can be cloned inside any project and used to build Debian and Redhat packages, as well as gems, apple packages and tarballs. This repo is currently under heavy development and in a state flux, and it should not be considered to have a formal API. However, every effort is being made to ensure existing tasks/behavior are not broken as we continue to iterate and improve upon it.
##Using the Packaging Repo
Several Puppet projects are using the packaging repo. They are:
- puppet
- facter
- puppet-dashboard
- hiera
- puppetdb
- razor
as well as several closed-source projects, including
- live-management
- console-auth
- console
Generally speaking, the packaging repo should be compatible with ruby 1.8.7,
ruby 1.9.3 and rake 0.9.x. To pull the packaging tasks into your source repo,
do a rake package:bootstrap
. This will clone this repo into the ext directory
of the project and make many packaging tasks available. The tasks are
generally grouped into two categories, package:
namespaced tasks and pl:
namespaced tasks.
package:
namespaced tasks are general purpose tasks that are set up to use
the most minimal tool chain possible for creating packages. These tasks will
create rpms and debs, but any build dependencies will need to be satisifed by
the building host, and any dynamically generated dependencies may result in
packages that are only suitable for the OS/version of the build host. However,
for rolling one's own debs and rpms or for use in environments without many
OSes/versions, this may work just fine. To build an rpm using the packaging
repo, do a rake package:rpm
. To build a deb, use rake package:deb
.
pl:
namespaced tasks rely on a slighly more complex toolchain for packaging
inside clean chroot environments for the various operating systems and versions
that Puppet supports. On the rpm side, this is done with
mock and for debs, we use
pdebuild and cowbuilder. For the most
part, these tasks are keyed to puppet infrastructure, and are used by the
Release Engineering team to create release packages. However, they can
certainly be modified to suit other environments, and much effort went into
making tasks as modular and reusable as possible. Several Puppet-specific
tasks are only available if the file '~/.packaging' is present. This file is
created by the pl:fetch
task, which curls two yaml files into 'team' and
'project' subdirectories from a separate build data
repository, which contains
additional settings/data specific to Puppet release infrastructure. By
default, the team data file is pulled from the 'dev' branch of the repo, and
the project data file is pulled from a branch named after the project (e.g. for
puppet, there is a branch named puppet with a build data file). The goal in
separating these data and tasks out is to refrain from presenting by
default yet more Puppet-specific tasks that aren't generally consumable by
everyone. To build a deb from a local repository using a pl
task, ssh into a
builder (e.g., one stood up using the modules detailed below) and clone the
source repo, e.g. puppet. Then, run rake package:bootstrap
and rake pl:deb
to create a deb, and rake pl:mock
to make an rpm (on a debian or redhat host,
respectively).
There is also a pe:
namespace, for the building of Puppet
Labs' Puppet Enterprise packages that have been converted to using this repo.
The pe:
tasks rely heavily on PL internal infrastructure, and are not
generally useful outside of this environment. To create packages, in the source
repository run rake package:bootstrap
, followed by rake pl:fetch
. These two
commands bootstrap the packaging environment and pull in the additional data
needed for PE building (see pl:fetch
notes above). Then, to make a debian
package, run rake pe:deb
, and to make an rpm, run rake pe:mock
. There are
also pe:deb_all
and pe:mock_all
tasks, which build packages against all
shipped debian/redhat targets. The pe:deb_all
task is not generally necessary
for developer use for building test packages; the pe:deb
task creates a
package that will work against virtually all supported PE debian versions.
The same is generally true for PE internal rpms, but because of variances in
build macros for rpm, rpms should generally be built with pe:mock_all
, and
then the desired version installed, or by building only for a specific
target. This is accomplished by passing MOCK= to the rake call, e.g.
rake pe:mock MOCK=<mock>
. The available mocks are listed in
ext/build_defaults.yaml
after final_mocks:
. For PE, the mocks are
formatted as pupent-<peversion>-<distversion>-<arch>
, e.g.
pupent-2.7-el5-i386
. To build for a specific target, set MOCK=<mock>
to
the mock that matches the target.
There are also sub-namespaces of :pl
and :pe
that are
worth noting. First, the :remote
namespace. Tasks under :remote
perform
builds remotely on internal builders from your local workstation. How they
work:
-
Run
pl:fetch
to obtain extra data from the build-data repo. The data includes the hostnames of builders to use for packaging. -
Create a git bundle of the local workspace and tar it up.
-
Create a build parameters file. The params file includes all the information about the build, including any values overridden with env vars, and the actual task to run, e.g.
rake pl:deb
. -
scp the git bundle and build parameters file to a temporary directory on the builder hostname assigned to that particular package build type.
-
ssh into the builder, untar the git bundle, clone it, and run
rake package:bootstrap
. -
ssh into the builder, cd into the cloned repo, and run
rake pl:build_from_params PARAMS_FILE=/path/to/previously/sent/file
. -
Maintain the ssh connection until the build finishes, and rsync the packages from the builder to the local workstation.
Note that these tasks require ssh access to the builder hosts that are
specified in the build-data file, and appropriate membership in the build
groups, e.g. to use mock on the builder, membership in the mock group. This is
a major hurdle, and is resolved with the jenkins
tasks below.
(Deprecated - see "dyamic jenkins task workflow" below)
Jenkins tasks are similar to the :remote:
tasks, but they do not require ssh
access to the builders. They do require being on the local network - the
jenkins instance that performs package builds is an internal server only,
accessible when connected via VPN or on-site. The jenkins tasks enable the
packaging repo to kick off packaging builds on a remote jenkins slave.
There are two workflows of jenkins tasks in the packaging repo. The first
workflow, which is used for creating individual platform packages on jenkins
(e.g. creating a deb with pl:jenkins:deb) relies on a job that exists on a
remote jenkins server. The tasks transmit information to the jenkins job, which
handles the rest. The data passed are the following:
-
$PROJECT_BUNDLE - a tar.gz of a git-bundle from HEAD of the current project, which is cloned on the builder to set up a duplicate of this environment
-
$BUILD_PROPERTIES - a build parameters file, containing all information about the build
-
$BUILD_TYPE - the "type" of build, e.g. rpm, deb, gem, etc The jenkins url and job name are obtained via the team build-data file from the build data repository
-
$PROJECT - the project we're building, e.g. facter, puppet. This is used later in determining the target for the build artifacts on the distribution server
-
$METRICS - a string of data points related to the build which are used for data analysis. Contents of this string are items which cannot be obtained from within the Jenkins job itself.
-
$DOWNSTREAM_JOB - The URL of a downstream job that jenkins should post to upon success. This is obtained via the DOWNSTREAM_JOB environment variable.
On the Jenkins end, the job is a parameterized job that accepts five parameters. Jenkins has the Parameterized Trigger Plugin, Workspace Cleanup Plugin, and Node and Label Parameter Plugin in use for this job. The workspace cleanup plugin cleans the workspace before each build. Two are file parameters, two string parameters, and a Label parameter provided by the Node and Label Parameter Plugin, as described above. When the pl:jenkins:* task triggers a build, it passes values for all of these parameters. The Label parameter is associated with the build type. This way we can queue the job on a builder with the appropriate capabilities just by assigning a builder the label "deb" or "rpm," etc. The job allows parallel execution of jobs - in this way, we can queue many package jobs on the jenkins instance, which will farm them out to builders that are slaves of that jenkins instance. This also allows us to scale building capacity simply by adding builders as slaves to the jenkins instance. The actual build itself is accomplished via a shell build task. The contents of the task are:
#################
SHA=$(echo $BUILD_PROPERTIES | cut -d '.' -f1)
echo "Build type: $BUILD_TYPE"
### Create a local clone of the git-bundle that was passed
# The bundle is a tarball, and since this is a project-agnostic
# job, we don't actually know what's in it, just that it's a
# git bundle.
[ -f "PROJECT_BUNDLE" ] || exit 1
mkdir project && tar -xzf PROJECT_BUNDLE -C project/
cd project
git clone --recursive $(ls) git_repo
cd git_repo
### Clone the packaging repo
rake package:bootstrap && rake pl:fetch
### Perform the build
rake pl:load_extras pl:build_from_params PARAMS_FILE=$WORKSPACE/BUILD_PROPERTIES
### Send the results
rake pl:jenkins:ship["artifacts"]
### If a downstream job was passed, trigger it now
if [ -n "$DOWNSTREAM_JOB" ] ; then
rake pl:jenkins:post["$DOWNSTREAM_JOB"]
fi
#################
To gather metrics related to a Jenkins build, the Groovy Postbuild plugin is used. For tasks carried out on the static Jenkins job, the script must be manually added to the job's configuration. The script in its entirety can be seen here.
The recommended and far simpler jenkins-based workflow is for initiating the "uber_build", or a package build for all of our target platforms.
The uber_build is invoked as "pl:jenkins:uber_build" or "pe:jenkins:uber_build" depending on if this is a FOSS or PE package.
This workflow doesn't actually use a static job on the jenkins-server. Instead it creates the jenkins jobs for you, on-demand. Specifically, it creates two jenkins-jobs, and can create an optional third.
The first job is a matrix job, the cells of which are individual package tasks for all of the build targets. This job takes four parameters:
-
$PROJECT_BUNDLE - a tar.gz of a git-bundle from HEAD of the current project, which is cloned on the builder to set up a duplicate of this environment
-
$BUILD_PROPERTIES - a build parameters file, containing all information about the build
-
$PROJECT - the project we're building, e.g. facter, puppet. This is used later in determining the target for the build artifacts on the distribution server
-
$METRICS - a string of data points related to the build which are used for data analysis. Contents of this string are items which cannot be obtained from within the Jenkins job itself. Note that the Groovy postbuild script needed for metrics gathering is dynamically passed to each job.
This first job clones the git bundle passed in as a parameter, then clones the packaging repo (rake package:bootstrap) and for every cell in its matrix performs a package build for a specific target (e.g. rake pl:deb COW=base-lucid-i386.cow). Once all cells in the matrix complete (either succeed or fail), this job automatically triggers the second of the new jobs as a downstream job.
To receive an email notification from jenkins about the status of the packaging job, pass NOTIFY= as an environment variable to the uber_build invocation, e.g.:
rake pl:jenkins:uber_build NOTIFY="foo@puppet.com bar@puppet.com"
The second job is an automatic repository creation task for this git repo.
Specifically, the job copies the git bundle from the packaging job and clones
it, and uses the git information in the git bundle to clone the packaging repo
and invoke the repository creation jobs pl:jenkins:rpm_repos
and
pl:jenkins:deb_repos
. The job will always be invoked, but will only actually
create repos if the upstream packaging job actually succeeded.
The third job is only created if the environment variable
DOWNSTREAM_JOB=<job_url>
was passed to the initial "pl:jenkins:uber_build"
invocation. This third job takes the value assigned to DOWNSTREAM_JOB
and
creates a proxy jenkins job with a single build step, a curl call to this
value, presumably a url to a jenkins job to trigger programmatically.
An important note about DOWNSTREAM_JOB
: DOWNSTREAM_JOB
in the dynamic jenkins
workflow is always invoked if it is passed in as an environment variable.
However, it is appended with an additional parameter, PACKAGE_BUILD_STATUS
,
which will be the string "success" if package and repo builds succeeded, or
"failure" if package or repo builds failed. By modifying the actual downstream
jenkins job to accept a string parameter of PACKAGE_BUILD_STATUS
, one can
switch on the success or failure of the packaging job, responding
appropriately. A second parameter, PACKAGE_BUILD_URL
is also appended to
DOWNSTREAM_JOB
, the value of which is the url of the packaging job itself.
This is to assist with tracing failures in a multi-jenkins environment. By
modifying the downstream jenkins job to accept a string parameter of
PACKAGE_BUILD_URL
, one can use the value to display the url prominently in
case of failure, for example.
E.g., a job url: http://jenkins.example.net/job/downstream/buildWithParameters?FOO=bar
in the success case will be transformed into
and in the failure case transformed into
Since jenkins will just drop parameters that are not configured in the job, accepting PACKAGE_BUILD_STATUS and PACKAGE_BUILD_URL in the downstream job isn't mandatory.
All 3 jobs are configured by default for removal by jenkins after 3 days, to avoid clutter.
The goal is to move toward migrating all of the jenkins tasks from the first workflow, using a static job that is called many times per package, to this second workflow of creating the jobs on demand.
It is possible to pass an integer argument to the uber_build
task which will be
used by the job as polling interval. If this value is passed in, the task will
query the dynamic job using the Jenkins API periodically until the build is
finished. Then, it will query the build to determine the SUCCESS/FAILURE status.
The job output will look something like:
Packaging SUCCESS
Repo SUCCESS
If for example the Packaging job had failed the output would look like this:
Packaging FAILURE
If a build fails then the rake task will terminate with a nonzero exit status
which can be used during CI or other automated contexts to detect and act on the
failure. This eliminates the need to pass a DOWNSTREAM_JOB
variable to the
uber_build job although it is still possible to do so.
For a listing of all available tasks and their functions, see the Task Dictionary at the end of this README.
A puppet module, puppetlabs-debbuilder, has been created to stand up a debian build host compatible with the debian side of this packaging repo. The rpm-side module, puppetlabs-rpmbuilder, will set up an rpm builder.
To remove the packaging repo, remove the ext/packaging directory or run rake package:implode
.
##Setting up projects for the Packaging Repo
The packaging repo requires many project-side artifacts inside the ext directory at the top level. facter and hiera are good examples. It expects the following directory structure in the project
- ext/{debian,redhat,osx}
each of which contains templated erb files using the instance variables specified in the setupvars task. These include a debian changelog, a redhat spec file, and an osx preflight and plist.
The top level Rakefile or a separate task file in the project should have the following added:
(this assumes RAKE_ROOT is defined in the top-level Rakefile using something like the following)
RAKE_ROOT = File.expand_path(File.dirname(__FILE__))
build_defs_file = File.join(RAKE_ROOT, 'ext', 'build_defaults.yaml')
if File.exist?(build_defs_file)
begin
require 'yaml'
@build_defaults ||= YAML.load_file(build_defs_file)
rescue Exception => e
$stderr.puts "Unable to load yaml from #{build_defs_file}:"
raise e
end
@packaging_url = @build_defaults['packaging_url']
@packaging_repo = @build_defaults['packaging_repo']
raise "Could not find packaging url in #{build_defs_file}" if @packaging_url.nil?
raise "Could not find packaging repo in #{build_defs_file}" if @packaging_repo.nil?
namespace :package do
desc "Bootstrap packaging automation, e.g. clone into packaging repo"
task :bootstrap do
if File.exist?(File.join(RAKE_ROOT, "ext", @packaging_repo))
puts "It looks like you already have ext/#{@packaging_repo}. If you don't like it, blow it away with package:implode."
else
cd File.join(RAKE_ROOT, 'ext') do
%x{git clone #{@packaging_url}}
end
end
end
desc "Remove all cloned packaging automation"
task :implode do
rm_rf File.join(RAKE_ROOT, "ext", @packaging_repo)
end
end
end
begin
load File.join(RAKE_ROOT, 'ext', 'packaging', 'packaging.rake')
rescue LoadError
end
Also in ext should be two files, build_defaults.yaml and project_data.yaml.
This is the sample build_defaults.yaml file from Hiera:
---
packaging_url: 'git@github.com:puppetlabs/packaging --branch=master'
packaging_repo: 'packaging'
default_cow: 'base-squeeze-i386.cow'
# Which debian distributions to build for. Noarch packages only need one arch of each cow.
cows: 'base-lucid-amd64.cow base-lucid-i386.cow base-natty-amd64.cow base-natty-i386.cow base-oneiric-amd64.cow base-oneiric-i386.cow base-precise-amd64.cow base-precise-i386.cow base-sid-amd64.cow base-sid-i386.cow base-squeeze-amd64.cow base-squeeze-i386.cow base-testing-amd64.cow base-testing-i386.cow base-wheezy-i386.cow'
# The pbuilder configuration file to use
pbuild_conf: '/etc/pbuilderrc'
# Alternate debian mirrors to build against (must be an array)
# The __DIST__ string is automatically replaced with codename of the cow being built, so when the squeeze cow is being built the deb_build_mirrors will be:
# deb http://apt.puppetlabs.com squeeze main dependencies
# deb http://somethingelse.com/debian squeeze
# This will happen for each cow during the build.
deb_build_mirrors:
- deb http://apt.puppetlabs.com __DIST__ main dependencies
- deb http://somethingelse.com/debian __DIST__
# Who is packaging. Turns up in various packaging artifacts
packager: 'puppetlabs'
# GPG key ID of the signer
gpg_key: '7F438280EF8D349F'
# Whether to require tarball signing as a prerequisite of other package building
sign_tar: FALSE
# a space separated list of mock configs. These are the rpm distributions to package for. If a noarch package, only one arch of each is needed.
final_mocks: 'pl-el-5-i386 pl-el-5-x86_64 pl-el-6-i386 pl-el-6-x86_64 pl-fedora-16-i386 pl-fedora-16-x86_64 pl-fedora-17-i386 pl-fedora-17-x86_64'
# The host that contains the yum repository to ship to
yum_host: 'yum.puppetlabs.com'
# The remote path the repository on the yum\_host
yum_repo_path: '/some/repo/'
# The host that contains the apt repository to ship to
apt_host: 'apt.puppetlabs.com'
# The URL to use for the apt dependencies in cow building
apt_repo_url: 'http://apt.puppetlabs.com'
# The path on the remote apt host that debs should ship to
apt_repo_path: '/opt/repository/incoming'
# The host that stores the tarballs for downloading
tar_host: 'downloads.puppetlabs.com'
# Whether to present the gem and apple tasks
build_gem: TRUE
build_dmg: TRUE
# Whether to execute the rdoc rake tasks prior to composing the tarball
build_doc: FALSE
# Whether to kick of a dynamic msi build job along side the uber_build
# If present, a dynamically generated jenkins job will be kicked off.
# The automation in puppet_for_the_win is used to build the msi with the
# following components.
build_msi:
puppet_for_the_win:
ref: 'origin/master'
repo: 'git://github.com/puppetlabs/puppet_for_the_win.git'
facter:
ref: 'refs/tags/2.1.0'
repo: 'git://github.com/puppetlabs/facter.git'
hiera:
ref: 'refs/tags/1.3.4'
repo: 'git://github.com/puppetlabs/hiera.git'
sys:
ref:
x86: 'origin/1.9.3-x86'
x64: 'origin/2.0.0-x64'
repo: 'git://github.com/puppetlabs/puppet-win32-ruby.git'
# Whether to present the Solaris 11 IPS packaging tasks
# This requires suitable IPS packaging artifacts in the project in ext/ips
build_ips: FALSE
# Whether this project is a PE project or not
build_pe: FALSE
# An optional task to execute pre-tarball composition. See the tasks in
# the 'pretasks' directory
pre_tar_task: 'package:vendor_gems'
This is the sample project_data.yaml file:
---
project: 'hiera'
author: 'Puppet'
email: 'info@puppet.com'
homepage: 'https://github.com/puppetlabs/hiera'
summary: 'Light weight hierarchical data store'
description: 'A pluggable data store for hierarcical data'
# file containing hard coded version information, if present
version_file: '/lib/hiera.rb'
# A string indicating the version strategy for the project (one of 'odd_even' or 'rc_final'), defaults to rc_final
# odd_even is a final release when the minor version is even, and a development release when it is odd
# rc_final is a final release when there is no rc at the end of the version string, and a development release when there is
version_strategy: 'rc_final'
# Boolean value of whether or not to automatically update the version file before packaging (defaults to false)
update_version_file: true
# files and gem\_files are space separated lists
# files to be packaged into a tarball and released with deb/rpm
files: '[A-Z]* ext lib bin spec acceptance_tests'
# space separated list of files to *exclude* from the tarball
# note that each listing in files, above, is recursively copied into the tarball, so
# 'tar\_excludes' only needs to include any undesired subdirectories/files of the 'files'
# list to exclude
tar_excludes: 'ext/packaging lib/some_excluded_file'
# Array of templates or globs of templates to evaluate. Note that without this key, the packaging will
# default to searching for any files in `ext` with the extension '.erb' and evaluate them. When this
# key is supplied, its values override the defaults, and all desired erb files must be specified with a path or glob.
templates:
- ext/redhat/project.spec.erb
- ext/templates/**/*.erb
# files to be packaged into a gem
gem_files: '{bin,lib}/**/* CHANGELOG COPYING README.md LICENSE'
# To exclude specific files from inclusion in a gem:
gem_excludes: 'lib/hiera/file_to_exclude.rb lib/hiera/directory_to_exclude'
# If gem name differs in some way from project name, e.g. only build gem for a client end
gem_name: hiera_the_gem
# If gem summary and/or description differs from general summary
gem_summary: 'A sub-set of the Hiera pluggable data store'
gem_description: 'A gem of the pluggable data store for hierarchical data'
gem_require_path: 'lib'
gem_test_files: 'spec/**/*'
gem_executables: 'hiera'
gem_default_executables: 'hiera'
# To add gem dependencies, indent.
# This is an example only, hiera doesn't really depend on hiera-puppet/json/facter
# For no specific version, leave version empty
gem_runtime_dependencies:
hiera-puppet: '1.0.0rc'
hiera-json:
gem_development_dependencies:
facter: '>= 1.6.11'
# To add gem dependencies which only apply to a specific platform, add the key "gem_platform_dependencies".
# The first key under the gem_platform_dependencies has to be a value that
# corresponds to a value of RUBY_PLATFORM. The subkeys are the same as the
# top-level gem dependency keys:
gem_platform_dependencies:
x86-mingw32:
gem_runtime_dependencies:
win32process: '~> 0.6.5'
gem_development_dependencies:
rake: '~> 0.9.0'
x86_64-darwin:
gem_runtime_dependencies:
CFPropertyList: '~> 2.2.4'
# rdoc options as an array
gem_rdoc_options:
- --line-numbers
- --main
- Hiera.README
For basic mac packaging, add an osx directory in ext containing the following files:
- a preflight.erb template for any pre-flight actions, perhaps removing the old package if present.
- a prototype.plist.erb that is templated into the pkginfo.plist file for the package. This is the one from puppet. Note that these variable names aren't mutable here, but there's no need to worry about their value assignment, it's done in the apple task:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>CFBundleIdentifier</key>
<string><%= @title %></string>
<key>CFBundleShortVersionString</key>
<string><%= @version %></string>
<key>IFMajorVersion</key>
<integer><%= @package_major_version %></integer>
<key>IFMinorVersion</key>
<integer><%= @package_minor_version %></integer>
<key>IFPkgBuildDate</key>
<date><%= @build_date %></date>
<key>IFPkgFlagAllowBackRev</key>
<false/>
<key>IFPkgFlagAuthorizationAction</key>
<string>RootAuthorization</string>
<key>IFPkgFlagDefaultLocation</key>
<string>/</string>
<key>IFPkgFlagFollowLinks</key>
<true/>
<key>IFPkgFlagInstallFat</key>
<false/>
<key>IFPkgFlagIsRequired</key>
<false/>
<key>IFPkgFlagOverwritePermissions</key>
<false/>
<key>IFPkgFlagRelocatable</key>
<false/>
<key>IFPkgFlagRestartAction</key>
<string><%= @pm_restart %></string>
<key>IFPkgFlagRootVolumeOnly</key>
<true/>
<key>IFPkgFlagUpdateInstalledLanguages</key>
<false/>
</dict>
</plist>
A file_mapping.yaml file that specifies a set of files and a set of directories from the source to install, with destinations, ownership, and permissions. The directories are top level directories in the source to install. The files are files somewhere in the source to install. This is the one from puppet 3.x:
---
directories:
# this will take the contents of lib, e.g. puppet/lib/\* and place them in
# /usr/lib/ruby/site\_ruby/1.8
lib:
path: 'usr/lib/ruby/site_ruby/1.8'
owner: 'root'
group: 'wheel'
perms: '0644'
bin:
path: 'usr/bin'
owner: 'root'
group: 'wheel'
perms: '0755'
'man/man8':
path: 'usr/share/man/man8'
owner: 'root'
group: 'wheel'
perms: '0755'
files:
# this will take the file puppet/conf/auth.conf and place it in
# /private/etc/puppet/, creating the directory if not present
'conf/auth.conf':
path: 'private/etc/puppet'
owner: 'root'
group: 'wheel'
perms: '0644'
'man/man5/puppet.conf.5':
path: 'usr/share/man/man5'
owner: 'root'
group: 'wheel'
perms: '0644'
'[A-Z]*':
path: 'usr/share/doc/puppet'
owner: 'root'
group: 'wheel'
perms: '0644'
-
package:apple
Use
PackageMaker
to create a pkg package inside a dmg. Requires 'sudo' privileges.build_dmg: TRUE
must be set inext/build_defaults.yaml
. Packages are staged in ./pkg/apple. See the Mac packaging section of Setting up projects for the Packaging Repo. -
package:bootstrap
Clone the packaging repo into this project. This task isn't actually in the packaging repo itself, but resides in the project. See Setting up projects for the Packaging Repo.
-
package:deb
Use
debbuild
to create a deb package and associated debian package artifacts from the repository. Requires all build dependencies be satisfied locally. Packages are staged in ./pkg/deb. -
package:gem Use the
rubygems/package_task
library to create a rubygem from the repository. Requiresbuild_gem: TRUE
and gem-related parameters be set inext/build_defaults.yaml
andext/project\_data.yaml
. The gem is staged in./pkg
. -
package:implode
Remove the packaging repo entirely from the project. This task isn't actually in the packaging repo itself, but resides in the project. See Setting up projects for the Packaging Repo.
-
package:ips
Use Solaris 11 pkg* tools to create a IPS package from the project. Packages are staged in
./pkg/ips/pkgs
. Requires allpkg
,pkgdepend
,pkgsend
,pkglint
, andpkgmogrify
. Currently only puppet, facter, and hiera have this capability. -
package:rpm
Use
rpmbuild
to create an rpm of the project. This will also make a source rpm. Requires all build dependencies by satisfied locally. Packages are staged in./pkg/rpm
. -
package:srpm
Use
rpmbuild
to create a source rpm of the project. Source rpm is staged in./pkg/srpm
. -
package:tar
Create a source tarball of the project. The tarball is staged in
./pkg
. -
package:update
Update the clone of the packaging repo by pulling from origin.
-
pl:build_from_params
Invoke a build from a build parameters yaml file. The parameters file should be created with
rake pl:write_build_params
. The settings in the build parameters file will override all values contained in./ext/build_defaults.yaml
and./ext/project_data.yaml
. -
pl:deb
Use pdebuild with cowbuilder to create a debian package and associated source artifacts from the default "cow", currently Debian Squeeze i386. Requires that pbuilder/cowbuilder be installed and set up with a Debian Squeeze cow. See the puppetlabs-debbuilder module for an easy way to set up a host for building with cows. The deb and source artifacts are staged in
./pkg/deb/squeeze
. -
pl:deb_all*
The same as
rake pl:deb
, but a package is built for every cow listed inext/build_defaults.yaml
on the linecows:<cows>
. The packages and associated source artifacts are staged in./pkg/deb/$distribution
, where $distribution is the Debian/Ubuntu codename of the cow that was used to build the package, e.g. "wheezy" or "precise." -
pl:ips
Invoke package:ips, but do so after pl:fetch and pl:load_extras, which load signing/certificate information. The resulting packages are signed. IPS packages are staged in
./pkg/ips/pkgs
-
pl:jenkins:deb
Perform
pl:deb
by posting a jenkins build, as described above. Once the build is complete, runpl:jenkins:retrieve
to retrieve the built packages. -
pl:jenkins:deb_all
Perform what is a effectively a
pl:deb_all
but in a different fashion.pl:deb_all
performs debian cow builds in serial with every cow listed in ext/build_defaults.yaml.pl:jenkins:deb_all
splits the cows listed, and posts a separatepl:jenkins:deb
job for every cow listed to the jenkins server, allowing jenkins to parallelize the building of packages for every cow. Executepl:jenkins:retrieve
to retrieve all packages. -
pl:jenkins:deb_repo_configs
On the distribution server generate a listing of all debian repositories that exist for the current SHA/ref of HEAD of the project repository. Then generate debian apt client configuration files for every existing repository. The distribution server is a web server, so the client configurations can be placed on a debian client in /etc/apt/sources.list.d/ and the client will be able to install the built packages via apt. Requires SSH access to the distribution server.
-
pl:jenkins:deb_repos
On the distribution server, generate debian apt repositories for every distribution containing any packages that are stored that match the current SHA/ref of HEAD of the project repository. Requires SSH access to the distribution server.
-
pl:jenkins:dmg
Perform
package:apple
by posting a jenkins build. Runpl:jenkins:retrieve
to retrieve the built packages. -
pl:jenkins:gem
Perform
package:gem
by posting a jenkins build. Runpl:jenkins:retrieve
to retrieve the built packages. -
pl:jenkins:mock
Perform
pl:mock
by posting a jenkins build. Runpl:jenkins:retrieve
to retrieve the built packages. -
pl:jenkins:mock_all
Perform what is effectively a
pl:mock_all
but in a different fashion.pl:mock_all
performs mock builds in serial with every mock listed in ext/build_defaults.yaml.pl:jenkins:mock_all
splits the mocks listed, and posts a separatepl:jenkins:mock
job for every mock to the jenkins server, allowing jenkins to parallelize the building of packages for every mock configuration. The mock build root is randomized by the packaging repo, avoiding conflicts with existing builds of the same mock configuration. To retrieve built packages, callpl:jenkins:retrieve
. -
pl:jenkins:post[uri]
Post to the jenkins server as specified in the team build_extras.yaml file, with the job uri specified.
-
pl:jenkins:retrieve[target]
Retrieve packages stored on the distribution server that have been built from the current SHA/ref of HEAD of the project repository. Optionally pass [target] to override the default, which is to retrieve the contents of the "artifacts" subdirectory. Other targets are "repos" and "shipped".
-
pl:jenkins:rpm_repo_configs
On the distribution server generate a listing of all yum rpm package repositories that exist for the current SHA/ref of HEAD of the project repository. Then generate yum client configuration files for every existing repository. The distribution server is a web server, so the client configurations can be placed on a redhat client in /etc/yum.repos.d/ and the client will be able to install the packages via yum install. Requires SSH access to the distribution server.
-
pl:jenkins:rpm_repos
On the distribution server, generate yum rpm repositories for every distribution containing any packages that are stored that match the current SHA/ref of HEAD of the project repository. Requires SSH access to the distribution server. The yum repos are created in a "repos" subdirectory of the standard builds location, e.g. /opt/jenkins-builds/$project/${SHA|ref}/repos, using everything currently in the "artifacts" subdirectory of the same location.
-
pl:jenkins:ship[target]
Take the packages staged in pkg/ and ship them to locations partially specified by data in the project build_extras.yaml file. The current paradigm is to ship the files to a subdirectory of /opt/jenkins-builds on the distribution server. The subdirectory is constructed with the project and SHA or ref of HEAD of the project repository. That is, if project HEAD is currently at the tag "1.2.3", then the directory that packages will be shipped to is /opt/jenkins-builds/$project/1.2.3/. If HEAD is a git SHA, then "1.2.3" will instead be that SHA. By default, all artifacts in pkg/ will be shipped to a "artifacts" subdirectory of the standard target. E.g. /opt/jenkins-builds/$project/1.2.3/artifacts. When a final shipping occurs, e.g. when shipping signed artifacts into production, a second subdirectory is created - "shipped" - and all artifacts that are shipped to production are also shipped here. This allows a historical archive of all shipped artifacts.
-
pl:jenkins:sign_all
Take all packages staged in pkg/ and sign them via the various signing tasks. All signing occurs on the distribution server:
- create a git-bundle of the project and rsync it to the distribution
- server ssh to the distribution server and clone the git-bundle, and clone
- the packaging repository rsync the contents of the local pkg/ directory
- into the pkg/ directory of the remote git project ssh to the distribution
- server and execute the following rake tasks:
- pl:sign_tar
- pl:sign_rpms
- pl:sign_deb_changes
- rsync the remote pkg/ directory contents to the local pkg/ directory
-
pl:jenkins:tar
Perform
package:tar
by posting a jenkins build. Runpl:jenkins:retrieve
to retrieve the built packages. -
pl:jenkins:uber_build
Create a jenkins job on the fly that performs an aggregate of build tasks. These include all the debian builds using
pl:deb COW=<cow>
, rpm builds withpl:mock MOCKS=<mock>
,package:tar
,package:apple
, andpackage:gem
if applicable. See jenkins-tasks above for more detail. -
pl:jenkins:uber_ship
An aggregate of retrieval, signing, and shipping tasks. Execute
pl:jenkins:retrieve
to retrieve any packages on the distribution server that were built from the SHA/ref of HEAD. Thenpl:jenkins:sign_all
to sign all packages. Finally,pl:uber_ship
,pl:remote:freight
, andpl:remote:update_yum_repo
.pl:jenkins:uber_build
combined withpl:jenkins:uber_ship
performs the entire build and release process for a project. -
pl:mock
Use
mock
to build an rpm package using the default mock distribution, Redhat Linux 5, i386. Requires that themock
package be installed. See the puppetlabs-rpmbuilder module for an easy way to set up a host for building with mock. Resulting rpm is staged in./pkg/el/rpm/5/(products | devel)/(i386 | SRPMS)
. The placement into the "products" or "devel" subdirectories is determined by the name of the package. If the package has a Release Candidate version, it is placed in "devel". Otherwise it is placed in "products". A Release Candidate is determined by parsing thegit describe
string, and searching forrc
after the version numbers. -
pl:mock_all
The same as
rake pl:mock
, but a package is built for every mock listed inext/build_defaults.yaml
on the linemocks:<mocks>
. Packages are staged in./pkg/(el | fedora)/$version/(products | devel)/(i386 | x86_64 | SRPMS)
. The subdirectories are dependent on the mock that is used. The task assumes that the mock configurations are the standard Puppet mock configurations that are generated by the puppetlabs-rpmbuilder module. -
pl:print_build_params
Print all build parameters to $stdout as they would be used in a package build. This prints data that is loaded from
ext/build_defaults.yaml
andext/project_data.yaml
, as well as whatever is overridden with environment variables. Useful for debugging problems with parameter values. -
pl:print_build_param[param]
Print a specific build parameter to $stdout as it would be used in a package build. This prints data that is loaded from
ext/build_defaults.yaml
andext/project_data.yaml
, as well as whatever is overridden with environment variables. Useful for debugging problems with parameter values. param should be the name of the parameter as a symbol, e.g. :ref or :version -
pl:remote:freight
Performs an ssh call to the package server that calls a server-side rake task. The rake task takes the debian packages that have (presumably) been shipped via
pl:ship_debs
and invokes freight with them, which places them in the apt repository. -
pl:remote:update_yum_repo
As with
pl:remote:freight
, this task performs an ssh call to the yum RPM package server, and invokes an existing server-side rake task. The task iterates through theel
andfedora
directories of the yum repository and re-creates package server metadata for rpms in the subdirectories. -
pl:ship_debs
Rsync pkg/deb/* to the "incoming" directory on the debian apt package repository server. Note: this task does not place the packages into production - it is more accurate to consider the packages "staged" on the repository server rather than actually shipped. The
pl:remote:freight
task takes the packages in the "incoming" directory and actually places them in the apt server. -
pl:ship_gem
Takes the built gem in pkg/ and pushes it to rubygems.com. This task assumes you have the appropriate rubygems.com access and config to push the gem.
-
pl:ship_ips
Takes the IPS packages in pkg/ips/pkgs/ and rsyncs them to a holding directory on a package download server. This is not a true IPS server yet, but just a basic file server. Eventually the goal is to have a true IPS package repository running.
-
pl:ship_rpms
Rsyncs the contents of pkg/el and pkg/fedora into the yum repository server. While these packages are available immediately for download by browsing the yum server directories directly, the yum repodata metadata has not been updated, and thus the packages are not available to yum clients. The
pl:remote:update_yum_repo
task updates the metadata, after which the packages will be available to yum clients. -
pl:sign_rpms
Sign the rpms staged locally under pkg/ with the gpg key user ID (e.g. email) specified in ext/build_defaults.yaml as
gpg_name
. This value can be overridden by passing GPG_NAME as an environment variable to the rake task. -
pl:sign_tar
Use gpg to create a detached signature of the tarball. By default this uses the gpg_key value specified in ext/build_defaults.yaml in the project. This can be overridden by passing GPG_KEY as an environment variable to the rake task.
-
pl:tag
Create a signed, annotated git tag of the current repository. Requires TAG be passed as an environment variable to the rake task, which is the value that will be used as both the tag message and the name of the tag. The gpg key that is used for signing is assumed from gpg_key in ext/build_defaults.yaml. This can be overridden by passing GPG_KEY as an environment variable to the rake task.
-
pl:uber_ship
A composite task that performs the following tasks:
pl:ship_gem
pl:ship_rpms
pl:ship_debs
pl:ship_dmgs
pl:ship_tar
pl:jenkins:ship
This is essentially a "ship all the things" task, but it is important to note that it does not update either yum or apt repo metadata on these respective servers - this has to be done viapl:remote:update_yum_repo
andpl:remote:freight
. -
pl:update_ips_repo
Take the packages in pkg/ips/pkg and rsync them to the IPS repository server specified in the build_extras.yaml file. Via ssh commands, call pkgrecv, pkgrepo, svcadm on the IPS repository server. Via ssh, restart the IPS repository service.
-
pl:write_build_params
Generate a yaml file with all the build properties that have been loaded from build_defaults.yaml, project_data.yaml, (optionally) build_extras.yaml(s) via
pl:fetch
, and any environment variables. This file can be used by the packaging repo as a single source of truth for build data viapl:build_from_params
. By default it is written to a temporary location and its location is printed to $stdout. To override the destination, pass OUTPUT_DIR as a environment variable to the task. By default, the name of the file will be either the git tag, if HEAD of the project repository is a tag, or the git sha of HEAD.