Pivotal OSS Build Schema and Instance Management for Apache HTTP Server

This program and the accompanying materials are made available under the terms of the under the Apache License, Version 2.0 (the "License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Overview

Builds are monolithic. These rely on many established components shipped with modern linux distributions which are reasonably stable and up to date. On Windows, every component must be built from source.

The build target is /opt/pivotal/webserver/httpd-2.4/ on Linux and Windows, but is relocatable. Each has a long pathname, e.g. httpd-2.4.25-$bldno/ which is symlinked from the 'expected' path. Non-release builds have additional pathname components such as -snapshot, -bleed, etc, and may have a TAG to distinguish two otherwise apparently identical builds.

Build {pkg} reflects the particular component, either the offical designation (httpd-2.4.25/), an indication of interim packages (httpd-2.4.26-rc1/), or the exact province of the file (httpd-2.4.26-dev-r1872199/).

Several {bld} schemas are expected; 'release' for the official packages, 'candidate' for packages anticipated for release, 'snapshot' for the working trees of the component’s current maintenance trees, and 'bleed' for next-major or minor development trees.

mak/                   Build script files

src/                    Complete build sources
   {pkg}/               Component source (for VPATH build)
   pkgs/                Source component package downloads
   {bld}-manifest-vars  Description of {pkg} names and versions for {bld}

bld/                    Build trees of components by build type
   {pkg}/               Component build tree (vpath or copied from src/)

dst/                    Resulting intermediate binary packages
   {httpd-pkg}{tag}/    Temporary installation for testing and relocation

pkg/                    Final binary package
   {httpd-pkg}{tag}/    Relocateable binaries for distribution

Platform Requirements

The development toolchain package group from the respective OS distribution is required to build this package. In addition, a number of stable, well-maintained components are leveraged from the OS distribution, rather than building and installing potentially conflicting versions.

On all platforms, pre-install the Text::Template package from CPAN to assure you are building OpenSSL with the latest tools. In order to run Makefile.test there is a list of CPAN dependencies required to invoke the perl-based Apache::Test framework. These can be found under src/httpdtest-trunk/README after first running the Makefile.gather phase. (An immediate goal of this toolchain is to automate the local provisioning of those dependencies if not already installed.)

RedHat or Fedora Linux development package dependencies:

For all steps the complete group of development tools is recommended, e.g.;

$ yum groupinstall 'C Development Tools and Libraries'

or simply;

$ yum groupinstall 'Development Tools'

Additional required tools which may not be included by default;

$ yum install cmake imake

For the actual Makefile.build phase, a number of system component -devel packages are needed.

$ yum install libuuid-devel expat-devel libxml2-devel lua-devel pcre-devel zlib-devel

These base packages are required for deployment (and are incidently installed with the -devel packages above; most are already present by default);

$ yum install libuuid expat libxml2 lua pcre zlib

Note that following RHEL 7 / Fedora 21, yum becomes the dnf utility. The package names have remained the same.

Ubuntu or Debian development package dependencies:

$ apt-get -y install \
    build-essential \
    cmake \
    git \
    subversion \
    xutils-dev\
    libexpat1-dev \
    lua5.3 \
    libtool \
    libpcre3-dev \
    libxml2-dev \
    zlib1g-dev \
    wget

Microsoft Windows dependencies

Microsoft Visual Studio 2017 or 2015
NASM Assembler
ActiveState or Strawberry Perl
unxutils or gnuwin32 Windows-native unix command line tools (Note mingw and cygwin are not supported)
Info-zip command line zip
curl and awk (or name gawk from unxutils as awk)
Subversion and GIT command line tools

Phase 1: Gather Sources

$ cd src/
$ make -f ../mak/Makefile.gather [BLD={type}] [GRP=complete]

BLD defines the build type: release - candidate - snapshot - bleed (case sensitive) where release is the default.

Gathers source code packages or (source checkouts) for all packages into the source tree, and generates a version and directory name manifest.

This will gather all components if GRP=complete is specified, otherwise the linux system package sources of expat, lua, pcre, libxml2 and zlib will not be gathered or compiled.

Each component is designed for persistence, if from git or svn it performs a fetch / update, if from a most recent release package, it fetches and unpacks the package. The source directory tree is designed for parallelism, the different build types may coexist in the same source tree.

This makefile is run first from the source directory root (e.g. src/), and must be performed only upon updates to the source packages. The resulting manifest can be compared to the previously created manifest to determine if the sources have been updated.

Makefile.preconfig should follow immediately when the manifest changes.

Phase 2: Preconfigure Sources

$ cd src/
$ make -f ../mak/Makefile.preconfig [BLD={type}]

Prepare configuration scripts of packages, particularly from source control where autoconf etc have not been invoked yet.

This makefile is run after Makefile.gather from the source directory root, and must be performed following updates to the source packages as indicated by manifest changes. Only source code packages corresponding to the specific BLD target are updated.

The result of this step is suitable for archive, or escrow and distribution to multiple build systems, beginning at the Makefile.build step.

Phase 3: Build Sources

$ cd bld/
$ make -f ../mak/Makefile.build [BLD={type}] [TAG={-suffix}]

Build all components described by the manifest into the intermediate/ temporary installation tree, using that intermediate tree as the component reference for later components.

TAG defines the target directory and package name suffix such as a datestamp, checkout revision, or continuous build revision number. By default there is no suffix tag.

This makefile is run after Makefile.gather and Makefile.preconfig and may be based on a snapshot of the build tree from those two previous steps from another continuous build job.

This makefile must be run from the build (not source) subdirectory, such as bld/. The build tree uses the same component directory names as the source tree. The components are initially installed into the DESTDIR which is the ../dst/httpd component directory name with the TAG variable suffixed. SRCDIR references the source tree (typically ../src) and would typically not need to be overridden.

The TARGET directory, /opt/pivotal/webserver/$(httpd_srcdir)$(TAG) would typically not be overridden, and refers to the anticipated installation path of the resulting package. Use this to ensure the generated suexec binaries are recognized as valid.

Phase 4: Test Source and Intermediate Installation

$ cd bld/
$ make -f ../mak/Makefile.test [BLD={type}] [TAG={-suffix}]

Test all components described by the manifest and the intermediate/ temporary installation httpd server.

This makefile must be run from the build (not source) subdirectory. Where a component has an integrated test target these are processed within the build tree. The Apache httpd perl test framework is invoked against the intermediate installation in the $DESTDIR path.

Phase 5: Package Installation Binaries

$ cd pkg/
$ make -f ../mak/Makefile.package [BLD={type}] [TAG={-suffix}]

Copy the intermediate/temporary installation httpd server and dependent binaries into the dst/webserver/ tree to rewrite configurations files and scripts with as relocatable paths, add the instance management scripts, split the debugging symbols from the binaries, and tar up the package.

This makefile is run from the pkg (not src, bld or dst) subdirectory. Note that running this in the dst subdirectory will wipe out the last build target directory; please use caution.

Distribute the resulting .tar.bz2 files as desired.

Installation Phase

Installing these binaries to a target machine consists of untarring the package, relocating references to the desired installation path and creating a symlink to use as the 'generic' reference to the now-current httpd.

$ mkdir -p /opt/pivotal/webserver
$ cd /opt/pivotal/webserver
$ tar -xjvf {pkgname}
$ ./httpd-2.4.29{tag}/bin/fixrootpath.pl
$ ln -sf httpd-2.4.29{tag} httpd-2.4

Packages may be installed in parallel; in order to switch the running httpd version, simply reassign the symlink to the desired version and restart the server instances.

Instance Creation

To create an instance /opt/pivotal/webserver/{hostname}, use the following commands;

$ cd /opt/pivotal/webserver
$ ./httpd-2.4/bin/newserver.pl --servername {hostname}

The resulting directory includes bin, conf, htdocs, cgi-bin, ssl and logs subdirectories. The bin directory includes an environment script for consuming the instance’s and then binaries distributed in httpd-2.4/bin, as well as an httpd control script httpdctl.

timani / oss-httpd-build