README for FTY

TL;DR summary

This repository allows to quickly checkout, build, test and develop the ecosystem of 42ITy™ project, including components of the project itself and customized or original dependency projects. There are some third-party dependencies and tools (suggested in details below) that this repository does not provide; they are expected to just be on your build system when the corresponding configure script runs.

To build the latest HEAD of each component’s default branch just run this:

$ git clone https://github.com/42ity/FTY && \
  ( cd FTY && make world )

Note that this would require an internet connection to check out all the repositories, and considerable time to build the ecosystem. To rebuild the checkouts already present on your system (e.g. to continue a broken build after installing dependencies), the ordinary make should suffice.

It is recommended to have installed ccache to speed up subsequent rebuilds, as well as certain other common dependencies and tools. For reference, you can find a list of additional packages needed on Debian and Ubuntu Linux distributions in the .travis.yml file.

Builds and tests should be executed as a non-root user, because:

some component self-tests expect in-accessibility of certain paths;
tasks such as make web-test-bios run daemons as a non-privileged account that would not see into `root’s home directory with build artifacts.

About the FTY repository

FTY is a "dispatcher" repository which links (as "git submodules") to the repositories used to build the 42ITy™ project. This includes some third party projects (git HEADs), forks of third-party projects customized for 42ITy™ needs, and of course lots of original development.

For a rebuild from scratch you’d need to compile and install the following components in order (this is automated by the accompanying Makefile):

gsl and zproject (optional — to refresh project definitions)
libzmq
libczmq (one of the supported czmq variants as discussed below)
malamute
cxxtools
tntdb
log4cplus
fty-proto
fty-common
other fty-* components as needed and possibly with dependencies on each other as defined in their profile.xml manifests for zproject
fty-core and fty-rest possibly come last

Note that the "git submodules" mechanism used by the FTY repository can only refer to specific Git commit IDs, not branches or tags, so when you first clone the repositories you will get a lot of "detached HEADs" in the component subdirectories. This suffices for automated builds with predetermined versions of the software. The FTY repository Makefile recipes and scripts include additional methods to add support for human developers as well, in order to both ease integration with their own forks of the components on GitHub, and to follow actual Git branches in the local workspace checkouts.

This repository tracks a couple of similar but separate branches:

master refers to HEADs of our projects that we might experiment upgrading into — such as the zeromq ecosystem at this time;
FTY branch is a bit more conservative and tracks the state of third-party sources used in our daily work and recent releases.

Making the ecosystem

The included Makefile automates building of project components, including the customized third-party projects, in the order of dependencies. It is ready for correct parallel compilation with GNU Make 3.81 or newer, and can take advantage of ccache if available on your system.

It is up to the developer to provide a build environment with appropriate third-party software dependencies which our project does not modify, nor track git HEADs, nor build (such as MySQL/MariaDB libraries to link against, or lua-5.1/5.2, or libneon, or libapr, maybe openssl etc.), as well as an internet connection to completely check out the source code to build.

The Makefile defines dependencies and recipes that can cause download or update of sources from designated remote repositories using git, so the basic way to build the 42ITy™ ecosystem using those component revisions that the FTY repository points to at any given time is to:

$ git clone https://github.com/42ity/FTY && \
  cd FTY && \
  make -j 4

To update sources from current state of remote Git branches and rebuild in proper ordering whatever has changed later on, you can use e.g. make world. You can also do this for initial build command, if you want to build right away with the current state of default branch in each component repository.

Also you can call a number of actions against specific components using a metaphor of subdirectory in the recipes, e.g. make recheck/fty-example. See the Makefile for different wildcard actions it defines (with percent character), using this query for the current version: egrep '/%:$' Makefile

When running parallel make -jN beware that if you spawn too many workers, the operating system might kill off some of them if it is insufficiently resourced, causing the build to fail not due to source code errors. In order to take advantage of parallel processing, you may be well served by running first a parallel and then a sequential build attempt: make -j 4 -k || make

Developer incremental build support

The Makefile provides a development-rebuild option, invoked like this:

$ make devel/fty-example

which takes effect both for components "prepped" as symlinks (e.g. by setting PREP_TYPE_fty-example=cloneln-src or cloneln-obj in the Makefile) and as copies of files (PREP_TYPE_component=clonetar-src or clonetar-obj). This is the default now, unless a recipe for some component defines a different PREP_TYPE because of its build-system quirks.

If the component was already configured, this recipe just runs a make in its custom build directory and so recompiles just the binaries whose sources you have changed right in the common checked-out Git workspace.

Of course, this only works as long as you have not added new source files, etc. — but that sort of change would anyway need a rebuild/fty-example. Other similar limitations may exist but were not yet found or documented.

As a fallback, this would prep/autogen/configure components that were not yet "configured", and would fall back to a full rebuild for components that are not "prepped" as symlinks.

The intended use-case looks like this, to streamline iterations of small changes during development:

$ gmake devel/fty-example && gmake memcheck/fty-example

…or this for custom runs (assuming one platform so /*/ is not ambiguous in the short example below):

$ gmake devel/fty-example && (cd .build/*/fty-example && src/test-me --arg -v)

Note	At this time, the `devel/*` recipe only looks at, and recompiles if needed, the specified component — not those it might depend on — so in case you edit lots of ecosystem, refresh such components explicitly:

$ gmake devel/{fty-proto,czmq}

Developer fty-rest support

This Makefile supports make web-test and make web-test-bios recipes to help with in-place testing of the fty-rest component during development.

The make web-test integration simply builds fty-rest with dependencies (including a copy of tntnet from the tracked fork) and runs it with the tntnet.xml generated by fty-rest (pointing to the freshly built copy of bios-web.so), as the developer’s user account on a dedicated port (8000) and using a dedicated directory for data files that the servlets might edit without conflict with the OS — at least as long as sources do not use any hardcoded paths. This recipe has the same effect as if developer went to the fty-rest workspace and executed make web-test there (using same-named recipe in that component), it just provides the depedencies built from the freshest of sources.

The make web-test-bios integration is intended for tests approaching the reality of our product, running in our generated and pre-configured OS image in a container or on a rack controller. As such, this recipe takes the same /etc/tntnet/bios.xml file (presence required) and environment variable files as referred by the production systemd unit tntnet@bios.service, and produces a configuration patched for the compPath reference to the freshly built copy of bios_web.so object with developed servlets. Then this recipe disables the running instance of tntnet@bios.service and uses sudo -E to start the freshly built webserver in its place as root, so it changes credentials according to configuration just as the packaged service does. As you can see, this recipe requires quite a bit of circumstances and intentional system setup to work; in particular this should keep your daily host operating system safe from experimental code ;)

Note	This repository includes an `admin-sudo` file that you can copy into the `/etc/sudoers.d` (running as `root`) in an instance of our OS image:

# cp admin-sudo /etc/sudoers.d/

Note	At this time, use of the patched `/etc/tntnet/bios.xml` from the OS image does not take into account any customizations from the `tntnet.xml` template in the `fty-rest` component sources under your development and testing.

Support for multiple CZMQ library versions

Note that as part of the project’s evolution and legacy, it used to require the czmq version 3.0.2, and since that release was obsoleted by upstream a while ago, we tracked our own fork with small fixes. Subsequently the code of 42ITy™ components where it mattered was updated to support either the CZMQ3 or CZMQ4 APIs, and now the Makefile in this repository allows to automate the builds against either our czmq-v3.0.2 fork or the upstream czmq-master from the GitHub, or using binary packages of zeromq stack as provided in your OS by other means (beware that these can lag behind respective upstream/master or upstream/stable branches and so can lack the features that 42ITy codebase might need). Our current goal is to drop the requirement of the obsoleted version and use the community-supported master branch, where we can collaborate on fixes for bugs that bite us all.

To choose the czmq version to build your FTY components against, you can define the CI_CZMQ_VER=<value> (as an exported environment variable, or as a make argument); suported values are 3 for our "czmq-v3.0.2" fork (it is also the default choice if no value is set explicitly), 4 for the upstream/master which is currently under the 4.x umbrella versions, and pkg for OS packages of the whole libsodium`libzmq`czmq+malamute stack.

Support for Address Sanitizer

The zeromq-related ecosystem of software projects has recently added support for building with the Address Sanitizer in supportive compilers such as the recent GCC versions. You can find more details about this technology at https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html and https://github.com/google/sanitizers/wiki/AddressSanitizer.

This can be enabled in Travis CI tests for projects generated with zproject and now for (Travis or manually executed) builds done with the Makefile provided by this FTY repository, by an exporting an environment variable usually before a memcheck/valgrind build and run:

$ export ADDRESS_SANITIZER=enabled
$ make memcheck/fty-example

Note that the actual compilation flags enabled by such toggle can depend on contents of the configure.ac scripts in each component project, and of course functionality of the actual implementation depends on the evolution level of the compiler in use for the build.

Also note that Leak Sanitizer (-llsan) is not currently available in the Travis CI operating environments.

Support for different operating systems

This Makefile has been verified on Debian 8 (devel images for the project build farm); it also passes on Ubuntu-based systems used in the Travis CI cloud. Much but not all of this code can also build under OpenIndiana, more with a purpose to test the approach to portability than to use it there (in the near term at least). Feel free to update the recipes if needed for other OSes and distros, and take inspiration from hooks made for and used by the OpenIndiana/SunOS integration.

Fetching all current sources and updating the HEADs

Note that when you initially check out a clone of this repository, you only get meta-data. To instantiate (or subsequently update) the actual source code for the components, as well as to update the reference to Git HEADs of the referenced components, run the ./sync.sh script in this workspace.

Note	If you intend not only to build the unmodified 42ITy codebase, but to also collaborate about changes and improvements, see the chapter below about setting up your forks on GitHub and how to automate definition of references to both your "origin" fork and our common "upstream" repository.

Note	In order for updates from the common codebase to come without conflicts, never work in a `master` branch (or other preferred branch in certain repos)! Use dedicated private branches for development of new features!

TODO: Add a regular job, or one triggered by commits to project repos, to run such updates and push new references to common FTY dispatcher repo’s upstream/master.

Adding a tracked repository

As new agents and components and perhaps tweaked third-party projects are added into the mix, either in the common Git organization or in your own set of FTY repositories forks, you can add and check out new Git submodules like this:

:; git submodule add https://github.com/42ity/fty-new-agent

or (to specify a default non-master branch right away):

:; git submodule add -b 42ity https://github.com/42ity/third-party-fork

Tracking and checkout of new repositories under https://github.com/42ity/ can be automated using ./sync-repos.sh script.

Note that after adding sources for a submodule, you’d likely want to reference its place in the dependency chain for the Makefile of this FTY workspace as well (perhaps among COMPONENTS_FTY_EXPERIMENTAL first, for skeleton component directories). Don’t forget to git add both the updated Makefile and the new component directory, and to set up your own developer fork for it as detailed below.

To maintain a repository that was forked off an original and actively developed third-party project, you may want to locally define another remote reference so as to track the upstream evolution. For this to work reliably, the 42ITy fork should use different branch names which are not equal to names or tags used by the original repository.

Adding a tracked repository for a fork of third-party project

At this time the convention for settling on a certain release of the upstream project is to (assuming you have the admin rights for 42ITy org on GitHub):

Fork it on GitHub under the 42ITy org;
Go to Travis CI web interface at https://travis-ci.org/profile/42ity/, click to "Sync account" and enable the new fork in sliders below;
Fork the 42ITy replica to your developer account (you can also use a fork of original upstream repository — but then take extra care about pull request targeting later on);
Clone it to a temporary local workspace, outside the FTY directory;
Check out the commit we settled on as the current baseline, usually some X.Y(.Z) release: git checkout 123cafe or git checkout release/1.2.3;
Nail it down for easier later comparisons or merging as we’d decide to move on to a new baseline: git checkout -b 1.2.3-release;
Branch off a stable line we’d use to cut an end-user release at some point: git checkout -b 1.2.3-FTY;
Branch off a "master-like" line we’d use to develop and stage improvements to this fork, like backported bugfixes or updated Travis integration recipes: git checkout -b 1.2.3-FTY-master — this stuff would be used in daily OS images and eventually passes from here to the X.Y(.Z)-FTY branch;
Put the new branching info to the 42ITy fork: git push --all origin
Go to GitHub web interface, select the new 42ITy component and go Settings / Branches, e.g.: https://github.com/42ity/libzmq/settings/branches
Pick the X.Y(.Z)-FTY-master branch as the default, click accept;
Prevent inadvertent changes to the non-master branches:
- Under Protected branches, choose the X.Y(.Z)-FTY and click that we want to "Protect this branch", "Require pull request reviews before merging" (and "Dismiss stale pull request approvals when new commits are pushed" under that), "Require status checks to pass before merging" (and "Require branches to be up to date before merging" as well as CI tests where defined) and finally "Include administrators" so at least two people are needed to push the changes to these branches (an admin can also unset this protection, when in a pinch).
- Repeat for X.Y(.Z)-release.
- Do not protect the branches we’d want to track from the original project, like the master for most of them, as you’d pull-push these occasionally (or even automatics would).
Finally, set up git submodule tracking (e.g. using the sync-repos.sh) in your checkout of the FTY repo;
Revise that the appropriate X.Y(.Z)-FTY-master is the default tracking branch in .gitmodules for FTY master branch, and X.Y(.Z)-FTY is for the FTY FTY branch;
Add corresponding commits to the FTY repos to begin tracking the project as submodule;
Update Makefile etc. as may be needed to track this component and maybe its build variants (like we have done to test various CZMQ versions);
Possibly update our other projects (project.xml files and Travis recipes) to build by default against our specified fork and branch, rather than the upstream master repositories which may e.g. no longer expose bugs that can plague our builds… or vice versa (new bugs are introduced by upstreams from time to time, that we don’t see in our OS images).
Push back the updates to the FTY repository on GitHub.

To use your local workspace for occasional synchronization from the original project, run the git-myorigin script detailed below to ensure that your upstream refers to the 42ITy fork, and origin is your own private fork. Then add tracking for the opensource reference to the original project and specify that you want certain branches (like master) to come from there:

:; cd <newfork>
:; git remote add opensource https://github.com/someorg/somerepo
:; git remote set-url --push opensource no_push
:; git branch --set-upstream-to=opensource/master master

While here, you can also make sure to push the FTY branches to your private repository, to use for PRs later on:

:; git branch --set-upstream-to=origin/1.0-FTY-master 1.0-FTY-master
:; git branch --set-upstream-to=origin/1.0-FTY-master 1.0-FTY-master

For projects (currently czmq) where branch and tag naming use the same string values, which is valid but confuses some git operations including checkouts, you might want to forbid pulling remote tags:

:; git config remote.opensource.tagOpt --no-tags

Given sufficient rights in the 42ITy org, you can later synchronize the master branch upstream changes, so we can keep track of all changes there (and of how far ahead/behind is our forked project), with:

:; cd <newfork>
:; git checkout master && git pull --all && \
   git merge opensource/master && git push upstream

Do not forget to git checkout X.Y(.Z)-FTY-master after doing this, or perhaps keep aside a workspace with a checkout of the FTY repository just for this job. In particular, take care to not add to the git-submodule tracking the commit IDs not pointing to the latest X.Y(.Z)-FTY(-master) state. The ./git-sub-branch-list can speed up such verification.

When updating the FTY repo itself and transplanting changes between the repositories, make sure that the FTY and master branches keep pointing where they should have (do not merge unexpected changes to .gitmodules). The git checkout master && git diff FTY .gitmodules trick can help here.

Generally a git checkout master && git diff FTY should only show the .gitmodules file and some different submodule references (commit IDs).

Adding new FTY components

When starting a new component, don’t hesitate to start with fty-example and its project.xml in particular to seed the generation of your new codebase in a way similar to our other components.

If your codebase uses features of C++11 or newer standard, see notes in the .travis.yml file (re-)generated for your component about requesting an appropriate build environment from the Travis CI farm, with a capable compiler.

If the new component delivers systemd services that should be manageable as part of the 42ITy™ product, consider updating the list of recognized services used in fty-core::tools/systemd and in fty-rest::systemctl.ecpp (or rather fty-rest::helpers.cc at this time).

Finally, although orthogonal to updating this repository, don’t forget to enable Travis CI for the new component and add or update some corresponding recipes on your build farm, if any.

Changing tracked repository data

If the submodule configuration needs updates due to evolution over time or because of initial-setup errors, such as that a different remote repository or default branch must be tracked, you may want to edit the .gitmodules file directly to set the details you need. It may be required to git deinit an existing working copy of the submodule and check it out again, to use the new repository tracking metadata — so before such operations do not forget to commit your changes and push them out into the GitHub fork. Alternately, local copies of repositories are just directories with special files — so you can just rename them to sit nearby, and as far as the Git software is concerned, by this action you’ve just nuked a checked out submodule and should simply re-init it again.

Developer "origin" forks vs. common "upstream" repos

Also note that if you clone FTY, the checked-out repositories will likely initially refer to the component repository URLs as an origin, while they are rather upstream for our context (and a real origin would be your development fork of each such component repo you collaborate on). In this case, change to the subdirectory of the component in question and run the git remote commands to rename references, for example:

:; echo 'GITHUB_USER="mygithubname"' > ~/.git-myorigin
:; ./git-myorigin */

aquette / FTY