hengestone / chirp

Message-passing for everyone

libchirp

Message-passing for everyone

Read the Docs

Supported by

On twitter @libchirp

BETA-RELEASE: 1.2.1

Features

• Fully automatic connection setup
• TLS support
  • Connections to 127.0.0.1 and ::1 aren't encrypted
  • We support and test with OpenSSL and LibreSSL
• Easy message routing
• Robust
  • No message can be lost without an error (in sync mode)
• Very thin API
• Minimal code-base, all additional features will be implemented as modules in an upper layer
• Fast
  • Up to 240'000 msg/s on a single-connection, in an unrealistic test:
    • Run on loopback
    • Nothing was executed
    • No encryption
    • Asynchronous
    • The test shows that chirp is highly optimized, but if the network delay is bigger star- or mesh-topology can improve throughput.
  • Up to 55'000 msg/s in synchronous mode

Install

Package-maintainers please use the tarball: https://1042.ch/42/ganwell/chirp/

libchirp is distributed as an amalgamation (only needs make and compiler). See DIST-README.rst. The information below applies to the git repository.

Unix

Dependencies:

• libuv
• openssl or libressl

Build dependencies:

• python3¹
• make
• gcc or clang

Documentation build dependencies:

• sphinx
• graphviz

Install to prefix /usr/local. (with docs)

cd build
../configure --doc
make
make check doc
sudo make install

Install to prefix /usr. (without docs)

cd build
../configure --prefix /usr
make
make check
sudo make install

Install to prefix /usr, but copy to package dir. (Package creation)

cd build
../configure --prefix /usr
make
make check
make install DEST=pkgdir

Note

On Mac you need to install clang-format using

brew install clang-format

How to create a source distribution

Building a source distribution is useful when you need to include libchirp in your project, but don't want to use it's build system. A source distribution can easily be compiled with just a make call.

cd build
../configure --dev --doc
make dist
ls dist

Windows

We want to support Windows, but we are currently not building on Windows. VS 2015 or newer should support all C99 feature we use.

Development

Test dependencies:

• cppcheck
• abi-compliance-checker

Unix

cd build
../configure --dev
make test

In development mode the make file has a help:

make

Chirp has a mode to debug macros:

../configure --dev
make clean all MACRO_DEBUG=True
gdb src/message_etest

This requires clang-format to be installed.

Running pytest manually with -s for example:

cd build
make all
pytest -s ../src

To build without TLS:

make all WITHOUT_TLS=True

Docker

If a tool is not available on your platform or you have a old version of cppcheck (cppcheck is known to behave very different across versions), you can use the docker based tests.

./ci/alpine.sh

Travis will also run this script, so you can also use it to reproduce errors on travis.

You can also run a shell.

./ci/alpine.sh shell

./ci/arch.sh shell

Note: Docker must have IPv6 enabled. Since we only need loopback, you can configure a unique local subnet. For some reason docker doesn't support loopback only anymore. I consider it a bug, the corresponding issue told me it isn't.

DOCKER_OPTS="--ipv6 --fixed-cidr-v6 fc00:beef:beef::/40"

If IPv6 is working in your docker, you don't have to change anything. We only need to loopback. The above is just how I solved the problem.

Windows

No development build available.

Check vs test

make check

Not instrumented (release mode), goal: checking compatibility

make test

Instrumented (dev mode), goal: helping developers to find bugs

Clang-format

We enforce a specific format using clang-format. To format all the code do in the build folder:

make format

You can use the clang-format vim plugin:

Plugin 'rhysd/vim-clang-format'
au FileType c ClangFormatAutoEnable

If you have a different version of clang-format than our CI, the result of clang-format might differ. To format using the ci do:

ci/alpine.sh shell
make format

Syntastic

By default vim will treat *.h files as cpp, but syntastic has no make-checker for cpp, so *.h would not get checked.

let g:syntastic_c_checkers = ['make']
au BufNewFile,BufRead *.h set ft=c

With this setting syntastic will check the following:

• Clang-based build errors
• Line length
• Trailing whitespaces

Clang complete

If you use clang complete, we recommend

let g:clang_auto_select     = 1
let g:clang_snippets        = 1
let g:clang_snippets_engine = 'clang_complete'

Changes

2018-12-12 - 1.2.1-beta

Read the Docs 1.2.1

• Do not enable auto_stop_loop in ch_chirp_run

2018-11-27 - 1.2.0-beta

• Make sending handshake independent from first read
• Pure low-level writes block the send-queue
• Only start a send_if_pending loop if is not active

2018-08-23 - 1.1.2-beta

• Fix warning about strncpy by using memcpy
• Inform user about purpose ch_chirp_set_auto_stop_loop

2018-03-31 - 1.1.1-beta

• Fix deadlock in ch_chirp_send_ts
• Fix bug when disabling signals

2018-03-20 - 1.1.0-beta

• Implement scatter-gather write API
  • Improves peak performance to 240'000 msg/s
• Allow build without TLS
• Timeout consistency
  • TIMEOUT
    • Send- and connect-timeout scaling in seconds. Send-timeout will be TIMEOUT seconds. Connect-timeout will be min(TIMEOUT * 2, 60) seconds.
  • REUSE_TIME
    • Time until a connection gets garbage collected. Until then the connection will be reused. Actual reuse time will be max(REUSE_TIME, TIMEOUT * 3).

2018-03-17 - 1.0.0-beta

• Rename config.ACKNOWLEDGE to config.SYNCHRONOUS (ABI break)
• Update to rbtree 0.7
• Correctly handle ENOMEM when connecting
• Always ch_cn_shutdown on conn init errors (used to just ch_free sometimes)
• Fix AF_INET assert
• Fix late send_ts_queue_lock destroy
• Do not log missing recv callback
• Add ch_chirp_t and ch_release_cb_t to ch_chirp_release_msg_slot (ABI break)
  • Allows to continue when message has been released
• Add ch_chirp_release_msg_slot_ts
  • Allows to release message in a different thread
• Recover from partial ch_chirp_init() properly
• Renaming buffer-handlers to message-slots (ABI break)
• Add _ssl_context to message for future use (ABI break)
• Move srand() to ch_libchirp_init
• ch_msg_free_data now also frees the upper-layer protocol header
• Missing lock in ch_at_allocated

2018-02-17 - 0.2.1-beta

• Remove unnecessary copy of reconnect_remotes datastructure
• Simplify iterations in gc_connections
• Send probe message the first time it is needed
• Fix memory leak in buffer pool
• Switch from uv_hrtime to uv_now

2018-01-03 - 0.2.0-beta

• Initial public beta release
• All functional features implemented
• Some performance/build features missing

Thanks

For letting me do this:

• Adfinis SyGroup

For helping me with the architecture:

• David Vogt @winged
• Sven Osterwalder @sosterwalder

For helping me with the documentation:

• Sven Osterwalder @sosterwalder
• David Vogt @winged

For reviewing my pull requests:

• Oliver Sauder @sliverc
• David Vogt @winged
• Tobias Rueetschi @keachi

Contributing

Please open an issue first. Contributions of missing features are very welcome, but we want to keep to scope of libchirp minimal, so additional features should probably be implemented in an upper layer.

Most valuable contributions:

• If you run continuous integration on your app, build chirp with CH_ENABLE_ASSERTS and report bugs.
• Contribute any kind of tests or fuzzing (if possible hypothesis based)
• Make bindings for your favorite language
• Make packages for your favorite distribution
• Promote libchirp

---

1. Script-headers can be patched to work with python2.↩