This implements a tool to accept the protocol emitted by pg_logfebe
(version PG-9.2.x/logfebe-1
) and send it to logplex using the
library logplexc.
It is necessary to download submodules and set GOPATH to build the program with the most convenience by writing:
$ git submodule init $ git submodule update $ make
Alternatively, if getting pg_logplexcollector for the first time, one
can initialize submodules upon git clone
and build by running:
$ git clone <pg_logplexcollector remote> --recursive $ cd pg_logplexcollector $ make
This is because git submodules are used to version and retrieve other
libraries, such as femebe and logplexc. Having done this though,
make all
(or just make
) will produce binaries in the bin
directory.
There exists a Makefile
target for setting up most of what one
needs to demonstrate the entire system end-to-end. It installs
everything into a subdirectory tmp
:
$ PGSRC=git-repo-directory-for-postgres make testdb $ ./bin/logplexd & https://127.0.0.1:44786 # (dynamically generated) $ LOGPLEX_URL=https://127.0.0.1:44786 \ SERVE_DB_DIR=tmp \ ./bin/pg_logplexcollector $ ./tmp/postgres/bin/postgres -D tmp/testdb
After this, one should be rewarded with printed HTTP requests, written
to standard output from logplexd
, forwarded by
pg_logplexcollector
, emitted by the configured pg_logfebe
and
the PostgreSQL server in which it resides.
For production use, two pieces of software must be configured:
pg_logfebe
, and pg_logplexcollector
.
Configuring pg_logfebe
requires Postgres 9.2.
It can be installed via make install
, per standard Postgres
extension mechanisms. As with all such extensions, the most important
thing to confirm is that the pg_config
program is both in path and
points to the correct binary Postgres installation. You may need a
developer package from a distributor, such as the
postgresql-server-dev-9.2
package on Debian and Ubuntu.
Having done this, configure postgresql.conf with something like the following:
shared_preload_libraries = 'pg_logfebe' logfebe.unix_socket = '/tmp/log.sock' logfebe.identity = 'a-logging-identity'
logfebe.unix_socket
must be an absolute path to the unix socket
pg_logfebe
will attempt to connect to deliver logs. When one sets
up pg_logplexcollector
, it must listen at that address.
logfebe.identity
is the non-secret 'identity' of the PostgreSQL
installation that will be delivered by pg_logfebe
so that
pg_logplexcollector
can determine what logplex token (which is
secret) to use.
pg_logfebe
delivers logs on a best-effort basis, so it is
relatively harmless to start Postgres at this point if
pg_logplexcollector
is not yet running.
Configuring pg_logplexcollector
consists of two concepts:
LOGPLEX_URL
: What logplex service to submit HTTP POSTs to.SERVE_DB_DIR
: What directory contains the 'serve database'
SERVE_DB_DIR
deserves more explanation:
In order to preserve the secrecy of logplex tokens and provide greater
security for tenants, pg_logplexcollector
ties together three
pieces of information:
- A non-secret identity (configured in
postgresql.conf
with the GUCpg_logfebe.identity
) - A secret logplex token
- A specific unix socket that will only accept connections for a given identity.
This is done by writing a JSON file into $SERVE_DB_DIR/serves.new
that looks like this:
{"serves": [ {"i": "identity-1", "t": "token-1", "p": "/path/unix.socket-1"}, {"i": "identity-2", "t": "token-2", "p": "/path/unix.socket-2"} ] }
One can confirm that the serves.new
file has been loaded by
watching it be copied to $SERVE_DB_DIR/serves.loaded
. At that
time, serves.new
, and any existing serves.rej
or
last_error
file, if any, will be removed.
If one submits invalid input, serves.new
is removed and
serves.rej
and a last_error
file are emitted for inspection.
serves.loaded
does not change in this case.
pg_logplexcollector
will check for serves.new
at various
arbitrary times. Right now it occurs every ten seconds.
Putting these together, an invocation of pg_logplexcollector
looks
like this:
$ SERVE_DB_DIR=/path/to/servedb LOGPLEX_URL=https://somewhere.com/logs \ ./pg_logplexcollector
pg_logplexcollector
logs client connections, disconnections, and
errors. The former is to help determine if one's configuration is
working as intended.
- Does not support HTTP client timeouts. Unfortunately this doesn't
look easy to do without implementing an entire Go
net/http
RoundTripper
. - Log formatting is not designed at all: it's just the first thing anyone has implemented.