Giters

containers / conmon-rs

An OCI container runtime monitor written in Rust

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Allow CRI-O logging to stdout

saschagrunert opened this issue · comments

commented

We right now use the default logger in CRI-O, which logs to systemd. It would be good to have at least a runtime monitor configuration to also log to stdout. This allows us to collect conmon and CRI-O logs in parallel.

Q: Do we wanna make stdout logging the default? Which benefits do we gain when logging to systemd?

commented

can we stay attached to stdout if we've daemonized? conmon-rs is a child of systemd, not cri-o, and we may lose the logs (we definitely would if cri-o restarts, but may unconditionally)

commented

Hm, setting the log driver to stdout works when CRI-O runs in foreground in my terminal, then both logs appear side by side. A CRI-O restart would probably loose that connection, but we could establish an IO reconnect on restore (or similar). 🤔

Connecting the journal logs to CRI-O containers is hard since we do not verbose output them per default. Having both logs in one dump (timing wise) seems reasonable to me.

commented

I think we can use multiple logging drivers in parallel and add a file-based logger later on. This can be used by CRI-O to stream the logs.

commented

what do you think about enabling verbose logs in the journal? I am open to a file based log driver, but it feels more idiomatic (and closer to conmon's behavior) to have it logged to journal

commented

what do you think about enabling verbose logs in the journal? I am open to a file based log driver, but it feels more idiomatic (and closer to conmon's behavior) to have it logged to journal

I'm wondering how this should work, for example sudo journalctl _COMM=conmonrs -o verbose outputs the following format:

Tue 2022-09-20 10:53:47.707490 CEST [s=3ec34f7a8bd34c4e97b1f8702442e55e;i=123a55b;b=8f6c219633ce46e3a1c4dbad7043bc47;m=1aeb85159;t=5e917f765a89a;x=73e177f314cf05b0]
    _UID=0
    _AUDIT_LOGINUID=1000
    _AUDIT_SESSION=1
    _SELINUX_CONTEXT=kernel
    _BOOT_ID=8f6c219633ce46e3a1c4dbad7043bc47
    _MACHINE_ID=4ef6243ad0d84d968b0a6e3060f537ba
    _HOSTNAME=nixos
    PRIORITY=6
    _GID=0
    _CAP_EFFECTIVE=1ffffffffff
    _SYSTEMD_CGROUP=/user.slice/user-1000.slice/session-1.scope
    _SYSTEMD_SESSION=1
    _SYSTEMD_OWNER_UID=1000
    _SYSTEMD_UNIT=session-1.scope
    _SYSTEMD_SLICE=user-1000.slice
    _SYSTEMD_USER_SLICE=-.slice
    _SYSTEMD_INVOCATION_ID=cdb953c9bfc841e1b4dca01c58f33d76
    _TRANSPORT=journal
    SPAN_NAME=backend
    SPAN_TARGET=conmonrs::server
    SPAN_CODE_FILE=conmon-rs/server/src/server.rs
    SPAN_CODE_LINE=155
    TARGET=conmonrs::rpc
    CODE_FILE=conmon-rs/server/src/rpc.rs
    CODE_LINE=68
    SYSLOG_IDENTIFIER=conmonrs
    MESSAGE=Got a version request
    _PID=114097
    _COMM=conmonrs
    _EXE=/home/sascha/git/conmon-rs/target/debug/conmonrs
    _CMDLINE=/home/sascha/git/conmon-rs/target/debug/conmonrs --runtime /nix/store/l5g06xdam7qkfyl6qb3nq671zmjqh7aa-runc-1.1.4/bin/runc --runtime-dir /run/containers/storage/overlay-containers/c9aa05bd97d7a82ca70a011007c4da37d93b7056ba7d5150a44c84134b5d9a8d/userdata --runtime-root /run/runc --log-level debug --cgroup-manager cgroupfs
    _SOURCE_REALTIME_TIMESTAMP=1663664027707490

We can modify the fields, for example via sudo journalctl _COMM=conmonrs --output-fields MESSAGE,TARGET -o verbose

> sudo journalctl -f _COMM=conmonrs --output-fields MESSAGE,TARGET -o verbose
Tue 2022-10-04 12:19:41.623065 CEST [s=d354219abea04e1bb07faa901751c6c0;i=156fd61;b=790f4abbb7fe49d49960c9d06be5c81e;m=3009940c5;t=5ea32cc5f6b95;x=385464dd220ee99d]
    TARGET=conmonrs::server
    MESSAGE=Set log level to: info
Tue 2022-10-04 12:19:54.212481 CEST [s=d354219abea04e1bb07faa901751c6c0;i=156fd6a;b=790f4abbb7fe49d49960c9d06be5c81e;m=3015959d3;t=5ea32cd1f84a3;x=a5af3e5885129085]
    TARGET=conmonrs::server
    MESSAGE=Set log level to: info

The format is kinda hard to read / trace.

commented

currently we print a line of a bunch of stuff to stdout, is there anything preventing us from formatting the output to journal the same way?

commented

currently we print a line of a bunch of stuff to stdout, is there anything preventing us from formatting the output to journal the same way?

Hm, we could fork the tracing journald lib to see if we can bypass the field conversion: https://github.com/tokio-rs/tracing/blob/master/tracing-journald/src/lib.rs

commented

Or we implement our own writer and write to journald, I think I can give this a try.