conmon 2.1.9 not working with podman 4.8.2 and ubuntu 22.04

Question

conmon 2.1.9 not working with podman 4.8.2 and ubuntu 22.04

dmdx86 opened this issue 8 months ago · comments

root@nas:~/conmon-2.1.8# cd ../conmon-2.1.9/
root@nas:~/conmon-2.1.9# make install
<snip>
root@nas:~/conmon-2.1.9# podman run --rm -it ubi9-init /bin/bash
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Same error on any other container, new or existing.

It is OK on 2.1.8

root@nas:~/conmon-2.1.9# cd ../conmon-2.1.8/
root@nas:~/conmon-2.1.8# make install
<snip>
root@nas:~/conmon-2.1.8# podman run --rm -it ubi9-init /bin/bash
[root@e9ded0c5b94a /]# 
exit
root@nas:~/conmon-2.1.8#

I compile and build podman/conmon/crun/buildah, do not use the versions in the Ubuntu repositories. Everything was good up until I updated conmon this morning.

This is trivial for me to reproduce, LMK if anything else is needed from me or if I am doing something dumb to cause this.

root@nas:~/conmon-2.1.8# podman version
Client:       Podman Engine
Version:      4.8.2
API Version:  4.8.2
Go Version:   go1.20.3
Built:        Thu Dec 14 16:53:00 2023
OS/Arch:      linux/amd64

Gregory · Answer 1 · Fri Dec 15 2023 07:30:37 GMT+0800 (China Standard Time)

I logged to say the same. I'm on archlinux, but all of my containers now fail to start with the same exact error.

Reverting back to 2.1.8 with podman 4.8.2 works as expected.

Client:       Podman Engine
Version:      4.8.2
API Version:  4.8.2
Go Version:   go1.21.5
Git Commit:   aa546902fa1a927b3d770528565627d1395b19f3-dirty
Built:        Wed Dec 13 17:07:26 2023
OS/Arch:      linux/amd64

Cameron Nemo · Answer 2 · Fri Dec 15 2023 10:49:43 GMT+0800 (China Standard Time)

Indeed it appears the release is broken. This is the results I get:

$ podman run --rm -it docker.io/alpine:edge sh
Error: runc: runc create failed: chdir c�Z: no such file or directory: OCI runtime attempted to invoke a command that was not found

And then in the working directory I see ctl and winsz sockets...

Everything works fine with conmon 2.1.8.

Dominique Martinet · Answer 3 · Fri Dec 15 2023 12:09:19 GMT+0800 (China Standard Time)

hit the same problem when trying to check my patch just now...
The problem was introduced in a recent static analysis fix ; the free is incorrect if the value comes straight from options.

I've fixed it in #476

Peter Upfold · Answer 4 · Fri Dec 15 2023 14:54:54 GMT+0800 (China Standard Time)

I can also reproduce this issue on Arch.

$ podman run -t ubuntu echo "Hello"

Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

Downgrading to 2.1.8 causes the issue to go away.

Benjamin Xiao · Answer 5 · Fri Dec 15 2023 19:39:29 GMT+0800 (China Standard Time)

Can confirm that applying #476 on top of Arch's 2.1.9 package seems to have fixed the issue. My containers now all properly start up again.

Benjamin Xiao · Answer 6 · Mon Dec 18 2023 13:05:34 GMT+0800 (China Standard Time)

Indeed it appears the release is broken. This is the results I get:
$ podman run --rm -it docker.io/alpine:edge sh
Error: runc: runc create failed: chdir c�Z: no such file or directory: OCI runtime attempted to invoke a command that was not found
And then in the working directory I see ctl and winsz sockets...

Everything works fine with conmon 2.1.8.

Today I noticed ctl and winsz in my root directory. Do these belong to podman? I was a bit freaked out when I saw these files.

Giuseppe Scrivano · Answer 7 · Mon Dec 18 2023 22:54:57 GMT+0800 (China Standard Time)

Today I noticed ctl and winsz in my root directory. Do these belong to podman? I was a bit freaked out when I saw these files.

conmon creates them

Cameron Nemo · Answer 8 · Tue Dec 19 2023 01:55:59 GMT+0800 (China Standard Time)

@giuseppe y'all might want to tag a bugfix release or at least update the 2.1.9 release notes (if that is that possible) to mention that it is broken. Seems that less-than-careful downstreams like NixOS are not aware of this bug.

Peter Hunt · Answer 9 · Tue Dec 19 2023 04:01:06 GMT+0800 (China Standard Time)

should be fixed in 2.1.10 now

d03j · Answer 10 · Tue Dec 19 2023 17:26:37 GMT+0800 (China Standard Time)

conmon creates them

Is this new? I also noticed them today. I would swear they did not use to be there before.

Dominique Martinet · Answer 11 · Tue Dec 19 2023 20:00:39 GMT+0800 (China Standard Time)

They're not supposed to be in your root directory (just some run dir that is used by podman as per the path in options); the bug just made them appear in whatever current directory was used at the time because the path was incorrectly freed and reused for something else.

You can safely delete them.

jwhitaker-a365 · Answer 12 · Wed Dec 20 2023 08:41:04 GMT+0800 (China Standard Time)

hey @lsm5 , any chance of getting 2.1.10 into obs kubic? :)

AntonioSun · Answer 13 · Tue Jan 02 2024 02:09:37 GMT+0800 (China Standard Time)

Problem still exist in Mac? Can someone double-check please?

As I just installed yesterday and it is still failing for me on mac:

$ podman run -t ubuntu echo "Hello"
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob sha256:005e2837585d0b391170fd9faf2e0c279d64ba0eb011cda8dedf28cb5839861e
Copying config sha256:da935f0649133cbea2f5ad83db14bf782aa5ee9ad17cd609253e3750201a9298
Writing manifest to image destination
Error: preparing container 6dc501b868f5994fbde0decaa223c0275d7cb67448310bff3c3352dbd576d6f3 for attach: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

$ brew info podman
==> podman: stable 4.8.2 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/4.8.2 (191 files, 56.7MB) *
  Poured from bottle using the formulae.brew.sh API on 2023-12-31 at 15:17:20
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/p/podman.rb
License: Apache-2.0 and GPL-3.0-or-later

$ uname -srm
Darwin 23.1.0 arm64

$ sw_vers
ProductName:		macOS
ProductVersion:		14.1.2
BuildVersion:		23B92