IPC::ShareLite store() error: Operation not permitted with mysql_ on fedora33

Question

IPC::ShareLite store() error: Operation not permitted with mysql_ on fedora33

dwreski opened this issue 4 years ago · comments

Hi,
I have a fedora33 install with mariadb-10.4.17 and munin-node-2.0.63 and trying to use the mysql_ plugin.

munin-run mysql_connections
IPC::ShareLite store() error: Operation not permitted at /usr/share/perl5/vendor_perl/Cache/SharedMemoryBackend.pm line 156.

This is effectively the same outcome for all plugins I've linked.

# pwd
/etc/munin/plugins
# ls -l mysql*
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_connections -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_innodb_log -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_innodb_queries -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:38 mysql_max_mem -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_network_traffic -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_performance -> /usr/share/munin/plugins/mysql_
lrwxrwxrwx 1 root root 31 Dec  6 10:37 mysql_slow -> /usr/share/munin/plugins/mysql_

The following are my /etc/munin/plugin-conf.d/00-default settings for mysql:

[mysql*]
   env.mysql /usr/bin/mysql
   env.mysqlopts -u munin --password=muninpass --host=localhost
   env.warning 2147483648
   env.critical 1073741824
   env.mysqluser munin
   env.mysqlpassword muninpass
   env.slave_io_running_warning 0.5
   env.slave_sql_running_warning 0.5
   env.seconds_behind_master_warning 3600
   env.seconds_behind_master_critical 86400
   env.wsrep_flow_control_paused_ns_warning 200000000
   env.wsrep_flow_control_paused_ns_critical 10000000000
   env.cachenamespace munin_mysql_pri
   env.mysqlconnection DBI:mysql:information_schema;host=127.0.0.1;port=3306

What more do I need to do to make this plugin work?

Lars Kruse · Answer 1 · Mon Dec 07 2020 06:57:19 GMT+0800 (China Standard Time)

Thanks for your report!

Did you try running the plugin via munin-run mysql_connections? Did you also try munin-run --ignore-systemd-properties mysql_connections?

In case the latter works (and the former fails): maybe the execution environment of your munin-node service restricts shared memory? (e.g. RemoveIPC=yes)

dwreski · Answer 2 · Mon Dec 07 2020 08:37:45 GMT+0800 (China Standard Time)

Thanks so much for your help.

It seems like it produces the IPC problem every third of fourth time I run "munin-run mysql_connections" (or any of the other mysql_ plugins). It apparently doesn't happen every time.

It also still occurs with the --ignore-systemd-properties. It requires ctrl-C to quit or it hangs indefinately.

# munin-run --ignore-systemd-properties mysql_connections
IPC::ShareLite store() error: Operation not permitted at /usr/share/perl5/vendor_perl/Cache/SharedMemoryBackend.pm line 156.

Here is the contents of the munin-node systemd service file.

# cat ./system/multi-user.target.wants/munin-node.service
[Unit]
Description=Munin Node
Documentation=man:munin-node(1) http://guide.munin-monitoring.org/en/latest/node/index.html
After=network.target network-online.target
PartOf=munin-asyncd.service

[Service]
Type=notify
ExecStart=/usr/sbin/munin-node --foreground
PrivateDevices=no
PrivateTmp=yes
ProtectHome=read-only
ProtectSystem=full
TimeoutStopSec=30s

[Install]
WantedBy=multi-user.target

Lars Kruse · Answer 3 · Sun Dec 13 2020 09:42:36 GMT+0800 (China Standard Time)

It seems like it produces the IPC problem every third of fourth time I run "munin-run mysql_connections" (or any of the other mysql_ plugins). It apparently doesn't happen every time.

that sounds interesting (in the weird way) :(

Thus we can rule out a configuration issue. Instead it seems to be some kind of resource usage collision.

Sadly I have no good understanding of the SharedMemoryBackend module. Thus I need to guess ...

Maybe multiple instances of this plugin are competing for a single resource or lock?

Maybe you could take a look at this discussion at serverfault? Does the recommended switch to the Cache::FileCache module fix your problem?

dwreski · Answer 4 · Mon Dec 14 2020 04:50:18 GMT+0800 (China Standard Time)

Hi, thanks for your continued support. I also saw this serverfault post, but wanted to be sure it wasn't otherwise fixed with an upstream version, etc. I've made the change to use FileCache instead of SharedMemoryBackend and it fixed it. That was from five years ago - I really thought there were more people using these plugins, particularly the ones that are included with the munin-node distro RPM.

There's also the other mysql_* plugins that are also included, but don't seem as capable. Makes it confusing for the end-user, though.

The only problem with the mysql_ plugin now is with mysql_innodb_queries producing no value, but I suspect that's another problem. I still have to troubleshoot that.

github-actions · Answer 5 · Sat Apr 17 2021 10:22:10 GMT+0800 (China Standard Time)

Stale issue message

Kim B. Heino · Answer 6 · Thu May 27 2021 19:07:52 GMT+0800 (China Standard Time)

It seems like it produces the IPC problem every third of fourth time I run "munin-run mysql_connections" (or any of the other mysql_ plugins). It apparently doesn't happen every time.

This sounds like SElinux problem. Can you check your audit.log file, or try to run it permissive mode?

You can also run "ipcs --shmems" and "ipcs --semaphores". Does it work when those lists are empty, and doesn't work when there is something owned by munin?