Dynamic log level change doesn't work the same for all environments

[Philip Schmid]

What happened?

The Tetragon doc has a Change log level dynamically section that explains how to dynamically change it using SIGRTMIN+x. However, SIGRTMIN isn't the same value for every environment:

The Linux kernel supports a range of 33 different real-time signals, numbered 32 to 64. However, the glibc
POSIX threads implementation internally uses two (for NPTL) or three (for LinuxThreads) real-time signals (see
pthreads(7)), and adjusts the value of SIGRTMIN suitably (to 34 or 35).

It depends on libc, and, e.g., Alpine uses musl: https://git.musl-libc.org/cgit/musl/commit/src/signal/sigrtmin.c?id=016a5dc1925a66c7d1ffc14b862b1342f399cfda

We would likely need to develop a proper tetra CLI command to set the debug level in a unified and environment-independent way.

I ran into this issue while deploying Tetragon on a Talos Linux environment. For reproduction, you could use https://github.com/isovalent/terraform-aws-talos/tree/main/example.

Relevant Tetragon part:

tetragon/cmd/tetragon/main.go

Lines 310 to 329 in e44f7df

 case tgsyscall.SIGRTMIN_20: // SIGRTMIN+20 
 	currentLevel := logger.GetLogLevel() 
 	if currentLevel == logrus.DebugLevel { 
 		log.Infof("Received signal SIGRTMIN+20: LogLevel is already '%s'", currentLevel) 
 	} else { 
 		logger.SetLogLevel(logrus.DebugLevel) 
 		log.Infof("Received signal SIGRTMIN+20: switching from LogLevel '%s' to '%s'", currentLevel, logger.GetLogLevel()) 
 	} 
 case tgsyscall.SIGRTMIN_21: // SIGRTMIN+21 
 	currentLevel := logger.GetLogLevel() 
 	if currentLevel == logrus.TraceLevel { 
 		log.Infof("Received signal SIGRTMIN+21: LogLevel is already '%s'", currentLevel) 
 	} else { 
 		logger.SetLogLevel(logrus.TraceLevel) 
 		log.Infof("Received signal SIGRTMIN+21: switching from LogLevel '%s' to '%s'", currentLevel, logger.GetLogLevel()) 
 	} 
 case tgsyscall.SIGRTMIN_22: // SIGRTMIN+22 
 	logger.SetLogLevel(defaultLevel) 
 	log.Infof("Received signal SIGRTMIN+22: resetting original LogLevel '%s'", logger.GetLogLevel()) 
 } 

Tetragon Version

v1.1.0

Kernel Version

6.6.29-talos

Kubernetes Version

v1.29.3

Bugtool

No response

Relevant log output

No response

Anything else?

No response