Xpra-org / xpra

We see that sometimes Xpra keeps starting Xorg-for-Xpra processes with constantly rising display numbers.
Then after 6 hours 1800 processes are running (every ~10 seconds a new one pops up) and the system is blocked.

Xorg-for-Xpra-S24055 -noreset -novtswitch -nolisten tcp +extension GLX +extension RANDR +extension RENDER -auth /p/home/jusers/.Xauthority -logfile /tmp/xpra_sockets_19601_8651gz3u/xpra/S24055/Xorg.log -configdir /tmp/xpra_sockets_19601_8651gz3u/xpra/S24055/xorg.conf.d/24055 -config /p/software/juwels/stages/2024/software/xpra/5.0.8-GCCcore-12.3.0/etc/xpra/xorg.conf -depth 24 -displayfd 12

I wonder if the reason for this is the timeout from write_displayfd
https://github.com/Xpra-org/xpra/blob/v5.0.8/xpra/platform/displayfd.py#L16
Instead of stopping Xpra when the timeout occurs, it continues to endlessly start new "Xorg-for-Xpra" processes.

It could also be from read_displayfd - but that would not match the 10-second-frequency of respawns.
https://github.com/Xpra-org/xpra/blob/v5.0.8/xpra/platform/displayfd.py#L50

We are using Xpra 5.0.8 on Linux systems (Redhat 8)
and xpra gets started by https://github.com/FZJ-JSC/jupyter-xprahtml5-proxy with the following command

xpra start --html=on --bind=/tmp/xpra_sockets_19601_8651gz3u/xpra-server,auth=none --socket-dir=/tmp/xpra_sockets_19601_8651gz3u --start=xterm -fa "DejaVu Sans Mono" -fs 14 --clipboard-direction=both --no-keyboard-sync --no-mdns --no-bell --no-speaker --no-printing --no-microphone --no-notifications --no-systemd-run --sharing --no-daemon

I'm confused, xpra only starts the vfb once on startup. Why would you get 1800 processes?

Thank you for your reply.

If xpra does not retry to start the vfb but would just exit with an error
then it might be jupyter-server-proxy which restarts the process over and over again. 🤔

But could it be that xpra does not clean up the Xvfb-for-Xpra process if it fails to start?
Perhaps here: https://github.com/Xpra-org/xpra/blob/v5.0.8/xpra/platform/displayfd.py#L37

I have to investigate more on this. Until now I just see this behavior and try to find out why it happens sometimes.
Unfortunately, this is quite rare and difficult to reproduce - at least until I know how to force it.

@jhgoebbert the

xpra/xpra/platform/displayfd.py

Line 16 in 7c3ed93

def write_displayfd(w_pipe, display, timeout=10):

function is used by xpra if xpra's own --displayfd command line option is used, typically so a process wrapping xpra can be told what display was allocated.

The one used for parsing the displayfd sent by the vfb subprocess is:

xpra/xpra/platform/displayfd.py

Line 50 in 7c3ed93

def read_displayfd(r_pipe, timeout=DISPLAY_FD_TIMEOUT, proc=None):

I assume that this is the one you refer to.
It is called from here:

xpra/xpra/x11/vfb_util.py

Lines 283 to 313 in 7c3ed93

    
           if use_display_fd: 
        
               def displayfd_err(msg): 
        
                   raise InitException(f"{xvfb_executable}: {msg}") 
        
               r_pipe, w_pipe = os.pipe() 
        
               try: 
        
                   os.set_inheritable(w_pipe, True)        #@UndefinedVariable 
        
                   xvfb_cmd += ["-displayfd", str(w_pipe)] 
        
                   xvfb_cmd[0] = f"{xvfb_executable}-for-Xpra-" + display_name.lstrip(":") 
        
                   def preexec(): 
        
                       os.setpgrp() 
        
                       if getuid()==0 and uid: 
        
                           setuidgid(uid, gid) 
        
                   try: 
        
                       # pylint: disable=consider-using-with 
        
                       # pylint: disable=subprocess-popen-preexec-fn 
        
                       xvfb = Popen(xvfb_cmd, executable=xvfb_executable, 
        
                                    preexec_fn=preexec, cwd=cwd, pass_fds=(w_pipe,)) 
        
                   except OSError as e: 
        
                       log("Popen%s", (xvfb_cmd, xvfb_executable, cwd), exc_info=True) 
        
                       raise InitException(f"failed to execute xvfb command {xvfb_cmd}: {e}") from None 
        
                   assert xvfb.poll() is None, "xvfb command failed" 
        
                   # Read the display number from the pipe we gave to Xvfb 
        
                   try: 
        
                       buf = read_displayfd(r_pipe) 
        
                   except Exception as e: 
        
                       buf = b"" 
        
                       log("read_displayfd(%s)", r_pipe, exc_info=True) 
        
                       displayfd_err(f"failed to read displayfd pipe {r_pipe}: {e}") 
        
               finally: 
        
                   osclose(r_pipe) 
        
                   osclose(w_pipe)

I think that you're right and the exception handler should probably kill the vfb at that point.

This problem should be pretty rare, after all the default is 20 seconds!

xpra/xpra/platform/displayfd.py

Line 13 in 7c3ed93

DISPLAY_FD_TIMEOUT = envint("XPRA_DISPLAY_FD_TIMEOUT", 20)

Please try 8a4a964 (untested).
You should be able to reproduce it more readily using XPRA_DISPLAY_FD_TIMEOUT=1.

Hi @totaam,
thank you very much for your patch! This looks good - it will ensure that no "zombie"-XServer are left behind. Perfect.
But I have not yet tested it because our systems are down at the moment.

In the meantime I am still trying to understand why Xpra gave up and exited in some situations in the first place.
At the moment we have the impression that the following is happening:

Xpra starts an XServer. This XServer searches for a free display by itself. Presumably via the entries in /tmp/.X11-unix. If it findes, for example, that the next free display is :4 it starts on :4.
Xpra then asks the XServer for the display number it has choosen and checks /tmp/<HOSTNAME>-<DISPLAYNUM> to see whether an Xpra socket file for display :4 already exists.
If this is the case - perhaps because another Xpra session of some other user did not clean up correctly and left the socket file behind (?) - Xpra concludes that the display number is already in use by some other Xpra server running on :4 and so it then exits.
In our case jupyter-server-proxy now detects that the Xpra process died exit and restarts it.

We cannot yet understand why this step 3 could lead to a loop in which 1800 XServers got started.
The only idea I have is, that Xpra started a new XServer on a new display but always connected to the fist XServer it started in the loop to end up in the same exit-situation. So your patch might also solve this.

I will come back to you as soon as I have tested your patch.

btw: We could see this behavior always then, when xpra list shows an INACCESSIBLE session.
I assume the reason for an INACCESSIBLE session is that some other user is running Xpra on that display or some socket-file has not been cleaned up.

[]$ xpra list
Found the following xpra sessions:
/tmp:
	INACCESSIBLE session at :1
	INACCESSIBLE session at :2
	LIVE session at :33
	INACCESSIBLE session at :4
	INACCESSIBLE session at :5

This then leads to the error when starting xpra:

xpra start --start=xterm --no-daemon
using systemd-run to wrap 'seamless' xpra server subcommand
Running scope as unit: run-r9384454326834c31a92b18ffc5554b26.scope
2024-06-06 15:23:48,043 Warning: cannot enable SSH socket upgrades
2024-06-06 15:23:48,043  No module named 'paramiko'
2024-06-06 15:23:48,046 no uinput module (not usually needed)
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
[..]
2024-06-06 15:23:50,367 xpra server initialization error:
2024-06-06 15:23:50,367  An xpra server is already running at '/tmp/jwvis02.juwels-5'

Xpra concludes that the display number is already in use by some other Xpra server running on :4 and so it then exits.

No, it will probe this socket to see if it belongs to a dead server or not.
If the socket's modified timestamp is recent, it will wait longer - potentially waiting for a server to complete its startup sequence.
(servers continuously update this timestamp to help other processes detect the socket as "alive")

Servers can take a long time to startup because of pyxdg and the loading of all the commands' icons.

I assume the reason for an INACCESSIBLE session is that some other user is running Xpra on that display or some socket-file has not been cleaned up.

xpra/xpra/common.py

Line 65 in e10a1e8

INACCESSIBLE = "INACCESSIBLE"

The only way for xpra to show this state is from here:

xpra/xpra/platform/dotxpra.py

Lines 118 to 119 in e10a1e8

    
           if err == errno.EACCES: 
        
               return SocketState.INACCESSIBLE

errno.EACCES typically means that the permissions on the socket do not allow the current user to connect to it.
Unless someone manually does something very weird with chmod, this means that the socket belongs to a different user.
I see that you're using /tmp for your sockets.
That's not a good location for a multi-user system, and even for a single user system, /tmp is special (sticky bit).
If you are unable to use XDG_RUNTIME_DIR, you may want to add users to the xpra group and use /run/xpra instead.

Thank you for all these great hints.

	if use_display_fd:
	def displayfd_err(msg):
	raise InitException(f"{xvfb_executable}: {msg}")
	r_pipe, w_pipe = os.pipe()
	try:
	os.set_inheritable(w_pipe, True) #@UndefinedVariable
	xvfb_cmd += ["-displayfd", str(w_pipe)]
	xvfb_cmd[0] = f"{xvfb_executable}-for-Xpra-" + display_name.lstrip(":")
	def preexec():
	os.setpgrp()
	if getuid()==0 and uid:
	setuidgid(uid, gid)
	try:
	# pylint: disable=consider-using-with
	# pylint: disable=subprocess-popen-preexec-fn
	xvfb = Popen(xvfb_cmd, executable=xvfb_executable,
	preexec_fn=preexec, cwd=cwd, pass_fds=(w_pipe,))
	except OSError as e:
	log("Popen%s", (xvfb_cmd, xvfb_executable, cwd), exc_info=True)
	raise InitException(f"failed to execute xvfb command {xvfb_cmd}: {e}") from None
	assert xvfb.poll() is None, "xvfb command failed"
	# Read the display number from the pipe we gave to Xvfb
	try:
	buf = read_displayfd(r_pipe)
	except Exception as e:
	buf = b""
	log("read_displayfd(%s)", r_pipe, exc_info=True)
	displayfd_err(f"failed to read displayfd pipe {r_pipe}: {e}")
	finally:
	osclose(r_pipe)
	osclose(w_pipe)

sometimes Xpra keeps on starting Xvfb-for-Xpra processes endlessly