GFX: graphic glitches when a monitor is more than 4096 pixels wide or high

[metalefty]

xrdp version

0.10.0-beta.3

Detailed xrdp version, build options

xrdp 0.10.0-beta.3
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2024 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-imlib2
      --enable-jpeg
      --enable-painter
      --enable-pam-config=freebsd
      --enable-pixman
      --enable-rfxcodec
      --enable-strict-locations
      --enable-vsock
      --localstatedir=/var
      --with-pkgconfigdir=/usr/local/libdata/pkgconfig
      --disable-devel-debug
      --disable-devel-logging
      --enable-fdkaac
      --disable-fuse
      --enable-ipv6
      --disable-mp3lame
      --enable-opus
      --x-libraries=/usr/local/lib
      --x-includes=/usr/local/include
      --prefix=/usr/local
      --mandir=/usr/local/share/man
      --disable-silent-rules
      --infodir=/usr/local/share/info/
      --build=amd64-portbld-freebsd14.0
      build_alias=amd64-portbld-freebsd14.0
      CC=cc
      CFLAGS=-O2 -pipe  -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing
      LDFLAGS= -lssl  -fstack-protector-strong
      LIBS=-L/usr/local/lib
      CPPFLAGS=-isystem /usr/local/include
      CXX=c++
      CXXFLAGS=-O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing  -isystem /usr/local/include
      PKG_CONFIG=pkgconf
      PKG_CONFIG_LIBDIR=/ssd/ports-wrkdir/home/meta/FreeBSD/ports/git/net/xrdp-devel/work/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig
      OPENSSL_CFLAGS=-I/usr/include
      OPENSSL_LIBS=-L/usr/lib -lcrypto -lssl
      CPP=cpp

  Compiled with OpenSSL 3.0.13 30 Jan 2024

Operating system & version

FreeBSD 14 but actually not OS specific

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

xorgxrdp 0.10.1

What desktop environment do you use?

Xfce but any

Environment xrdp running on

any

What's your client?

Win 11 mstsc.exe

Area(s) with issue?

Graphic glitches

Steps to reproduce

Start an xrdp session at least one monitor has more than 4096 pixels wide or high.
We can start an RDP session larger than the physical monitor if specifying width/height via command line options.

1. Press Win+R shortcut key
2. Run mstsc /w:4160 /h:2160 /v:xrdp-server.example.com to specify screen
3. Login with valid credentials

✔️ Expected Behavior

Works fine as well if the width or height is less than or equal to 4096 pixels.

❌ Actual Behavior

No response

Anything else?

This issue is similar to #2063 but there's no protocol error. #2063 happens if the total screen width is more than 8192. However, this issue happens when one of the monitors has a width of more than 4096.

Screenshot:

[metalefty]

@jsorg71 Does librfxcodec encoder support more than 4096 pixels per screen/encoder?

[jsorg71]

Oh no, the history only had 64x64 64x64 tiles so that is 4096x4096 pixels. I don't think it's a codec limitation, just an implementation limit. We can improve that.

[metalefty]

@jsorg71 Could extend librfxcoec to support screens larger than 4096x4096 pixcels?

[rowlap]

Is this limit set by the various 64 and 4096 constants in https://github.com/neutrinolabs/librfxcodec/blob/devel/src/rfxencode.h ? We see a fair few matches searching for 4096, so this doesn't look like a trivial patch to a #define.

[jsorg71]

I don't have a 4K monitor to test but I created neutrinolabs/librfxcodec#65 to possibly fix this.

[jsorg71]

I was able to edit an rdp file to get a single monitor > 4K and it seems to work ok.
@metalefty are you able to test?

[rowlap]

@jsorg71 please see the repro steps above; we can just pass /w and /h to simulate ~any arbitrary screen size.

[jsorg71]

thanks @rowlap I am able to test now. I also went back to devel branch to make sure I can repro the issue(which I should have done first) and I can so I think it's fixed.

[metalefty]

Works fine for me now.

[metalefty]

Merged neutrinolabs/librfxcodec#65.

[metalefty]

I will close this after updaitng submodule.