support for rm2fb
raisjn opened this issue · comments
when running rm2fb (as a package from https://github.com/toltec-dev/toltec), the behavior is the following:
- start rmview - connect and the screen displays properly
- the pen trail shows up correctly
- no further updates to the display are shown
when running rm2fb, there are two binaries with the paper plugin: remarkable-shutdown (hosting rm2fb server) and xochitl (which has been re-configured to send its updates to the server). is it possible that rmview is using xochitl (instead of remarkable-shutdown?)
PS: what logs / debugging information should i attach? all I have so far is the output of rmview which doesn't contain much information.
from @SyntaxBlitz on discord
I'm having some trouble on rM2.. it connects, shows the current screen on my computer, then nothing can write to the framebuffer anymore until I restart the device. clear-full-screen messages are still making it through (when I open a notebook and swipe left/right, or when I press the button to sleep), but nothing is actually drawn. just frozen on the initial screen (on the desktop client and on the actual device). pen location makes it through to the desktop client but no screen updates
i believe the issue is rm2fb because i uninstalled rm2fb, reboot the device and rmview works as expected
Thank you for the report! I will try to take a look.
It is particularly interesting because you use QMetaObject::invokeMethod, which is precisely the way in which I find sendUpdate (in the sense that it grabs the appropriate metaobject and eventually resolves to a call to the qt_static_metacall function that I emulate to find the address), so we certainly ought to be finding the same function.
when running rm2fb, there are two binaries with the paper plugin: remarkable-shutdown (hosting rm2fb server) and xochitl (which has been re-configured to send its updates to the server). is it possible that rmview is using xochitl (instead of remarkable-shutdown?)
The library finds out which process is controlling the framebuffer by looking for who has /dev/fb0 open. If you let xochitl open /dev/fb0 there may be some issue, but since this happens in the EPFramebuffer initialization routines I was assuming that you patch it prior to that.
In case you don't want to build a copy yourself (I do apologize for the build infrastructure at the moment! If you would like to but have any issues please let me know and I'll try to improve it), here is a version where the fb logic can be overridden for debugging. Could you try running it with LIBQSGEPAPER_FBCON_PID=<pid of rm2fb>?
PS: what logs / debugging information should i attach? all I have so far is the output of
rmviewwhich doesn't contain much information.
Unfortunately, most debugging is via gdb and snooping on the Unix socket used to send information around at this point, so if the process didn't die with an informative error there's not a lot of logs per se.
If you could send me the specific xochitl and remarkable-shutdown binaries you used (and any other shared objects needed to run them), it would make debugging quite a bit easier.
from @SyntaxBlitz on discord
I'm having some trouble on rM2.. it connects, shows the current screen on my computer, then nothing can write to the framebuffer anymore until I restart the device. clear-full-screen messages are still making it through (when I open a notebook and swipe left/right, or when I press the button to sleep), but nothing is actually drawn. just frozen on the initial screen (on the desktop
client and on the actual device). pen location makes it through to the desktop client but no screen updates
I suppose this might be caused by my hooking of sendUpdate if I ended up hooking xochitl and broke the frida interception (e.g. if it does something pc-relative, since it gets relocated by my own interception).
Thanks!
I tried the supplied binary and supplied the PID of remarkable-shutdown process, still no luck.
for rm2fb: can download the server and client.so files from: https://github.com/ddvk/remarkable2-framebuffer/releases/tag/v0.0.2
the short is: LD_PRELOAD=/path/to/server.so.1.0.0 /usr/bin/remarkable-shutdown to run server, to run client: LD_PRELOAD=/path/to/client.so.1.0.0 /usr/bin/xochitl (make sure to give full paths to all binaries).
i tested on 2.5 version of xochitl:
md5s are
- fec600ccae7743dd4e5d8046427244c0 /usr/bin/remarkable-shutdown
- aaa77163501618b0cfae54843505df88 /usr/bin/xochitl
Oh, dear, I think I see the issue. You have replaced the framebuffer that the EPFramebuffer draws from with your own shared memory. Then I have replaced it again with my own shared memory. Your clients write into your shared memory which is no longer mapped in the process, but EPFramebuffer and my server read from my shared memory which is never updated. The damage tracking is working fine, it's just that the repainted areas are being drawn from the old backing store.
I apologize for the delay---I had been looking just at your swtfb.cpp where you actually send updates, and did not realize that server/main.cpp is also doing things.
I don't know what a great fix that wouldn't involve an annoying amount of special-casing is. Since you are already setting up shared memory and already intercepting all surface damage, I think the best bet (unless you have any better ideas) is probably to provide a vnc backend that gets the information directly from your server without having to deal with the injection (I think you can easily make an LD_PRELOAD'able .so which adds this---it just needs a call to register_backend in its .ctors---and the vncs.c would need to be made a little bit smarter about allowing the user to choose a backend when multiple are available---I'd love a patch for this ;)). I would also be amenable to modifying libqsgepaper-interact to work with rm2fb-taken-over'd processes, but I'd need you to provide some means of identifying that the process has been modified by your library, such as magic numbers somewhere convenient (before/after the QImage of the framebuffer/its vtable, most probably). Let me know how you'd like to proceed.
(I do appreciate your zero-copy approach. Given how much work the software tcon thread is doing, I'm becoming more convinced that one or two extra copies would be fairly harmless, though, which is why I'm thinking of Wayland---I rather like the idea of having a standard protocol and a fairly decent Xorg DDX that already has support for XDamage reporting :). With appropriate buffer sizing optimizations and perhaps protocol changes, I think that wl_shm can probably get down to one extra copy, anyway, unless you're doing fancy things with subsurfaces (and perhaps sometimes even then). It's a bit annoying that the big wayland compositor libraries like wlroots are all so tightly tied to EGL/linux-dmabuf, though.)
(Oh, this is a bit tangential to the issue at hand, but as to compositor design---IIRC wayland recently came up with a fullscreen unredirect protocol extension to let fullscreen clients draw directly to the scanout buffer---so potentially that would provide the same zero-copy-ness).
my preference here is to avoid modifying rm2fb because there's already some set of people using it. oth - since its been installed via a package manager, it should be easy enough to roll out updates. in order of preference (based on least amount of modification)
-
the rm2fb process can be identified because it's opened fb0 and its cmdline will contain
remarkable-shutdown. until rM team removes that binary, its the one we plan on using. -
we can also add an environment variable to the binary (or otherwise add some magic values somewhere)
-
for the vnc backend option, maybe one method would be to add another message queue in rm2fb where updates will be sent to, then in this repo, we read from that queue and the shared memory in a new backend type.
my preference here is to avoid modifying rm2fb because there's already some set of people using it. oth - since its been installed via a package manager, it should be easy enough to roll out updates. in order of preference (based on least amount of modification)
That makes sense.
1. the rm2fb process can be identified because it's opened fb0 and its cmdline will contain `remarkable-shutdown`. until rM team removes that binary, its the one we plan on using.
I'd rather not rely on this, simply because remarkable-shutdown may be used for other things as well. So on the principle of being the least likely to accidentally trash something if run in an unknown environment, I'd prefer to have a bit of a stronger indication.
However, I could also imagine letting the user indicate that they are using rm2fb in which case the normal "looking for who is on /dev/fb0" process works.
2. we can also add an environment variable to the binary (or otherwise add some magic values somewhere)
Magic values around the framebuffer itself are probably the most reasonable thing to special-case support for in libqsgepaper-snoop.a. But see my comment below.
3. for the vnc backend option, maybe one method would be to add another message queue in rm2fb where updates will be sent to, then in this repo, we read from that queue and the shared memory in a new backend type.
That makes sense. It occurs to me that since you already have the framebuffer mapped, though, that perhaps the simplest thing to do is to add a config parameter to libqsgepaper-snoop.a that allows disabling grabbing the framebuffer, since I understand that you already have some way of allowing other processes to acquire the framebuffer. Then we could write another vnc backend that gets the damage updates via patching sendUpdate ourselves through my library, but gets the framebuffer from you.
If there are easily-locatable magic values in the patched process, then this can be combined with the normal rm2 backend and libqsgepaper-snoop.a can just decide dynamically whether to steal the framebuffer or to nab a file descriptor for it from wherever you are stashing it. Or, if you decide to prefer to avoid that, it can be an option for the user via backend selection as described above.
However, I could also imagine letting the user indicate that they are using rm2fb in which case the normal "looking for who is on /dev/fb0" process works.
That makes sense. It occurs to me that since you already have the framebuffer mapped, though, that perhaps the simplest thing to do is to add a config parameter to libqsgepaper-snoop.a that allows disabling grabbing the framebuffer, since I understand that you already have some way of allowing other processes to acquire the framebuffer. Then we could write another vnc backend that gets the damage updates via patching sendUpdate ourselves through my library, but gets the framebuffer from you.
this sounds good to me. the framebuffer is just opened as shared mem in /dev/shm/swtfb.01. i think rmview can detect if rm2fb is running (systemctl is-active rm2fb) and set the right config param to tell the snooper to read the framebuffer from the correct shared mem (or even accept the location as a config param)
btw: i like how you obtain the framebuffer without using magic bytes, nice work!
Hey, any update on this? If I understand this correctly, then launchers like remux and oxide require rm2fb (I am using remux on the rM2). If I do uninstall all of these, rmview works nicely, but not with them installed.
no updates, but looking at the code, i think adding a new backend is more appropriate. we will update rm2fb to emit dirty events, then we can have an rm2fb backend
I apologize for the lack of updates---I've been a bit busy and not had time to finish doing RM2 things, unfortunately.
I just got back to looking at this today. It looks like the sendUpdate hooking that libqsgepaper-snoop does right now will still work, and in fact even the address calculation for the fb address should work---we just need to turn off the "map a new shared memory framebuffer over the one used by the process" code. This is pretty easy---I just implemented the most important bit in libqsgepaper-snoop this morning. This binary should work with rm2fb if you run it as:
LIBQSGEPAPER_SNOOP_REQUESTED_FD=<fd in the target process that corresponds to the swtfb> /path/to/downloaded/server/binary
(If you have something else touching the mxs framebuffer, you can try making sure that it gets the right process via LIBQSGEPAPER_FBCON_PID=<pid of rm2fb server>)
It would presumably also work to grab the shared memory via the normal shmem mechanisms, rather than pulling it out of your process via a Unix socket, but this is a slightly smaller code change, seems to work with rm2fb, and should support other options as well.
I may also write a bit of code in backend-qsg.c to speak with systemd and get the appropriate pid and fd.
awesome!
It would presumably also work to grab the shared memory via the normal shmem mechanisms, rather than pulling it out of your process via a Unix socket, but this is a slightly smaller code change, seems to work with rm2fb, and should support other options as well.
how are people supposed to know the fd to pass to LIBQSGEPAPER_SNOOP_REQUESTED_FD=<fd in the target process that corresponds to the swtfb>? it seems more reasonable to just use open() (or shm_open) on /dev/shm/swtfb.01
how are people supposed to know the fd to pass to
LIBQSGEPAPER_SNOOP_REQUESTED_FD=<fd in the target process that corresponds to the swtfb>? it seems more reasonable to just use open() (or shm_open) on/dev/shm/swtfb.01
I was going to just grab it from reading /proc/<pid>/fd/ in backend-qsg (since one can get the pid from systemd over dbus), but that is indeed an issue. I can add support for using some other open()'d file in the client process instead, but since I don't know of any other consumers of the API (or specifically, such consumers that want support for this) I thought this might be easier. (Since one needs to speak to systemd/get the pid of rm2fb anyway to know whether or not to run this code path.)
ok, that makes sense - it is just some code that needs to connect the fd. i thought this was going to be something everyone has to do.
Ah, yes, that would have been a problem. The reason I haven't pushed the libqsgepaper code for release yet is that I was going to write that this evening---I just wanted to update that the tiny modification worked.
(I'll also need someone to test the wiring-fds-up code on a real device, since my emulator of course doesn't use systemd to launch the rm2fb.)
plenty of people will be glad to help, thanks!
The binary here ought to work automatically on systems where rm2fb is started through a systemd unit rm2fb.service. Otherwise, run the vnc server with QSG_RM2FB_PID=<pid of rm2fb server>.
If someone with real hardware can test with (1) no rm2fb at all, (2) toltec systemd-activated rm2fb, (3) rm2fb run without systemd (via the environment variable mentioned above), I'll release this and ping the maintainer of rmview about the update.
If you do end up adding a separate change tracking export API to rm2fb, I'll might also write a tiny (simpler) backend for that one as well.
case 1: can not connect, server side it says it started server and is listening on ports but nothing happens when running rmview
case 2: i'm able to successfully connect in case 2 and can disconnect / restart the standalone server and reconnect successfully.
case 3: i can not connect. once i start standalone server, it look like it is ready for connections but rmview connections do not work.
after trying case 1 or 3, i can no longer successfully run case 2.
Hmm, that is very strange. All of the cases work in my VM.
Could you please send me the output of ls -l /proc/<pid of rM-vnc-server>/fd/ in each case, and ls -l /proc/<pid of rm2fb server>/fd/ in any case where rm2fb is running?
What exactly do you mean by "nothing happens when running rmview"? Do even the normal messages that libvncserver prints to stdout on client connect not appear? If not, I am particularly surprised---I can't think of anything that should let it get as far as starting the server thread but then block so as to prevent the connection from being accepted. You can check as well from the device---nc localhost 5900 ought to cause a client connection message on the server stdout and an RFB 003.008 header should show up.
If it gets as far as starting the server thread but then does the old behavior with the framebuffer, that makes more sense. Although, case 1 really should always work---if systemd does not tell it that rm2fb is running (or if it sees that the process controlling the framebuffer does not have the shared memory file open)---it will fall back to precisely the old behavior.
how are you checking it is running with systemd? is it isactive or isenabled?
it does start the vnc server and when client connects it does say connected but nothing shows up. i will post what you asked for tomorrow AM
Thank you! Oh, I realized that the contents of /proc/<pid of rM-vnc-server>/environ may also be useful.
That behavior makes somewhat more sense. I am still not sure what would cause it in the first case, but the third makes sense.
It is checking via the ActiveState property in dbus on the unit, and then it gets the pid of the rm2fb server process by means of GetProcesses. Then, it searches in the /proc/pid/fd directory of that process for a file descriptor which points to a path with a basename of swtfb.01. So, if the rm2fb is not actually running there is very little chance it will think it is.
@ raisjn Do you mind sharing the toltec programs you had installed during case 1?
@raisjn ^ (sorry, copy/paste error)
@pl-semiotics FWIW, it seems to be working for me in all 3 cases.
i ran through test cases this morning. it's likely an error on my part in case 3.
case 1: removed rm2fb via opkg, rebooted, ran standalone server - success
case 2: installed rm2fb via opkg, rebooted, ran standalone server - success. disabled and re-enabled standalone then reconnected, success
case 3: install rm2fb via opkg, start using LD_PRELOAD=path/to/server.so /usr/bin/remarkable-shutdown. run standalone PID passed in. try to connect with rmview config (that worked above):
reMarkable: ~/ QSG_RM2FB_PID=502 ./rM2-vnc-server-standalone
Using backend libqsgepaper-snoop
28/02/2021 12:53:07 Listening for VNC connections on TCP port 5900
28/02/2021 12:53:07 Listening for VNC connections on TCP6 port 5900
...
reMarkable: ~/ ls -l /proc/502/fd/
lrwx------ 1 root root 64 Feb 28 12:50 0 -> /dev/pts/1
lrwx------ 1 root root 64 Feb 28 12:50 1 -> /dev/pts/1
lrwx------ 1 root root 64 Feb 28 12:50 2 -> /dev/pts/1
lrwx------ 1 root root 64 Feb 28 12:50 3 -> /dev/shm/swtfb.01
lrwx------ 1 root root 64 Feb 28 12:50 4 -> /dev/fb0
lr-x------ 1 root root 64 Feb 28 12:50 5 -> /dev/urandom
lrwx------ 1 root root 64 Feb 28 12:51 6 -> socket:[9942]
lrwx------ 1 root root 64 Feb 28 12:53 7 -> socket:[11459]
reMarkable: ~/ ls -l /proc/670/fd
lrwx------ 1 root root 64 Feb 28 12:53 0 -> /dev/pts/2
lrwx------ 1 root root 64 Feb 28 12:53 1 -> /dev/pts/2
lrwx------ 1 root root 64 Feb 28 12:53 10 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Feb 28 12:53 11 -> socket:[11463]
lrwx------ 1 root root 64 Feb 28 12:53 2 -> /dev/pts/2
lr-x------ 1 root root 64 Feb 28 12:53 3 -> /home/root/.cache/libqsgepaper-snoop/e6b476be908dee36483fae4a0d9a2650177431f594786184b2ff71a1de59e557
lrwx------ 1 root root 64 Feb 28 12:53 4 -> socket:[11458]
lrwx------ 1 root root 64 Feb 28 12:53 5 -> /dev/shm/swtfb.01
lrwx------ 1 root root 64 Feb 28 12:53 6 -> /dev/input/event0
lrwx------ 1 root root 64 Feb 28 12:53 7 -> /dev/input/event1
lrwx------ 1 root root 64 Feb 28 12:53 8 -> /dev/input/event2
lrwx------ 1 root root 64 Feb 28 12:53 9 -> socket:[11462]
reMarkable: ~/ cat /proc/670/environ
QSG_RM2FB_PID=502LS_COLORS=no=01;37:fi=01;37:di=01;34:ln=01;36:pi=01;32:so=01;35:do=01;35:bd=01;33:cd=01;33:ex=01;31:mi=00;37:or=00;36:SSH_CONNECTION=::ffff:10.11.99.5 41852 ::ffff:10.11.99.1 22HISTCONTROL=ignoreboth:erasedupsQT_QPA_EVDEV_TOUCHSCREEN_PARAMETERS=rotate=180EDITOR=vimSSH_AUTH_SOCK=/tmp/dropbear-94b53c5a/auth-4349c032-7USER=rootPAGER=lessPWD=/home/rootFCEDIT=vimHOME=/home/rootSSH_CLIENT=::ffff:10.11.99.5 41852 22TMUX=/tmp//tmux-0/default,490,0HISTFILE=/home/root/.bash_historySSH_TTY=/dev/pts/0VISUAL=vimTERM=screenSHELL=/bin/shQMLSCENE_DEVICE=epaperTMUX_PANE=%1SHLVL=2PROMPT_COMMAND=history -aELIDED_PATH=$(echo -n "${PWD/#$HOME/\~}" | awk -F "/" '{
if (length($0) > 50) { if (NF>4) print $1 "/" $2 "/.../" $(NF-1) "/" $NF;
else if (NF>3) print $1 "/" $2 "/.../" $NF;
else print $1 "/.../" $NF; }
else print $0;}')QT_QPA_PLATFORM=epaperLOGNAME=rootPATH=/opt/bin:/opt/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbinPS1=\[\033[00;37m\]reMarkable\[\033[01;31m\]: \[\033[01;37m\]$(echo -n "${PWD/#$HOME/\~}" | awk -F "/" '{
if (length($0) > 50) { if (NF>4) print $1 "/" $2 "/.../" $(NF-1) "/" $NF;
else if (NF>3) print $1 "/" $2 "/.../" $NF;
else print $1 "/.../" $NF; }
else print $0;}')/\[\033[00m\] HISTSIZE=-1_=./rM2-vnc-server-standalonereMarkable: ~/
reMarkable: ~/
EDIT: after case 3 fails, I need to restart the tablet for case 2 to succeed again.
i think case 3 isn't as important as 1 or 2 and am fine with it not working (for me) since it would mostly affect rm2fb devs who are in the middle of active development and want to use rmview. EDIT: it would also affect people who want to stream KOReader using ddvk's installer (but streaming xochitl would be fine)
i am curious why i can't get it working - i can try the test cases again later a few more times and check if i am doing something wrong.
So first...drooling over your shape drawing....
Anyway, one difference between my setup and your setup for case #3 is I downloaded the rm2fb server+client from ddvk's repo directly, rather than from opkg. Maybe that could make a difference somehow?
@LeifAndersen it could be, i'll try it out again later with the release binaries. i did a quick uninstall of rm2fb and used my locally compiled rm2fb binaries and it still didn't work.
if it works for other people, then its likely something with my setup
Another possibility, did you leave remux or any other launchers installed?
@raisjn I think your problem may be that rmview is actually starting another vnc server itself, as I don't see anything (on a quick inspection) in its source that suggests it detects when one is already running. This one, it starts without the necessary environment in case 3, so it ends up remapping the framebuffer in the non-rm2fb mode, which breaks things.
(One might think that this would cause different behavior, where the VNC client works but the actual display stops updating. I think that rmview is looking for messages on stderr to decide that the server has started up, or something, and is never initiating a vnc connection in the third case. Looking at the output of your second case, something is actually wrong that is causing the version of the server in rmview to refuse to start---the "bad cache preamble 0" line---so it sees the error message, thinks that it has started, and then ends up connecting to the existing vnc server. The bad cache preamble line would seem to indicate that the "detect a process that has already had its framebuffer stolen and do the right thing" logic is not working quite right, but it's clearly mostly working since the "stop and restart the vnc server" case worked. Possibly there is some strange interaction due to rmview using a different version of the server executable? Although I've never changed the magic numbers that that is looking for, so I'm not quite sure why that would be.)
yes, i think you are exactly right (that the issue is using rmview). i just connected through regular VNC viewer on test case 2 and 3 and everything works fine (doh, why didn't i realize rmview is starting server
@pl-semiotics @raisjn sorry I haven't followed this discussion closely, is there anything I can change in rmview that would resolve compatibility with rm2fb?
I believe we have something ~working, so I just need to finish a tiny bit of cleanup/publishing/making a new release binary and hand it over to you. I wanted to try to add the new input protocol quickly at the same time to reduce overhead, but unfortunately I've gotten unexpectedly busy IRL and haven't had the time to design and implement something decent yet. I hope I'll have a little time to poke at things on Friday, at which point I may just drop in the "send an up immediately after each event" as a stopgap measure since I don't have anything that'll be too upset by that yet.
W.r.t. to these changes for rm2fb: if rm2fb is being run via the toltec systemd unit, it'll Just Work(TM). If rm2fb is being run a different way, you'll need to set an environment variable with its PID when launching the vnc server.
@pl-semiotics that's great! Just ping me when you make progress and I'll integrate all the necessary changes asap.
Thanks for your amazing work!
I came here after posting in #8 thinking that it is a software compatibility problem. @bordaigorl thankfully pointed out the issue with rm2fb which is indeed the case. Perhaps put a notice in the main README? Preferably with instructions how to disable packages with opkg.
(I literally installed opkg for the first time yesterday and rm2fb is one of the first things I had to install. Now I removed rm2fb because I do not know how to disable it without removing...)
Oh dear, I thought we had the rm2fb-compatible executable in rmview by now---I apologise for the miscommunication & delay. I shall coordinate with @bordaigorl to make sure the fix is disseminated appropriately as soon as the 2.6/2.7 fix is done.
Oh dear, I thought we had the rm2fb-compatible executable in rmview by now---I apologise for the miscommunication & delay. I shall coordinate with @bordaigorl to make sure the fix is disseminated appropriately as soon as the 2.6/2.7 fix is done.
Is there some progress, please?
I apologise again for the delay in resolving this issue. I believe that these binaries should work on both rm1 and rm2, both with and without rm2fb in the latter case, under the new system software revisions. If people with hardware/rm2fb could corroborate this, I will go ahead and make an official release.
Sorry for not having time to test it before. It works now and does not freeze! Thanks a lot @pl-semiotics !
Oh no, I have just updated to software version 2.9 and got #10...