Image display without Ueberzug
yingzhu146 opened this issue · comments
This plugin is literally all I ever wanted. Can't tell you how pumped I am. However, I'm, like so many, running Mac so this doesn't work right now.
(Not telling you anything new here probably but ueberzug isn't available on Mac)
Looks like #14 is a very good possible solution.
This plugin is literally all I ever wanted. Can't tell you how pumped I am.
Glad to hear it! It really warms my heart.
However, I'm, like so many, running Mac so this doesn't work right now.
(Not telling you anything new here probably but ueberzug isn't available on Mac)
I actually still had some hope that Ueberzug would work on Mac; evidently, it does not. This (more cross-platform image display) now becomes a bit of a priority for this project, then.
Looks like #14 is a very good possible solution.
What in there is a good possible solution? The image providers framework or support for hologram.nvim? (Remember that hologram.nvim only supports the Kitty protocol for now.)
Regardless, I'm actually experimenting with making my own ueberzug-esque image display method for terminals, and one of my goals is to be quite cross-platform -- so that could be of help. I might need some help soon for actually testing things on Mac, though.
This plugin is literally all I ever wanted. Can't tell you how pumped I am.
Glad to hear it! It really warms my heart.
However, I'm, like so many, running Mac so this doesn't work right now.
(Not telling you anything new here probably but ueberzug isn't available on Mac)I actually still had some hope that Ueberzug would work on Mac; evidently, it does not. This (more cross-platform image display) now becomes a bit of a priority for this project, then.
Looks like #14 is a very good possible solution.
What in there is a good possible solution? The image providers framework or support for hologram.nvim? (Remember that hologram.nvim only supports the Kitty protocol for now.)
Hologram.nvim - because it supports kitty. alacritty/alacritty#910 (comment) interesting discussion here, @dankamongmen built the really awesome https://github.com/dankamongmen/notcurses and sounds like kitty is the best solution available for terminal images right now
Regardless, I'm actually experimenting with making my own ueberzug-esque image display method for terminals, and one of my goals is to be quite cross-platform -- so that could be of help. I might need some help soon for actually testing things on Mac, though.
More than happy to test - just as disclaimer, I don't do CISC anymore, so I'm on an arm64 M1.
Again thanks a million, this can plugin can literally become huge for all data science people! Just needs Mac support, some good testing, a cool logo and a discord channel and maybe a YouTube walkthrough - looks like that's a good recipe for fast plugin adoption
happy to help if you have any Notcurses questions!
Hologram.nvim - because it supports kitty. alacritty/alacritty#910 (comment) interesting discussion here, @dankamongmen built the really awesome https://github.com/dankamongmen/notcurses and sounds like kitty is the best solution available for terminal images right now
Gotcha. Just need to be careful because we don't wan't to force people to use Kitty (that's why I initially went with Ueberzug) -- I think what we are developing with multiple possible image providers is the way to go for that, so that we can support cool things like the Kitty image protocol as well as providing methods which work on most terminal emulators, such as Ueberzug and viu.
More than happy to test - just as disclaimer, I don't do CISC anymore, so I'm on an arm64 M1.
Thanks! Once I have something concrete I'll give an update here.
Again thanks a million, this can plugin can literally become huge for all data science people! Just needs Mac support, some good testing, a cool logo and a discord channel and maybe a YouTube walkthrough - looks like that's a good recipe for fast plugin adoption
That's the plan! One step at a time, though :P
Hologram.nvim - because it supports kitty. alacritty/alacritty#910 (comment) interesting discussion here, @dankamongmen built the really awesome https://github.com/dankamongmen/notcurses and sounds like kitty is the best solution available for terminal images right now
I'd also consider WezTerm quite strong, along with Foot on Wayland. the author of iTerm2 agreed literally just this morning to make the necessary changes to support Notcurses-style graphics, so look for improvement there this year. Xterm is faithful, if slow, much like MLterm. Contour is an up-and-comer.
if you use Notcurses, you'll be able to target all of them through one API, and also the Linux console framebuffer on top of that. on terminals which don't have pixel graphics, it'll fall back seamlessly to Unicode sextants. doing all of this is non-trivial, so if you really want a terminal-portable solution, i ummm don't mean to spam your issue-tracker, but there's really only one game in town =]. like i said, happy to help with any questions or troubles.
I'd also consider WezTerm quite strong, along with Foot on Wayland. the author of iTerm2 agreed literally just this morning to make the necessary changes to support Notcurses-style graphics, so look for improvement there this year. Xterm is faithful, if slow, much like MLterm. Contour is an up-and-comer.
Interesting -- these things have advanced much more than I thought.
if you use Notcurses, you'll be able to target all of them through one API, and also the Linux console framebuffer on top of that. on terminals which don't have pixel graphics, it'll fall back seamlessly to Unicode sextants. doing all of this is non-trivial, so if you really want a terminal-portable solution, i ummm don't mean to spam your issue-tracker, but there's really only one game in town =]. like i said, happy to help with any questions or troubles.
Yeah, notcurses will definetly be one of our image providers. I'm sold :)
(I hope it works fine from inside neovim, what with its stdout trickery... will check later.)
I think we'll still keep our image provider abstraction, though, because I also want to support things like Ueberzug, which I'd say are a bit outside of notcurses' domain. But even then, notcurses will definetly be of great help!!
Just cheering from the sidelines here guys, thanks a ton to both of you, you're both legends! might even be able to revive/redirect the discussion upstream wrt to terminal graphics (neovim/neovim#12991) if we can show a cool implementation here, possibly switching to notcurses, always easier to show value with a working implementation that solves a use case.
(I hope it works fine from inside neovim, what with its stdout trickery... will check later.)
you definitely hand off a FILE* to notcurses_init(). it will dump most output there, but also look for a true terminal to do a few queries. so if you've redirected stdout, that's no problem; notcurses will render to stdout (presumably some pipe or something), and walk up to /dev/tty to find the actual terminal type.
unless it's doing something truly bizarre, it should either work, or i can make it work. care to elaborate on the stdout worries?
you definitely hand off a FILE* to notcurses_init(). it will dump most output there, but also look for a true terminal to do a few queries. so if you've redirected stdout, that's no problem; notcurses will render to stdout (presumably some pipe or something), and walk up to /dev/tty to find the actual terminal type.
Such foresight.
unless it's doing something truly bizarre, it should either work, or i can make it work. care to elaborate on the stdout worries?
I'm 90% sure it's just a redirection -- the tricky bit is that they do some extra Python magic for remote plugins (which is the mechanism we use) in sys.stdout to try to redirect prints to neovim's message system, so getting the actual stdout is not trivial (but it's also not impossible, if I'm not mistaken we already do it).
I'm 90% sure it's just a redirection -- the tricky bit is that they do some extra Python magic for remote plugins (which is the mechanism we use) in
sys.stdoutto try to redirect
alright; you seem to know what's going on in the world, so i'll let you be. feel free to mail me directly at nickblack@linux.com, or use our matrix channel, or just file a bug under "userquestions" rather than banging your head against anything. if we need do something weird to deal with this case, we'll do it, as you're not the first person doing in-editor stuff like this. hack on!
oh and though i'm not sure i'm following you completely, if you're willing to work at a C-ish level but need Python for the environment, the cffi-style wrappers are pretty much exactly that, so it would save you a round of going outside python. maybe. python is a mystery to me.
oooh i just saw your little infobox swing in from the right in your README's demo. that's fly as hell! you're going to love notcurses; it's built for exactly that kind of thing. go nuts.
alright; you seem to know what's going on in the world, so i'll let you be. feel free to mail me directly at nickblack@linux.com, or use our matrix channel, or just file a bug under "userquestions" rather than banging your head against anything. if we need do something weird to deal with this case, we'll do it, as you're not the first person doing in-editor stuff like this. hack on!
Thanks!! Actually, before you go, I'm struggling to find some documentation for the Python bindings of notcurses; is there any, or should I just base myself on the manpages?
oh and though i'm not sure i'm following you completely, if you're willing to work at a C-ish level but need Python for the environment, the cffi-style wrappers are pretty much exactly that, so it would save you a round of going outside python. maybe. python is a mystery to me.
Cool, thanks; seems like something I might need.
Actually, just found the notcuses.lib submodule, which seems to have direct bindings to the C library, so I think I'm OK with docs now.
ok so you will be using python? yeah, documentation there is going to be very sparse =[. so far as the wrappers are trivial, yeah, the manual pages are pretty complete. notcurses.h is maybe the best place to start. hrmm. there's also the book; it's a bit out of date, but most of the necessary changes are trivial: https://nick-black.com/htp-notcurses.pdf
Thanks!
In this case, it seems natural to try to use Python, since pretty much the whole plugin already uses it (some Python libraries were necessary); but, thankfully, the notcurses package gives us direct bindings to the C API, which I'm selecting as the first plan -- it neatly fits in with our current code, and follows the same pattern as the other fancy output features we currently provide (that they depend on some specialized package installable through PyPI, and so we can easily detect if we can use that feature or not).
From what I'm seeing, I feel like the manpages are enough, but I'll check out the book if I have trouble.
In this case, it seems natural to try to use Python, since pretty much the whole plugin already uses it (some Python libraries were necessary); but, thankfully, the notcurses package gives us direct bindings to the C API, which I'm selecting as the first plan -- it neatly fits in with our current code, and follows the same pattern as the other fancy output features we currently provide (that they depend on some specialized package installable through PyPI, and so we can easily detect if we can use that feature or not).
are things going alright, or has the sad state of our Python support derailed you? =\ the newer ones in python/ are supposedly superior to those in cffi/.
are things going alright, or has the sad state of our Python support derailed you?
I had to take a little "break" from this project, only doing minor stuff.
Don't worry, I'm quite persistent at dealing with sub-optimal Python libraries -- though I admit I was running into a bit of an issue with creating C structs from Python (but I didn't try much, tbh, I was already a bit short on time).
the newer ones in python/ are supposedly superior to those in cffi/.
Interesting. Will take a look once I'm properly back (which will only be in a few days, I'm afraid).
Hey @dccsillag no pressure but given that really nobody has yet produced a viable alternative to magma.nvim I just wanted to swing by and say I'd still very excited about approaches addressing this issue 😇
Oh, so sorry. This will defintely get fixed, don't worry -- IMO it's the thing in this repo with the highest priority.
However, I've been quite busy, and haven't had much time to tinker with magma-nvim lately. But I'll try to have this solved soon-ish.
Thanks so much!
Can I ask about the current state of MacOS support for images? I've got some time and would be happy to try help make this request a reality.
Sure! Your help is very welcome.
The crux of the issue is that we need to support ways to display images other than Ueberzug. Here are some possible ways to do it (without including terminal-specific protocols):
-
Unfortunately, it appears to be quite tricky to make notcurses work from within Neovim. But maybe a determined individual with time as yourself can make it happen!
-
One of the really appealing things about this is that it also provides a way to display images within standard terminals. It's absolutely not perfect, with the resolution dropping significantly, but I was very surprised at how decent it was, so it should be a good fallback. And, besides that, it seems to implement a couple of terminal-specific protocols.
BTW, this is what https://github.com/joouha/euporie uses. -
It looks like we'll definitely want to support this sometime in order to be able to properly display images from a non-terminal Neovim UI (e.g., Neovide). But it's still a bit in its infancy. Nevertheless, maybe this is the time to contribute to that as well in order to get things working more easily.
The current abstraction we have to implement these things are Canvass. It's somewhat documented, but if you have any questions feel free to ask. It might also happen that we need to change this abstraction in order to support some of the methods I listed above.
Now, besides all this, there is the apparent issue of #48; but it shouldn't be too tricky and I expect to solve later today (or maybe sometime tomorrow) in a reasonable manner.
All in all, it's quite possible, but will require some work. Thank you for your interest! I'm looking forward to advancing this front.
Okay so notcurses definitely seems like the most powerful, but possibly also overkill for what we need. Busy installing it now to give it a go and see what I can make it do.
timg was super simple but unfortunately tmux doesn't support sixels tmux/tmux#1613 so I'm not sure what to do since I'm in tmux all the time (and I imagine I'm not the only one).
Will keep on looking around.
I thought timg had an ANSI-only output as well?
Alacritty doesn't support sixel (yet), but I was able to get this:
Sure, it's not best (you can barely see the axis numbers!), but it seems like a reasonable fallback, and is plain ANSI.
(Also, I've found it a bit better than viu's output:)
At the end of the day, I doubt that tmux is going to support any sort of terminal image protocols, since tmux is basically an entirely separate terminal emulator. If your goal is to support tmux, then a good first step would be to look at what image protocols tmux would support.
Another side to approach this would be to basically reimplement Ueberzug but for MacOS. The key things that you would need to do for this are:
- Figure out the window corresponding to the current terminal
- Figure out the position of a given terminal cell in pixels, from the origin of the terminal's window
- Create a child window in the terminal's window containing the image we want to show.
In Ueberzug, (1) and (2) are the tricky steps: For (1), it tries to look at the process tree, and look for a window with a common PID, but if the terminal emulator does not use a single process for each window then this fails. For (2), Ueberzug can be pretty accurate by assuming that there are no additional UI components, other than the terminal grid (i.e., it would already break on, say, terminal tabs). (3) happens to be rather trivial by using X11.
In Mac, (1) and (3) will be trickier. Personally, I'm not too sure if all this work would be worth it, there's probably some simpler way.
Maybe @dankamongmen has some insight about images from tmux?
About images on tmux, the ideas here might be of help: https://gist.github.com/krtx/533d33d6cc49ecbbb8fab0ae871059ec
About images on tmux, the ideas here might be of help: https://gist.github.com/krtx/533d33d6cc49ecbbb8fab0ae871059ec
the tmux people are on the record about not wanting images sent through, iirc
Reimplementing Ueberzug is probably not feasible for me given my skill level and knowledge of the area, but I'll take a look and see if there're some easy wins that I can get started on. I'd very much like to see macos support happen so I'll keep digging.
Again, about images on tmux: you might want to take a look at https://github.com/csdvrx/sixel-tmux (also, its rant was an interesting read).
My recommendation is to set the tmux problem aside, and focus on getting some image protocol(s) working.
I've renamed this issue so that it's more accurate to what the actual problem is.
Alternative should be taken into consideration for Linux as well since the ueberzug stop maintained the project. So sad as of the reason he mentioned for a good opensource project.
seebye/ueberzug@b00c268
I just stumbled upon this plugin repository. I guess there was no (cross-platform) alternative found for ueberzug?
Seems like this new plugin could help make this a reality
So this is linked from the README as a place to look for alternatives; as a Mac user, is currently possible to use Magma, or not? I am confused about whether there is some workaround presented here or not.
Using another terminal seems to work (e. g. kitty), if that's a sacrifice you're willing to make
Chiming in to note that SIXEL support was merged into Ranger in April 2023 and it's been working well for me in Arch Linux using the foot terminal emulator. It looks like iTerm2 also supports this format, so this could resolve the issue for Mac users well.
Hah, I went down the rabbit hole on this a little, and came to the conclusion that adding support for notcurses might be the best way to handle image display cross-platform (it supports both kitty and sixel formats, can be installed on a wide range of operating systems, and is supposedly very fast).
After seeing some sample code for using notcurses as a matplotlib backend, I started to think it might not be too hard to add support to magma-vim.
As I was looking for pointers on how to use the notcurses python API, I stumbled upon this dankamongmen/notcurses#2308 and to my surprise it was started by @dccsillag, the creator/original maintainer of this repo!
It looks like the conversation stopped after running into some neovim-specific issues in getting the images to render. Is this discourse topic perhaps the best place to pick up the discussion?
how about ueberzugpp
