Command output with ninja build system broken
marwing opened this issue · comments
Bug description
When generating a ninja based build system instead of a Makefile based one the display of ninjas output during build doesn't work as it should. There are two components to this issue:
- Ninjas output only shows up when the command exits
- All output is displayed next to each other in a single line
To Reproduce
Steps to reproduce the behavior:
- Create a CMake project or use an existing one
- Make sure you have ninja installed
- open in vim/neovim and run :CMakeGenerate Debug -GNinja
- run :CMakeBuild
- wait for command to finish and see output
Expected behavior
I would expect ninjas output to be either shown as in a normal terminal or as when piped through a command like cat.
For reference run the following commands in the build folder:
ninjaninja | catBehavior with minimal .vimrc
Same as above with empty vimrc/init.vim
Screenshots
Actual:
Expected normal:
Expected cat:
Other info
- OS: ArchLinux 5.18.14
- Vim/Neovim version: NVIM v0.8.0-dev-726-gb971547c54
- Vim-CMake version (5th line in
:help cmake): 0.9.0 / d62f24d - CMake version (
cmake --version): 3.23.2
Additional context
Ninja outputs carriage returns instead of newlines to only have one line of output. Vim-CMake removes carriage returns and waits for the entire line before sending it to the console window. I could experimentally patch the first part by just disabling the relevant bit of code but don't know what that entails other than the intended (code is probably there for a reason), the second part takes some more knowledge of the code than I have to do right probably.
Hi, thanks a lot for reporting this.
I had indeed overlooked the situation with build systems other than Makefile-based ones. There are a number of things I assumed to be universal to command outputs which are not. Here's a summary of the differences between Makefile and Ninja generators I've encountered so far. These should perhaps end up being separate issues to be tackled independently.
Line endings
As you noticed, Ninja does not output newline (NL) characters, only carriage return (CR) characters. Neovim channels convert NLs into empty strings, which are used to delimit lines of output (:help channel-lines). CRs are left untouched instead.
First of all, Vim-CMake relies on the above logic to identify lines, which results in the first issue you mention: Ninja's output only shows up when the command exits. To fix that, we should have Vim-CMake print stdout data as it comes in, without waiting for line delimiters. I'm sure I wanted to avoid this mechanism for a reason, which now I'm forgetting, and which I've also forgotten to document. But it could be to ensure compatibility on multiple OSes. I'll have to think about this further, and I'll also have to figure out how Vim behaves in this situation.
Second, Vim-CMake strips CRs from stdout data, as I had not contemplated that some commands use CRs to overwrite lines. Also here I don't know exactly what would be the consequence of not stripping CR characters, but again I think it's some compatibility thing (maybe the terminal emulator Neovim uses in Windows translates CR into NL). I'll have to look into this as well.
For the second problem, forcing Ninja to output multiple lines could be a way to work around this problem (perhaps it should even be a configurable feature?). But I can imagine that this is not a very portable solution, I don't think Ninja has a built-in options to do that.
Stdout line width
I also observed that Ninja truncates stdout lines, probably to ensure that each line fits into a single terminal row. And I guess it achieves this by checking the width property of the pseudo-terminal where the command runs. In general, the width of the pseudo-terminal is not the same as that of the Vim-CMake window. Even this is not a major issue, we might want to find a way to fix this.
Build targets
Vim-CMake uses the help target to retrieve a list of available build targets. When running cmake --build <build_dir> --target help, there are a few differences between the output for Makefile systems and that for Ninja systems—e.g. the format is different, and Ninja systems also return a lot more entries.
We could address this by having a dedicated parser for each type of output, i.e. one for Makefile systems, one for Ninja systems, etc. However I don't think this is very scalable and maintainable. A cleaner option would be to use native CMake functionalities to get a list of build targets, but unfortunately there isn't an obvious CMake command to do that—we'd have to write our own script that reads CMake properties (e.g. based on https://stackoverflow.com/a/62311397/13297923).
I went a little bit into too much detail perhaps, as I'm curious to hear your thoughts on these issues, if you have any. I'll try to find time soon to split this issue into multiple tickets that can be addressed independently and in parallel. I'll also then prioritize this issue over other feature requests. And as always contributions are welcome!
Thanks for the extensive response. That is indeed more feedback than I expected but I do have some thoughts on these issues. I do have to prefix them with a disclaimer though: I know next to nothing about the APIs and details surrounding embedded terminals in vim and neovim, especially considering the differences between vim's and neovim's versions.
Line endings
Both neovim and vim do the right thing when running :terminal ninja -C <path to build dir> in a ninja based buildtree. Based on this observation I would assume the problem is in how Vim-CMake buffers/passes lines to the terminal buffer. Ideally there was a way to just open a terminal in something like a managed mode (for it to not close automatically/display a command exited message) and then just run a command in said terminal buffer. Alternatively I would just pass all data as soon as is available as I already know the terminal knows how to handle it. I don't know how this interacts with channels however so this might not work as I expect.
For stripping CRs the only thing I can think of that might remotely have something to do with compatibility would be Windows using CRLF as the platform line endings but I would expect the respective terminal implementations to handle this just fine.
Last I checked ninja didn't have an option to force multi-line output but you can force it rather easily by not allocating a pty to stdout (e.g. piping to a command). Ninja then outputs multiple lines instead (and also doesn't truncate lines) but you also loose colored output for compiler warnings (and probably a few other things).
Stdout line width
Both neovim and vim can automatically set the size for their terminal from the width of the window. Example neovim (:help nvim_open_term()):
nvim_open_term({buffer}, {opts}) *nvim_open_term()*
Open a terminal instance in a buffer
[...]
Note: to directly initiate the terminal using the right size,
display the buffer in a configured window before calling this.
For instance, for a floating display, first create an empty
buffer using |nvim_create_buf()|, then display it using
|nvim_open_win()|, and then call this function. Then
|nvim_chan_send()| can be called immediately to process
sequences in a virtual terminal having the intended size.
According to this it seems you just have to open the window before creating the terminal in the buffer.
In vim I would expect this to work similar based on :help 'termwinsize'.
In addition vim (and probably also neovim based on the wording of the note above) has the option to set/overwrite the size so you could query the necessary information beforehand and set it manually.
Build targets
A separate parser for every build tool does in fact seem unmaintainable and using a CMake feature would be way nicer. Fortunately CMake has a feature for this. It is called cmake-file-api(7) and is meant to be used by 'clients' to retrieve semantic information about the buildsystems CMake generates. So exactly what we want, especially with the Object Kind "codemodel". Unfortunately usage requires parsing of JSON data which is easy enough in neovim but I don't know about vim (maybe fall back to jq for filtering with something like this: jq '.configurations[].targets[] | select(.projectIndex | contains(0)) .name' codemodel-v2-xxxxxxxxxxxxxxxx.json.
Ping?
Pong! :) Apologies for the long radio silence, it's a busy period. And thanks a lot for the extensive comment.
Ideally there was a way to just open a terminal in something like a managed mode (for it to not close automatically/display a command exited message) and then just run a command in said terminal buffer.
Vim-CMake used to work in a similar fashion (up to v0.6.2). It used to run a terminal with a simple bash script running inside of it, where the script would wait for some command string on the stdin and would send some "end of command" text on the stdout. Very hacky and rather not portable.
Now, Vim-CMake runs commands as Vim/Neovim jobs, and sends the commands' output to an "echo" terminal (see :h nvim_open_term() for Neovim for instance). This method gives us more control over how to run jobs, how to parse their output, and what to display in the terminal. And it decouples the execution of commands from the visualization of text in the terminal, which I think makes the architecture of Vim-CMake more modular and would make unit testing easier (if we're ever to add unit tests).
Alternatively I would just pass all data as soon as is available as I already know the terminal knows how to handle it.
This would be the preferred way, unless there's a third options which we don't know about. We should just fix our filtering rules for data to send to the terminal, and make sure that this yields a consistent result on Unix and Windows and in Neovim and Vim (for now actually Vim on Windows does not seem to produce any good results at all, but that's another problem).
I'd suggest we just fine tune our text filtering routine, and also configure the job's PTY correctly so that lines are not truncated when not necessary. Feel free to have a go at this, otherwise I'll do it, but I won't have time these next few weeks. In any case, I'll make sure to test the changes thoroughly on all platforms/editors.
As for the problem with retrieving the build targets consistently, CMake's file API sounds like a promising option, thank you for pointing that out. I opened #62 for that.
No worries, we all know busy periods.
Shell scripts are indeed not a good solution so the second option is probably the better one. The question then is do we need any filtering at all? Does CMake output/make buildsystems output anything that would break Vim-CMake or vim/neovim?
Configuring the pty seems easy by opening the terminal after the window. Both vim and neovim the use the window size by default if I interpret the docs correctly. Resizing the window should maybe be supported in the future too but I would focus on getting some initial support in first to make ninja usable.
Unfortunately I don't know enough about the necessary parts of vim/neovim and vim-cmake to work on this right now so I would leave this to you for now. If I find more time to read and understand the system and terminal code I would come back to this if you haven't started yet. This may take a while for me too though so I don't know if I get to this before you do.
I have left a comment on #62 though.
Then let's agree that I work on this one and you take up #62, would that be okay?
Works for me