Full compliance with completing-read
minad opened this issue · comments
Here we collect a list of icompatibilities with the Emacs completing-read API
-
Completion boundaries are not supported (#258).
-
"Dynamic completion table" support is incomplete (#114)
completion-table-in-turnis unsupported (#211)completion-table-dynamicandcompletion-table-with-cacheare unsupported
-
completion-ignored-extensionsis unsupported (#172) -
Emacs flex uses metadata adjustment in minibuffer.el to overwrite the sort function. In my opinion this is a big hack and should be removed from upstream Emacs. But it exists and it is good to keep it in mind. See #477 discussion. This should not be fixed in my opinion since it is an Emacs misfeature. Someone dares to propose the removal of this feature to the Emacs devel list? ;)
-
Require match is not implemented completely #429 -
Default history is not implemented completely #314
Anything else where Selectrum is incompatible?
To be transparent, there are a few standard Emacs features which are not implemented in Selectrum (mostly because I was unaware they existed) but which do work in Icomplete: for example, many of the completion-* user options such as completion-ignore-case. I do not see any design reason these features cannot all be incorporated into Selectrum eventually.
The README contains this paragraph. We could also maintain the list of incompatibilities somewhere else.
Anything else where Selectrum is incompatible?
- There is
completion-ignored-extensionswhich we can think about supporting in some way (#172) - For getting candidates from dynamic completion tables they are only called once at initialization. They also get passed an empty string, while in default completion the passed string can change based on completion-boundaries (#114)
The README contains this paragraph.
The part about completion-ignore-case isn't true anymore now that we use completion-styles by default.
Completion boundaries are next 😅 😉
Ich muss weg 🏃 ;)
"Dynamic completion table" support is incomplete, this is related to completion boundaries?
I think this is covered by:
For getting candidates from dynamic completion tables they are only called once at initialization. They also get passed an empty string, while in default completion the passed string can change based on completion-boundaries (#114)
?
Okay, but there has been some confusion about this before. Does this only happen in combination with completion boundaries or are there other dynamic tables? The way I understand it now is that the completion table is re-queried with the input string if the input string changes, but this depends on the completion boundaries and happens only if completion boundaries are specified by the table. With this feature one can support dynamic queries, file completion, consult-ripgrep etc.
There are a few tables which aren't correctly handled without using boundaries, for example the one in the linked issue uses minibuffer-contents. In Selectrum we only initialize the table once so these don't work correctly, too.
There are a few tables which aren't correctly handled without using boundaries, for example the one in the linked issue uses minibuffer-contents. In Selectrum we only initialize the table once so these don't work correctly, too.
Ah right. These are the 💩 tables since they violate purity. Do such tables exist in the wild? I think most realistic completion tables don't bend the API in that way.
Do such tables exist in the wild?
I'm not sure, some do but probably rare and not in Emacs itself AFAIK.
@clemera I toyed a bit around with default completion. I wonder if it would make sense to change Selectrum such that it will be fully compliant by simply wrapping the default completion and only replacing the UI. This will give full compliance for free. See the discussion before in #211 with @DarwinAwardWinner (#211 (comment)).
I implemented a super simple prototype, where a completion buffer is opened above the minibuffer. The candidates are shown in reverse order. This certainly takes a bit getting used to, maybe the Selectrum default UI is better. I don't know. But this is basically only a simple speed test to see if the result would still be reasonably snappy. See https://gist.github.com/minad/faa4be403a8999e59f630fe8a8ac6a7e.
So, my understanding is that icomplete-mode essentially works this way: it's just some additional "UI" added to default completion. Maybe we should look at how icomplete computes its list of matches within the default completion framework.
Yes, my prototype works more or less like icomplete works. Icomplete displays the candidates as overlay in the minibuffer. However icomplete has some downsides, in contrast to Selectrum.
- Icomplete uses cycling. I prefer using a candidate index as Selectrum does.
- Icomplete uses slow sorting, but this could be fixed upstream or via an advice to
completion-all-sorted-candidates, compare with themy-sortfunction. - The Icomplete UI is not flexible. One has to use icomplete-vertical to modernize this.
- Icomplete uses a delay. Selectrum does not. My prototype also doesn't use a delay and still seems to be reasonably snappy.
- Icomplete does not optimize completion style highlighting. Orderless supports filtering without highlighting and separate highlighting. I am using this in the prototype too. This is relevant for performance.
For me the question is if the approach taken by Selectrum should be changed. By wrapping default completion we automatically get support for all completion tables for free. Selectrum would not have to provide its own read file function. Right now Selectrum uses its own processing pipeline, where the candidates are not regenerated every time. The Selectrum pipeline is really fast, but there are so many issues one has to work around. The question is if using default completion in combination with an auto-updating UI (overlay or via buffer/display-action) is a better approach in the long term.
@minad I like your prototype, it isn't as fast as Selectrum but not bad and it could be improved with some additional caching. Emacs completion is written in a way which assumes the default UI in a lot of places, avoiding all the problems at once definitely sounds attractive. I'm not against changing things in fundamental ways if it turns out to be the better approach (even if it means we have to admit icomplete authors were right all along 😆) Though for the case of file completions I think it probably make sense to keep things like they are, we improve on default file completion in a lot of ways and in this specific case we also have no compatibility problems AFAIK.
I see there's a noticeable pause after the close paren if you do C-h i g (Elisp)Eval. However, even icomplete has the same pause I assume this is the time it takes Emacs to parse the Elisp manual and generate the TOC for it, so I think that's just unavoidable if you want to show completions. Regular unadorned emacs completion doesn't have this pause until you press TAB to request completions. In practice, no one is ever going to type an info node without completion, so they're always going to have to wait, but it's obviously much less jarring to wait after pressing TAB than when you're in the middle of typing. Anyway, no agenda here, just making some observations of how this interacts with my pet example for dynamic completion.
(By the way, also pay attention to how the list of completions changes dynamically after you type the opening paren.)
@clemera I pushed an update to the prototype. It replicates the selectrum ui with the overlay and allows now to scroll through the candidates and select.
I like your prototype, it isn't as fast as Selectrum but not bad and it could be improved with some additional caching.
I wonder if there is still room for performance improvements. Where do you think it is possible to add caching? I don't think this is possible since you always have to recompute the candidates. I did some profiling. The sorting is pretty optimized and costs only 2% for M-x. Most of the cost is due to completion-all-completions (70%). I wonder why the Selectrum approach is more snappy, since there you also have to filter always.
Though for the case of file completions I think it probably make sense to keep things like they are, we improve on default file completion in a lot of ways and in this specific case we also have no compatibility problems AFAIK.
Yes, if the Selectrum file selection is fundamentally better, it may make sense to keep this. But I find it very appealing to not having to reimplement all the completion tables. Everything should just work thanks to the default completion.
Very nice! For me the slowdown is mostly noticeable when there are few input chars, I experimented a bit and caching the value for the empty prompt helps when deleting the prompt and starting to provide new input, after the first char it is already fast enough for me so there would only be an initial delay when providing new input.
Another example where the slowdown is significant is insert-char. For insert-char sorting really becomes a major bottleneck. Sorting can probably be optimized even more, alternatively one could simply disable it for more than ~10k items.
Like I mentioned in the other thread I like the idea of some kind of hybrid, by default we can support all tables/completions and then we can handle tables/commands which are known to not need recomputation or benefit from special treatment (like file completions) differently.
Like I mentioned in the other thread I like the idea of some kind of hybrid, by default we can support all tables/completions and then we can handle tables/commands which are known to not need recomputation or benefit from special treatment (like file completions) differently.
Yes, this makes sense. But for most tables probably the standard treatment will just be good enough? This is at least what I think the prototype shows. In the case of file handling, the partial completion, which Selectrum implements, is not possible with the default file completion table. (Edit: not true, the built-in partial-completion style works) Therefore it definitely makes sense to keep special treatment for this.
I hacked a bit more on the mini completion system (https://gist.github.com/minad/faa4be403a8999e59f630fe8a8ac6a7e). I find it quite neat that one gets a completion system with only 300 loc.
There is one slightly different design I tried - instead of only keeping the index of the current candidate, I also updated completion-all-sorted-completions, such that the current candidate is always at the top, like Icomplete does, but keeping a Selectrum-like UI. This way one can reuse all the existing minibuffer functions which use the sorted list. However this does not work well together with the Selectrum concept of the prompt selection, which I like. For this reason I guess it is just better to not reuse the sorted candidates functionality and keep Selectrum like this with respect to how it handles candidate selection.
But I think I would like to see Selectrum developing a bit more in the direction of reusing the default completion pipeline, since this is just a simpler approach to achieve compliance with arbitrary completion tables. Anf if one goes with a hybrid design one can still optimize all the special cases and enhance them like file completion.
Yes most tables don't need it, that is nice. Only the big ones which would get slow when sorting after each input and some special cases we would like to extend like file-completions.
But I think I would like to see Selectrum developing a bit more in the direction of reusing the default completion pipeline, since this is just a simpler approach to achieve compliance with arbitrary completion tables. Anf if one goes with a hybrid design one can still optimize all the special cases and enhance them like file completion.
It seems the main difference is that we currently sort completion tables once and then refine that sorted collection, instead we would need to refine first (using selectrum-refine-candidates-using-completions-styles which already uses completion-all-completions) and do this on each input change and afterwards sort. Kind of what we do for the internal dynamic tables selectrum--read supports (with the difference that the table already handles refinement).
It seems the main difference is that we currently sort completion tables once and then refine that sorted collection, instead we would need to refine first (using selectrum-refine-candidates-using-completions-styles which already uses completion-all-completions) and do this on each input change and afterwards sort. Kind of what we do for the internal dynamic tables selectrum--read supports (with the difference that the table already handles refinement).
Yes, somehow like that. But there is some more internal magic going on, e.g., boundaries. By setting everything atop of default completion all these things just seem to work. However one cannot implement a hybrid using this approach.
I experimented by putting the following at top of the body of selectrum--read:
(when (functionp minibuffer-completion-table)
(setq candidates (lambda (input)
(setq-local selectrum-refine-candidates-function
#'selectrum-candidates-identity)
(nconc
(completion-all-completions
input
minibuffer-completion-table
minibuffer-completion-predicate
(length input)
(completion-metadata input
minibuffer-completion-table
minibuffer-completion-predicate))
nil))))Using this completion-table-in-turn works and also your example from supporting completion-boundaries in #448.
Oh, okay. That's an interesting idea to reuse the selectrum dynamic completion function.
Like I mentioned in the other thread I like the idea of some kind of hybrid, by default we can support all tables/completions and then we can handle tables/commands which are known to not need recomputation or benefit from special treatment (like file completions) differently.
There has been some comment on the emacs-devel mailing list that the completion style initials also works with file completions. You can try this with default completion or Vertico.
(setq completion-styles '(initials))
(setq completion-category-overrides nil)
(setq completion-category-defaults nil)This does not seem to work with Selectrum since it does not use the file completion table. But I don't know the exact reason. I am not saying that this should be supported (I find it pretty useless tbh), it is probably only for hardcore flex/initials fans. I just want to point it out.
Thanks, we could integrate this feature in a way similar to how we handle partial completions, triggering when there are no matches, though I agree that it is probably not worth it.
To give some more info: We use the built-in file completion table, but in Selectrum the refinement happens on the results, styles like initials or partial-completion do not only filter but also change the gathering process of candidates which is why they need to be handled specially if we want to allow them.
To give some more info: We use the built-in file completion table
Yes, I know. Selectrum does not read the file system itself. Could you explain precisely where Selectrum improves on the default file completion table? I only looked briefly at the code. As soon as #532 is merged you could still use the normal table?
You probably could but you would give up on the following improvements: Caching the results per directory (to make it fast for large dirs and slow file systems/connections), the handling of tramp connections (don't auto connect if a connection hasn't been established yet), improved env var completion (showing values of the variables as annotations) and automatically falling back to partial-completion results for incomplete paths.
Okay, thanks! These are good improvements. Is it realistic to get some of these improvements upstreamed into Emacs? Since this is where I think they should actually be handled. But I understand that in the short term the enhancements are good to have as is in Selectrum.
The improvements are mostly nice in combination with the different UI, with default completion you don't need to prevent tramp connections because it only happens after you press TAB already, the caching could help with the default UI too but is also less of a problem. Partial-completion style can be configured to be used for file completions, we needed to integrate it for Selectrum as otherwise the partial completion wouldn't work (as the filtering is done afterwards).
The improvements are mostly nice in combination with the different UI, with default completion you don't need to prevent tramp connections because it only happens after you press TAB already, the caching could help with the default UI too but is also less of a problem.
This is not entirely true. There is also Icomplete which is part of Emacs. So I assume there is some motivation to have this fixed upstream.
Partial-completion style can be configured to be used for file completions, we needed to integrate it for Selectrum as otherwise the partial completion wouldn't work (as the filtering is done afterwards).
This is hardly an advantage but rather a result of the custom Selectrum pipeline.
improved env var completion (showing values of the variables as annotations)
What is actually improved here? If I use default completion and press "$ TAB" when selecting a file, the completions buffer also shows the environment variables including values. (Ah, now I understand! This works in my setup since I have Marginalia activated.)
So all in all I think the real improvements are the better tramp handling (caching and connection opening) and I would like to see this upstream. At least I find it kind of nice that you can almost work with the standard facilities and still get a decent experience.
This is not entirely true. There is also Icomplete which is part of Emacs. So I assume there is some motivation to have this fixed upstream.
Okay, I only thought about default UI, not sure which parts could be implemented generically probably there needs to be some icomplete specific handling?
This is hardly an advantage but rather a result of the custom Selectrum pipeline.
The advantage I see is to have partial-style completions integrated into to regular file completions in a performant way (with the UI we have).
Okay, I only thought about default UI, not sure which parts could be implemented generically probably there needs to be some e icomplete specific handling?
I have no idea tbh. As you are aware, Icomplete is very simple internally (as is Vertico). It just shows the completions-all-completions in a post-command-hook. That's it. I don't know where one would want to add special handling - in the completion table itself such that it detects an interactive UI (like Icomplete, Selectrum, Vertico) and then adjusts its behavior accordingly? There is the variable non-essential which may be interesting - Icomplete uses this (Vertico does not yet). This variable prevents Tramp from opening connections I think, or at least prevents password prompts. A variable similar to that could be used to inform the completion table.
The advantage I see is to have partial-style completions integrated into to regular file completions in a performant way (with the UI we have).
Okay, maybe what Selectrum does is more performant - I have not observed that. Partial-completions just work in the same way in default completion and Vertico. But you have to activate it as completion style obviously. If this is an disadvantage or not? I think it is rather an advantage since you have to explicitly opt-in to the completion style.
This variable prevents Tramp from opening connections I think, or at least prevents password prompts.
Ah yes I also discovered this variable in the past, there is always a variable in Emacs somewhere 😆
Partial-completions just work in the same way in default completion and Vertico. But you have to activate it as completion style obviously. If this is an disadvantage or not? I think it is rather an advantage since you have to explicitly opt-in to the completion style.
I haven't tried it, the difference probably only becomes apparent when using slow connections/file systems but if it is fast enough that is good! We could add a variable to opt-in but so far no one has complained ;)