Looking for advice on dealing with consult tofu characters

[Simon Manning]

Hey jimeh, I noticed you use this siren-flx-rs-score-consult-fix to support consult with flx-rs, and I'm running into a similar issue trying to get the compiled version of hotfuzz to play nicely: axelf4/hotfuzz#12

I tried doing something similar to your function with:

(defun hotfuzz--fix-consult-candidates (orig-fun string candidates &rest args)
  (let ((newcands (mapcar #'fussy-without-tofu-char candidates)))
    (apply orig-fun string newcands args)))

(advice-add 'hotfuzz-filter :around 'hotfuzz--fix-consult-candidates)

but ran into issues with vertico with this change when using consult with multiple sources, e.g.:

Debugger entered--Lisp error: (args-out-of-range [(:name "Project Buffer" :narrow 98 :category buffer :face consult-buffer :history buffer-name-history :state consult--buffer-state :enabled #f(compiled-function () #<bytecode 0x1980d94e0aa5>) :items #f(compiled-function () #<bytecode -0x358dac4159a18cf>)) (:preview-key (:debounce 0.4 any) :name "Project File" :narrow 102 :category file :face consult-file :history file-name-history :state consult--file-state :new #f(compiled-function (file) #<bytecode 0x5c1cf97f793e43c>) :enabled #f(compiled-function () #<bytecode 0x1991e5075901>) :items #f(compiled-function () #<bytecode 0x5dc1600af97cc19>))] -2097110)
  consult--multi-group([(:name "Project Buffer" :narrow 98 :category buffer :face consult-buffer :history buffer-name-history :state consult--buffer-state :enabled #f(compiled-function () #<bytecode 0x1980d94e0aa5>) :items #f(compiled-function () #<bytecode -0x358dac4159a18cf>)) (:preview-key (:debounce 0.4 any) :name "Project File" :narrow 102 :category file :face consult-file :history file-name-history :state consult--file-state :new #f(compiled-function (file) #<bytecode 0x5c1cf97f793e43c>) :enabled #f(compiled-function () #<bytecode 0x1991e5075901>) :items #f(compiled-function () #<bytecode 0x5dc1600af97cc19>))] #("*Help*" 0 6 (multi-category (buffer . "*Help*") face consult-buffer)) nil)
  apply(consult--multi-group ([(:name "Project Buffer" :narrow 98 :category buffer :face consult-buffer :history buffer-name-history :state consult--buffer-state :enabled #f(compiled-function () #<bytecode 0x1980d94e0aa5>) :items #f(compiled-function () #<bytecode -0x358dac4159a18cf>)) (:preview-key (:debounce 0.4 any) :name "Project File" :narrow 102 :category file :face consult-file :history file-name-history :state consult--file-state :new #f(compiled-function (file) #<bytecode 0x5c1cf97f793e43c>) :enabled #f(compiled-function () #<bytecode 0x1991e5075901>) :items #f(compiled-function () #<bytecode 0x5dc1600af97cc19>))] #("*Help*" 0 6 (multi-category (buffer . "*Help*") face consult-buffer)) nil))
  #f(compiled-function (&rest args2) #<bytecode -0xae28048c891503f>)(#("*Help*" 0 6 (multi-category (buffer . "*Help*") face consult-buffer)) nil)

I'm guessing this has something to do with the order in which I'm stripping these characters. Wondering if you might have any clues? :)

[Daniel Mendler]

See also axelf4/hotfuzz#12 (comment) for a simpler fix.

[Jim Myhrberg]

@sjrmanning apologies for the late response. The fussy package now have a few helper functions to strip out consult tofu characters.

I've adopted it for a simpler fix in commit e74a3bd. And I also submitted a PR to fussy for flx-rs specifically.

However, in my testing it seems like hotfuzz can deal with the tofu characters now, including the native module. As I saw no difference in results when I did strip them out compared to leaving them in, unlike how flx-rs behaved.

[Jim Myhrberg]

I'll be closing this issue, as it's not directly related to this repo, and I believe there's good solutions to solve the question being asked here now :)