Quotes in comments affect highlighting of non-commented code in racket-hash-lang-mode
ashton314 opened this issue · comments
Characters like ' in comments seem to affect the highlighting of the rest of the buffer. Clearly, what happens in comments should stay in comments. :)
((alist-get 'racket-mode package-alist)) ((emacs-version "29.1.50") (system-type darwin) (x-gtk-use-system-tooltips UNDEFINED) (major-mode help-mode) (racket--el-source-dir "/var/folders/k3/4nd7wv0s6z9bm2h5p95ftrbw0000gn/T/elpaca.SRjyFh/elpaca/builds/racket-mode/") (racket--rkt-source-dir "/var/folders/k3/4nd7wv0s6z9bm2h5p95ftrbw0000gn/T/elpaca.SRjyFh/elpaca/builds/racket-mode/racket/") (racket-program "racket") (racket-command-timeout 10) (racket-path-from-emacs-to-racket-function UNDEFINED) (racket-path-from-racket-to-emacs-function UNDEFINED) (racket-browse-url-function racket-browse-url-using-temporary-file) (racket-documentation-search-location "https://docs.racket-lang.org/search/index.html?q=%s") (racket-xp-after-change-refresh-delay 1) (racket-xp-mode-lighter (:eval (racket--xp-mode-lighter))) (racket-xp-highlight-unused-regexp "^[^_]") (racket-repl-buffer-name-function nil) (racket-submodules-to-run ((test) (main))) (racket-memory-limit 2048) (racket-error-context medium) (racket-repl-history-directory "/var/folders/k3/4nd7wv0s6z9bm2h5p95ftrbw0000gn/T/elpaca.SRjyFh/racket-mode/") (racket-history-filter-regexp "\\`\\s *\\'") (racket-images-inline t) (racket-imagemagick-props nil) (racket-images-keep-last 100) (racket-images-system-viewer "open") (racket-pretty-print t) (racket-use-repl-submit-predicate nil) (racket-pretty-print t) (racket-indent-curly-as-sequence t) (racket-indent-sequence-depth 0) (racket-pretty-lambda nil) (racket-smart-open-bracket-enable nil) (racket-module-forms "\\s(\\(?:module[*+]?\\|library\\)") (racket-logger-config ((cm-accomplice . warning) (GC . info) (module-prefetch . warning) (optimizer . info) (racket/contract . error) (racket-mode-debugger . info) (sequence-specialization . info) (* . fatal))) (racket-show-functions (racket-show-pseudo-tooltip))) (enabled-minor-modes (auto-composition-mode) (auto-compression-mode) (auto-encryption-mode) (auto-fill-mode) (auto-save-mode) (blink-cursor-mode) (buffer-read-only) (electric-indent-mode) (file-name-shadow-mode) (font-lock-mode) (global-eldoc-mode) (global-font-lock-mode) (indent-tabs-mode) (isearch-fold-quotes-mode) (line-number-mode) (menu-bar-mode) (mouse-wheel-mode) (semantic-minor-modes-format) (shell-dirtrack-mode) (show-paren-mode) (tool-bar-mode) (tooltip-mode) (transient-mark-mode)) (disabled-minor-modes (abbrev-mode) (auto-fill-function) (auto-save-visited-mode) (buffer-face-mode) (button-mode) (cl-old-struct-compat-mode) (column-number-mode) (comint-fontify-input-mode) (compilation-minor-mode) (compilation-shell-minor-mode) (completion-in-region-mode) (context-menu-mode) (cursor-face-highlight-mode) (defining-kbd-macro) (display-line-numbers-mode) (eldoc-mode) (electric-layout-mode) (electric-quote-mode) (elpaca-no-symlink-mode) (elpaca-ui-live-update-mode) (global-display-line-numbers-mode) (global-hl-line-mode) (global-prettify-symbols-mode) (global-semantic-highlight-edits-mode) (global-semantic-highlight-func-mode) (global-semantic-show-parser-state-mode) (global-semantic-show-unmatched-syntax-mode) (global-semantic-stickyfunc-mode) (global-visual-line-mode) (header-line-indent-mode) (hl-line-mode) (horizontal-scroll-bar-mode) (hs-minor-mode) (isearch-mode) (jit-lock-debug-mode) (lock-file-mode) (lost-selection-mode) (next-error-follow-minor-mode) (overwrite-mode) (paragraph-indent-minor-mode) (prettify-symbols-mode) (racket-hash-lang-repl-mode) (racket-smart-open-bracket-mode) (racket-xp-mode) (read-extended-command-mode) (semantic-highlight-edits-mode) (semantic-highlight-func-mode) (semantic-mode) (semantic-show-parser-state-mode) (semantic-show-unmatched-syntax-mode) (semantic-stickyfunc-mode) (sh-electric-here-document-mode) (shell-highlight-undef-mode) (size-indication-mode) (tab-bar-history-mode) (tab-bar-mode) (temp-buffer-resize-mode) (text-scale-mode) (treesit-explore-mode) (treesit-inspect-mode) (undelete-frame-mode) (url-handler-mode) (use-hard-newlines) (view-mode) (visible-mode) (visual-line-mode) (window-divider-mode) (xref-etags-mode))
Here's a sample Rhombus file that exhibits the behavior:
#lang rhombus
fun factorial(n :: Int) :: Int:
if n == 0
| 1
| n * factorial(n - 1)
// The quote shouldn't affect the next block
// But everything after this gets font-lock-string-face
fun factorial_fast(n :: Int, acc :: Int) :: Int:
if n == 0
| acc
| factorial_fast(n - 1, n * acc)
factorial_fast(5, 1)
The ' in the word shouldn't in the comment makes the rest of the file (except for the numbers) get the font-lock-string-face. If you delete the single quote, everything goes back to normal.
If you use elpaca, here's an elpaca-test macro to reproduce the behavior:
;; -*- lexical-binding: t; -*-
(elpaca-test
:interactive t
:early-init
(setq byte-compile-warnings '(not obsolete))
(setq inhibit-startup-screen t)
:init
(elpaca (racket-mode :host github :repo "greghendershott/racket-mode"))
(elpaca-wait)
(find-file "hash_lang_highlight_bug.rhm")
(insert "#lang rhombus
fun factorial(n :: Int) :: Int:
if n == 0
| 1
| n * factorial(n - 1)
// The quote shouldn't affect the next block
// But everything after this gets font-lock-string-face
fun factorial_fast(n :: Int, acc :: Int) :: Int:
if n == 0
| acc
| factorial_fast(n - 1, n * acc)
factorial_fast(5, 1)")
(save-buffer)
(racket-hash-lang-mode)
(racket-xp-mode))Dump that into a buffer, and then run eval-buffer, and a new Emacs session should pop up in a clean environment, install racket-mode, and load the buffer. (You might have to hit q once Elpaca is done installing racket-mode to see the buffer.) As soon as you close the new Emacs session, Elpaca will clean everything up.
Thanks for the report!
If you M-x describe-char in those places the racket-token property is correct (e.g. "symbol" or "constant" etc. not "string"). So this is related to something else jumping in and doing font-lock.
I think this broke when fixing #669 to accommodate things like highlight-indent-guides by allowing some normal font-lock to happen, via font-lock-keywords matchers, and calling the default font-lock-fontify-region as well as doing our own thing.
I believe what I overlooked was that syntax-propertize-function must be set not to nil -- which means "do the default thing", which means "use the char syntax table for stuff like quotes to do font-lock" -- but instead to something like #'ignore (Emacs lisp that's ~= void in Racket), meaning do absolutely nothing.
At least that fixes the problem for me in a few quick tests. I might sleep on this to see if I can think of any other angles, before committing.
Thanks again for pointing this out!
Good to know—thank you for looking at this, Greg!
I'm at an awkward level of Emacs proficiency: I can kinda stumble around with Elisp a bit and read basic code, but my knowledge of Emacs internals (e.g. how font-lock works) is pretty lacking—so my help will be limited. 😅 I'm still very happy to help test stuff!
Oh no worries. I was thinking out loud in my comment. Your bug report was excellent.
Besides which, font-lock is probably the most complicated aspect of Emacs I've encountered so far. There are so many variables and configurable functions (~= parameters in Racket). So many layers.
Indeed after thinking about this more, I believe a better fix is to set font-lock-keywords-only true in our major mode initialization. And for belt+suspenders (just in case some minor-mode @#$s with that value), also avoid calling the general font-lock-default-fontify-region and instead call the more limited font-lock-fontify-keywords-region.
That also fixes your reported problem. And that's both clearer and more resilient... I think. :)
font-lock is probably the most complicated aspect of Emacs I've encountered so far
OK, well, if the maintainer racket-mode says an Emacs thing is complicated, it's complicated. :) I'm glad my bug report was helpful! I just wish I could actually monkey around in the code to maybe come up with a solution on occasion.
I noticed the same problem happens with #' for Rhombus/Shplait code:
#'thingy
Running describe-character on the ' (thanks for the tip!) shows the face as font-lock-string-face but the racket-token is [operator]. I imagine it's the same root cause ultimately, no?
(Happy Thanksgiving tomorrow! I hope you have a good, relaxing, and filling holiday!)
(Happy Thanksgiving tomorrow! I hope you have a good, relaxing, and filling holiday!)
Thanks!
I merged a commit to fix this -- as well as to more thoroughly review the situation.
Originally I believed that racket-hash-lang-mode would handle everything via tokens from the back end, and no "normal" font-lock would also be involved.
Then when I addressed #669 I didn't fully understand or appreciate what kinds font-lock it could co-exist with (and what would be required to do so properly) vs. what kinds of font-lock were just flat-out incompatible.
I think I sorted through all that, now. The size of the commit is mostly trying to handle and comment/document that in general.
I noticed the same problem happens with
#'for Rhombus/Shplait code:#'thingyRunning
describe-characteron the'(thanks for the tip!) shows the face asfont-lock-string-facebut theracket-tokenis[operator]. I imagine it's the same root cause ultimately, no?
Yes. I just installed shplait and your example looks correct to me with the new commit now.
The bad behavior is definitely gone now. Thanks!
Alas, all the pretty colors are gone now too. Before fun, if, etc. would get highlighted with racket-xp-binding-lang-use-face, but that doesn't seem to be happening any more.
Is there something I can do to get the colors back? E.g. can I ask the racket backend to annotate things like fun or if or match as keywords to get some of the nice colors back? Right now the racket-token property for almost everything is [symbol]. Is this a limitation of how each language is implemented?
It's off by default because now it's just one of a couple possible approaches to adding extra colors.
You can enable via customization variable racket-xp-add-binding-faces.
See racket-hash-lang-module-language-hook doc string, toward the end, for discussion.
In other words you can keep using racket-xp-mode to add binding faces in racket-hash-lang-mode. If you always want this, you could (setq racket-xp-add-binding-faces) from a racket-hash-lang-mode-hook. Or, if you only want that for some hash-langs, you could enable this (or not) in racket-hash-lang-module-language-hook.
Another choice is that, you can use font-lock-add-keywords to add patterns. You can borrow some of the ones from classic racket-mode. You could make a new set of patterns for rhombus or for X.
Something like the "dynamic" approach of binding faces feels more general and better.
On the other hand, a fixed list of keywords to highlight for a given lang, could work better in some cases; it's definitely the classic Emacs approach.
p.s. A drawback of the binding faces is that, if check-syntax finds an error in your program, you get no colors until it's corrected. e.g. https://racket.discourse.group/t/racket-mode-users-want-to-try-racket-hash-lang-mode/2372/7?u=greghendershott
I think a third, dynamic-ish way might be possible: Making old-fashioned font-lock-keywords regexp patterns from the list of completion candidates. That's close to the same set of things as binding sites that get colored.
That way, even if racket-xp-mode found an error in your program, you could still get more colors (just like you still get completion candidates) based on info from previous non-error checks.
It wouldn't be perfect but maybe that would be OK if this approach were used as a "best effort fallback" only during the check-syntax error state, and then was replaced by the binding arrows colors when not in error. (Just thinking out loud here.)
However completions for rhombus don't work well quite yet (https://racket.discourse.group/t/imports-as-completion-candidates-for-hash-langs/2511/6?u=greghendershott), so that's moot for you at least in the near future.