Empty method definition with only a comment seems to confuse syntaxsuggest

[Yusuke Endoh]

class C
  def foo
    # comment
  end

  def bar
    "some literal"
  end

  def baz
  end

  def qux
  end

  def quux
  end
end
end # extra end

Expected:

$ ruby error.rb 
error.rb: --> error.rb
Unmatched `end', missing keyword (`do', `def`, `if`, etc.) ?
>  1  class C
> 18  end
> 19  end # extra end
error.rb:19: syntax error, unexpected `end' (SyntaxError)
end # extra end
^~~

Actual:

$ ruby error.rb 
error.rb: --> error.rb
Unmatched `end', missing keyword (`do', `def`, `if`, etc.) ?
   1  class C
>  4    end
> 10    def baz
> 11    end
  18  end
error.rb:19: syntax error, unexpected `end' (SyntaxError)
end # extra end
^~~

I haven't identified the exact condition to reproduce this issue, but I am wondering that an empty method definition with only comments would confuse the heuristics.

[Richard Schneeman]

Very interesting. Thanks for the report. I do some special stuff to handle comments

syntax_suggest/lib/syntax_suggest/clean_document.rb

Lines 70 to 85 in e0c53bb

 # ## Comments and whitespace 
 # 
 # Comments can throw off the way the lexer tells us that the line 
 # logically belongs with the next line. This is valid ruby but 
 # results in a different lex output than before: 
 # 
 #     1 User. 
 #     2   where(name: "schneems"). 
 #     3   # Comment here 
 #     4   first 
 # 
 # To handle this we can replace comment lines with empty lines 
 # and then re-lex the source. This removal and re-lexing preserves 
 # line index and document size, but generates an easier to work with 
 # document. 
 # 

. So they're actually stripped out when the document is re-lexxed. This doc is equivalent to:

class C
  def foo

  end

  def bar
    "some literal"
  end

  def baz
  end

  def qux
  end

  def quux
  end
end
end # extra end

If you want to see individual steps you can run it with this flag:

$ SYNTAX_SUGGEST_DEBUG=1 ruby scratch.rb

The first few steps aren't that interesting, but step 5 is:

    Block lines: 5..9 (expand) 

   1  class C
   2    def foo
   3
   4    end
>  5
>  9
  10    def baz
  11    end
  12
  13    def qux
  14    end
  15
  16    def quux
  17    end
  18  end
  19  end # extra end

On the next expansion it pops lines 5-9 and expands in both directions them to 3-12:

    Block lines: 3..12 (expand) 

   1  class C
   2    def foo
>  3
>  4    end
> 10    def baz
> 11    end
> 12
  13    def qux
  14    end
  15
  16    def quux
  17    end
  18  end
  19  end # extra end

So the issue looks like it stops that expansion up before it should.

syntax_suggest/lib/syntax_suggest/block_expand.rb

Lines 55 to 60 in e0c53bb

 expanded_lines = AroundBlockScan.new(code_lines: @code_lines, block: block) 
   .skip(:hidden?) 
   .stop_after_kw 
   .scan_neighbors 
   .scan_while { |line| line.empty? } # Slurp up empties 
   .lines 

I'll have to play with it some. Either we can change the input document (hiding comment lines for example) or we can change some logic in that scanner. Usually fixing one problem breaks an existing case, but often there's some good path forward.