LanguageProcessor.expectedAt() often gives no suggestions despite working fine in similar situations
willplatt opened this issue · comments
I'm trying to use LanguageProcessor.expectedAt() to provide users with suggestions to complete the current thing they are typing or suggestions for what to type next. However, I often get no suggestions back, despite getting multiple good suggestions in similar contexts. I don't see much of a pattern to it, which makes it hard to come up with a workaround for and it's not good enough just to have no suggestions.
Here's an example with version 3.5.2 (I don't know how to use version 4.0.0-rc.4):
Sentence.agl:
namespace org.example.sentence.agl
grammar Sentence {
sentence = sentenceSequence | DOLLAR | WORD;
sentenceSequence = sentence sentence+;
leaf DOLLAR = '$';
leaf WORD = "[-a-zA-Z]+";
skip WHITESPACE = "\s+";
}
Suggestions.java:
import net.akehurst.language.agl.processor.Agl;
import net.akehurst.language.api.processor.CompletionItem;
import net.akehurst.language.api.processor.LanguageProcessor;
import java.io.InputStream;
import java.util.List;
import java.util.Scanner;
import java.util.stream.Collectors;
import static java.nio.charset.StandardCharsets.UTF_8;
public class Suggestions {
public static void main(String[] args) {
String grammarString = readResource(Suggestions.class.getResourceAsStream("Sentence.agl"));
LanguageProcessor languageProcessor = Agl.INSTANCE.processor(grammarString, null, null);
List<CompletionItem> initialSuggestions = languageProcessor.expectedAt("", 0, 100);
List<CompletionItem> suggestionsAfterDollar = languageProcessor.expectedAt("$", 1, 100);
List<CompletionItem> suggestionsAfterWord = languageProcessor.expectedAt("p", 1, 100);
List<CompletionItem> suggestionsAfterWordAndSpace = languageProcessor.expectedAt("p ", 2, 100);
List<CompletionItem> suggestionsAfterTwoWords = languageProcessor.expectedAt("p q", 3, 100);
System.out.print("initialSuggestions: ");
printSuggestions(initialSuggestions);
System.out.print("suggestionsAfterDollar: ");
printSuggestions(suggestionsAfterDollar);
System.out.print("suggestionsAfterWord: ");
printSuggestions(suggestionsAfterWord);
System.out.print("suggestionsAfterWordAndSpace: ");
printSuggestions(suggestionsAfterWordAndSpace);
System.out.print("suggestionsAfterTwoWords: ");
printSuggestions(suggestionsAfterTwoWords);
}
private static String readResource(InputStream resourceStream) {
return new Scanner(resourceStream, UTF_8).useDelimiter("\\A").next();
}
private static void printSuggestions(List<CompletionItem> suggestions) {
System.out.println(suggestions.stream().map(CompletionItem::getText).collect(Collectors.joining(", ")));
}
}
Output:
initialSuggestions: DOLLAR, WORD
suggestionsAfterDollar:
suggestionsAfterWord:
suggestionsAfterWordAndSpace:
suggestionsAfterTwoWords: DOLLAR, WORD
The initial suggestions and suggestions after two words are good, but I don't understand why there are no suggestions in the other cases. One word or two words or a dollar, they are all sentences and they can all be extended with further words and dollars, so why are the suggestions different?
The issue goes away if I change the sentence rule to sentence = (DOLLAR | WORD)+; but that isn't the point. This is a simplified example and it can also be helpful to have the sentenceSequence label if I want to treat these sentences differently from the simpler sentences after parsing.
what is your problem re using the later releases ?
Well I haven't found any documentation for the later releases and 3.5.2 seems to be the latest production-worthy release in Maven Central. I did have a quick try with 4.0.0-rc.4 back when I created this issue but couldn't figure out how to do anything with it. From what I recall things seemed to be significantly re-arranged and hardly any of the classes I had been using with 3.5.2 existed anymore (whether that's because they were simply renamed or completely refactored into other classes) so it wasn't clear where to begin.
Sorry if documentation is slow to come, however code completion in an IDE should provide most of what you need.
i.e. it was not hard to update your example to the 4.0.0-rc.4, see below.
I think it gives the expectedAt result you are looking for.
There are some major changes in API for v4.0.0. and it is not fixed in rc.4, but IDE code completion should always find the classes you are after and indicate function names and parameters.
Correct rc.4 is not really production ready, but expectedAt is a bit of a work in progress, especially in v3.x.
Sorry for the limitations and slow support (this is essentially a home projects of mine, done in my spare time)
Feel free to pay for support if you need something better than I can offer in free time ;-)
Changes for v4.0.0-rc.4
package test;
import net.akehurst.language.agl.processor.Agl;
import net.akehurst.language.api.processor.AutomatonKind;
import net.akehurst.language.api.processor.CompletionItem;
import net.akehurst.language.api.processor.LanguageProcessor;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Scanner;
import java.util.stream.Collectors;
class Suggestions {
public static void main(String[] args) {
AutomatonKind ak = AutomatonKind.LOOKAHEAD_1;
String grammarString = readResource(Suggestions.class.getResourceAsStream("/Sentence.agl"));
LanguageProcessor languageProcessor = Agl.INSTANCE.processorFromString(grammarString, null, null,null);
List<CompletionItem> initialSuggestions = languageProcessor.expectedAt("", 0, 100,ak);
List<CompletionItem> suggestionsAfterDollar = languageProcessor.expectedAt("$", 1, 100,ak);
List<CompletionItem> suggestionsAfterWord = languageProcessor.expectedAt("p", 1, 100,ak);
List<CompletionItem> suggestionsAfterWordAndSpace = languageProcessor.expectedAt("p ", 2, 100,ak);
List<CompletionItem> suggestionsAfterTwoWords = languageProcessor.expectedAt("p q", 3, 100,ak);
System.out.print("initialSuggestions: ");
printSuggestions(initialSuggestions);
System.out.print("suggestionsAfterDollar: ");
printSuggestions(suggestionsAfterDollar);
System.out.print("suggestionsAfterWord: ");
printSuggestions(suggestionsAfterWord);
System.out.print("suggestionsAfterWordAndSpace: ");
printSuggestions(suggestionsAfterWordAndSpace);
System.out.print("suggestionsAfterTwoWords: ");
printSuggestions(suggestionsAfterTwoWords);
}
...
output:
initialSuggestions: DOLLAR, WORD
suggestionsAfterDollar: DOLLAR, WORD
suggestionsAfterWord: DOLLAR, WORD
suggestionsAfterWordAndSpace: DOLLAR, WORD
suggestionsAfterTwoWords: DOLLAR, WORD
Thanks for this. Yes, that was easy to modify for 4.0.0-rc.4; maybe it was a different program I got stuck trying to modify. It's good to see the suggestions are much better.
I understand if it's just a hobby project. I really appreciate it though; very powerful yet much less fuss than other parsers I tried boasting similar capabilities. I'm using it for the corpus query language in #LancsBox X. Once there is a production-worthy release with good suggestions I should be able to add autocomplete suggestions for our users :)
I guess you are incorporating it into an editor?
Out of interest, which editor?
I've currently focused on web editor integration, but want to do a java one also, so I'm interested in what java editors are possible targets
Not exactly an editor but I have single line text fields with syntax highlighting and I've implemented my own suggestion popups. I'm using RichTextFX—it focuses on low-level functions like styling rather than being a fully fledged code editor. RSyntaxTextArea would be a more feature-rich and mature Java code editor, but for Swing rather than JavaFX. MonacoFX might be what you want for JavaFX, although I only just discovered it now. Web-based editors can also be used in JavaFX via a WebView, as in this (somewhat dated) example.