Old features removed?

[William Platt]

I'm trying to migrate from version 3.5.2 to 4.0.1 but with how things have been re-organised I'm not sure if some features have been removed or I just can't figure out how to access them. I don't know Kotlin but I've been using the source code to try to make sense of things.

Parsing invalid strings

I need to provide syntax highlighting even when user-provided strings are invalid according to my grammar. In 3.5.2 I would get the longest match and it would work well for syntax highlighting up to the invalid part of the string:

try {
	return LANGUAGE_PROCESSOR.parse(QUERY, queryString).tokensByLine(0);
} catch (ParseFailedException e) {
	SharedPackedParseTree longestMatch = e.getLongestMatch();
	if (longestMatch == null) {
		return LANGUAGE_PROCESSOR.scan(queryString);
	} else {
		return longestMatch.tokensByLine(0);
	}
}

However, in 4.0.1 when the parsing fails I don't get an exception with some parse information I can use, I just get a null parse tree.

SharedPackedParseTree parseTree = LANGUAGE_PROCESSOR.parse(queryString, LANGUAGE_PROCESSOR.parseOptionsDefault()).getSppt();
if (parseTree == null) {
	return LANGUAGE_PROCESSOR.scan(queryString);
}
return parseTree.tokensByLine(0);

Unfortunately, LANGUAGE_PROCESSOR.scan(queryString) gives inaccurate results (I think because alphabetic characters can match multiple terminals in my grammar but which one it should actually match depends on context, which scan() doesn't seem to take into account).

Code completion

I was looking forward to the improvements in this area, but currently I can't get any suggested completions. I used to be able to get suggested completions like so:

List<CompletionItem> suggestions = LANGUAGE_PROCESSOR.expectedAt(queryString, position, 100);

Now the suggestions are always an empty list:

List<CompletionItem> suggestions = LANGUAGE_PROCESSOR.expectedItemsAt(queryString, position, 100, LANGUAGE_PROCESSOR.optionsDefault()).getItems();

Any help would be appreciated, but until then I can stick with the old version.