Magic command API redesign
jonsequitur opened this issue · comments
Update: These changes are now available in VS Code Insiders.
Magic command API redesign
Background and motivations
In the early days of .NET Interactive, providing magic command features as seen in Jupyter required a parser for a magic command grammar that would be independent of the various languages (C#, F#, PowerShell) that .NET Interactive supports. A POSIX command line-style grammar was a fairly clear choice, and System.CommandLine was robust and ready to use, so rather than build something custom, we took a dependency on System.CommandLine. This API has served .NET Interactive well for several years now. But as System.Commandline's design is being reset for inclusion as a core .NET library, some of the features that .NET Interactive uses are likely to be removed. At the same time, .NET Interactive's input system is undergoing a long-planned set of improvements whose ergonomics can be much better if we support features that are well outside of what a command line parser would include.
The largest impacts of these changes will be on people who've written extensions to .NET Interactive that customize magic commands. We are attempting to minimize impacts on notebook users by maintining backwards compatibility to the greatest extent possible, but it's important to recognize that some breaking changes are unavoidable unless we want to bring the entirety of the System.CommandLine codebase into .NET Interactive, which has a high cost for maintainability and concept complexity.
So what's changing?
Options and arguments are now just parameters
The terms option and argument were chosen based on common Gnu/POSIX naming conventions for the command line. Separating the magic command API from the command line use case is an opportunity to choose the more commonly-recognized term parameter. Both named parameters (i.e. options) and unnamed parameters (i.e. arguments) will now use the same type, KernelDirectiveParameter
.
What's being removed?
The following features of System.CommandLine are not currently planned to be reimplemented by the new magic command parser.
-
A magic command can no longer have multiple unnamed parameters (i.e. arguments).
-
Optionally, a single unnamed parameter can be allowed if the magic command is configured to allow it. It is not required of API users to configure their magic commands to have a name-optional parameter (
KernelDirectiveParameter.AllowImplicitName
).The following two lines are equivalent, assuming
--name
is the parameter whose name is optional:#!fruit --name apple #!fruit apple
Similarly, the following are equivalent:
#!fruit --color red --name apple #!fruit --color red apple
Relative ordering between parameters is not significant, so the following are equivalent:
#!fruit --color red apple #!fruit apple --color red
-
Inputs (e.g.
@input
and@password
) are now only valid for parameter values. They can no longer be used to supply subcommands or parameter names.Assuming a
#!fruit
magic command with two options,--color
and--name
, neither of which allow an implicit parameter name, the following would be invalid:#!fruit @input:"Which parameter are you setting?" red
-
The allowed parameter name prefix
/
(used for Windows-style options) will no longer be supported. The only allowed prefixes for parameter names will be-
and--
. -
System.CommandLine allows three ways to separate an option name from its argument:
--option argument
,--option=argument
, and--option:argument
. Only the first (space-separated) syntax will now be supported. -
POSIX-style option bundling (e.g.
git clean -fdx
which is equivalent togit clean -f -d -x
) will no longer be supported. -
The POSIX-style
--
delimiter, used as an escape in POSIX command lines, will no longer be supported. -
Minimum arities are no longer supported. Parameters can be required but a minimum arity can't be specified. Only maximum arities (now specified using
KernelDirectiveParameter.MaxOccurrences
) are supported. -
Magic command help (requested using the
-h
or--help
options) is no longer supported. The new parser will be used to provide hover text, which is more consistent with language services provided by other languages in Polyglot Notebooks.
What's being added?
-
Inline JSON can now be used to specify parameter values. (Multi-line magic commands are still not allowed, so JSON must be all on one line.)
#!fruit --name apple { "color": "red", "varieties": ["Macintosh", "Honeycrisp"] }
Variable sharing and input tokens are supported as before, but are not allowed within inlined JSON.
-
Inline JSON can also be used to configure expressions such as
@input
and@password
.#!connect mssql --data-source @input:{ "prompt": "Please choose a database", "type":"list", "recall": true, "provider": "AzureAD", "clientId": "abc", "tenantId": "def", "subscriptionId": "xyz" }
-
Type hints can be specified directly on a
KernelDirectiveParameter
. (Previsouly, they were inferred from the generic parameter of the targetArgument<T>
orOption<T>
.) -
More granular diagnostic squiggles are now provided. The System.CommandLine parser is unaware of character positions in the original text. This isn't knowable in a command line because the arguments to be parsed are sent to a .NET application entry point already broken into an array. This limitation need not apply in a notebook.
-
Richer completion support is now possible (e.g. within an argument, or within JSON).
Handling magic command invocation
With System.CommandLine beta 4 (which is the most recent version that .NET Interactive depends on), the code that handles the execution of a magic command was specified using the Command.Handler
property or the Command.SetHandler
method, which mainly did two things:
-
Called a user-specified delegate passing strongly-typed parameters.
-
Parsed those strongly-typed parameters from string input.
The Command.Handler
API has been a significant pain point in System.CommandLine and has seen a number of breaking changes over the last few years. The magic command API redesign is an opportunity to replace a couple of its responsibilities with existing, stable APIs.
-
Since there's already an API for kernels to handle
KernelCommand
-derived commands, magic commands will now use the same approach. On invocation, magic commands will be parsed intoKernelCommand
implementations and passed toKernel.SendAsync
. For custom magics, these will typically be commands defined by the magic command author. All of the existing command handling APIs, including middleware, will work with these commands just like they do with existing commands such asSubmitCode
.This has the additional effect of allowing magic command behaviors to be invoked directly by sending the corresponding
KernelCommand
, bypassing the magic command syntax. For example,#r nuget
is now parsed into anAddPackage
command, making the following two examples functionally equivalent:#r "nuget: Plotly.NET, 5.0.0"
var command = new AddPackage( packageName: "Plotly.NET", packageVersion: "5.0.0"); await kernel.SendAsync(command);
You can think of a magic command as a gesture for the notebook user while the
KernelCommand
is its underlying API, which will often be more ergonomic for programmatic usage. -
Rather than using a custom binding and serialization implementation to create
KernelCommand
instances from parsed magic commands, .NET Interactive will now use JSON serialization. The new parser has the ability to serialize parsed magic commands into JSON. Magic commands are then deserialized using System.Text.Json. Any custom deserialization you might need can be configured using standard System.Text.Json attributes on your custom command type.
With all of that in mind, here's a simple example of the new API:
var fruitDirective = new KernelActionDirective("#!fruit");
kernel.AddDirective<ExampleCommand>(
fruitDirective,
async (ExampleCommand command, KernelInvocationContext context) =>
{
// magic command logic goes here
});
SubmitCode.Parameters
There are a few kernels whose code submissions can be parameterized. For example, naming a result when using T-SQL and KQL kernels requires using the kernel-specifier magic and providing an additional --name
parameter
This design predates the kernel picker UI element in Polyglot Notebooks. There has been no way to specify the query name when using the kernel picker UI alone, and the strong coupling of this parameter to the parsing of the magic command made it difficult to address. While the new magic command API redesign doesn't fix this issue, it does create a way to supply these parameters without the use of a magic command.
A new Parameters
property has been added to the SubmitCode
command. When SubmitCode
includes a kernel specifier magic, the SubmitCode.Parameters
property will be populated using any parameters that the user included, but these properties can also be set directly. This creates an opportunity for the UI to augment SubmitCode
commands in the future.
I'm glad to see this being worked on! Bravo! 😃
It's been on the back burner for a minute and is a very complex change. I'll be updating the details above as more of the design details are worked out. Please give us any thoughts or questions that occur to you.
This is great to know about in this level of detail. This makes a lot of sense and I love the clarity on implicit parameters as well. I think this change makes the overall system more approachable with consistent terminology, in addition to being a necessary change due to the commandline dependency.
I know we're still gathering community feedback on this, but do we have a rough idea of when an implementation might be available to use in a preview version of the extension?
@IntegerMan It's available now in VS Code Insiders.
I'm poking around with the preview and I think I'm missing some aspects of how all the pieces connect - specifically with providing a KernelDirectiveParameter and getting that parameter later.
So, I can define a parameter like this:
KernelDirectiveParameter variableNameParameter = new("name")
{
Description = "The name of the variable to reflect",
Required = true,
AllowImplicitName = true
};
and then I can declare a directive as follows:
KernelActionDirective reflectDirective = new("#!reflect")
{
Description = "Reflects the internal state of the object",
Parameters = { variableNameParameter }
};
and register it with the C# kernel:
kernel.AddDirective(reflectDirective,
(command, context) => {
// Send a message
context.DisplayStandardOut("Reflecting internal state of the object... ");
string variableName = "TODO"; // TODO: How do I get this?
// Return the object
return Task.CompletedTask;
}
);
But how do I get the value of the variable out of the context or command? Previously we could create a Command, pass it our Option values, then call SetHandler on that command. It doesn't appear that you can pass a KernelDirectiveParameter into a Command in the same way.
I'm probably missing something simple here.
If you use the Kernel.AddDirective<T>
overload and use a command type that you define, then the command instance passed to the handler will be deserialized using System.Text.Json. The JSON is built based on the magic command, with e.g. a parameter named --init-script
targeting (by convention) a property named InitScript
. You can customize as needed using System.Text.Json attributes.
Here's an example:
Awesome, @jonsequitur. I missed that the properties were getting hydrated via serialization.
Also, I was getting stuck because I had copied that example too literally and had a constructor parameter for my custom command's constructor when one wasn't needed.
The end result was that the magic command simply executed without stopping due to some form of hidden error trying to instantiate the command instance.
I can at least get the property out in my command handler now, so I believe tomorrow I should be good to wire things together for my actual logic without issues.
Thank you again.