dotnet / interactive

Update: These changes are now available in VS Code Insiders.

Magic command API redesign

Background and motivations

In the early days of .NET Interactive, providing magic command features as seen in Jupyter required a parser for a magic command grammar that would be independent of the various languages (C#, F#, PowerShell) that .NET Interactive supports. A POSIX command line-style grammar was a fairly clear choice, and System.CommandLine was robust and ready to use, so rather than build something custom, we took a dependency on System.CommandLine. This API has served .NET Interactive well for several years now. But as System.Commandline's design is being reset for inclusion as a core .NET library, some of the features that .NET Interactive uses are likely to be removed. At the same time, .NET Interactive's input system is undergoing a long-planned set of improvements whose ergonomics can be much better if we support features that are well outside of what a command line parser would include.

The largest impacts of these changes will be on people who've written extensions to .NET Interactive that customize magic commands. We are attempting to minimize impacts on notebook users by maintining backwards compatibility to the greatest extent possible, but it's important to recognize that some breaking changes are unavoidable unless we want to bring the entirety of the System.CommandLine codebase into .NET Interactive, which has a high cost for maintainability and concept complexity.

So what's changing?

Options and arguments are now just parameters

The terms option and argument were chosen based on common Gnu/POSIX naming conventions for the command line. Separating the magic command API from the command line use case is an opportunity to choose the more commonly-recognized term parameter. Both named parameters (i.e. options) and unnamed parameters (i.e. arguments) will now use the same type, KernelDirectiveParameter.

What's being removed?

The following features of System.CommandLine are not currently planned to be reimplemented by the new magic command parser.

A magic command can no longer have multiple unnamed parameters (i.e. arguments).
Optionally, a single unnamed parameter can be allowed if the magic command is configured to allow it. It is not required of API users to configure their magic commands to have a name-optional parameter (KernelDirectiveParameter.AllowImplicitName).

The following two lines are equivalent, assuming --name is the parameter whose name is optional:
```
#!fruit --name apple 
#!fruit apple
```
Similarly, the following are equivalent:
```
#!fruit --color red --name apple
#!fruit --color red apple
```
Relative ordering between parameters is not significant, so the following are equivalent:
```
#!fruit --color red apple 
#!fruit apple --color red
```
Inputs (e.g. @input and @password) are now only valid for parameter values. They can no longer be used to supply subcommands or parameter names.

Assuming a #!fruit magic command with two options, --color and --name, neither of which allow an implicit parameter name, the following would be invalid:
```
#!fruit @input:"Which parameter are you setting?" red
```
The allowed parameter name prefix / (used for Windows-style options) will no longer be supported. The only allowed prefixes for parameter names will be - and --.
System.CommandLine allows three ways to separate an option name from its argument: --option argument, --option=argument, and --option:argument. Only the first (space-separated) syntax will now be supported.
POSIX-style option bundling (e.g. git clean -fdx which is equivalent to git clean -f -d -x) will no longer be supported.
The POSIX-style -- delimiter, used as an escape in POSIX command lines, will no longer be supported.
Minimum arities are no longer supported. Parameters can be required but a minimum arity can't be specified. Only maximum arities (now specified using KernelDirectiveParameter.MaxOccurrences) are supported.
Magic command help (requested using the -h or --help options) is no longer supported. The new parser will be used to provide hover text, which is more consistent with language services provided by other languages in Polyglot Notebooks.

What's being added?

Inline JSON can now be used to specify parameter values. (Multi-line magic commands are still not allowed, so JSON must be all on one line.)
```
#!fruit --name apple { "color": "red", "varieties": ["Macintosh", "Honeycrisp"] }
```
Variable sharing and input tokens are supported as before, but are not allowed within inlined JSON.

Inline JSON can also be used to configure expressions such as @input and @password.

#!connect mssql --data-source @input:{ "prompt": "Please choose a database", "type":"list", "recall": true, "provider": "AzureAD", "clientId": "abc", "tenantId": "def", "subscriptionId": "xyz" }

Type hints can be specified directly on a KernelDirectiveParameter. (Previsouly, they were inferred from the generic parameter of the target Argument<T> or Option<T>.)
More granular diagnostic squiggles are now provided. The System.CommandLine parser is unaware of character positions in the original text. This isn't knowable in a command line because the arguments to be parsed are sent to a .NET application entry point already broken into an array. This limitation need not apply in a notebook.
Richer completion support is now possible (e.g. within an argument, or within JSON).

Handling magic command invocation

With System.CommandLine beta 4 (which is the most recent version that .NET Interactive depends on), the code that handles the execution of a magic command was specified using the Command.Handler property or the Command.SetHandler method, which mainly did two things:

Called a user-specified delegate passing strongly-typed parameters.
Parsed those strongly-typed parameters from string input.

The Command.Handler API has been a significant pain point in System.CommandLine and has seen a number of breaking changes over the last few years. The magic command API redesign is an opportunity to replace a couple of its responsibilities with existing, stable APIs.

Since there's already an API for kernels to handle KernelCommand-derived commands, magic commands will now use the same approach. On invocation, magic commands will be parsed into KernelCommand implementations and passed to Kernel.SendAsync. For custom magics, these will typically be commands defined by the magic command author. All of the existing command handling APIs, including middleware, will work with these commands just like they do with existing commands such as SubmitCode.

This has the additional effect of allowing magic command behaviors to be invoked directly by sending the corresponding KernelCommand, bypassing the magic command syntax. For example, #r nuget is now parsed into an AddPackage command, making the following two examples functionally equivalent:
```
#r "nuget: Plotly.NET, 5.0.0"
```
```
var command = new AddPackage(
    packageName: "Plotly.NET",
    packageVersion: "5.0.0");
await kernel.SendAsync(command);
```
You can think of a magic command as a gesture for the notebook user while the KernelCommand is its underlying API, which will often be more ergonomic for programmatic usage.
Rather than using a custom binding and serialization implementation to create KernelCommand instances from parsed magic commands, .NET Interactive will now use JSON serialization. The new parser has the ability to serialize parsed magic commands into JSON. Magic commands are then deserialized using System.Text.Json. Any custom deserialization you might need can be configured using standard System.Text.Json attributes on your custom command type.

With all of that in mind, here's a simple example of the new API:

var fruitDirective = new KernelActionDirective("#!fruit");

kernel.AddDirective<ExampleCommand>(
    fruitDirective,
    async (ExampleCommand command, KernelInvocationContext context) =>
    {
        // magic command logic goes here
    });

`SubmitCode.Parameters`

There are a few kernels whose code submissions can be parameterized. For example, naming a result when using T-SQL and KQL kernels requires using the kernel-specifier magic and providing an additional --name parameter

This design predates the kernel picker UI element in Polyglot Notebooks. There has been no way to specify the query name when using the kernel picker UI alone, and the strong coupling of this parameter to the parsing of the magic command made it difficult to address. While the new magic command API redesign doesn't fix this issue, it does create a way to supply these parameters without the use of a magic command.

A new Parameters property has been added to the SubmitCode command. When SubmitCode includes a kernel specifier magic, the SubmitCode.Parameters property will be populated using any parameters that the user included, but these properties can also be set directly. This creates an opportunity for the UI to augment SubmitCode commands in the future.

I'm glad to see this being worked on! Bravo! 😃

It's been on the back burner for a minute and is a very complex change. I'll be updating the details above as more of the design details are worked out. Please give us any thoughts or questions that occur to you.

This is great to know about in this level of detail. This makes a lot of sense and I love the clarity on implicit parameters as well. I think this change makes the overall system more approachable with consistent terminology, in addition to being a necessary change due to the commandline dependency.

I know we're still gathering community feedback on this, but do we have a rough idea of when an implementation might be available to use in a preview version of the extension?

@IntegerMan It's available now in VS Code Insiders.

I'm poking around with the preview and I think I'm missing some aspects of how all the pieces connect - specifically with providing a KernelDirectiveParameter and getting that parameter later.

So, I can define a parameter like this:

KernelDirectiveParameter variableNameParameter = new("name")
{
    Description = "The name of the variable to reflect",
    Required = true,
    AllowImplicitName = true
};

and then I can declare a directive as follows:

KernelActionDirective reflectDirective = new("#!reflect")
{
    Description = "Reflects the internal state of the object",
    Parameters = { variableNameParameter }
};

and register it with the C# kernel:

kernel.AddDirective(reflectDirective, 
    (command, context) => {
        // Send a message
        context.DisplayStandardOut("Reflecting internal state of the object... ");

        string variableName = "TODO"; // TODO: How do I get this?

        // Return the object
        return Task.CompletedTask;
    }
);

But how do I get the value of the variable out of the context or command? Previously we could create a Command, pass it our Option values, then call SetHandler on that command. It doesn't appear that you can pass a KernelDirectiveParameter into a Command in the same way.

I'm probably missing something simple here.

If you use the Kernel.AddDirective<T> overload and use a command type that you define, then the command instance passed to the handler will be deserialized using System.Text.Json. The JSON is built based on the magic command, with e.g. a parameter named --init-script targeting (by convention) a property named InitScript. You can customize as needed using System.Text.Json attributes.

Here's an example:

interactive/src/Microsoft.DotNet.Interactive.Jupyter/ConnectJupyterKernel.cs

Line 27 in 3bfd099

}

Awesome, @jonsequitur. I missed that the properties were getting hydrated via serialization.

Also, I was getting stuck because I had copied that example too literally and had a constructor parameter for my custom command's constructor when one wasn't needed.

The end result was that the magic command simply executed without stopping due to some form of hidden error trying to instantiate the command instance.

I can at least get the property out in my command handler now, so I believe tomorrow I should be good to wire things together for my actual logic without issues.

Thank you again.